Proteina: Scaling Flow-based Protein Structure Generative Models
Authors: Tomas Geffner
Kieran Didi
Zuobai Zhang
Danny Reidenbach
Zhonglin Cao
Jason Yim
Mario Geiger
Christian Dallago
Emine Kucukbenli
Arash Vahdat
Karsten Kreis
Venue: ICLR 2025 Oral (2025)
Keywords: protein structure generation
de novo protein design
flow matching
fold class conditioning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We present a novel flow-based protein backbone generative model that uses a new scalable transformer architecture and conditions on fold class labels.

Abstract: Recently, diffusion- and flow-based generative models of protein structures have emerged as a powerful tool for de novo protein design. Here, we develop *Proteina*, a new large-scale flow-based protein backbone generator that utilizes hierarchical fold class labels for conditioning and relies on a tailored scalable transformer architecture with up to $5\times$ as many parameters as previous models. To meaningfully quantify performance, we introduce a new set of metrics that directly measure the distributional similarity of generated proteins with reference sets, complementing existing metrics. We further explore scaling training data to millions of synthetic protein structures and explore improved training and sampling recipes adapted to protein backbone generation. This includes fine-tuning strategies like LoRA for protein backbones, new guidance methods like classifier-free guidance and autoguidance for protein backbones, and new adjusted training objectives. Proteina achieves state-of-the-art performance on de novo protein backbone design and produces diverse and designable proteins at unprecedented length, up to 800 residues. The hierarchical conditioning offers novel control, enabling high-level secondary-structure guidance as well as low-level fold-specific generation.

Published as a conference paper at ICLR 2025
PROTEíNA: SCALING FLOW-BASED PROTEIN STRUCTURE GENERATIVE MODELS
Tomas Geffner!
Kieran Didi
Zuobai Zhang! 1,2.3,+*
Danny Reidenbach'
Zhonglin Cao'
Jason Yim
Mario Geiger'
Christian Dallago'
Emine Kucukbenli'
Arash Vahdat'
Karsten Kreis
NVIDIA Mila
Québec Al Institute
Université de Montréal "Massachusetts Institute of Technology
Project page: https://resset nvid com/1 abs/ /tt.//eeaar/ppootiei.m.
ABSTRACT
Recently, diffusion- and flow-based generative models of protein structures have emerged as a powerful tool for de novo protein design. Here, we develop Proteina, new large-scale flow-based protein backbone generator that utilizes hierarchical fold class labels for conditioning and relies on a tailored scalable transformer ar- chitecture with up to 5x as many parameters as previous models. To meaningfully quantify performance, we introduce new set of metrics that directly measure the distributional similarity generated proteins with reference sets, complementing existing metrics. We further explore scaling training data to millions of synthetic protein structures and explore improved training and sampling recipes adapted protein backbone generation. This includes fine-tuning strategies like LoRA for protein backbones, new guidance methods like classifier-fre guidance and autoguidance for protein backbones, and new adjusted training objectives. Protefna achieves state-of-the-ee performance on de novo protein backbone design and produces diverse and designable proteins at unprecedented length, up 800 residues. The hierarchical conditioning offers novel control, enabling high-level secondary-structure guidance as well as low-level fold-specific generation.
INTRODUCTION
De novo protein design, the rational design of new proteins from scratch with specific functions and properties. a grand challenge molecular biology (Richardson & Richardson, 1989; Huang et al., 2016; Kuhlman & Bradley, 2019). Recently, deep generative models emerged as a novel data- driven tool for protein design. Since protein's function mediated through its structure, a popular approach is to directly model the distribution of three-dimensional protein structures (Ingraham et al., 2023; Watson et al., 2023; Yim et al., 2023b; Bose et al., 2024; Lin & Alquraishi, 2023), typically with diffusion- or flow-based methods (Ho et al., 2020; Lipman et al., 2023). Such protein structure generators usually synthesize backbones only, without sequence or side chains, in contrast to protein language models, which often model sequences instead (Elnaggar al., 2022; Lin et al., 2023; Alamdari et al., 2023), and sequence-to-strucuue to-structure folding models like AlphaFold (Jumper et al., 2021).
Previous unconditional protein structure generative models have only been trained small datasets, consisting of no more than half million structures at maximum (Lin et al., 2024). Moreover, their neural networks do not offer any control during synthesis and are usually small, compared to modern generative AI systems in domains such as natural language, image or video generation. There, we have witnessed major breakthroughs thanks scalable neural network architectures, large training datasets, and fine semantic control (Esser et al., 2024; Brooks et al., 2024; OpenAI, 2024). This begs the question: can we similarly scale and control protein structure diffusion and flow models, taking lessons from the recent successes of generative models in computer vision and natural language?
Here, we set out to scale protein structure generation and develop a new flow matching- based protein backbone generative model called Proteina. In vision and language modeling, generative models are typically prompted through semantic text or class inputs, offering enhanced controllability. Anal- ogously, we enrich our training data with hierarchical fold class labels following the CATH Protein
"Core contributor.
"Work done during internship at NVIDIA.

---

Published as a conference paper at ICLR 2025
Flow 1/+
Figure I: Proteina. We use flow-matching and learn a flow to transform a Gaus- sian distribution over ini- tial protein backbone coordi- nates (residues' C.. atoms) into realistic protein struc- tures. Proteina relies on a scalable transformer based architecture and can be con- ditioned on hierarchical fold class labels improved con- trollability and complex pro- tein structure design tasks.
oo X
Xt=1
Noisy intermediate X;
Protein Backbone Transformer
Flow vector field v?
(Optional) fold class label
Structure Classification scheme (Dawson al., 2016). Our novel hierarchical fold class conditioning offers both high-level control, for instance over secondary structure content, as well as low-level guid- ance with respect to specific fold classes. This can be leveraged, for instance, dramatically increase the number of B-sheets in generated proteins. We also explore scaling the training data and train on up to 21 million protein structures, a 35x increase of training data compared to previous work.
Next, we develop scalable transformer architecture. We opt for a non-equivariant design inspired by recent diffusion transformers in vision (Peebles & Xie, 2023; Ma et al., 2024). For boosted per- formance, we optionally include triangle layers, powerful albeit computationally expensive network component common in the protein literature (Jumper et al., 2021). Crucially, though, our models also achieves top performance without any triangle layer-based pair representation updates. This allows us to train on very large proteins and generate backbones of up to 800 residues, while maintaining des- ignability and diversity, significantly outperforming all previous works. Further, while non-equivariant diffusion models have recently been used as part of AlphaFold3 (Abramson et al., 2024), equivariant methods have been dominant in the unconditional protein structure generation literature. We show that large-scale coneeiivari flow models also succeed on unconditional protein structure generation. We train versions of Proteína with more than 400M parameters, more than 5 5x larger than RFDiffu- sion (Watson et al., 2023), to the best of our knowledge the largest existing protein backbone generator.
Protein structure generators are typically evaluated based on their generated proteins' diversity, novelty and designability (see Sec. 3.5). However, none of these metrics rigorously scores models at the distribution level, although the task of generative modeling is to learn a model of a data distribution. Hence, we introduce new metrics that directly score the learnt distribution instead of individual samples. Similar to the Fréchet Inception Distance in image generation (Heusel et al., 2017), we compare sets of generated samples against reference distributions in non-linear feature space. Since our feature extractor is based on a fold class predictor, we further quantify models' diversity over fold classes as well as the similarity of the generated class distribution compared to reference data's 's classes.
Further, we adjust the flow matching objective to protein structure generation and explore stage-wise training strategies. For instance, using low-rank adaptation (LoRA, Hu al. (2022)) we fine-tune Proteína models on natural, designable proteins. We also develop novel guidance schemes for hierarchical fold class conditioning and successfully showcase autoguidance (Karras et al., 2024) to enhance protein designability. Experimentally, Proteína achieves state-of- ofethe-rr protein backbone generation performance, vastly outperforming all baselines especially in long chain synthesis, and we demonstrate superior control compared to previous models through our novel fold class conditioning.
We present Proteína, novel flow-based generative protein structure foundation model using a new scalable non equivarrian transformer architecture, which we scale to more than 400M parameters. (ūi) We incorporate hierarchical fold class conditioning into Proteína and develop tailored training algorithms and guidance schemes, leading to unprecedented semantic controllability over protein structure generation. In particular, we showcase fold-specific synthesis as well as controlled eahancement of 8-sheets in generated structures. (iii) We introduce several new protein structure generation metrics to complement existing metrics and to better analyze and compare existing models. (fv) We scale training data to almost 21M high-quality synthetic protein structures, and show successful training of models with very high designability on such large data. (v) We achieve state-of-the-an designable and diverse protein backbone generation performance for unconditional and fold class-conditional generation as well as motif-scaffolding. Thanks to our efficient transformer architecture, we scale to an unprecedented length of 800 residues, still producing diverse and designable proteins, vastly outperforming previous works. (vi) For the first time, we demonstrate LoR A-bsed fine-tuning and autoguidance for flow-based protein structure generative models.
Main contributions: (i)
2

---

Published as a conference paper at ICLR 2025
Figure 2: Proteína Samples. Designable backbones generated unconditionally by MMs model (<250 residues).
BACKGROUND AND RELATED WORK
Proteína relies on flow-matching (Lipman al., 2023; Liu et al., 2023; Albergo & Vanden-Eijnden
2023), which models probability density path (x,) that gradually transforms an analytically tractable noise distribution (Pt=a) into a data distribution (Pt=1), following time variable E [o. 1]. Formally, the path pr (x) corresponds to a flow thx that pushes samples from po to pt via p. = [<>]+*po where * denotes the push-forward equation (ODE) dx, v t t)ttt, defined through learnable vector field vī (x,, t) with parameters 0. Initialized from noise Xg ~ Po Xx). this ODE simulates the flow and transforms noise into approxi- mate data distribution samples. The probability density path (x() and the (intractable) ground-truth vector field uy (xx) are related via the continuity equation Öp, (xx) /0t Vx. (x()x)((x)))
practice, the flow is modelled via an ordinary differential
To learn vΩ (xx,t) t) one can employ conditional flow matching (CFM). In CFM, conditioned on data samples X1 ~ Pi (x1), we construct conditional probability paths pe Pe(x, (x1) for which the corresponding ground-truth conditional vector field (xx/xx) is analytically tractable for simple distributions (xx). like Gaussian noise. The CFM objective then corresponds to regressing the neural network-defines approximate vector field v (x,, t) against ur (xx(x1), where the intermediate samples x are drawn from the tractable conditional probability path pr (Xe x and we marginalize over data X1 via Monte Carlo sampling. Since in expectation the CFM objective results in the same gradients as directly regressing against the intractable marginal ground-truth vector field u, (xx). vf (x,, t) learns an approximation of the ground- truth u, (x:).
In practice, the conditional probability paths are defined through an interpolant that connects noise Xo and data samples X1 and constructs intermediate x, via interpolation. We rely on the rectified flow (Liu et al., 2023) (also known as conditional optimal transport (Lipman et al., 2023)) formulation, using a linear interpolant x, = fx + (1 x)x; and the regression target dxtp (xo xx)/dd /dt = X1 Xo. See Sec. 3.2 for our exact instantiation of the CFM objective. Flow-matching is related to diffusion mod- els (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) 1), see App. J; for Gaussian flows the frameworks become equivalent up to reparametrizztions (Kingma & Gao, 2023; Albergo et al., 2023).
Related Work. Two seminal works on protein backbone generation with diffusion models are Chroma (Ingraham al., 2023) and RFDiffusion (Watson et al., 2023), the latter fine-tuning RoseTTAFold (Baek et al., 2021). FrameDiff (Yim et al., 2023b) performs frame-based (Jumper al., 2021) Riemannian manifold diffusion (Huang et al., 2022; Bortoli et al., 2022) to model residue rotations. These works were followed by FoldFlow (Bose et al., 2024) and FrameFlow (Yim et al., 2023a), leveraging Riemanning flow matching (Chen & Lipman, 2024). Meanwhile, Genie (Lin & Alquraishi, 2023) and others (Trippe et al., 2023) generate protein backbones diffusing only residues' Ca coordinates. Proteus (Wang et al., 2024) builds on top of Frfme introducing efficient triangle layers. Recently, FoldFlow2 (Huguet et al., 2024) and Genie2 (Lin et al., 2024) extended training data to the AFDB, although with significantly less data than Proteína. MultiFlow (Camphell et al., 2024), building FrameFlow, jointly generates sequence and structure. Related, Protpardelle (Chu et al., 2024) and the concurrent Pallatom (Qu et al., 2024) generate fully atomistic proteins. The latter uses similar non-equivariant transformer architecture like AlphaFold3 (Abramson et al., 2024), also related to Proteína's 's architecture. Meanwhile, masked language models have been trained on structure tokens, with ESM3 (Hayes et al., 2024) being the most recent and prominent model. Chroma showed classifier- guidance with respect to fold classes. In contrast, we, for the first time, leverage classifier-free guidance using large fold class annotations, and perform thorough quantitative analyses.
PROTEÍNA
3.1 SCALING PROTEIN STRUCTURE TRAINING DATA WITH FOLD CLASSES
Most protein structure generators have been trained on natural proteins, using filtered subsets of the PDB (Berman et al., 2000), resulting in training set sizes in the order of 20k. Recently, some works (Lin et al., 2024; Huguet al., 2024; Qu et al., 2024) relied on the AFDB (Varadi et al., 2021) and in-
3

---

Published as a conference paper at ICLR 2025
Truining Datusets
20,814,855
Mainly helices
DRrM Class Hienrchy
Figure 3: Dataset Statisties. (a) Dataset size comparisons. (b) Sunburst plot of the hierar- chical fold class labels our largest dataset Dam, depicting the hicrarchical label structure and the relative sizes of the three hicrarchical class cat- egories C, A, and T.
10"
Dausi
2,091,500 clustens
Mainly fishheet
69.7 labeled
Mixed a/0
No clustering
99.9 % labeled
10
0/3
588,318
C-levol
225,399
A-level
(4)
PDB
Drs P21M
(b)
I-level
cluded synthetic AlphaFold2 structures (Jumper et al., 2021). Genie2 (Lin et al., 2024) used the largest dataset, i.e. =0.6M synthetic structures. Inspired by the data scaling success of generative models in areas such as image and video generation and natural language synthesis (Brooks et al., 2024; Esser et al., 2024; OpenAl, 2024), we explore scaling protein structure training data even further. The entire AFDB extends to x21 4M structures, orders of magnitude larger than its small subsets used in previous works. However, not all of these structures are useful for training protein structure generators, as they contain low-quality predictions and other unsuitable data. Our main Proteína models trained two datasets, denoted as Drs and D2IM. the latter newly created (data processing details in App. M):
1. Foldseek AFDB clusters Drs: This dataset corresponds to the data that was also used by Genie2, MMseqs2 and the based on sequential filtering and clustering of the AFDB with the et al., 2023). This data structure- based Foldseek (van Kempen et al., 2024; cluster representatives only, i.e. only one structure per cluster. Like Genie2, we use protein lengths between 32 and 256 residues in our main models, leading 588,318 structures in total.
2. High-quality filtered AFDB subset D2IM: We filtered all 2214M AFDB structures for proteins with max. residue length 256, min. average PLDDT 85, max. pLDDT standard deviation of 15, max. coil percentage of 50%, and max. radius of gyration of 3nm. This led to 00,77,4885 structures. We further clustered the data with MMseqs2 (Steinegger & Söding, 2017) using 50% sequence similarity threshold. During training, we sample clusters uniformly, and draw random structures within.
We use DES as, the best of our knowledge, it represents the largest training dataset used in any previous flow- or diffusion-based structure generators. With D2IM we are pushing the frontier of training data scale for protein structure generation. fact, DZIM 35x larger than Drs (see Fig. 3).
Hierarchical fold class annotations. Large-scale generative models in the visual domain typically rely on semantic class- or text-conditioning to offer control or to effectively break down the generative modeling task into set of simpler conditional tasks (Bao et al., 2022). However, existing protein structure diffusion or flow models are either trained unconditionally, or condition only on partially given local structures, for instance in motif scaffolding tasks (Yim al., 2024: Lin et al., 2024).
We propose, for the first time, to instead leverage fold class annotations that globally describe protein structures, akin semantic class text labels of images. We use The Encyclopedia of Domains (TED) data, which consists of structural domain assignments to proteins in the AFDB (Lau et al., 2024b;a). TED uses the CATH structural hierarchy (Dawson al., 2016) to assign labels, where C ("class"") describes the overall secondary-struccure content of domain, A ("architecture") groups domains with high structural similarity, T ("topology/oold)) further refines the structure groupings, and H ("homol-
aa)
(c)
(b)
(d)
(0
Figure 4: Long Proteina Samples. Chain lengths in (a)-(g): [300, 400, 500, 600. 700, 800, 800]. (a) "Mixed a/3"-guided (b) "Mainly 3"-guided. (c) "Mixed a/8"-guided Others unconditional. All samples designable.
4

---

Published as a conference paper at ICLR 2025
ogous superfamily") labels are only shared between domains with evolutionary relationships. Since we are strictly interested in structural modeling, we discard the H level and leverage only C, A, and level labels. We assign labels to the proteins in all datasets, but since TED annotated not all of AFDB, some structures lack CAT labels. Moreover, some labels are less common than others (see Fig. 3); we only consider the main "mainly a", "mainly 8",, and "mixed o/3" C classes. See App. M for details.
3.2 TRAINING OBJECTIVE
We model protein backbones' residue locations through their C. atom coordinates, similar to Lin & Alquraishi (2023); Lin et al. (2024). Note that many works instead leverage so-called frames (Jumper et al., 2021), additionally capturing residue rotations. However, this requires modeling generative process over Riemannian rotation manifolds as well as ad hoc modifications to the rotation generation schedule during inference, which are not well understood (Yim et al., 2023a; Bose et al., 2024; Huguet et al., 2024). We purposedly avoid such representations to not make the framework unnecessarily complicated, and prioritize simplicity and scalability, relying purely on Co backbone coordinates.
Consider the vector of a protein backbone's ' 3D Co coordinates x E R&L,, where is the number of residues. Denote the protein's 's fold class labels as {Cx, Ax, Tx &CAr, and the binned pairwise distance between residues! i and j as (x), Using x = tx + (1 t) Proteína's 's objective then
min e.0-
(xe, x(xx), (Cx, Ax, (CAT) ex-
e)1
Main conditional flow-matching loss, see Sec. 2.
(1)
1(t 0.3) L2
Do,c (x) log Ph.y (x,, (x,, xxx,) {Cx, Ax, Tx Txlcar)
Optional auxiliary binned distogram loss.
Similar to Abramson et al. (2024); Qu et al. (2024), we optionally include cross entropy-based distogram loss, which discretizes pairwise residue distances into 64 bins. The distogram is predicted via a prediction head attached to our architecture's pair representation and only used if this pair representation is updated (see Sec. 3.3). This loss is generally used only for 2 0.3. We also train for self-conditioning, conditioning the model on its own clean data prediction xxx:) x, (1 ()vf(xx 4,6 0, (Cx, Ax. Tx )CT) with probability 0.5. Furthermore, we design a novel (-sampling distribution, p(t) (0, 1) + 0.98 ((1.,, 1.0), tailored to flow matching for protein backbone generation (motivation and discussion in App. K, visualization in Fig. 20, ablation studies in App. L).
Fold-class conditioning. Our fold class labels describe protein structures at different levels of detail, and we seek the ability to both condition on varying levels of the hierarchy, and to also run the model unconditionally. To this end, we propose to hierarchically drop out different label combinations during training. Specifically, with p = 0.5 we drop all labels ((0. 0. CAT),, with p 0.1 we only show the label ((Cx. 0, CAT)) with p = 0.15 we drop only the label ([Cx, Ax, (CAT) and with p = 0.25 we give the model all labels ((Cx. Ax, Tx Tx )CAT). The drop probabilities are chosen such that, on the one hand, we learn a strong unconditional model without any labels. On the other hand, the number of categories increases along the hierarchy, such that we focus training more on the increasingly fine A and T classes, as opposed to conditioning only on the coarser C labels (Fig. 3). Moreover, our approach enables frese guidance (Ho & Salimans, 2021) for all possible levels during inference, combining the unconditional model prediction with any of the label-conditioned predictions (guidance weight w, App. I). Note that, while most training proteins have only a single label, if a protein has multiple domains and corresponding hicrarchical labels, we randomly feed one of them to the model.
3.3 SCALABLE PROTEIN STRUCTURE TRANSFORMER ARCHITECTURE
While previous protein structure generators typically use small equivariant neural networks, we take inspiration from language and image generation (Peebles & Xie, 2023; Ma et al., 2024; Esser et al., 2024) and design a new streamlined non-equivariant transformer, see Fig. 5. It constructs residue chain and pair representations from the (noisy) protein coordinates, the residue indices, the sequence sepa- ration between residues and the (optional) self self-conditioning input. The residue chain representation is processed by stack of conditioned and biased multi- head self-attention layers (Vaswani et al., 2017), using a pair bias via the pair representation, which can be optionally updated, too. At the end, the up- dated sequence representation is decoded into the vector field prediction vē model Proteína's flow.
5

---

Published as a conference paper at ICLR 2025
Chore Superace Reprocntation
(b) Crose Supuence Conditiuiing
(c) Create Pair Reprractttionn
(d) Naraal Netwonk Proceoing Stade
X
ldk.
lid..
Sequence Kepc
Sequence Cood.
Pair Repe
Ers.
Diets
Pair Dios
Enc.
Concal
Concei.
Concat
Adaptive Blased
MLLP
Adtypivo
Linex
MulssHeed Ameettinn
Adaitie Transtionn
Puir Uplate
(optional)
Seguenee
Registons
Sequence Cord.
Zero
Pair
Zero Pad
Sequenece Repe
Paii Reppe
Concal
Coacat
Concor
Sequesce
Sequesee
Vector field v
Painsise diskanzoi (optiomal)
(e) Mdaaptie Blased Muti--Head Aneatioo and Adaptire Trunstion
Secuence Repe
Linsue INN
Splin
Panalll Attention Heads
Scale
Sodarack
Linesr IIN
Concrr Linear
Aulaptive Scale
Adaattive LS
Linerr Switi U
Adeestve Scale
EN
Raps
Lincr
Spli
Sequence Coad
Pai Rege
LN Linor
Figure 5: Proteina's transformer architecture. (a)-(c) We first create sequence representation, sequence conditioning features, and pair representation. (d) They processed by conditioned and biased (through the pair representation) multi-head attention layers, described in (e). We variant of QK normalization, applying LayerNorm (LN) to the Q and K inputs the attention operation, before the mulli-head split. Optionally, the pair representation can be updated. See App. N for the Pair Update, Adaptive LN, and Adaptive Scale modules.
related architecture has recently been introduced by AlphaFold3 (Abramson et al., 2024), and is used concurrently in Pallatom (Qu et al., 2024). Our design features some additional components: (i) As discussed, we condition on hierarchical fold class labels. They are fed to the model through concate- nated learnable embeddings, injected into the attention stack via adaptive layer norms, together with the f embedding. (ii) Following best practices from language and vision, we extend our sequence rep- resentation with auxiliary tokens, known as registers (Darcet et al., 2024), which can capture global in- formation or act as attention sinks (Xiao et al., 2024) and streamline the sequence processing. (iii) We use a variant of QK normalization (Dehghani et al., 2023) to avoid uncontrolled attention logit growth. While our models are smaller than the large models in vision and language, we train with relatively small batch sizes and high learning rates, where similar instabilities can occur (Wortsman et al., 2024). (iv) All our attention layers feature residual connections miithttions-withouut, we were not able to train stably (AlphaFold3 is ambiguous regarding their use of such residuals). (v) We use triangle multiplicative lay- (Jumper et al., 2021) optional add-on only to update the pair representation. While triangle lay- ers have been shown to boost performance (Jumper et al., 2021; 1 Lin et al., 2024; Huguet et al., 2024), they highly compute and memory intensive, limiting scalability. Hence, in Proteína avoid their usage as the driving model component and carry out most processing with the main transformer stack.
AlphaFold3 showed that non-equivariant diffusion models can succeed in protein folding, but they rely on expressive amino acid sequence and MSA embeddings. We instead learn the distribution of protein structures without sequence inputs. For this task, the best of our knowledge, almost all related works used equivariant architectures, aside from the concurrent Pallatom (Qu et al., 2024) and Prot- pardelle (Chu al., 2024). To nonetheless learn equivariance, we center training proteins and augment with random rotations; in App. E we show that our model learns an approximately SO(3)) vector field. We train models with up to Rx400M parameters in the transformer and x 17M in the triangle layers, which, we believe, represents the largest protein structure flow or diffusion model.
3.4 SAMPLING
New protein backbones can be generated with Proteína by simulating the learnt flow's ODE, see Sec. 2. Since our flow is Gaussian, there exists connection between the learnt vector field and the corresponding score s(x.) == Vx, log pr (x.)) (Albergo et al., 2023; Ma et al., 2024),
1 (x, 2) -x
where we use è as abbreviation for all conditioning inputs (see Sec. 3.2). This allows us to construct stochastic differential equation (SDE) that can be used as a stochastic alternative to sample Proteína,
dx, vx( d) + g()ss (x,, (xx, 229(()7 dWe.

---

Published as a conference paper at ICLR 2025
TIM Berrel C.A.E 3,0039
Besa Damel
Rosensoon Fold C.A.T.
Immuncglabulin Domain
Rare class POB, CUmmION in AFDB C.A.T
Mainly Beta
Figure 6: Fold class-conditional Generation with M model (Sec. 4.3). All samples are designable and correctly re- re-classified (App. D). The used C.A.T fold class conditioning codes given below fold names.
where W, is Wiener process and g(t) scales the additional score and noise terms, which corresponds to Langevin dynamics (Karras et al., 2022). Crucially, we have introduced a noise scaling parameter y. For y=1, the SDE has the same marginals and hence samples from the same distribution as the ODE (Karras et al., 2022; Ma et al., 2024). However, it is common in the protein structure generation literature to reduce the noise scale in stochastic sampling (Ingraham et al., 2023; Wang et al., 2024; Lin et al., 2024). This is not a principled way to reduce the temperature of the sampled distribution (Du et al., 2023). but can be beneficial empirically, often improving designability the cost of diversity. Fold label conditioning is done via classifier-free guidance (CFG) (Ho & Salimans, 2021), and we also explore autoguidance (Karras et al., 2024), where model guided using a "bad" version of itself. In unifying formulation, we can write the guided vector field as
e guiidd (xe, 2) v. + (1- w) [11 vvv 0) 8, (x,, a)
where w 2 defines the overall guidance weight and E [0. interpolates between CFG and autoguidance. An analogous equation holds for the scores sy é). To the best of our knowledge, no previous works explore CFG or autoguidance for protein structure generation. More details in App. I.
3.5 PROBABILISTIC METRICS FOR PROTEIN STRUCTURE GENERATIVE MODELS
Protein structure generators are scored based on their samples' designability, diversity and novelty (see App. F). However, designability relies auxiliary models, ProteinMPNN (Dauparas et al., 2022) and ESMFold (Lin et al., 2023), with their own biases. Morcover, we cannot necessarily expect to maximize designability by learning better generative model, because not even all training proteins are designable (Lin et al., 2024). Next, diversity and novelty are usually only computed among designable samples, which makes them dependent the complex designability metric, and diversity and novelty do otherwise not depend on quality. Therefore, we propose new probabilistic metrics that offer complementary insights. We suggest more directly quantify how well model matches relevant reference distribution. Specifically, we first train fold class predictor (+(-Ix) with features 6xx) for all CAT hierarchy levels (Sec. 3.1). Leveraging this classifier, we propose three new metries:
Fréchet Protein Structure Distance (FPSD). Inspired by the FID score (Heusel et al., 2017), we embed generated and reference structures into the feature space of the fold class predictor and measure the Wasserstein distance between the feature distributions, modeling them as Gaussians. Defining the generated and the reference set of protein structures as {x)
gen and (x) respectively, have
FPSD (PSD((x)gem; hx) (|(()))) -1 (o000) 112 (Z((0())))) + Eat00m (0(0)) ((000)a)))
An accurate fold class predictor must learn expressive feature representation of protein structures. Hence, we argue that these feature embeddings must be well-suited for fine- ane-grained reasoning about protein structure distributions, making fold class predictor an ideal choice embedding model.
Fold Jensen Shannon Divergence (fJSD). We also directly compare the marginal predicted categori- cal fold class distributions of generated and reference structures via the Jensen Shannon Divergence,
(JSD(()) (xrr == 10 x (SD(Exx-1x)p.. (tto: (xx
Note that we can evaluate this fJSD metric at all levels of the predicted CAT fold class hierarchy, allowing us to measure distributional fold class similarity at different levels of granularity. In practice, in this work report the average over all levels in the interest of conciseness.
Fold Score (IS). Inspired by the Inception Score (Salimans et al., 2016), we propose a Fold Score
[S((x)eeu == exp (Ex (x) DKI. (= (xo
P
7

---

Published as a conference paper at ICLR 2025
Table 1: Proteina's 's unconditional backbone generation performance compared to baselines. All models and baselines luned for designability via noise scaling or inference rotation annealing, not sampling full distribution. For metric evaluation details see App. F and App. G. Best scores bold, sccond best underlined.
Model
Design- ability (8)1
Diversity
Novelty
FPSD
BSD PDBL AFDBL
Sec. Struct. %
Clussert
TM-S
PDB]
AFDBL
PDB
AFDBI
(C/A/T))
(a/
Unconditonal generation MC
denotes
model sariont, y the noive for Protetna
FramcDiff
65.4
0.39 (126)
0.40
0.73
0.75
194.2
258.1
2.4/5578/ 23.35
1.04
1.42
61./11.22
FoldFlow (base)
96.6
0.20 (98)
0:45
0.75
0.79
601.5
566.2
1.0//.79/9.77
3.18
3.10
87.5/0.4
FoldHow (stoc.
97.0
0.25 (121)
0.44
0.74
0.78
543.6
520.4
1.21/2.9/
01/99/11: 11.59
3.60
2.71
86.1
FoldFlow (OT)
97.2
0.57
0.41
431.4
414,1
1.35/3.1// 13:62
2.90
2.32
82.7/2.0
FrameFlow
(236)
0.36
124.9
199.9
2.52/588/225
ESM3
22.0
0.58 (64)
0.42
933.9
855.4
3.19/671/17.71
1:53
0.98
64.5/8.5
Chroma
74.8
0.51 (190)
0.38
0.69
0.74
189.0
184.1
2.34/4.95/
18:15
1.00
1.08
69.0/ 12.5
RFDiffusion
94.4
0.46 (217)
0.42
0.71
0.77
253.7
252.4
/5.06/ 19.83
1:21
1.13
67.3
Proteus
94.2
0.22(100)
0:45
0.74
0.76
225.7
226.2
2.26 16.22
1.41
1.37
73.1
Gonie2
(281)
0.38
313.8
1.55/366//
72.7
Mas 7-0.5
98.2
(239)
0.37
411.2
392.1
/.9//511/
16.79
1.96
1.53
Maaa 1-0.45
96.4
(306)
0.36
0.69
0.25
388.0
368.2
2.06/ 5.32/ 19:05
1.65
1.23
69.176.9
Mas. 7-0.5
91.4
0.71 (323)
0.35
0.59
0.75
380.1
359.8
2.10/ 5.18/ 19.07
1.55
1.15
7.2
Mg 9:445
(292)
0.36
322.2
306.2
1,80/4.22/
1.84
1.36
71.3/5.
Mane 3=0.3
99.0
0.30 (150)
0.39
0.81
0.84
280.7
319.9
2.05
5.900
1,66
1.81
62.2/99
MaiM. 3-0.6 0.6
84.6
0.59 (294)
0.35
0.72
0.77
280.7
301.8
2.31 5.76/ 30.11
0.89
0.95
58.7 12.0
MESRA. y-0.5
96.5
0.43 (208)
0.38
0.75
0.78
274.1
336.0
2.40/6.26/ 26.93
0.79
0.93
54.3 13.0
A higher score is desired. The IS is maximized when individual sample's i class predictions p (-x) are sharp, while the marginal distribution Px- has high entropy and covers many classes. Hence, this score encourages diverse generation, while individual samples should be of high quality to enable confident predictions under the classifier. The iS can also be evaluated for all CAT levels.
Our new metrics are probabilistic and directly score generated proteins at the distribution level, offer- ing additional insights. They can help model development, but are not meant as optimization targets to rank models. A protein designer in practice still cares primarily about designable, diverse and novel proteins. Therefore, we did not indicate bold/underlined scores for these metries in the evaluation ta- bles in Sec. 4. The new metrics are evaluated with 5,000 samples in practice. In App. G, we provide de- tails and extensively validate the new metrics on benchmarks, to establish their validity and sensitivity.
4 EXPERIMENTS
We trained three main Proteína models (M), all with the possibility for conditional and unconditional generation (Sec. 3.2): (i) Model Mys is trained on Dps with 200M parameter transformer and 15M parameters in triangle layers. (ii) The more efficient Mas is trained on Dps with a 200M parameter transformer without any triangle layers nor pair representation updates. (iii) MZ1M is trained on DZIM with a 400M parameter transformer and 1 15M parameters in triangle layers. Details in App. O.
4.1 PROTEIN BACKBONE GENER BENCHMARK
In Tab. 1, we compare our models' performance with baselines for protein backbone generation (see Sec. 2). We select all appropriate baselines for which code was available, as we require to generate samples to fairly evaluate metries and follow a consistent evaluation protocol (described in detail in Apps. F and G). We did not evaluate Genie, as it is outdated since Genie2, and we were not able to compare to the recent FoldFlow2, as no code available. We also evaluated ESM3 as a state-of-the thet masked language model that can also produce structures. Baseline evaluation and experiment details in Apps. O and P. All models and baselines in Tab. are adjusted for high designability via rotation annealing or reduction of the noise scale during inference. Tab. I findings:
Unconditional generation. (i) MPs can be tuned during inference for different designability, diversity and novelty trade-offs (varying 7). It outperforms all baselines in designability and diversity, while performing competitively on novelty, only behind Genie2 and FrameFlow for AFDB novelty (model samples in Fig. 2). (ii) Mg still reaches 93.8% designability and outperforms all baselines on diversity, despite not using any expensive triangle layers and no pair track updates- in contrast
Table 2: Proteina's 's and Chroma's 's fold class-conditional backbone generation performance.
Model
Design- ability (5)t
Diversity
Novelty vs. AFDRJ
FPSD vs. ARDR
SS (C) (C/A/TI
FISD vs. PDR. AFDR
Sec. Stract. % (o/
Cluster?
TM-Sc.
PDR]
PDRL
Fold closs conditional generation with Proteitu model Map and CFG with guislance weight u and noise scalc
0.4.
Chrooma
$7.0
0.65 (186)
0.37
0.68
0.73
157.8
131.0
2.36/5.11 /9.82
0.81
0.77
70.27 11.1
91.4
(262)
0.34
0.22
0.81
121.1
127.6
2.30/693/3L.11
0.58
0.52
57.17 13.7
M 1.5
89.2
0.57 (252)
0.33
0.77
0.81
106.1
113.5
2.58/7.36/322
0.19
0.47
56.07 14.
Mag 2.0
(225)
1030
2.67/7.5533374
15.7/15.7
8

---

Published as a conference paper at ICLR 2025
Table 3: Proteína's and GENIE2's backbone generation performance when evaluated to sample full distribution,
L.e. no noise or temperature reduction. Metric details in Apps. F and G. Best scores bold, second best underlined.
Model
Design- abilit 0437
Diversity
Novelly va PDOL
FPSD vs. AFDB,
68
BSD vs.
Sec. Seniet &
Cluster
TM Sc.
AFDOL
PDBL
(C/A/T)
PDBL
AFDBL
ta/ 53
Unconditiona generasiion Mt deanies Proteinu modil voriont, Sunpling for Protein preforoned sning generative ODE (Ape i. for GENIE wish their appvooch
Gasic?
19.0
0.1 (77)
0.35
1.66
0.72
104.7
20.94
2.24 22244449/1 1283
0.75
0.16
65.0/75
Maa
19.6
8393 (91)
0.32
1.66
B.74
85.
21.41
15111588/1 /51//58//235
0.59
0.9
482/152
Mom
35.4
0.6 (115)
0.34
0.74
0.79
50.14
44.98
2.51 25116646 30.65
0.32
023
557/118
Musa
44.2
0.58 (129)
0.35
0.75
68.56
138.5
261/7.193 38.64
082
47.2/134
Rdd condenndiioa peneiaation with Proteila model My and CFG with gaidunce woiglt w. Sompling is prrformud d using geveraiie ODE (App. I.
24.2
74 (00)
0.29
0.73
0.79
41.46
19:45
26646
266/675/2614
6464
0.40
0.12
14.71147
to all existing models. (iii) M2IM achieves state-of-thh-an art 99.0% designability, while generating less diverse structures. This is expected, as it is trained on the very large, yet strongly filtered D2IM- Models trained on Prs exhibit higher diversity, because no radius of gyration or secondary structure filtering was used during data curation. With DIIM we were able to prove that one can create high- quality datasets, much larger than Drs. from fully synthetic structures that can be used for training generative models producing almost entirely designable structures. Furthermore, our discussed findings represent an important proof that non-equivariant architectures can achieve state-of-the-ae performance on protein backbone generation. All baselines use fully equivariant networks.
PDB-LoRA MLORA. We used LoRA (Hu et al., 2022) to fine- fine-tune_ Mys on a small dataset of only des- ignable proteins from the PDB (App. M.I). As expected, designability improves, diversity decreases, FPSD and fJSD with respect to PDB decrease, and FPSD and fJSD with respect to AFDB increase. This experiment showcases how model that is trained only on synthetic data can be successfully fine- tuned on natural proteins, and the metrics validate that the generated samples indeed are closer to the PDB in distribution. Moreover, the amount of 3-sheets doubles, an important aspect, due to the under- representation of 3-sheets in many protein design models. To the best of our knowledge, this is the first time that such LoRA fine-tuning has been demonstrated for protein structure flow or diffusion models.
Fold-Clase conditional generation and new metries. Next, we evaluate our fold class- conditiona model MMp as well as Chroma, the only baseline that also supports class-conditional sampling (see Tab. 2). We feed the labels from the empirical label distribution of Drs to the models. This enforces diversity across different fold structures, which reflected in the metrics. Compared to unconditional generation, our conditional model achieves state-of-the- ahr TM-Score diversity, while also reaching the best FPSD, fS and fJSD scores, thereby demonstrating fold structure diversity (fS) and better match in distribution to the references (FPSD, fJSD). Moreover, this is achieved while maintaining very high designability. Further, the effect is enhanced by classifier-fi alas-conndiiooing also significantly improves the 8-shect content of the generated backbones. Note that, however, the model does not improve novelty. Novelty can be at odds with learning a better model of the training distribution- the goal of any generative model as it rewards samples completely outside the training distribution. That motivates our new metrics, which are complementary, as clearly shown in the dlass-conditooning case. Chroma has very poor designability and is outperformed in TM-score diversity and the number of designable cluster. Moreover, we show in App. D. that, in contrast to Proteína, Chroma fails to perform accurate fold class- specissc generation by analyzing whether generated proteins correspond to the correct conditioning fold classes.
guidance (w 2 1.0). Fold
Full distribution modeling. Most models use temperature and noise scale reduction or rotation schedule annealing during inference to increase designability the cost of diversity. In Tab. 3, we analyze performance when sampling the entire distribution instead, comparing to Genie2 also sampled at full temperature. Genie2 produces the least designable samples. Mys performs overall on-par with or better than Genie2, but M2IM has much higher designability and LoRA fine-tuning also gives a big boost. Moreover, almost all new distribution metrics (FPSD, IS, (JSD) are significantly improved over Tab. 1, we now sample the entire distribution. This is only fully captured by our new metrics.
Deligabbligg
55-
53
Uncendtooal greeetion
Controonal grnertion
100
1.25
2/0 Ausouidacce Weists
iis
Figure 7: Designability of ManM ODE samples with autoguidance.
Autoguidance. In Fig. 7, we show a case-study of autoguidance (Karras et al. (2024), see App. I) for protein backbone generation with our M2IM model in full distribution mode (ODE), using an early training checkpoint as "bad" guidance checkpoint. We can significantly boost designability, up to 70% in conditional generation, far surpassing the results in Tab. 3. To the best of our knowledge, this the first proof of principle of autoguidance in the context of protein structure generation.
9

---

Published as a conference paper at ICLR 2025
4.2 LONG CHAIN GENERATION
While our main models are trained on proteins of up to 256 residues, we fine-tune the Mps model on proteins of up 768 residues (App. O for details). In Fig. 8, show our model's performance on long protein backbone generation of up to 800 residues (samples in Fig. 4.). While Genie2 exhibits superior diversity at 300 residues, beyond that Proteína significantly out- performs all baselines by large margin, achieving state-of-the-ard results. At very long lengths, all baselines collapse and cannot produce diverse designable proteins anymore. In contrast, for our model most generated backbones are designable even at length 800 and we still generate many diverse proteins, as measured by the number of designable clusters. To the best of our knowl- edge, no previous protein backbone generators successfully trained on proteins up to that length.
Divessiry (HDsigmnble Clusters 1
is possible for us because Mpsstt does not use any expen- sive triangle layers and no pair
90-
70-
Proteino
60-
Proteus
track updates, relying only on our novel efficient transformer,
Designbiiity
60
Genic2
FoldFlow OT
REDiffision
whose scalability this experi- ment validates. We envision that such long protein back- bone generation unlocks new large-scale protein design tasks. Note that long length genera- Figure 8: Proteína long backbone generation performance (also App. O.5).
30
FrameFlow
10
10
Chroma
PrameDifff
400 500
700
800
400
500
600
700
800
ESM-3
Number of Residues
Number of Resisuus
tion can be combined with our novel fold class conditioning, too, offering additional control (Fig. 4).
4.3 FOLD CLASS-SPECIFIC GUIDANCE AND INCREASED 8-SHEETS
problem that has plagued protein structure generators for a long time that they typically produce much more a- a-helices than 3-sheets (Tabs. 1 and 3). Our fold class conditioning offers address this without the need for fine-tuning (Huguet et al., 2024). In Tab. 4, we guide the Mas model with respect to the main high-level C level classes that determine secondary structure content (details App. O). When guiding into the "mixed oz/3" and especially "mainly 3 classes, f9-hhets increase dramatically in contrast to unconditional baselines in Tab. 1. Importantly, the samples remain designable. As we restrict generation to specific classes, diversity slightly decreases as expected, but we still generate diverse samples.
tool to
"mainly a" generation and also compared to all
Aside from C-level guidance to achieve controlled secondary structure diversity, we can also guide with respect to inter- esting or relevant
Table 4: Guiding Proteina into the C-Ievel classes.
Novelly
Street
ability
Fabbeek1
TM.Sc..
PDB]
AFDB]
Uncondiioona
(305)
and T-level classes. In Fig. 6, we show examples of guidance into different fold classes from the CAT
MMinly "Manly
0.37
(215)
"M
97.8
0.42 (207)
0.37
0.73
0.78
20.5
hierarchy, demonstrating that Proteína offers unprecedented control over protein backbone generation. We would also like to point to App. D, where we extensively validate that our novel fold class condi- tioning correctly works by re-classifying generated conditional samples with our fold class predictor.
Further Proteína samples in App. A. Proteína also achieves state-of-ther the performance in motif- scaffolding (App. B). Speed and efficiency analysis in App. C.2. More experiments in Apps. E and I.
CONCLUSIONS
We have presented Proteína, a foundation model for protein backbone generation. features novel fold class conditioning, offering unprecedented control over the synthesized protein structures. In compre- hensive unconditional, class-conditional and motif scaffolding benchmarks, Proteína achieves state-of- the-art performance. Our driving neural network component is scalable non-equivariant transformer, which allows us to scale Proteína to synthesize designable and diverse backbones up to 800 residues. We also curate 21M-si high-qualis dataset from the AFDB and, scaling Proteína to over 400M pa- rameters, show that highly designable protein generation is achievable even when training on synthetic data such unprecedented scale. For the first time, we demonstrate not only classifier-free but also au- toguidance as well as LoRA-base fine-tuning in protein structure flow models. Finally, we introduce new distributional metries that offer novel insights into the behaviors of protein structure generators. We hope that Proteína unlocks new large-scale protein design tasks while offering increased control.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We ensure that our data processing, network architecture design, inference-ting time sampling, sample evaluations, and baseline comparisons are reproducible. Our Appendix offers all necessary details and provides comprehensive explanations with respect all aspects of this work.
In addition to Sec. 3. in App. M we describe in detail how our Drs and D2IM datasets are created, processed, filtered and clustered, which includes the hierarchical CAT fold class labels that use. Dataset statistics are given in Fig. 3, which can serve as reference. Additional tools that we use during data processing and evaluation, such as MMseqs2 (Steinegger & Söding, 2017) and Foldseek (van Kempen et al., 2024; we cite them accordingly. Hence, our data processing pipeline is fully reproducible. Next, our new transformer architecture is explained in detail in Sec. 3.3 and App. N, with detailed module visualizations Figs. and 24 and network hyperparameters in App. O. Inference time sampling is described in Sec. 3.4 with additional algorithmic details in App. I. The corresponding sampling hyperparameters are provided in App. O. Furthermore, how we evaluate the traditional protein structure generation metrics is explained in detail in App. F and App. F.I, while our newly proposed metrics (Sec. 3.5) are validated and explained in-depth in App. G. Moreover, to ensure our extensive baseline comparisons are also reproducible, the corresponding details are described in App. P.
et al., 2023), are publicly available and
For model and code release please see Proteina's GitHub repository https: /ttptuu com/ NVIDIA-Digit BtllggiiaalBiipooooooooo Bitpt//iii/poo/aaa///ooeoeea as well our project page httpss//research. nvidia http:///aaaaggeeaarrrrrrreeeeeee
ETHICS STATEMENT
Protein design has been a grand challenge of molecular biology with many promising applications benefiting humanity. For instance, novel protein-based therapeutics, vaccines and antibodies created by generative models hold the potential to unlock new therapies against disease. Moreover, carefully engineered enzymes may find broad industrial applications and serve, for example, as biocatalysts for green chemistry and in manufacturing. Novel protein structures may also yield new biomaterials with applications in materials science. Beyond that, deep generative models encoding a general understanding of protein structures may improve our understanding of protein biology itself. However, it is important to be also aware of potentially harmful applications of generative models for de novo protein design, for instance related to biosecurity. Therefore, protein generative models generally need to be applied with an abundance of caution.
ACKNOWLEDGMENTS
We would like to thank Pavlo Molchanov, Bowen Jing and Hannes Stärk for helpful discussions. We also thank NVIDIA's compute infrastructure team for maintaining the GPU resources we utilized. Last, not least, thanks to all computational colleagues who make their tools available, to all experimental colleagues who help advancing science by making their data publicly available, and to all those who maintain the crucial databases we build our tools on, like the RCSB PDB, the AFDB, and UniProt.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung. Michael O'Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvilè Zemgulyte, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridgland, Alexey Cherepanov, Miles Congreve, Alexander Cowen-Rivers, Andrew Cowie, Michael Fig- urnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisun- daram, Catherine Tong. Sergei Yakneen, Ellen D. Zhong, Michal Zielinski, Augustin Zidek, Victor Bapst, Pushmeet Kohli, Max Jaderberg, Demis Hassabis, and John M. Jumper. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630:493-50M 2024. 2, 3, 3,
11

---

Published as a conference paper at ICLR 2025
Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X. Lu, Nicolo Fusi, Ava P. Amini, and Kevin K. Yang. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, 2023. 1
Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint (tti::2333008797,, 2023, 3, 6. 35, 39, 40
Michael Samuel Albergo and Eric Vanden-Eijnded Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations (ICLRR, 2023. 3, 35, 39, 40
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 929-947, 2024. 24
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee. Jue Wang, Qian Cong, Lisa N. Kinch, R. Dustin Schaeffer, Claudia Millan, Hahnbeom Park, Carson Adams, Caleb R. Glassman, Andy DeGiovanni, Jose H. Pereira, Andria V. Rodrigues, Alberdina A. van Dijk, Ana C. Ebrecht, Diederik J. Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K. Rathinaswamy, Udit Dalwadi, Calvin K. Yip, John Burke, K. Christopher Garcia, Nick V. Grishin, Paul D. Adams, Randy J. Read, and David Baker. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557):871-8877 2021. 3
Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu, Why are conditional generative models better than unconditional ones? arXiv preprint arXiv:2212.00366, 2022.
Inigo Barrio- Heriid-Hernanezz, Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron L. M. Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering predicted structures at the scale of the known protein universe. Nature, 622:637-64 2023. 4, 11, 43, 44
Helen M. Berman, John D. Westbrook, Zukang Feng, Gary Gilliland, Talapady N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The protein data bank. Nucleic Acids Research, 28(1):235 42, 2000. 3
Valentin De Bortoli, Emile Mathieu, Michael John Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Joey Bose, Tara Akhound Sudeeh, Guillaume Huguet, Kilian Fatras, Jarrid Scctor-Brooks Cheng- Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael M. Bronstein, and Alexander Tong. sto(3)-stcchastic flow matching for protein backbone generation. In The Twelfth International Conference on Learning Representations (ICLR), 2024. 1, 3, 5, 27, 28, 48
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr. Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https: /wpp/aipa...////////////.. ctm./cccoo/ooo research/ video- video-generate
generat ion dolooooooeee models-as asss wwwwwrlllee ld- simulators. 1,4
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Proceedings of the s1 International Conference on Machine Learning (ICML), 2024. 3
Ricky T. Q. Chen and Yaron Lipman. Flow matching on general geometries. In The Twelfth International Conference on Learning Representations (ICLR), 2024. 3
Alexander E. Chu, Jinho Kim, Lucy Cheng, Gina El NesT, Minkai Xu, Richard W. Shuai. and Po-So Ps-Ssu Huang. An all-atom protein generative model. Proceedings of the National Academy of Sciences, 121 (1(7))x2311500122 1500121, 2024. 3,
12

---

Published as a conference paper at ICLR 2025
Jose M Dana, Aleksandras Gutmanas, Nidhi Tyagi, Guoying Qi, Claire O'Donovan, Maria Martin, and Sameer Velankar. Sifts: updated structure integration with function, taxonomy and sequences resource allows 40-fold increase in coverage of structure-based based annotations for proteins. Nucleic acids research, (D811)):422-D48) 2019. 44
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Representations (ICLR), 2024. 6
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai. Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et Robust deep learning- protein sequence design using proteinmpnn. Science, 378(661 :):4655)):46666 2022. 7, 28
Natalie L. Dawson, Tony E. Lewis, Sayoni Das, Jonathan G. Lees, David A. Lee, Paul Ashford, Christine A. Orengo, and Ian P. W. Sillitoe. Cath: expanded resource predict protein function through structure and sequence. Nucleic Acids Research, 45:D289 D295, 2016. 2, 4, 31, 44
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner. Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenat- ton, Lucas Beyer, Michael Tschannen, Anurag Arnb,, Xiao Wang. Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov. Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023. 6
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: compositional generation with mergy--based diffusion models and meme. In International Conference on Machine Learning (ICMI), 2023. 7
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Toward understanding the language of life through self supervipd learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(10):7112-712) 2022. 1
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transform- ers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024. 1, 4, 5, 23, 40, 41
Margherita Grandini, Enrico Bagli, and Giorgio Visani. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.0577 2020. 32
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with a language model. bioRxiv, 2024. 3
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017. 2, 7, 29
Jonathan Ho and Tim Salimans. Classifier-Fres Diffusion Guidance, NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5, 7, 36, 37, 40
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in 3, 3.
Neural Information Processing Systems (NeurIPS), 2020.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference
Learning Representations (ICLR), 2022. 2, 9, 47
13

---

Published as a conference paper at ICLR 2025
Chin-Wor Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron Courville. Rieman- nian diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
3
Po-Ssu Huang, Scott E. Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537:320-322 2016. 1
Guillaume Huguet, James Vuckovic, Kilian Fatras, Eric Thibodeau-Laufen Pablo Lemos, Riashat Islam, Cheng-Harr Hao Liu, Jarrid Rector- Rector-Brooks, Tara Akhound -Sadegh, Michael Bronstein, Alexander Tong, and Avishek Joey Bose. Sequence-augmmetted backbone generation. arXiv preprini urXiv:2405.20315. 2024. 3, 5, 6, 10
augmented se(3)-flow --flww matching for conditional protein
John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang. Vincent Xue, Fritz Obermeyer, Andrew Beam, and Gevorg Grigoryan. Illuminating protein space with programmable generative model. Nature, 623: 120:1070-1078, 2023. 1, 3, 7, 26
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583-6555 2021. 1, 2, 3, 4, 5, 6, 45
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems (NewrIPS), 2022. 7, 40
Tero Karras, Miika Aittala, Tuomas Kynkääänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. arXiv preprint arXiv:2406.025500 2024. 2, 7, 9, 36, 37, 40, 48
Hyunbin Kim, Milot Mirdita, and Martin Steinegger. Foldcomp: a library and format for compressing and indexing large protein structure sets. Bioinformaties, 39t4), 03 2023. 43
Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014. 46
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023. 3, 40
Brian Kuhlman and Philip Bradley, Advances in protein structure prediction and design. Nat. Rev. Mol. Cell Biol., 20:681-60 1677, 2019.
Patrick Kunzmann and Kay Hamacher. Biotite: unifying open source computational biology framework in python. BMC bioinformatics, 19:1-8, 2018. 28
Gilles Labesse, N Colloc'h, h, Joël Pothier, and J-P Mornon. P-sea: a new efficient assignment of secondary structure from ca trace proteins. Bioinformatics, 13(3):291-29) 1997. 28
A. M. Lau, N. Bordin, S. M. Kandathil, I. Sillitoe, P. Waman, J. Wells, C. A. Orengo, and D. Jones. The encyclopedia of domains (ted) structural domains assignments for alphafold database v4 [data set]. Zenodo, 2024a. 4
A. M. Lau, N. Bordin, S. M. Kandathil, I. Sillitoe, V. P. Waman, J. Wells, C. A. Orengo, and D. T. Jones, Exploring structural diversity across the protein universe with the encyclopedia of domains. bioRxiv, 2024b. 4, 45
Yeqing Lin and Mohammed Alquraishi. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. In Proceedings of the 40th International Conference Machine Learning (ICML), 2023. 1, 3,
14

---

Published as a conference paper at ICLR 2025
Yeqing Lin, Minji Lee, Zhao Zhang, and Mohammed AlQuraishi. Out of many, one: Designing and scaffolding proteins at the scale of the structural universe with genie arXiv preprint arXiv:2405.15489 2024. 1, 4. 5, 6, 7, 19, 23, 24, 27, 28, 34, 44
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637: --30, 2023. 1, 7, 28
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Repre- sentations (ICLR), 2023. 3, 35,
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023. 3, 35, 39
Loredana Lo Conte, Bart Ailey, Tim Hubbard, Steven Brenner, Alexey G Murzin, and Cyrus Chothia. Scop: a structural classification of proteins database. Nucleic acids research, 28(1): 257-259. 2000.
Nanye Ma, Mark Goldstein, Michael Albergo, Nicholas M. Boffi, Eric Handen-Fijnden and Saining Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740. 2024. 2, 5, 6, 7, 36, 40
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2024. 1.4
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 39
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) 2023. 2,
Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky Q Chen. Multisample flow matching: Straightening flows with minibatch couplings. In Proceedings of the 40th International Conference Machine Learning (ICML), 2023. 40
Wei Qu, Jiawei Guan, Rui Ma, Ke Zhai, Weikun Wu, and Haobo Wang. P(all-atom) is unlocking new path for protein design. bioRxiv, 2024. 3, 5,
Janes S. Richardson and David C. Richardson. The de novo design of protein structures. Trends in Biochemical Sciences, 14(7):304-309 1989. 1
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 2.29
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 45
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In International Conference on Machine Learning (ICML), 2015. 3,
Yang Song, Jascha Sohl- DickkDiitt,,,, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference Learning Representations (ICLR), 2021. 3, 39, 40
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for
the analysis of massive data sets. Nat Biotechnol. 35: 15:1622-1028 2017. 4, 11, 43, 44
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568: 127063, 2024. 42
15

---

Published as a conference paper at ICLR 2025
Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research (TMLR), 2024. 40
Brian L.. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference Learning Representations (ICLR), 2023.
Michel van Kempen, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron L. M. Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nat Biotechnol., 42:243-246, 2024. 4, 28,
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Zídek, Tim Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, and Sameer Velankar. Alphafold protein structure database: Massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 50:D439-D444, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 5
Sameer Velankar, José M Dana, Julius Jacobsen, Glen Van Ginkel, Paul J Gane, Jie Luo, Thomas J Oldfield, Claire D'Doova,, Maria-Jesus Martin, and Gerard Kleywegt. Sifts: structure integra- tion with function, taxonomy and sequences resource. Nucleic acids research, 41(D1):D433-449, 2012. 44
Chentong Wang, Yannan Qu, Zhangzhi Peng, Yukai Wang, Hongli Zhu, Dachuan Chen, and Longxing Cao. Proteus: Exploring protein structure generation for enhanced designability and efficiency. bioRxiv, 2024. 3, 7
Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahem, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel. Samuel J. Pellock. Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature, 120:1089-118 2023. 2, 19
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference Learning Representations (ICLR), 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024.
Jason Yim, Andrew Campbell, Andrew Y. K. Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S. Veeling, Regina Barzilay, Tommi Jaakkola, and Frank Noé. Fast protein backbone generation with se(3) flow matching. arXiv preprint arXiv:2310.05220 2023a. 3, 5, 27
Jason Yim, Brian L. Trippe. Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. SE(3) diffusion model with application to protein backbone generation. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023b. 1, 3, 27, 28
16

---

Published as a conference paper at ICLR 2025
Jason Yim, Andrew Campbell, Emile Mathieu, Andrew Y. K. Foong, Michael Gastegger, Jose Jimenez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S. Veeling, Frank Noe, Regina Barzilay, and Tommi Jaakkola. Improved motif-scaffolding with SE(3) flow matching. Transactions on Machine Learning Research, 2024. 4
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In The Eleventh International Conference on Learning Representations, 2023, 31
17

---

Published as a conference paper at ICLR 2025
Appendix
Additional Proteína Sample Visualizations
B Motif-Scaffolding with Proteína
19
B.1 Motif-Scaffolding Implementation
19
B.2 Motif-Scaffolding Results
19
C Scaling and Efficiency Analysis
23
C.I Scaling Flow Matching Training
23
C.2 Model Parameters, Sampling Speed Memory Consumption
24
D Validating Fold Class Conditioning via Re-Classification
26
D.1 Re-Classification Analysis of Fold
Chroma Sampling
26
E Equivariance Analysis
27
F Established Metrics: Designability, Diversity, Novelty & Secondary Structure Foldseek Commands for Cluster Diversity and Novelty Calculations
27
29
New Metrics: FPSD, IS and FJSD
29
G.1
Motivation
29
G.2
Metric Definition
29
G.3 Fold Classifier Training
31
G.4 Metric Validation
................................
32
Analysis and Validation Metries Calculations
H.l Fine-grained Diversity Evaluations
34
H.2 Evaluation of Metrics for Reference Datasets
Del.............................
34
H.3 Statistical Variation of Metrics
35
Sampling, Autoguidance and Hierarchical Fold Class Guidance
35
I.1
ODE and SDE Sampling
35
1.2 Classifier Guidance and Autoguidance
36
1.3 Guidance with Hierarchical Fold Class Labels
38
1.4 Step Size and Stochasticity Schedules..
39
J On the Relation between Flow Matching and Diffusion Models
39
K New Time Sampling Distribution
40
L Ablation Studies
41
Sampling Distributions
41
L.2 Stochasticity Schedules g(t)
42
L.3 QK Layer Norm, Registers and RoPE Embeddings
42
M Data Processing
43
M.1 PDB Processing, Filtering and Clustering
43
M.2 AlphaFold Database Processing, Filtering and Clustering
43
18

---

Published as a conference paper at ICLR 2025
M.3 CATH Label Annotations for PDB and AFDB
44
N Additional Neural Network Architecture Details
45
Experiment Details and Hyperparameters
46
0.1 Trained Proteína Models
46
0.2 Unconditional Generation Experiments
46
0.3 LoRA Fine-tuning on PDB
47
0.4 Conditional Generation Experiments
47
0.5 Long Length Generation Experiments
48
0.6 Autoguidance Experiments
48
P Baselines
48
A ADDITIONAL PROTEÍNA SAMPLE VISUALIZATIONS
In Fig. 9, we show additional protein backbones generated by Proteína, covering the entire chain length spectrum of our model. These samples generated without any conditioning. Furthermore, in Fig. 10 we show additional fold class-conditioned samples and in Fig. 12 are visualizations of successful motif-scaffolding.
Note that in all figures all shown samples are designable, according to our definition of designability (see App. F).
B MOTIF-SCAFFOLDING WITH PROTEÍNA
To validate the performance of Proteína in conditional tasks beside fold conditioning, we implement motif-scaffolding capabilities for Proteína and test its performance on the RFDiffusion benchmark (Watson et al., 2023).
B. MOTIF-SCAFFOLDING IMPLEMENTATION
To enable Proteína to perform motif-scaffolding, we add two additional features to our model via embedding layers: the motif structure (with coordinates set the origin for residues that not part of the motif) and motif mask (1 for positions that are part of the motif, 0 for positions that are not). In addition, center the data (x1) and the noise (xn) not based the overall centre of mass, but only on the center of mass calculated over the motif coordinates.
At inference time, sampling is initialized as before but again centered based on the center of mass calculated over the motif coordinates. We train model with 60M parameters in the transformer layers and 12M parameters multiplicative triangle layers, using the same dataset and motif training augmentation as Lin et al. (2024). We use batch size of 5 and add an additional motif structure auxiliary loss with weight of 5 in addition to the losses discussed in 3.2. Compared to Lin al. (2024), in addition to specifying the structural constraints of the motif in the model pair representation, we encode the masked motif coordinates the sequence representation. Thus, given conditional motif coordinates, the model is tasked with inpainting designable scaffold. At inference time sample with a reduced noise scale, as done in the unconditional case, using y = 0.5.
B.2
RESULTS
Motif-scaffolding performance is judged by common criteria outlined in previous work (Lin et al., 2024; Watson et al., 2023):
For each problem the benchmark set by Watson et (2023), 1000 backbones are generated.
For each backbone, 8 ProteinMPNN sequences are generated with fixed sequences in the motif region following the convention of Lin al. (2024)
19

---

Published as a conference paper at ICLR 2025
100
150
200
250
300
300
300
400
400
400
500
500
500
600
600
700
700
800
Figure 9: Unconditional Proteína Samples. The numbers helow the proteins denote the generated proteins' number of residues. All shown proteins are designable.
All 8 sequences per backbone are fed to ESMFold. The predicted structures are used compute the scRMSD, which is the C.MRMSD between the designed and predicted backbone, as well as the motifRMSD, which is the full backbone RMSD between the predicted and the ground truth motif.
20

---

Published as a conference paper at ICLR 2025
50 Sandwich 2.60 2.60
100 Alpha-Beta Plaits C.A.T:3.30.70 1.30.70
125 Roll
Helix Hairpins C.A.T: 1.0.287
C.A: 3.10
125 Nuclear Transport Factor 2; Chain A
175 Transferase (Phospho- transferase): Domain I C.A.T: 1.10.510
150 Beta Barrel 2.40
175 Rossmann Fold C.A.T: 3.40.50
200 Up-Down Bundle C.A: 1.20
200 DHBP synthasc C.A.T: 3.90.870
225 3-Layer (aba) Sandwich 3.4:
225 Jelly Rolls C.A.T: 2.60.10
250
250 Alpha-Beta Complex C.A:3.90
300
400 Mainly Beta C:
Immunoglobulin-like C.A.T: 2.60.40
Propeller CLA: 2.115
500
600
700
800 Mainly Alpha C.1
TIM Barrel 3.20.20
Mainly Beta C.2
Mixed Alpha-Beta Beta C:3
Figure 10: Fold Class-Conditions Proteina Samples. The numbers below the proteins denote the generated proteins' number of residues. Morcover, we show the fold class and the corresponding C.A.T fold class code for conditioning. All shown proteins are designable and correctly re-classified into the conditioning fold class.
21

---

Published as a conference paper at ICLR 2025
Table Number of unique successes on the RFDiffusion benchmark for 4 different methods, each generating 1000 backbones.
Task Name
Proteina
Genie2
REDiffusion
FrameFlow
6E6R long
713
-15
381
110
6E5R medium
417
272
151
99
IYCR
249
134
7
149
Jong
6EXZ med
43
54
25
110
7MRX_122
35
6E6R sbort
56
STRV uedd
22
23
10
21
3XX
14
STPN
IQJG
18
STRV
SYUL
5
4ZYP
6EX2_shott
IPRW
SIUS
SWN9
2
2KL8
4JHW
0
A backbone is classified success when one of the sequences generated for it has an scRMSD < 2A, motifRMSD < 1Å, PLDDT > 70, and pÅE < 5.
All successes are clustered via hierarchical clustering with single linkage and a TM-score threshold of 0.6 reach the final number unique successes.
Looking at the performance over the entire benchmark (Tab. 6 and Fig. 11), that Proteína has highest number of unique successes overall in the benchmark (2094 compared to 1445 for the second-best method Genie2) and is the sole best method in 8 tasks (compared with the second-best method Genie2 that wins in tasks).
Investigating the performance for each task individually (Tab. 5 and Fig. 13), we that Proteína outperforms mostly on easy and medium tasks, whereas the hardest tasks with or O successes still seem challenging. Successful designs shown in Fig. 12.
Table Number of unique successes summed over whole RFDiffusion benchmark number of times method was the sole best method different methods, each generating 1000 backbones. See Fig. 11 for bar chart of these results.
Task Name
Proteina
Genie2
FrameFlow
RFDiffusion
#Sucoesses total #Tasks as best method
2094
1445
1099
889
8
4
1
Total unique successes
Tasks as best method
2000
1500
6-
1000
2-
Proteina
Genie2
FrameFlow REDiffusion
Proteina
Genie2
FrameFlow REDiffusion
Figure 11: Motif-scaffolding results. Number of unique successes summed over the whole RFDif- fusion benchmark and number of times a method was the sole best method for different methods, each generating 1000 backbones. These the same numbers in Tab. 6.
22

---

Published as a conference paper at ICLR 2025
IBCF
3IXT
4ZYP
STRV-long
5WN9
SYUI
6EXZ-med
7MRX-128
Figure 12: Examples of successful designs the motif-scaffolding benchmark. All shown samples satisfy the criteria for task success. The task specification is given below the proteins. Motif residues are shown yellow.
RFOtifasion
Frameflow
Genied
Proteina (ours)
Numbor of uniuue Succosna
(but of 1000
6E6R Jong
6B22 lang
SYCK
STRY jang
PWAX 320-
BEGR shor-
7HIUX 85
THRX 60
1010
SYUI
42P
short
3PR0
5US
10CK-
SHIND
4000
SEX2
Drfenent scaffols ng tasts define by diferentt conditoning mobel structures
Figure 13: Motif-scaffolding results. Numbers of unique successes on the RFDiffusion benchmark,
following the success definition and clustering methodology from Genie2 (Lin al., 2024).
C SCALING AND EFFICIENCY ANALYSIS
C. SCALING FLOW MATCHING TRAINING
In Fig. 14, we study the optimization of Proteína's flow matching objective as function of the number of parameters, using Proteína models without triangular multiplicate layers, scaling the novel non-equivariant transformer architecture. We trained models various sizes between #660 and x440M parameters, and we find that thereby validating the scalability our architecture. This observation is in line with recent work on state-of-the-an image generation (Esser al., 2024), leveraging similar flow matching approach.
consistently improve the loss as scale the model size,
23

---

Published as a conference paper at ICLR 2025
040
59M
59M
94M
94M
0.39
191M
0.39
191M
80m matching loss
Now matoh ng 015
0.38
0.38
0.37
0.36
0.36
100000
200009 300000
400000
400000
600000
800000
1000000
training
training
Figure 14: Left: Flow matching loss over the course of training for differently sized Proteína models (number of model parameters given at the top right). Batch size 5 for all. Right: The same training curves, but we emphasize that when scaling the model the flow matching loss reaches similarly low values (gray dashed line) significantly faster.
Table 7: Unconditional backbone generation performance of the additional, smaller Mis Mm Proteína model, side-by-side with the other, larger models that we trained. Results partly copied from Tab. 1.
Design- ability (86)
Divensiyy
Nowelty vs. AFDBL
FPSD
Striet, (a/
Clustert TM-Sc.
PDBL
PDBJ
AFDBL
ICIAIT
PDBL APDE]
Unconditional generation. MC dosotes the Proteina model variant, and y is the notee soale for Proteho
Mrs. -0.45
96.4
0.63 (305)
0.36
0.69
0.75
388.0
368.2
206/5332/
19.05
166
1.23
68.1/6.9
Man 0.45
9188
0.2
0.36
1.80472/
7..3/5.5
Ma 7.445
94.8
0.35 (273)
0.35
0.72
0.78
322.3
323.3
2.215591/2288 /2..8
1.33
1.24
8.0
C.2 MODEL PARAMETERS, SAMPLING SPEED AND MEMORY CONSUMPTION
To compare the parameter counts of different models as well as the practical implications of these parameter counts such as memory consumption and sampling speed, we conduct three analyses:
1. Models are sampled with batch size 1 and the sampling time is measured. This is run an A6000-48GB GPU for comparison with previous works (Lin et al., 2024). See Tab. 8 and Fig. 15.
2. For all tested models, we determine the largest supported batch size that fits into GPU memory and does not result in out-of out-of-memory errors. This is executed on an A100-8008 80GB GPU. See Tab. 9,
Models are sampled with their maximum batch size and the sampling time measured, normal- ized with respect the batch size. This is executed on A A100-80GB GPU. See Tab. 10.
Each of the linked tables shows all models' number of parameters.
As part of these experiments, we use an additional model MMga'll which only contains around 60M parameters (similar to RFDiffusion), but still performs very competitively, outperforming most baselines like RFDiffusion (Tab. 7). As one would expect due to the smaller model size, does perform slightly worse than larger state-of-the-art models, though, showing slightly worse diversity and novelty. The training and sampling of this model follows the setting from M with the main difference being the number of parameters. For all our models, we leverage the fact that our transformer-basee architecture is amenable to hardware optimisations and leverage the torch compilation framework (Ansel et al., 2024) speed up training and inference. The inference numbers depicted here for Proteína account for inference time of the compiled model.
Looking at sampling time for single protein generation (batch size 1) on an A6000-48GB (Tab. 8), we see that the runtime of Proteína depends on whether we use triangle layers or not: Proteína models with triangle layers are still faster than state-of-the-arr tools like RFDiffusion and Genie2, are slower than FrameFlow at all lengths and slower than Chroma longer lengths. However, Proteína models without triangle layers are a lot faster and perform competitively even with much smaller models like FrameFlow (with Mamall running faster than FrameFlow for all lengths). Note that compare with RFDiffusion, Genie2, FrameFlow and Chroma, as these represent the most competitive baselines.
In practice, one performs inference batch-wise. To compare the performance of Proteína in this setting, we determined the maximum batch size for each method A A100-80GB GPU (Tab. 9)
24

---

Published as a conference paper at ICLR 2025
Table 8: Sampling time [seconds] for different methods batch size samples varying length (the
numbers in the top row indicate protein backbone chain length) on an A6000-48GB GPU.
Method
Model parameters
Inference sleps
200
300
400
500
Genie2
15.7M
1000
48
75
135
233
356
536
740
961
RFDiftusion
PrameFFoo
17.4M
100
13
18
Chreena
18.5M
500
36
42
49
69
59M
400
12
191M
400
15
23
32
54
Mes
208M
400
119
188
273
529
MarM
397M
400
24
54
102
159
230
310
408
1300
800
Runtine (4)
Runtime (c)
10
FrardFle
Chreea
800
Mis
300
Mys
100
200
(10
1003
1300
500
180
200
110
130
700
500
Chain Length
Chain Length
Figure 15: Single sample runtimes. The runtimes for different models for batch size 1 on A6000 GPU. Different scales used for y-axis (left linear, right logarithmic). The same data is shown in Tab, 8.
Table Maximum batch size during inference different methods samples varying length (the numbers the top row indicate protein backbone chain length) on an A100-80GB 80GB GPU.
Method
Model parameters
Inference steps
100
200
300
Genic2
7M
1000
204
51
22
12
4
3
Apgeal
59M
400
1599
416
200
194
72
46
36
25
MES
Mis
208M
400
199
55
26
14
3
MrM
397M
400
157
44
20
Table 10: Sampling time [seconds] for different methods at max batch size for varying lengths (the numbers in the top row) on an A A100-80GB GPU. The time is obtained by dividing the total runtime by the hatch size.
Method
Model parameters
Inference steps
100
200
300
400
500
600
700
800
Genie2
15.7M
1000
27.74
65.47
117.59
183.67
257.63
373.40
$26.00
690.67
Chroma
18.5M
500
9.56
17.58
26.31
30.84
35.17
59M
400
0.29
0.94
2.01
3.38
5.26
7.33
9.97
14.44
MrS
191M
400
0.59
1.88
3.87
6.54
9.96
14.04
18.87
24.33
Mips
208M
400
3.74
13.05
28.31
50.14
80.89
125.33
173.25
229.00
Man
and then determined the normalized sampling times per sequence in this batch setting by dividing the overall batch runtime by the batch size (Tab. 10). No numbers were reported for RFDiffusion and FrameFlow since these methods do not support batched inference, limiting the batch size to 1.
Even with Proteína having more parameters than the baselines, we see that Proteína models with triangle layers can fit similar batch sizes to Genie2. On the other hand, Proteína models without triangle layers can fit very large batches, up to 1.6k proteins length 100 for M
Looking at the per-sequence sampling time in the max batch size setting (Tab. 10), we see that Proteína benefits strongly from batched inference, especially for models without triangle layers and shorter sequence lengths. This enables fast batched sample generation, with less than I second per chain for short chain lengths.
Our overall conclusion from these experiments is that even though investigated model size scaling in this work, this scaling does not come at a cost in terms of inference efficiency, thanks to our efficient and scalable architecture. Our models support batches as large larger than the baselines and can be sampled as fast as or faster than the baselines, meanwhile leading to state-of oftte-of-hhearr t-thh-art protein backbone generation performance (see main paper).
25

---

Published as a conference paper at ICLR 2025
Table 11: Fold cans-conditioned generation: We report the generated proteins' re-llassification probabilities of the correct fold class label that was used during conditioning.
Sctup
T
a
a/3 /
Common
Regular
Race
Common
Regular
Rare
Classifier.free guidance (a
0.0), guidance weight (P (Protetna model 27
0.3)
Proteina, 0.0
0.585
0.017
0.450
0.128
0.014
0.000
0.032
0.002
0.000
Proteina, w 0.5
0.914
0.479
0.784
0.437
0.204
0.119
0.336
0.114
0.006
Proteina, w
0.986
0.961
0.701
0.334
0.209
0.010
Proteina, w=5
0.993
0,962
0.977
0.772
0.363
0.242
0.611
0.225
0.012
Proteina, w-26 2.0
0.992
0.975
0.976
0.788
0,383
0.233
0.638
0.230
0.012
Proteina, w 2.5
0.993
0.997
0.842
0.366
0.224
0.012
Chroma
0.888
0.486
0.644
0.240
0.007
0.000
0.133
0.002
0.000
D VALIDATING FOLD CLASS CONDITIONING VIA RE- CI-CLSSSIIIATINNN
To analyze whether our fold class conditioning correctly works, we re-classify generated conditional samples with our fold class predictor and validate whether the generated samples correctly correspond to their conditioning classes; see Tab. 11. We use dlassffier-free guidance on the model M with a noise scale of 7 = 0.3, which yields the best re-classification probabilities. We guide the model to generate 100 samples for each C-level class, 30 samples for each A-level class, and 2 samples for each T-level class. The generated samples are then evaluated using our fold classifier (trained in App. G.3.2) to predict the probability that they belong to the correct class.
We group the classes by their frequency in the training set and calculate the average re- re-classification probability for each group. Specifically. there are three C-level classes: "Mainly Alpha", "Mainly Beta", and "Mixed Alpha/Beta". For A-level classes, we divide them into three categories: 9 classes with over 500K samples (common). 13 classes with 10K-500K samples (regular), and 17 classes with fewer than 10K samples (rare). For T-level classes, we have 31 classes with over 100K samples (common), 237 classes with 5K-100K samples (regular), and 958 with fewer than 5K samples (rare).
As shown in Tab. 11, Proteína can accurately produce the main C classes. At A- and T-level, where we have increasingly fine spectrum of classes (Fig. 3), the task becomes more challenging, and on average common folds are generated better than rare ones. Considering the imbalanced label distribution with many rare classes. this result is expected. Moreover, re-classification accuracy generally increases with guidance weight w, validating our tailored CFG scheme (Sec. 3.2). We conclude that while rare classes can be challenging, as expected, the conditioning generally works well for the three C and the common A and T classes.
D. RE-CLASSIFICATION ANALYSIS OF FOLD CLASS-CONDITIONAL CHROMA SAMPLING
As discussed in the main text, we also evaluated Chroma (Ingraham al., 2023) on fold class- conditional generation (also see App. P). Chroma uses its own CATH fold class label classifier to guide its generation when conditioning on fold classes. We repeated the re-classification analysis for Chroma and report its correct re re-classification probabilities in the last row in Tab. 11. We find that Chroma generally performs poorly compared Proteína. While Proteína can guide into the three main C classes with almost 100% success rate, Chroma struggles to reliably guide into these high-level classes. Furthermore, when guiding with respect to the more fine-grained A and T classes, Chroma's 's success plummets. This means that Chroma cannot reliably perform fold class conditioning, in contrast to Proteína.
We would also like to comment on Chroma's results in Tab. 2, where it performs competitively with Proteína. This is because the designability, diversity and novelty metrics do not actually test whether correct protein structures given the labels were generated, but these metrics only score the overall set of generated backbones, irrespective of their labels. Only the re-classification analysis conducted here specifically tests the fold class conditioning capabilities in fine-grained manner.
26

---

Published as a conference paper at ICLR 2025
E EQUIVARIANCE ANALYSIS
In this section we study whether our transformer architecture learns rotationally equivariant vector field. Since the optimal vector field known to be rotationally equivariant, studying this may yield insights into our method's performance and behavior. We study this empirically for our MMs model the unconditional sampling setting by comparing clean-sample predictions on rotated versions of noisy/diffused backbone x,. More specifically, we compute three metrics. The first one is given by
(t)))=
E Xx R- R-Unif(SO((3)
RMSD Rx(x), (x(R")))
x:)]
(5)
where (xx) X + (1-11) vxx,, 0) is the clean sample prediction (since we use the velocity parameterization). This metric compares the outputs our model with respects to two inputs (noisy backbones) that are the same up to rotation R. A perfectly equivariant model is guaranteed to achieve (t) = 0, as the two outputs would also be equal up to the same rotation R. For a non equivariant model, however, would have 8 (t) > 0, with greater values corresponding to "less cquivariant" models.
The second metric we consider is given by
(t))) E
RMSD (*(x)), xt)
(6)
where U in Eq. (6) is the rotation that optimally aligns x(xx) and x(R' xt), that is, U arg min AcsO( ||x(xx,) A*(R*
This metric has two interesting properties. First, a perfectly x,) (x,xxx)]
equivariant model satisfies U R and (t) = O. And second, E*(() S (t), with the two metries being close when the optimal rotation U = R. Approximately equivariant models should achieve low values for this metric. Additionally, for approximately equivariant models the gap in E*(() S E" (t) should be small.
Finally, the third metric is given by
{()) E RMSD (x(x)), (RT xx)]
In contrast to the first two metrics, E(t) is minimized by rotationally invariant models (in fact, E(t 0 only for such models). In contrast, equivariant or approximately equivariant models should produce larger values for this metric. Intuitively, approximately equivariant models should satisfy (*())= < Et).
Results for all three metrics as a function of are shown in Fig. 16. It can be observed that, while greater than zero, our model achieves (L) R E" (L) < 0.5À for all I. This confirms that while our model does not learn perfectly equivariant vector field, it is approxiamtely equivariant, thanks to the random rotation augmentations applied to clean samples during training. Additionally, as expected for approximately equivariant models, {t) is considerably higher than the other two metrics.
may also informative to consider the notion of designability (see App. F), which (broadly) deems backbone designable if there exists a sequence that folds into a structure withing 2 (RMSD) of the original backbone. The metric E" shows that rotating our model predictions accordingly (on rotated inputs) yields RMSDs values below 0.5A significantly below the "similarity" threshold used to measure designability.
F ESTABLISHED METRICS: DESIGNABILITY, DIVERSITY, NOVELTY & SECONDARY STRUCTURE
We evaluate models using a set of metrics previously established in the literature, including des-
ignability, diversity, novelty, and secondary structure content. These metrics are computed across 500 samples, which include 100 proteins at each of the following lengths: 50, 100, 150, 200, and 250.
A fact leveraged by many existing methods, which rely on rotationally equivariant architectures (Yim al., 2023h;a; Lin & Alquraishi, 2023; Lin et al., 2024; Bose al., 2024).
27

---

Published as a conference paper at ICLR 2025
5
4
Error (A)
3
8
1
0.2
0.4
0.6
0.8
Figure 16: Equivariance analysis. E, and Ed from the captions measure different types of errors, and are formally defined in Eqs. (5), (6) and (7). For a perfectly equivariant model, the green (E") and orange (E") lines would be exactly zero for all t. Approximately equivariant models achieve low values for ~" and E", and large values for E. Our model, despite not being equivariant by construction, follows such a trend.
Designability. A protein backbone is considered designable if there exists an amino acid sequence which folds into that structure. Our evaluation of designability follows the methodology outlined by Yim et al. (2023b). For each backbone generated by a model, we produce eight sequences using ProteinMPNN (Dauparas al., 2022) with a sampling temperature of 0.1. We then predict a structure for each sequence using ESMFold (Lin et al., 2023) and calculate the root mean square deviation (RMSD) between each predicted structure and the model's original structure. A sample classified as designable if its lowest CMMSD-referse to as the self-consistency RMSD (scRMSD) under 2 The overall designability of model is computed the fraction of samples that meet this criterion.
Diversity (TM-score). We evaluate diversity in two different ways. The first measure of diversity we report follows the methodology from Bose al. (2024). For each protein length specified above, we compute the average pairwise TM-score among designable samples, and then aggregate these averages across lengths. Since TM-scores range from zero to one, where higher scores indicate greater similarity, lower scores are preferable for this metric.
Diversity (Cluster). The second measure of diversity follows the methodology from Yim et al. (2023b). The designable backbones are clustered based then computed by dividing the total number of clusters by the number of designable samples, that is, (number of designable clusters) (number of designable samples). We perform clustering using Foldseek (van Kempen et al., 2024). Detailed commands for this process are provided in App. F.1. Since more diverse samples imply
TM-score threshold of 0.5. Diversity is
clusters, higher scores are preferable for this metric.
Novelty. This metric assesses a model's ability to generate structures that are distinct from those in predefined reference set. For every designable structure we compute its TM-score against each structure in the reference set, tracking the maximum score obtained. We then report the average of these maximum TM-scores (lower is better). In this work consider two reference sets: the PDB, and Drs (Sec. 3.1), the AlphaFold DB subset used by Lin et al. (2024) to train Genie2 (this set was also used to train our Mys models). These metrics measure how well model can produce samples that lack close analogs within the reference sets. We use Foldseek (van Kempen et al., 2024) to evaluate the TM-score of backbone against these two databases. Detailed commands for this process are provided in App. F.I.
We use Biotite's (Kunzmann & Hamacher, 2018) implementation of the P-SEA algorithm (Labesse al., 1997) to analyze the secondary structure content of designable backbones. Specifically, we calculate the proportions of alpha helices (a), beta sheets (3), and coils (c) in each sample. The results reported normalized values: a/(a + 3 + c) for alpha helices, 3/ + B + <) for beta sheets, and c/ (a + 3 + c) for coils. In the main paper, we sometimes only report the and 3 percentages in the interest of brevity.
Secondary structure content.
28

---

Published as a conference paper at ICLR 2025
F.1 FOLDSEEK COMMANDS FOR CLUSTER DIVERSITY AND NOVELTY CALCULATIONS
Diversity (Cluster), As mentioned above we use Foldseek to cluster sets of designable backbones. The command used is
fol dseek easy-cl cstee <path_samples> <path_tmp>/ren /res <path_tmp> --alignmen-tyype --cov-mode --min-seq-id -- -tmscore-threshoLLee 0.5
where <path_samples
directory with all designable samples stored in PDB format and <path_tmpp> is a directory for temporary files during computation.
Novelty. We Foldseek evaluate the TM-score of protein backhone against a reference set. We store the reference sets as Foldseck databases, For the PDB we use Foldseek's 's precomputed database, and create our own for Drs. We use the following Foldseek command compute max TM-scores
foldseek easy-searcn <path_sample> <database_pate path>> <out_ file>
<tmp_path> bath> -aliigment-tyye exxhasstive-sseret --tsscore-hrrshho 0.0 --ax-seqq 10000000000 format-output query, target, a alntmscore, 1ddt
where @eath_sappee> is the path of the generated structure PDB file, settabase_pathle the path to the Foldseek database, and fout_file> and <tmp_path> specify the output file and directory for temporary files.
G NEW METRICS: FPSD, FS AND FJSD
G.1 MOTIVATION
Protein structure generators are typically evaluated based on designability, diversity and novelty, Designability measures whether the generated structures can be realistically designed, though with biases inherent in folding and inverse-folding models. While generating diverse and novel proteins is important, these metrics may overlook the quality of the samples-speccffcalll specifically, how closely they resemble realistic proteins. Besides, none of these metrics directly evaluates models the distribution level, failing to measure how well model aligns with reference or target distribution.
To address these limitations, we propose three new metrics that score the learnt distribution rather than individual samples. First, introduce the Fréchet Protein Structure Distance (FPSD), which compares sets of generated samples to a reference distribution a non-linear feature space, drawing inspiration from the Fréchet Inception Distance (FID) used in image generation (Heusel et al., 2017). Second, we define the Fold Score (fS), similar to the Inception Score (Salimans et al., 2016), which evaluates both the quality and diversity of generated samples using a trained fold classifier, Finally, present the Fold Jensen-Shannon Divergence (fJSD) to quantify the similarity of generated samples to reference distributions across predicted fold classes. All new metrics defined in detail in App. G.2. They all rely on fold classifier for protein backbones, Ps (- x), described in App. G.3.
G.2 METRIC DEFINITION
Fréchet Protein Structure Distance (FPSD). The Fréchet Protein Structure Distance (FPSD) measures the distance between two distributions over protein backbones, the one defined by generative model and a target reference distribution, leveraging non-linear feature extractor ((x) (in practice, we use the last layer of fold classifier (+).) see App. G.3).
Let {x)pen and hee denote two distributions over protein backbones, defined by a generative model and reference distribution, respectively. We compute the FPSD between these distributions by mea- suring the Fréchet Distance between the two Gaussian densities defined as N (H+(())) E(o()))
and N (H(()()) he E (()))) In practice this metric is computed following a two-step process:
1. Compute the mean and covariance over features d(x) for the generative and reference distributions ((((x)))) = Ex~ [xx)], Σ(o(0) = Ex~ [(o(x) - (²((()),, (o(x) - [4(0())pmm ((((())) Exx (0(x)] {d((0)) (onxxxell((xx ((000))) (1((())))))
29

---

Published as a conference paper at ICLR 2025
2. Measure the Fréchet Distance between the two resulting Gaussian distributions
FPSD((x) (xher) (0(0000))mm0((0))))))) + (E(((x))g + E(((0))er (0000)))
(1o00)) E(000)a)))
Here, [[()))))) (x(x)) her l22 represents the distance between the mean feature vectors, and the trace term captures the differences covariance matrices. The FPSD reflects how closely the generated structures resemble the reference distribution, as measured by distributional similarity in continuous feature space, with lower values indicating greater similarity.
Protein Fold Score (fS). The Protein Fold Score (fS) measures the quality and diversity of generated structures by evaluating how well they align with known fold classes.
Let {x]gen represent the distribution of generated structures, and pa (-1x) denote the predicted probability distribution over fold classes for structure x. The fS is computed in two steps:
Compute the marginal distribution over fold classes p+(-) (x (-(()))))
Calculate the Protein Fold Score
[S((x) = exp (Ex- [DeL
where DKL represents the Hallback-Leibler divergence. This score captures the average diver- gence between the label distribution of each generated sample and the marginal distribution over labels, reflecting both quality and diversity.
A higher fS indicates that the generated protein structures not only of high quality individually, but also exhibit a diverse range of fold classes, capturing the richness of the generated distribution. Note that fS is calculated separately at the different levels of the label hierarchy, i.e., separately for the C-, A- and T-level classes.
Protein Fold Jensen-Shannon Divergence (fJSD). The Protein Fold Censen-Shannor Divergence (IJSD) quantifies the similarity between the predicted label distribution of generated protein structures and that of a reference set, both derived from the same fold classifier.
Let gen and (x) (ef represent the distributions of generated and reference structures, respectively, and let (-Ix) denote the predicted probability distribution over fold classes for a structure x. The IJSD metric computed in two steps:
1. Compute the marginal predicted distribution over fold classes for the generative and reference distributions
Pgen
E. [P (px( and Prer()) (x (1 (-o(-1x))
2. Calculate the Protein Fold Jensen-Shannon Divergence
FJSD( ( (x)gen (x (x)rer 10 x Dis(Pgen (-ll)rrr))))
(8)
where Dis denotes the Jensen-Shannor divergence, defined as
1 Dis(PIIQ) Px (KL()))))
1
(9)
with 1 Q). In our case, P represents the distribution Pgea and Q represents the distribution for the reference set Prer(-). Since the Jensen-Shannon divergence upper bounded by 1, we multiply it by factor of 10 for easier reporting of the results.
Lower values fISD indicate that the predicted label distribution of generated proteins closely aligns with that from the reference set, reflecting higher fidelity to the expected fold classes. In contrast to FPSD, which measures the similarity between the generated and reference distributions continuous feature space, fJSD measures the similarity in the categorical label space from the fold classifier. As we empirically find that fJSD values calculated for the C-, A-, and T-level label distributions yield the same ranking across different methods, we decide to report the final metric values the average of the fJSD scores at the C-, A+, and T-levels. We note, however, that this metric can be reported separately for each of the C, A, and T-levels.
30

---

Published as a conference paper at ICLR 2025
Reference Datasets. To evaluate FPSD and fISD, construct two reference datasets, one for the PDB and another one for the AFDB. For the PDB reference set, we curate a high-quality single-chain dataset by applying several filters the PDB: minimum residue length of 50, maximum residue length of 256, a resolution threshold of 5.0 A, a maximum coil proportion of 0.5, and a maximum radius of gyration of 3.0 nm. We then cluster the dataset based on a sequence identity of 50% and select the cluster representatives, resulting in 15,357 samples. For the AFDB reference set, we directly use the Foldseck AFDB clusters, denoted by Dps in the main text.
These metrics are evaluated independently of existing metrics based on a different set of generated samples. We randomly sample 125 proteins at each length from 60 to 255 residues, with a step size of 5. We use all the 5,000 produced samples, without any designability filter, for evaluation.
G.3 FOLD CLASSIFIER TRAINING
A crucial aspect of defining the new metrics is developing an accurate fold classifier (+ which embeds lipha-carbon-only structures into the feature space (xx). In this subsection we give details behind the classifier we use, including the dataset it is trained on and its architecture.
G.3.1 DATASET PROCESSING
For training the classifier, CATH structural labels are utilized for protein domain annotation (Dawson al., 2016) which includes C (class), A (architecture), and (topology/fold) labels. We exclude (homologous superfamily) labels to ensure that our classification is based solely on structures.
We extract chains from the PDB dataset, with structures filtered to include a minimum length of 50 residues, a maximum length of 1000 residues, and maximum oligomeric state of 10. We also discard proteins with a resolution worse than 5Á and those lacking CATH labels. This results 2 214,564 structures, categorized into 5 C classes, 43 A classes, and 1,336 T classes. The dataset randomly divided into training, validation, and test sets at a ratio of 8: 1:1, ensuring that at least one protein from each class is included in the test set whenever possible. While the paper primarily focuses on the three main C-level classes ("mainly alpha". "mainly beta", "mixed alpha/beta"), as they the most interesting and relevant to our study, we still train the classifier all C-level classes. This ensures the metrics are universally applicable and can be used for future analyses involving any of the C-level classes.
total of
Given that the CATH database annotates protein domains, some proteins may have multiple domains, thus multiple CATH labels. For these proteins, we randomly sample one domain label the ground truth during training and encourage the model to predict equal probabilities for the labels of all domains. During testing, predicting any of the correct labels is considered a good prediction.
G.3.2 GEARNET-BASER FOLD CLASSIFIER
To build the fold classifier Po (1x), we utilize an SE(3)-invariant network, GearNet (Zhang et al., 2023), as our feature extractor d(x).. GearNet is geometric relational graph convolutional network specifically designed for protein structure modeling, making it ideal for tasks such as protein classi- fication and fold prediction. While the original GearNet architecture processes both structural and sequential data, we modify it to focus solely on predicting fold classes based on structure. The model components detailed as follows:
1. Input and Embedding Layer:
Each C. atom is treated node, and the node features are constructed by concatenating a 256-dimes 556-dimensional atom type embedding (generally corresponding the C. atom embedding) with 256-dimensional sinusoidal positional embedding based on sequence indices.
2. Graph Construction: Sequential relations are established by connecting neighboring atoms within relative sequence distance between -2 an 2, with each relative distance treated as a distinct relation type. Spatial relations connect atoms within a Euclidean distance of 10A. In total, the graph uses five sequential relation types and one spatial relation type, allowing the model to capture diverse interaction patterns between residues based on both sequence proximity and spatial context.
multi-relation graph is built using both sequential and spatial information.
31

---

Published as a conference paper at ICLR 2025
Table 12:
Summary of metric validation experiments.
Setting
Distribution
Expected Results
Results
Protein Fold Score (f5)
Balanced dataset
Diverse and halanced label distribution
High IS
Fig. 17 (blue)
Homogeneous dataset
Homogeneous label distribution
Low IS
Fig. 17 (red)
Imhalanced dataset
Diverse but imbalanced lahel distribution
Medium IS
Fig. 17 (green)
Imbalanced noisy dataset
Noisy and imbalanced distribution
Decreasing IS
Fig. 17 (green)
Unseen noisy datasct
Noisy distribution with unseen samples
Decreasing fS
Fig. 17 (orange)
Fréchet Protein Structure Distonce (PPSD) and Protein Fold Jensen Jensen-Shauion Divergence ((SSD)
Disjoint split datasets
Different structure distributions
High FPSD and (ISD)
Fig. 18 (blue)
Random split datasets
Similar structure distribations
Low FPSD and BSD
Fig. 18 (green)
Random split noisy datasets
Noisy distributions with samples
Increasing FPSD and USD
Fig. 18 (green)
Unseen random split noisy datasets
Noisy distributions with unseen samples
Increasing FPSD and DSS
Fig. 18 (orange)
3.0
3,05
12
11.00
500
337.75
10-
Fold Score
288
Fold Score
Fold Score
400-
Balanced
26-
3.0
Homogencous
Imbulanced
24
200
Unseon
22-
100
2.95
3.53
20
0-11
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Noise scale
Noise scale
Noise scale
Figure 17: Fold Scores (C/A/T) metrics on balanced, homogeneous, imbalanced and unseen subsets
of the PDB dataset, with varying levels of Gaussian noise (0.0 0.4A) applied on the latter two.
3. Edge and Message Passing: Edge features are generated using radial basis function (RBF) to capture spatial distance-based relationships, along with relative sequential positional encoding between atoms. Both features are 128diimensional. Additionally, we incorporate clockwise angular features to break reflection symmetries.
4. Relational Graph Convolution Layers: The model includes 8 layers of Geometric Relational Graph Convolution, each aggregating information from neighboring atoms using the node and edge features. These layers employ MLPs to process inputs and update node representations, ensuring that the model captures different types relational patterns between atoms.
5. Output and Prediction: After the convolutional layers, the atom features are aggregated using sum pooling to create global protein representation. This global feature is further refined through an MLP layer. For classification, the model includes separate output heads for predicting three levels of CATH labels: T, A, and C, with output sizes of 1336, 43, and 5 classes, respectively.
Throughout the model, a dropout rate of 0.2 is applied to prevent overfitting, and leaky ReLU activation functions with a slope of 0.1 are used. The model is trained using the Adam optimizer with a learning rate 0.0001, distributed across 8 GPUs with a batch size of and a gradient accumulation step of 2. Training is run over 70,000 parameter update steps. On the test set, the model achieves a Micro Accuracy (Grandini et al., 2020) of 97.8% at the T-level, 98. 1% at the A-level, and 99.2% at the C-level. Given the highly imbalanced nature of the CATH classes, we also report Macro Accuracy, achieving 94.0% at the T-level, 97.5% at the A-level, and 95.6% at the C-level. These results demonstrate that the classifier is highly effective in accurately predicting the fold labels of protein structures.
G.4 METRIC VALIDATION
To validate the effectiveness of our metrics, create two sets of experiments observe the behavior of iS and FPSD, fJSD under different settings. We summarize these experiments, together with their expected results, in Tab. 12.
Protein Fold Score (IS) Validation. Using the PDB training dataset, we create three subsets to assess the behavior of the Protein Fold Score. All experiments are repeated with 20 different random seeds.
32

---

Published as a conference paper at ICLR 2025
Disjoint
Random
1500
3-
Unseen
FPSD
1000
BSD
2-
500
452.44
1-
0-
0.2
Noise scale
Noise scale
Figure 18: FPSD and fISD metrics on (i) the fold-disjoint and (ii) random splits the PDB training set, and (iii) random split of the unseen PDB set, with Gaussian noise (0.0 to 0.4A) applied to the latter two.
1. Fold Mlass-Bllanced Subser: We randomly sample 300 T-level classes and then randomly sample approximately 16 proteins per class (with replacement) to create total of 5,000 samples. This subset tests whether I'S rewards a diverse, realistic, and class-balanced structure distribution.
2. Homogeneous Subser: We randomly sample 4 T-level classes and 1,250 proteins per class. This subset is designed test whether fS penalizes distributions lacking fold diversity.
3. Fold Class-lmbalancedd Subset: We randomly sample 5,000 proteins from the PDB dataset. Given PDB's inherent class imbalance (Fig. 3), this random sampling leads to diverse but imbalanced distribution, so we expect this to lead to "intermediate" values for the metric.
Results for these three subsets are shown in Fig. 17. The results show exactly the expected behavior for the IS metric, with the Fold Subset the lowest, and the Fold
Subset obtaining the highest score, the Homogeneous Subset standing in between these two extremes.
We additionally assess whether metric is robust to noisy structures and structures unseen in the classifier's training dataset. For the former, we continue using the previously defined Fold Class- Imbalanced Subset and gradually add Gaussian noise to all structures, with the noise scale increasing from 0.0 to 0.4A. We expect the fS score to decrease as the scale of the noise increases. For unseen structures, we randomly sample 5,000 structures from the full PDB dataset, without applying the CATH label filter, and apply Gaussian noise in the same manner.
We evaluate the Fold Score C/A/T on these noisy datasets, with the results shown in Fig. 17 (green and orange curves). As expected, as the noise scale increases, the quality of protein structures declines, leading to reduced classifier confidence and a corresponding gradual decrease in the Fold Score.
Overall, we find the Protein Fold Score is able to effectively measure the realism, diversity. and balance of distributions over protein structures. remains robust to unseen samples and deteriorates gracefully for noisy samples, effectively detecting the lower quality of the noisy samples.
Fréchet Protein Structure Distance (FPSD) and Protein Fold Jensen-Shannen Divergence (fJSD) Validation. Similar to the iS validation, we curate two dataset splits based on the PDB training set, and measure FPSD and fISD for the two splits.
1. Fold Class-Disjond Split: We randomly draw 5,000 samples for each split, ensuring no overlap at A-level classes between the two splits. This setup tests whether FPSI) and fJSD can distinguish different distributions. We exclude T-level classes here, they are too fine-grained to produce sufficiently distinct distributions. We expect this split to yeild large values for both metrics.
2. Random Split: We randomly sample 5,000 proteins for each split from the PDB dataset. Since constraints are applied during the split, both datasets are expected to follow the same distribution, and thus expect low values for both metries for this split.
We show results in Fig. 18, where splits. Specifically, the Fold Class- Maass-Disjoint Split yields a FPSD of 452.44 and fJSD 4.21, while the Random Split yields significantly lower values for both metrics; 10 for FPSD and 0 for fJSD.
observe that the metrics behave expected for both
33

---

Published as a conference paper at ICLR 2025
Table 13: FrameFlow's, RFDiffusion's Genie2's and three variants Proteina's cluster diversity values with
designable clusters in parentheses under different scoee clustering thresholds. Best scores are bold.
Threshold
0.1
0.2
0.5
0.6
0.9
FrameFlow
0.011 (23)
(24)
0.07 (30)
0.237 (105)
(232)
(358)
(414)
(440)
0.997 (442)
RFDiffusion
0.946 (22)
0.046 (22)
0.048 (23)
0.127 (60)
0.447 (211)
0.720 (340)
0.834 (394)
0.887 (419)
0.936(44))
0.060
0.060
0.060
(284)
0,978 (466)
(472)
1,000
Mrs. 40
0.101 (49)
0.101 (49)
0.118 (57)
0,302 (146)
0.639 (308)
0.852 (411)
0.943
0.981 (473)
0.995 (480)
Mig" 0.45
0,083 (39)
0.089
0.095 (45)
0.257 (121)
0.626 (294)
0.820 (385)
(431)((43)
0.960 (460)
0.993 (466)
Myn.. 7-0.3
0.056 (28)
0.060 (30)
0.064 (32)
0.147 (73)
0.305 (151)
0.442 (219)
0.569 (282)
0.723 (358)
0.901 (446)
FrameFlow
RFDidtusion
Ganie2
Mrrs y 0.45
Mg y 0.45
Mass y 0.3
TM-Score
0.4
0.2
500
100
150
200
250
Number Residues
Figure 19: Pairwise TM-Score distributions of FrameFlow, RFDiffusion, Genie2 and three variants of Proteína across different residue lengths, with lower TM-Scores indicating better performance.
Additionally, we apply the same process used in the fS validation to create noisy and unseen dataset splits for testing the FPSD and fJSD metrics, computing them between the noisy and original datasets. The results Fig. 18 (green and orange) indicate that as the noise scale increases, protein structure quality deteriorates, leading to increasignly higher FPSD and IJSD values.
In summary, FPSD and fJSD effectively recognize similarities and differences between structure distributions, remain robust to unseen samples and detect increasingly noisy samples.
H ANALYSIS AND VALIDATION OF METRICS CALCULATIONS
H.1 FINE-GRAINED DIVERSITY EVALCATIONS
Cluster-based diversity with different thresholds. We evaluate the cluster-based diversity metric under varying clustering thresholds for three of our best models and the most relevant and competitive baselines Genie2, RFDiffusion, and FrameFlow-as shown in Tab. 13. For looser thresholds, Mrs (7=0.45) model outperforms the others. However, with very strict clustering thresholds, all models and baselines produce highly diverse results, covering wide range of distinct clusters.
Distribution of pairwise TM-scores. The other diversity metric that we use in this work are average pairwise TM-scores. Here, we analyze the distributions of the pairwise TM-scores for samples generated by our models and the three most relevant basclines mentioned above. We draw violin plots across different lengths in Fig. 19. The results demonstrate that models maintain reasonable TM-score distributions, showing no signs of mode collapse.
H.2 EVALUATION METRICS FOR REFERENCE DATASETS
To provide reference values for our results in Tab. 1, we report metrics for two representative protein structure databases: the PDB (natural proteins) and the AFDB (synthetic proteins predicted by AlphaFold2). We use the representative subsets of the PDB and AFDB processed in App. G.2 as our reference distributions. Following the protocols outlined in App. F and App. G, we sample from these reference datasets and evaluate all metrics. The results, presented in Tab. 14, show novelty values close to and very low FPSD and fJSD values, as expected. The designability of the two reference datasets aligns with previously reported results in Lin et al. (2024), with the AFDB exhibiting lower
34

---

Published as a conference paper at ICLR 2025
Table 14: Reference metrics by sampling two reference datasets (AFDB and PDB) introduced App. G.2.
Dotaset
Diveritty
Novelty
HPSD
USD
Sec. Street
ability (R)
Cluster
TM-Se.]
AFDR]
PDR.
AFDRL
(C/A/TM
PDR]
AFDRL
PDB ref.
67.2
0.62 (208)
0.31
0.99
0.87
3,433
71.20
291/9.087 90.45
0.05
0.51
35.0/16.8
AFDB ter.
33.6
091 (154)
0.29
0.76
1.00
73,02
3,011
271 /657/
$4.88
0.53
0.02
449/127
Table 15: Proteina's unconditional generation results under different random sampling seeds.
Seed
Design- ability (3
Diversity
Novelty vs.
FPSD va.
FSD vs.
Sec. Serict, %
Cluster
TM-Sc.l
PDBL
AFDRL
PDR
AFDB.
(C/A/TY
PDRL APDB]
(o (2)
Ulcrondiional generation with Pronzina model Myt" and noise solle
0.45
93.6
0.62 (294)
0.36
0.70
0.76
322.2
306.1
1.80/472/
18.59
188
1.36
7..56.11
92.8
0.36
0.75
327.0
313.0
1.78.4.647 18.83
1.5
1.39
70.776.2
0.60
0.36
0.37
1.7874607
1.88
1.40
71.1/5.2
93.0
0.57 (263)
0.37
0.71
0.26
316.6
300.3
1.7914.70/
18.93
1.83
1.38
71.6/49
Mean 93.7
0.60 (279)
0.36
0.20
0.76
321.9
306.1
13074.68/
18.82
1.84
1:37
71.75.55
0.02 (12)
0.01
0.0270.06/0.0
0.03
0.08
0.5/0.6
designability due the use of sequence clustering avoid over-represented clusters. Additionally, we observe that the reference datasets display higher diversity than all models and baselines. This suggests that existing models still have room for improvement optimizing diversity.
Note that our models and the baselines are able to achieve much higher designability among the gen- erated proteins due to noise or temperature scaling during inference (or rotation schedule annealing), effectively modifying the generated distribution towards more well-structured protein backbones.
H.3 STATISTICAL VARIATION OF METRICS
To assess the statistical stability of our results, we select the model MMga (y=0.45) for efficiency and repeat the metrics evaluations five times using different random seeds. The results, presented in Tab. 15, show minimal variance across all metrics, indicating robustness of our evaluation process.
I SAMPLING, AUTOGUIDANCE AND HIERARCHICAL FOLD CLASS GUIDANCE
Here, we provide more details about Proteína's inference-time sampling as well as our new hierarchi- cal fold class guidance and autoguidance.
1.1 ODE AND SDE SAMPLING
Generally, flow- dow-matching (Lipman et al., 2023; Albergo & Vanden-Eijnden, 2023; Liu et al., 2023; Albergo al., 2023) models like Proteína rely on a vector field u, (x,) that describes the probability flow between noise and data (see Sec. 2). The default way to generate samples is to solve the flow's ODE
dx, u, (x, dt
(10)
from t 0 to 1, initialized from random noise. In our case, the noise corresponds to a standard Gaussian prior with unit variance over the 3D C. coordinates of the protein backbone. As discussed in Sec. 2, the flow's intermediate states x, are practice constructed through an interpolant between data X1 p(x)) and the Gaussian random noise distribution ~ (0, T), which takes the general form
x
02 + O,E
(11)
for the time-dependent scaling and standard deviation coefficients a and dr, respectively. In our work. we rely on the rectified flow (Liu et al., 2023) (also known as conditional optimal transport (Lipman et al., 2023)) formulation, corresponding to a linear interpolant
X tx1 + (1 - t)
(12)
between noise e and data samples X1. This leads to the corresponding marginal vector field at intermediate x,
0 (E[tx + E[ - e = E[x||x/] E[c[x.].
(x())) = Ot
(13)
35

---

Published as a conference paper at ICLR 2025
Further, we have that (see, e.g., proof in Ma et al. (2024))
St (x+) = Vx, log (ogp:((x) (x;)
- [E(c|xe]- -
(14)
This allows derive relation between un (xx) and (x,)::
nn (x+)
E [x1 xx] B[c|xx]]
X (1 -) E(c|xi]
X 1
(15)
B[c|xx]
t t
X +
1
s (x1);
t
(xx) tx,( Xt
(16)
Furthermore, given the score of the probability path pe (xx), we can now obtain the Langevin dynamics SDE
dx, = sx,x) + vad,,
(17)
with the Wiener process W₁. This SDE, when simulated, in principle samples from (x(x) at any fixed We can combine this with the flow's ODE (Eq. (10)) to obtain the SDE
dx;- = (x.)dd + (())s ddt + 2g(() dW.
(18)
which, for any (() 2 O, now simulates the stochastic flow along the marginal probability path pe (xx)) from t 0 1 with the stochastic paths due the Langevin dynamics component.
In practice, we model (xx)) by the learnt neural network v (x,, ē) with parameters 0 which we can use to obtain the corresponding learnt score s (x,, é), using Eq. (16) above. As in the main paper, represents all conditioning information we may use in practice. Hence,
dx, = vg (x,, d)dt + g (t)) (x,, o) + 2gg()) dW.
(19)
Importantly, in practice the velocity and score are neural network- approximations with respect to their ground truths, and the SDE is numerically discretized. Therefore, different choices of gt),, which scales the stochastic Langevin component, can lead to different results in practice. For g()) we recover the default flow ODE.
Moreover, we introduce a noise scale y, as is common in generative models of protein structures, and in practice use the generative SDE
dx; (x,, djdt + g(t)s (x,, a)dt + 22((t) dW..
(20)
For y 1, we simulate the "proper" marginal probability path, while lowering the noise scale often reduces the diversity of the generated results, oversampling the model's main modes. Although not principled, this can be empirically beneficial in protein structure generation, as the tails of the distribution consist of undesired samples that, for instance, may not be designable. Since models operate directly in 3D space, reduced noise injection results in more globular and well- structured backbones that tend to be more designable. In the main paper, default ODE for generation (corresponding to g(t) 0) or the SDE with a reduced noise scale y < 1 and some stochasticity schedule g(t) 0. Moreover, in practice it can be sensible to only simulate with (1) > 0 up to some cutoff time / < 1, due to the diverging denominator in Eq. (16), required to calculate the score from the vector field when using g())
typically either use the
I.2 CLASSIFIER-FRER GUIDANCE AND AUTOGUIDANCE
As mentioned in the main text, are leveraging both classifier-fis free guidance (CFG) (Ho & Salimans, 2021) and autoguidance (Karras et al., 2024) in selected experiments this paper. To the best of our knowledge, neither of the two methods have been explored in flow- or diffusion-based protein
36

---

Published as a conference paper at ICLR 2025
structure generation. In both approaches, different scores are combined to obtain "higher quality" score that leads to improved samples.
Let us assume we have access to the densities pt (xx) and p (xx). Now let us define the "guided" score (Karras et al., 2024)
Pl p (xx) )
p (xx))
guided (x,) == Vx, log (x1) == Vx. log $t
Pt
(xx)
(21)
where 2 0 denotes the guidance weight. In practice, p1f (xx) corresponds to the density primarily interested in and for = 1 we recover st squided (x,) Xx, px But we now choose density p (x+) that is more spread out than yf (x+). then, for w > 1, the term
pr (x+)
(22)
Ps (x+)
is typically > for X2 ~ pl (xx) and the overall score is essentially scaled according to the ratio between p (xx) and (xx). This can be leveraged to construct guided squuided (x.) that emphasizes
the difference between pf (x:) and p (xe). as we now explain (see Karras et al. (2024) for details).
In classifier-free guidance (Ho & Salimans, 2021), pr"(x (xx) corresponds to a conditional density and pf (xx) to a unconditional one and w > 1 emphasizes the conditioning information, leading to samples that are more characteristic for the given class, and often also improving quality. Autoguidance (Karras et al., 2024) disentangles the effects of improved class adherence and improved quality and instead uses for p (xx) "bad" version of p (xx) that is trained for fewer steps uses smaller, less expressive network. Due to the maximum likelihood-
objectives of diffusion and flow models their learnt densities generally tend to be mode covering and can be somewhat broader than the ideal target density. Hence, even the "good" p (x₂) will usually not be a "perfect" model that models the distribution of interest perfectly and the "bad" p. (xx) will make the same errors like p." (xx). ). but stronger and the density will be even broader. Guidance with w > 1 then emphasizes the quality difference between p (x:) and p (xx) and can result in sharper outputs, essentially extrapolating in distribution space beyond p. (x,) towards the true desired distribution of interest.
CFG also often improves quality because similar effect happens there, just entangled with the conditioning: The unconditional density used in CFG also represents broader density than the conditional one, which means the guided score does not only emphasize the class conditioning. but also pushes samples towards modes, which often correspond to less diverse, but high-quality samples. Note that autoguidance is general and can be used both for conditional and unconditional generation, whereas classifier-fi free guidance contrasts conditional and unconditional model and hence is only applicable when such a conditional model is available.
Let us now derive the exact guidance equation used in our work. Decomposing the logarithm term in Eq. (21) yields
quided st
(x,) == w V xx log p (xx) + (1 - Vx. log pf (xi),
(23)
or short
st guided (x+) (x;)
== w s (x+) (1 w (x:).
(24)
Inserting Eq. (16) that relates the score and the vector field, we find that the vector fields obey the analogous equation
guided (x,) == w (x,) + (1 wu, (x²).
(25)
In practice, we use learnt conditional models. And that case, we can now introduce the interpolation parameter E [0, 1] that interpolates between classifier-free guidance and autoguidance in a unified formulation (analogous to Karras et al. (2024) in their Appendix B.2). We get
O. g.guided (x., 2) = w wv (xx, + (1 - w) (1- a)vy (x.. 0) + a 9 (x,, Vi
(x, 8)
(26)
where (xe: é) is the main model with conditioning č, (x,, 0) corresponds to the unconditional version, and v d,bad (x,,e) denotes the "bad" model required for autoguidance. For 0, get regular classifier-ffe free guidance, while for a = 1, we get regular autoguidance. In that case, setting ? = 0, i.e. autoguidance for unconditional modeling, is still applicable. In this paper, do not use
37

---

Published as a conference paper at ICLR 2025
any intermediate but only explore either pure CFG or pure autoguidance, though. An analogous formula can be written for the scores,
A. guided
Xt, c) = w i (x,, a) + (1 - w) [11 (1- (x,, 0) + ass bad (x a)
(27)
required in the generative SDE in Eq. (20).
Note that when apply self-conditioning (Sec. 3.2) during sampling generally feed the same clean data prediction
x(x,) x, + (1 (x.,
(28)
conditioning to all different models of the guidance equations (Eqs. (26) and (27)). Self- conditioning is optional, though, since we train with the self-conditioning input in only 50% of the training iterations. Proteína be used both with and without self-conditioning.
1.3 GUIDANCE WITH HIERARCHICAL FOLD CLASS LABELS
In order to be able apply classifier-free guidance during inference, one typically learns model that can be used both as a conditional and unconditional one, by randomly dropping out the condi- tioning labels during training and feeding a corresponding d-embedding that indicates unconditional generation. As discussed in detail in Sec. 3.2, we drop out our hierarchical fold class labels in hierarchical manner, thereby enabling guidance with respect to all different levels of the hierarchy.
Here, we summarize the corresponding guidance equations. Note that we do not explicity indicate the time step Ê conditioning as well as the self- self-guidance conditioning, keep the equations short and readable.
T-level guidance. we guide with respect to the finest fold class T, we use
Augudded
(xe,, 1Cx, Axi Tx xxtaar) vg (xt. {Cx, Ax. Tx)cAr)
Ve
(29)
+ (1 - w) (1 a)v. (0. 0, 0) 0)CAT) + a vv h. (x,, (Cxx Ax, Tx)CAr))
and correspondingly for the score s; guidance strength. For autoguidance, we have a = 1, and for CFG we have = O. Note that we also feed the "coarser" C- and A-level labels that are the parents of the T-level label in the hierarchy.
d.guided (xe; {Cx, Ax, Tx (CAT). As mentioned above, w is the
A-level guidance. If we guide with respect to the fold class A, we use
v O.guided/
(xx, (Cx, Ax, 0)ATT) = w (xe. {Cx, Ax, ()CT))
(30)
+ + ww) [1
a)v; (xe, {0, 0, OJCAT) + a v 9.bed (.bod((e, (xe,
{Cxx Ax. Ax.0)CaT)]
and correspondingly for the score si g.guidted (xe: {Cx, Ax. ()CAT).
C-level guidance. If we guide with respect to the fold class C, we
O guided
(x,, {Cx, 0.0AAA)) vv (xe. {Cx, 0, 0}CAT)
(31)
(x,, + (1 w)) [11 ax) {0, 0, ()CA)) + v (. (x (Cx, 0, 0, 0.0AT))
and correspondingly for the score s.
giguided (xe, {Cx, 0, (.A)))))
No fold class guidance. autoguidance in its unconditional setting, we have = 1 and
do not guide with respect fold class, but we still want to apply
autoguided (x4, {0, 0, 0)xx)) = w vl (x., {0, 0, cxr) + (1 - w g. (x,, (0, 0, 0 ()xr),
(32)
and correspondingly for the score S autoguided (x,, {0, 6, ()xx))
In practice, for the "bad" models required for autoguidance, use early training checkpoints of our main models. We do not train separate, smaller dedicated models just for the purpose of autoguidance, but this would be interesting future endeavor.
38

---

Published as a conference paper at ICLR 2025
1.4 STEP SIZE AND STOCHASTICITY SCHEDULES
As discussed in the previous section, sampling Proteína involves simulating the SDE
dx, vv, (x,, o)dt + g(t)s o)dt 22g(t) dWp
(33)
from - 0 to t= 1, where the vector field and score can also be subject to guidance. This is exactly Eq. (3) in the main text, repeated here for convenience. In practice, we simulate the SDE using the Euler-Maruyama method detailed in Algorithm 1. For all our experiments,
use N 400 discretization steps and g()) = 1/(t + 0.01) for E [0, 0.99] and g(t) = 0 for t (0.99, 1) (we empirically observed that numerically simulating the SDE may lead unstable simulation for t close to 1. We avoid this by switching to the ODE, setting g(t) = O, for the last few steps). We explore multiple values for the noise scaling parameter 7, which leads different trade-off's between metrics (see Tabs. and 3). We discretize the unit interval using logarithmically spaced points. More precisely, in PyTorch code (Paszke et al., 2019), we get No 11,... -r. (N] by the following three steps
1.0 torch. logspace (-2, 0, nsteps 1) fflipp (0)
t t - torch. min (t)
t / Corch.max (t),
where the last two operations ensure that 4o 0 and 4 = 1.
Algorithm 1 Buler-Maruyama numerical simulation scheme
Input: Number of steps N
Input: Discretization of the unit interval to 4 < <t < ... < IN =1
Input:
Stochasticity schedule g(t)
Input:
Noise scaling parameter y
Input:
Conditioning variables ē
Xo I), for n = to N do
Cn T)
6n = ln- in-1
(x), Xt, Xtn + Vin (x + ((n (X,, 20 sn - / 220gg(1n)7
end for Output: X1
J ON THE RELATION BETWEEN FLOW MATCHING AND DIFFUSION MODELS
question that frequently comes up is the relation between flow matching (Lipman et al., 2023; Liu al., 2023; Albergo & Vanden-Fijnden 2023) and diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021). For Proteína, we opted for flow seitching-basee approach, but in protein structure generation, both approaches have been leveraged the past. Hence, here we discuss the two frameworks.
Crucially, we would first like point out that we are using flow matching couple the training data distribution (the protein backbones for training) with a Gaussian noise distribution, from which the generation process is initialized when sampling new protein backbones after training. this case, i.e. when coupling with Gaussian distribution, flow matching models and diffusion models can in fact be shown be equivalent up to reparametrizations. This because diffusion models generally use Gaussian diffusion process, thereby also defining Gaussian conditional probability paths, similar to the Gaussian conditional probability paths in flow matching with a Gaussian noise distribution.
For instance, when using Gaussian noise distribution, one can rewrite the velocity prediction objective used in flow matching as a noise prediction objective, which is frequently encountered in diffusion models (Ho et al., 2020). Different noise schedules in diffusion models can be related to different time variable reparametrizations flow models (Albergo et al., 2023). Most importantly, for Gaussian flow matching, can derive relationship between the score function Vx, log p. (x.)) of
39

---

Published as a conference paper at ICLR 2025
the interpolated distributions and the flow's velocity (see Eq. (2) well as App. L.). The score is the key quantity in score-based diffusion models (Song et al., 2021). Using this relation, diffusion-lit stochastic samplers for flow models can be derived, as well as flow-like deterministic ODE samplers for diffusion models (Ma et al., 2024). In conclusion, we could in theory look at our Proteína flow models equally as cosee- diffusion models. With that in mind, from a pure performance perspective flow matching-basse perform similarly well when coupling with a Gaussian noise distribution. In practice, performance boils down to choosing the best training objective formulation, the best time sampling distribution to give appropriate relative weight to the objective (see Sec. 3.2). etc. these aspects dictate model performance, independently of whether one approaches the problem from a diffusion model or flow matching perspective.
approaches and diffusion-basee approaches should in principle
In fact, we directly leverage the connections between diffusion and flow models when developing our stochastic samplers (see App. I.1) and guidance schemes. Both classifier guidance (Ho & Salimans, 2021) and autoguidance (Karras et al., 2024) were proposed for diffusion models, but due to the relations between score and velocity, we can also apply them to our flow models (to the best of our knowledge, our work is the first to demonstrate classifier-free free guidance and autoguidance for flow matching of protein backbone generation). Please see App. 1.2 for all technical details regarding guidance in Proteína.
Considering these relations, why did we overall opt for the flow matching formulation and per- spective? (i) Flow matching can be somewhat simpler implement and explain, as it is based on simple interpolations between data and noise samples. No stochastic diffusion processes need to be considered. (ii) Flow matching offers the flexibility be directly extended to more complex interpolations, beyond Gaussians and diffusion-like methods. For instance, we may consider optimal transport couplings (Pooladian et al., 2023; Tong et al.. 2024) to obtain straighter paths for faster generation or we could explore other, more complex non- -au-saas noise distributions. We plan to further improve Proteína in the future and flow matching offers more flexibility in that regard. At the same time, when using Gaussian noise, all tricks from the diffusion literature still remain applicable. (iii) The popular and state-of-the-arr large-scale image generation system Stable Diffusion 3 is similarly based on flow matching (Esser et al., 2024). This work demonstrated that flow matching can be scaled to large-scale generative modeling problems.
We would like to point out that the relations between fow-matching and diffusion models have been discussed in various papers. One of the first works pointing out the relation is Albergo & Vanden- Fijnden (2023) and the same authors describe a general framework in Stochastic Interpolants (Albergo et al., 2023), unifying a broad class of flow, diffusion and other models. Some of the key relations and equations can also be found more concisely in Ma al. (2024). The relations between flow matching and diffusion models have also been highlighted in the Appendix of Kingma & Gao (2023). The first work scaling flow matching large-scale text-to-image generation is the above mentioned Esser et al. (2024), which also systematically studies objective parametrizations and time sampling distributions, similarly leveraging the relation between flow and diffusion models.
K
NEW TIME t SAMPLING DISTRIBUTION
crucial parameter in diffusion and flow matching models is the sampling distribution p(t), which effectively weighs the objective (Eq. (1) for Proteína). Enhanced sampling of closer to f = 1 encourages the model to focus capacity on synthesizing accurate local details, which are generated at the end of the generative process, while sampling more at smaller t can improve larger-scale features. In image generation it is common to increase sampling at intermediate (Karras al., 2022; Esser et al., 2024), but this is not necessarily a good choice for protein structures- even slightly perturbing a structure could lead to unphysical residue arrangements and bond lengths. Hence, as discussed in Sec. 3.2 we designed a new & sampling function focusing more on large t,
p(t) 0.02 1) + 0.98 1.1...
where -(. is the Beta distribution, to encourage accurate local details. We mix in uniform sampling to avoid zero sampling density when t - O. In Fig. 20, we show our novel distribution, naive uniform distribution, and the logit-normal distribution that recently achieved state-of-the the-trr image synthesis in a similar rectified flow objective (Esser et al., 2024). Ablations can be found in App. I..
40

---

Published as a conference paper at ICLR 2025
2.0
Mixture(0.02 Uniform 0.98 Beta(1.9, 1))
Uniform
Logil-nurmal(O M
1.5
P()
1.0-
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 20: t-sampling distributions. We show our novel f-sampling distribution from Sec. 3.2 that mixes a Beta and uniform distribution, a naive uniform distribution, and the logit-normal distribution that recently achieved state-of-the-al image synthesis in a similar rectified flow objective (Esser et al., 2024).
2.70
Mixturetc.00
Beta/1.5,
Detal1.6:11
Diversit (Clustee)
Betall. 9, 11
Betal3,
2.55
0.39
Logk-normal(0.1)
Model1.99
Diversity
0.32
2.40
0.
5.5
57 Designability
Designability
Figure 21: Siiggaablitt-Diverrsii trade-offs achieved when training a "small" model using different t-sampling distributions. The curves obtained by sampling each trained model for multiple noise scaling parameters y between 0.25 and 0.55.
L ABLATION STUDIES
This section presents multiple ablations we carried out while developing the model. We tested multiple distributions to sample the time during training (App. L..), different stochasticity schedules g(() (App. I.2),, and explored various architectural choices (App. I..3). We note that these ablations were done at different stages during model development, and are thus not always directly comparable between each other, nor with the results presented in the main paper, as they were carried out with different (often small) models or sampling schemes. However, these ablations informed our decisions while developing model and training regime.
L.1 SAMPLING DISTRIBUTIONS FOR f
We consider multiple choices for the (-samlling distribution p()): the mode (1.29) distribution from Esser al. (2024), the Logit-normal (1, 0) distribution from Esser et al. (2024), Beta 1). Beta (1.6, 1). Beta (1.9, 1). Mixture (0. 20w002+Unnfrmmm of these we trained a "small" model (30M parameters, no pair updates) the Drs dataset for 150K steps, using 4 GPUs with batch size of 25 per GPU. Noting that training losses are not directly comparable for different (-sampling distributions, designability- diverrsty trade-off they achieve when sampled under different noise scaling parameters y. Results shown in Fig. 21, where show diversity metrics (cluster diversity and TM-score diversity, see App. F for details) as a function of designability. The curves are obtained by sweeping the noise scaling parameter y between 0.25 and 0.55. We can observe that the mixture distribution
38* 1)), and Uniform (0, 1). For each
compare the models by studying the
41

---

Published as a conference paper at ICLR 2025
consistently achieves the best trade-offs for both diversity metrics. Note that the training run using the uniform distribution displayed somewhat unstable behavior, producing nan values during training, so we did not sample it.
We performed this ablation early during model development, using the stochasticity schedule g(t) = (1-t)/(t++ 0.01) and uniform discretization of the unit interval (with 400 steps). We emphasize that these results are not comparable with the ones the main papar and other sections the Appendix, as they were obtained using a significantly smaller model, trained for less steps using less compute, and sampled using a different numerical simulation scheme.
L.2 STOCHASTICIT SCHEDULES g(()
In addition to the schedule s(1) = 1/(l + 0.01) presented in App. 1.4, used for all re- sults we report in the main paper, we also tested the schedules 91-x(t) = (1 t)/ 0.01) and Itan in Fig. 22, where it can be observed that Itan and 91-t inject significantly less noise for times ules. Results for deisgnability, diversity and novelty (w.r.t. PDB) are shown in Fig. 23. can be observed that the stochasticity sched- ule used has strong effect in the model's 's final performance, with g()) leading better results
2 tan ((1 (11-t) g). A comparison of these three schedules is shown
1. We sampled and evaluated our Mrs model for these three sched-
102
g(t)
9an(t))
than Itan (t) and 9n (t))(t). Note that, in princi- ple, for y = 1 all stochasticity schedules yield the same marginal distributions during the sam- pling process. In practice, however, the SDE is simulated numerically, and we scaling parameter y < (common in diffusion and flow-based generative methods for protein backbone design). These two factors have non- trivial interaction with the stochasticity sched- ulc, explaining the differences in results for the different (t) considered.
101
g (t)
10a
10:
noise
10
10-3
0.0
0.2
0.4
0.6
0.8
1.0
t
Figure 22: Different stochasticity schedules tested function of t.
L.3 QK LAYER NORM, REGISTERS AND ROPE EMBEDDINGS
We also ablated several choices for the architecture, with an interesting one being the addition or removal of pair updates with triangular multiplicative layers (model Mys against model MM These two models were compared in the main text in Tab. 1. While the use of pair updates with triangular multiplicative updates leads to better performance, it also has a negative impact on the model's scalability. In Tab. I we observed that Mpo model is still competitive while being significantly more computationally efficient, which enabled us to scale to protein backbones of up to 800 residues, discussed in the main text.
Other architectural choices we ablated involved the use of QK layer norm, registers, and rotary positional embeddings (RoPE) (Su et al., 2024) for the attention in the network's trunk. These changes only affect the architecture, so training losses are directly comparable. Training small models (see App. L. observed that the use of registers and QK layer norm led to slightly improved training losses, so we included them in our final architecture. The use of RoPE embeddings, on the other hand, led to a slight increase in the training loss, so we did not include it in our final architecture.
*For numerical stability we compute Stan (t) = 5(0)+0.0T where C(t) cos ((1 () and S(I)
C(t)
sin ((1 t)
42

---

Published as a conference paper at ICLR 2025
0.70
g(t)
Diversity (Qlstte)
0.65
Diversity (TM-score)
91. +l0)
0.60
0.37
0.55
g(t)
0.45
91 :(4)
0.35
0.80
0.85
0.75
0.50
0.85
0.90
Designabiiity
Designability
0.72
gie
Novelty (PDB)
0.71
91 :10)
0.68
0.75
0.80
0.85
0.90
0.95
Designabllity
Figure 23:
Results of sampling Mrs under three different schedules, g(),, Itan (() and g1-)(!).
Curves are obtained by sweeping the noise scaling parameter y between 0.3 and 0.5.
M DATA PROCESSING
M.1 PDB PROCESSING, FILTERING AND CLUSTERING
For PDB datasets, we use metadata from the PDB directly to filter for single chains with lengths 50-256, resolution below 5Å and structures that do contain non-standard residues. We also include chains from oligomeric proteins. We include structure-based filters, namely max. coil proportion of 0.5 and max. radius of gyration of 3.0nm. Together, this leads 114,076 protein chains.
For LoRse fine-tuning, we prepare a subset of the dataset above with only designable structures. For this, we feed all 114,076 chains from the first dataset through the designability pipeline (Protein- MPNN, ESMFold), only keeping chains that have scRMSD below 2A. With this, we reduce the dataset size to 90,423 proteins, indicating that 79.26% of the samples from the original PDB dataset described above were designable.
M.2 ALPHIAFOLD DATABASE PROCESSING, FILTERING CLUSTERING
Processing the full AFDB- AFDB-Uniprot Uniprot dataset takes more than 20TB in disk space and includes around 214M individual file objects, both of which make hard work with this data. To remedy that, leverage FoldComp (Kim et al., 2023) as tool to enable efficient storage and fast access. FoldComp leverages NeRF (Natural Extension Reference Frame) encode structure information in bytes per residue and allow fast compression/deoopressioi scheme with minimal reconstruction loss.
Combined with FoldComp use the MMseqs2 database format (Steinegger & Söding, 2017) filter the AFDB-UniProd database into custom databases based on our filters and allow fast random-access and on-the-fly decompression training time. These filters include sequence length, pL.DDT values (mean and variance over the sequence), secondary structure content and radius of gyration.
In addition, we want to avoid random sampling these databases since, after filtering, these datasets tend to be biased towards certain fold families. We therefore either utilise pre-computed clustering of the AFDB such as the databases according to sequence similarity via MMseqs2. We then use the mapping from cluster representative to cluster members to define a Py Torch Sampler that iterates over all clusters during one epoch, picking random cluster member for each cluster.
clusters Stario-Hernandes et al., 2023), or cluster the
43

---

Published as a conference paper at ICLR 2025
Via this data processing pipeline, we prepare the different datasets that are described in the main section of the paper:
1. High-quality filtered AFDB subset, size 22MM, D2IM: We filtered all =214M AFDB structures for a max. length of 256 residues, min. average pLDDT of 85, max. pLDDT standard deviation of 15, max. coil percentage of 50%, and max. radius of gyration of 3nm. After additional subsampling this led to 20,874.485 structures. We further clustered the data with MMseqs2 (Steinegger & Söding, 2017) using a 50% sequence similarity threshold. During training, we sample clusters uniformly and draw random structures within.
2. Foldseek AFDB clusters, only representatives, size 0..M Drs: This dataset corresponds the data that was also used by Genie2 (Lin et al., 2024), based on sequential filtering and clustering of the AFDB with the sequence-based MMseqs2 and the dtructure-based Foldseek (van Kempen et al., 2024; Garrio-Hernander et al., 2023). This data uses cluster representatives only, i.e. only one structure per cluster. Like Genie2, we use length cutoff at 256 residues and pLDDT filter of 80, as well as minimum length cutoff of 32 residues. We found that with this processing we obtained 588,318 18 instead of 588,571 AFDB structures compared to Genie2. We attribute this difference to two reasons: first, some pLDDT values that are listed as 80 in the AFDB are rounded to 80 and do not equal 80 when computed as an average directly from the per-residue pLDDT data, leading to samples that are in the dataset used by Genie2 but not in our dataset. Second, the AFDB clustering (Barri-Hermander Hermandez et al., 2023) was done on version 3 of the AFDB, whereas in version 4 some of the structures were re-predicted with better confidence values. Several structures that were excluded in the data used by Genie2 due to having low PLDDT values at the time of version 3 had significantly improved PLDDT values in version 4. This leads samples that are in our version of the dataset but not in Genie2's.
During long-length fine-tuning, we extend Drs by progressively increasing the maximum length considered, up to a maximum chain length of 768 residues.
M.3 CATH LABEL ANNOTATIONS FOR PDB AND AFDB
In order to make our model more controllable via methods like Classifier-Fre Guidance (CFG), we leverage hierarchical CATH fold class labels (Dawson et al., 2016) for both experimental and predicted structures. These labels come in a hierarchy with multiple different levels (also see Fig. 3 in the main text):
Class (C): describes the overall secondary structure content of the protein domain, similar to SCOPe class (Lo Conte al., 2000).
Architecture (A): describes how secondary structure elements are arranged in space (for example sandwiches, rolls and barrels).
Topology (T): describes how secondary structure elements are arranged and connected to each other.
Homologous superfamily (H): describes how likely these domains are evolutionarily related, often supported by sequence information.
Since we are mostly interested in structural features for guidance, we focus on the CAT levels of the hierarchy, discarding the H level. In addition, we focus mostly on the three major C classes ("mostly alpha", "mostly beta" and "mixed alpha/beta") and ignore the smaller special classes ("few secondary structure", "special").
PDB CATH labels: To obtain CATH labels for the individual PDB chains that we use as data points, we leverage the SIFTS resource (Structure Integration with Function, Taxonomy and Sequences resource) (Velankar et al., 2012), which regularly updated and provides residue-level mappings between UniProt, PDB and other data resources such as CATH (Dana et al., 2019). For each of our samples, we map from the PDB ID and chain ID to the corresponding UniProt ID via the cabb_chain_uiirot tsv. gz mapping, and from there to the corresponding CATH IDs and CATH codes via the páb sar__chhh____aaeeeeeeeeeeeeeeeee cath www.unnnntttttee .2 mapping. Some chains have more than one domain and then also more than one CATH code. For these, we use all CATH codes available (we randomly sample the conditioning CATH code fed to the model during training in those cases). We then truncate the labels to remove the H level and end up with CAT labels only.
44

---

Published as a conference paper at ICLR 2025
(a) Pair Update
Liscar
Sequence Repe.
INN
Ouse Sam
Pall Reps.
Muliplicuive Friengl: Layer
Multifliative Trangle Layer
RoLU Linear
Pair Kepe
(b) Adaptive LayerNorm (LN)
(c) Adaptive Scale
Inpec
LX too skift or scalet
Outpe:
lepui
Oupat
Conditioing
IN
Sigmeid
Conditonias
Siymoid
Figure 24: Additional modules of Proteína's transformer architecture. (a) Pair Update. (b) Our adaptive
LayerNorm (c) Adaptive Scale.
AFDB CATH labels: To obtain CATH labels for the individual AFDB chains that data points, we leverage the TED resource (The Encyclopedia of Domains) (Lau et al., 2024b) to map from the AFDB UniProt identifier the corresponding CAT/CATH codes. Again all available CATH codes and remove the H-level information if it is present.
N ADDITIONAL NEURAL NETWORK ARCHITECTURE DETAILS
Visualizations of the additional Pair Update, Adaptive LayerNorm (LN), and Adaptive Scale modules are shown in Fig. 24.
When creating the pair representation (see Fig. (c)), the pair and sequence distances created from the inputs x,, xxx)) and the sequence indices are discretized and encoded into one-hot encodings. Specifically, for the pair distances from X₁ we use 64 bins equal size between 1 and 30Å with the first bin being < LÂ and the last one being >30A. For the pair distances from x(x,)
128 bins of equal size between 1Â and 30A with the first bin being <1Å and the last one being >30A. For the sequence separation distances we use 127 bins for sequence separations [<-63, -3,, -2, -61, 61, 62, 63, >63]. As shown in Fig. this pair representation can be (optionally) updated throughout the network using pair update layers. These feed the sequence representation through linear layers to update the pair representation, which is additionally updated using triangular multiplicative updates (Jumper et al., 2021), shown in Fig. 24. While powerful, these triangular layers are computationally expensive; hence, we limit their use in our models. In fact, shown in Tab. 16, our M model completely avoids the use of these layers (leading to a much more scalable model), while Mps and M21M use and triangular multiplicative update layers, respectively. In this work we did not explore the of triangular attention layers (Jumper et al., 2021), )), as these are even more memory and computationally expensive, limiting the models' scalability.
We generally use 10 register tokens all models when constructing the sequence representation. Sequence conditioning and pair representation zero-padded accordingly.
The MLP used when creating the sequence conditioning (see Fig. (b)) corresponds to a Linear- LaL---Linarr-SiiGii
architecture (Shazeer, 2020).
Specific architecture hyperparameters like the number of layers, attention heads and embedding sizes used during training of different Proteína models can be found in App. O.
45

---

Published as a conference paper at ICLR 2025
O EXPERIMENT DETAILS AND HYPERPARAMETERS
This section provides details about our model architectures as well as training and sampling configu- rations for our experiments.
O.1 TRAINED PROTEÍNA MODELS
Tab. 16 presents the hyperparameters used to define the three architectures considered in this paper, giving details about number of layers, dimensions of each feature, and number trainable parameters, among others. It also offers details used, the number of training steps, and the batch size per GPU. All our models were trained using Adam (Kingma, 2014) with 31 = 0.9 and B2 0.999. We use random rotations to augment training samples.
the training of
models, such the number of GPUs
Table 16: Hyperparameters for Proteina model training.
Hyperparameter
Pre-training
Fine-tuning
MEs
M
MamM
MiDRA
Mung
Proteina Architecture
initialization
random
random
random
Mys
Mp-ei
sequence repr dim registers
768
768
1024
768
768
10
10
10
10
10
sequence cond dim sinusoidal enc dim idx. sinusoidal enc dim
512
512
512
512
512
256
256
256
256
256
128
128
128
128
fold emb dim
256
256
256
256
256
pair repr dim
512
512
512
512
512
separation dim
128
128
128
128
128
pair distances dim (xx)
64
64
64
64
64
pair distances dim (8(x)) )
128
128
128
128
128
pair distances min (A)
1
1
1
1
pair distances max (A)
30
30
30
30
30
allention heads
tranformer layers
triangle layers
5
trainable parameters
200M
200M
400M
7M
200M
Proteina Training
steps
200K
360K
180K
11K
220K/80K
batch size GPU
4
10
4
6
2/1
GPUs
128
96
128
32
128
grad. acc. steps
2
1/2
O.2 UNCONDITIONAL GENERATION EXPERIMENTS
This section presents precise details for all results for unconditional generation shown in Tabs. 1 and (not including LoRA fine-tuning, covered in App. O.3). All experiments follow the sampling algorithm described in App. I.4.
Tab. shows results obtained for our Mrs and M models under multiple noise scales. 5 We sampled the MEs model without self-conditioning, since we observed that this yielded better trade- offs between designability, diversity, and novelty in the unconditional setting. On the other hand, we used self-conditioning with the Ma model, since observed it often led to slightly improved performance. The noise scales 7 E (0.35, 0.45, 0.5} shown for Mrs were chosen to show different points in the Pareto front between different metrics, while still retaining high designability values. On the other hand, M was sampled for a single noise scale (7 = 0.45) to show that even without pair track (i.e., updates the pair representation, yielding significantly improved scalability) model still performs competitively.
Tab. also shows results for the MaIM model for different noise scales 7 E {0.3, 0.6). These two runs have different purposes. The one with lower noise scale aims to show that we can achieve
'While the M model was trained for 360k steps, we observed better designability trade-offs for earlier training checkpoints. Therefore, for that model, we show results after 80k steps.
46

---

Published as a conference paper at ICLR 2025
Protas
Grnied
xdMMSD)
Foldflow
RIDittutton
Frusseflow
PramaDif
ESM
Numberr Residlaes
600
300
400
Namber Blsiiitass
500
600
700
800
Figure 25: scRMSD values for long protein generation, with zoomed-out view (left, y-axis: 0 Å to 30 Å) and zoomed-in view (right, y-axis: 0 to 5 A).
extremely high designability values by training large dataset filtered for high quality structures (to achieve this we use oef-ccondiioning for this run), while the other one attempts to show better trade-offs between different metrics (achieved without self conditioning).
Finally, Tab. 3 shows results for ODE sampling for the MMs and M21M models (both runs with self-conditioning, which yields better designability values). These runs can be observed to produce significantly better values for our new metrics, FPSD, fS and fISD. This is expected since scaling the noise term in the SDE is known to modify the distribution being sampled.
O.3 LORA FINE-TUNING ON PDB
To enhance model's ability generate both designable and realistic samples, curate high- quality, designable PDB dataset as outlined in App. M.1. We then fine-tune our best unconditional model, Mrs, on this processed dataset. To prevent overfitting and enable efficient fine-tuning, apply LoRA (Hu et af., 2022) with rank of and a scaling factor of 32, introducing trainable low-rank decomposition matrices into all embedding and linear layers of the model. This reduces the number of trainable parameters to 7M, significantly lower than the original 200M parameters. The complete training configuration is detailed Tab. 16. For inference, we observe that enabling self-conditionin consistently improves designability, so we adopt it for this model.
0.4 CONDITIONAL GENERATION EXPERIMENTS
For conditional generation, we follow the same schedule as unconditional sampling. with self- conditioning enabled, we explore the effect of classifier free guidance (CFG) by sweeping the guidance weight among 1.0, 1.5, and 2.0 for the model Mesnd with noise scale of 0.4.
find it improves designability and re-classification probabilities. In Tab. 2,
During sampling, we account for the compatibility the (protein chain length, CATH code) combi- nations to avoid generating unrealistic lengths for certain classes. We divide the lengths into buckets ranging from 50 to 1,000, with a bucket size of 25, An empirical label distribution is then constructed for each length bucket based the datasets Drrs and D2IM- For each length in conditional sampling, we randomly select a CATH code from the empirical distribution corresponding to that length.
Class-specific Guidance. To showease the utility of guidance different hierarchical levels, we also perform class-specific guidance where we guide the model only via class labels ("mainly alpha", "mainly beta", "mixed alpha/beta")) to control secondary structure content in samples while still maintaining high designability and diversity. We sample the model conditionally with a guidance weight of 1, and a noise scale of y = 0.4 for the conditional classes and ς = 0.45 for the unconditional case. With this configuration, we generate samples of length 50, 100, 150, 200, and 250 with 100 examples each, totaling 500 samples, which are used to report designability, diversity, novelty, and secondary structure content in Tab. 4.
47

---

Published as a conference paper at ICLR 2025
0.5 LONG LENGTH GENERATION EXPERIMENTS
For the long length generation results in Fig. 8 we first take the model M after 360K steps and fine-tune it for long-length generation; for this, we train it for 220K steps on the AFDB Cluster representatives filtered minimum average pLLDT 80, minimum length 256 and maximum length 512. We then train it for 80K more steps on the same dataset, but with the maximum length increased to 768, We sample this final model MMong with a noise scale of 7 = 0.35 and 400 steps to generate samples of lengths 300, 400, 500, 600, 700 and 800 with 100 examples per length. These samples are then subject to the previously described metric pipeline calculating designability and diversity. Similarly. for Fig. 8, we sample these lengths from each baseline in accordance with App. P. Also see Fig. 25 for scRMSD plots of Proteína and the baselines for the long protein generation experiment.
In addition, we combine these long length generation capabilities of our model with class-specific guidance (i.e. conditional sampling of the model while providing labels at the C level of the CATH hierarchy) to obtain large proteins with controlled secondary structure content.
O.6 AUTOGUIDANCE EXPERIMENTS
In Fig. 7, we show both conditional and unconditional sampling of our model, M21M, using the full distribution mode (ODE). The checkpoint at 10K training steps serves as a "bad" guidance checkpoint, corresponding the "reduced training time" degradation discussed in the original paper (Karras et al., 2024). For both conditional and unconditional sampling, we apply self self-conditioning while keeping all other inference configurations consistent with those described earlier.
P BASELINES
In this section, briefly list the models that sampled for benchmarking and the sampling configurations we used.
We used the code from the Genie2 public repository. We loaded the base checkpoint that was trained for 40 epochs. The noise scale was set to for full temperature sampling and 0.6 for low temperature sampling. The sampling was in the provided docker image.
Genie2:
RFDiffusion: We used the code from the RFDiffusion public repository. The sampling was run in the provided docker image. Default configurations of the repository were used for sampling.
ESM3: We followed the instruction in the ESM3 public repository to install and load the publicly available weights through the HugginFace API. When sampling structures, set the temperature to 0.7, and number of steps to be L where L is the length of the protein sequence. It is noteworthy that ESM3 performs relatively poorly on metrics evaluating unconditional generation. This may be expected as ESM3 is trained on many metagenomic sequences which are less designable.
We used the code from the FoldFlow public repository. When sampling the FoldFlow- base model, we set both the ot-plan and st stochast setticcpattt in the ration to False. When sampling the FoldFlow- -TT model, we set the ot plan to True. Lastly, when sampling the FoldFlow-SFN, model, we set both the et plan and the stochastis pa th to True, with the noise noise_scall set to 0.1. For all three models, we set the configuration flow_matcher so3. yields a performance closest to the results reported in the FoldFlow paper (Bose et al., 2024).
FoldFlow:
configu-
to 5 as empirically found that such setting
FrameFlow: We installed FrameFlow from its public repository. The model weights are downloaded from Zenodo. Default settings are used for unconditional sampling.
Chroma: We used the code from the Chroma public repository. Model weights were downloaded through the API following the instructions. Default settings were used for unconditional sampling. For conditional sampling using CAT labels, we used the default ProclassCondi tioner provided in the repository to guide the generation.
FrameDiff: We used the code from the FrameDiff public repository, using the public weights from the paper located in . /tweeihtt//t/// s/pper weight s. ht.. The default configuration of the repository was used for sampling. The sampling was run in the provided conda environment.
48

---

Published as a conference paper at ICLR 2025
We used the code from the Proteus public repository, using the public weights from the paper located in //weights /pappapeaaa. seaper_weiie weights pt. The default configuration of the repository was used for sampling. The sampling was run in the provided conda environment.
Proteus:
49


Reviews:
[{"review_id": "S0ROfvO6uU", "reviewer": "Reviewer_Yrch", "summary": "This paper introduces Proteína, a foundational model for generating protein backbones with innovative fold class conditioning for precise control over structure. Proteína achieves state-of-the-art results, synthesizing diverse backbones up to 800 residues long using a scalable transformer model. It leverages a 21M-sized dataset from the AFDB and, at over 400M parameters, can generate highly designable proteins even with synthetic data. The model pioneers classifier-free, autoguidance, and LoRA-based fine-tuning in protein structure generation, and introduces novel metrics for assessing generator behavior.", "questions": "1. Given that the non-equivariant model still relies on random rotations as data augmentation, is the choice to adopt a non-equivariant transformer a fundamental advantage over equivariant models, or simply a practical compromise? In other words, is it reasonable to expect that, with the current dataset and model architecture, introducing equivariance into the model could further improve performance?\n2. The three newly introduced metrics (FPSD, fS, fJSD) each seem to reflect certain distributional similarities between the generated data and the reference dataset. Could the authors clarify the specific focus of each metric? Additionally, is there a possibility of merging these metrics to produce a more unified and intuitive measure?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces novel metrics that address previously omitted distribution-level aspects of protein generation, which is both valuable and innovative, allowing for a more comprehensive evaluation of model performance. Additionally, the scaling of both training data and model aligns with the evolution of the field of protein generation.\n2. The paper proposes an innovative $t$ sampling method that effectively captures the unique characteristics of protein data. This is also the first application of classifier-free guidance (CFG) and autoguidance in protein structure generation.\n3. The methodology is presented with commendable clarity and detail.", "weaknesses": "In line 119, a partial derivative seems mistakenly written as a total derivative, and the divergence is incorrectly labeled as a gradient. I believe the right form of the continuity equation should be like $\\partial p_t(\\boldsymbol x_t)/\\partial t=-\\nabla_{\\boldsymbol x_t}\\cdot(p_t(\\boldsymbol x_t)\\boldsymbol u_t(\\boldsymbol x_t))$.\nAdditionally, the differential symbol should be formatted in upright type, as $\\mathrm{d}$, to follow standard conventions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLugYss5Xq", "reviewer": "Reviewer_KGtK", "summary": "The paper introduces Proteina, which is a flow-matching based protein structure generative model. The authors make efforts to explore the scalability of the new structure and the training data which results in the largest model with impressive performance. The model introduces several interesting techniques for boosting performance including the recent proposed auto-guidance. The paper proposes several new evaluation metrics with extensive baselines to demonstrate the effectiveness of the Proteina.", "questions": "refer to above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is certainly well written and I do enjoy the reading. \n\n2. The paper makes several very interesting yet important explorations and observations.  For example, though AF3 already observes the Equivariant vs Non-equivariant properties, it would be nice to further explore the scalability with non-equivariant transformers; The auto guidance parts of generation also provides some new insights into the protein structure generation; Studying protein structure generation in scale is also an important research direction for the community. \n\n3. I appreciate the efforts of providing comprehensive and sound evaluations for the protein structure generation tasks. Specifically, apart from the previously used simple quality/diversity measurements, the introduced FPSD/FjSD/fs could provide a fresh evaluation view at the distributional level. And I believe it could make more impact in facilitating future research.", "weaknesses": "1. Though with a scaled structure, it would be better to understand the training in a more systematic way, e.g. scaling laws. The trained flow matching model in general could still obtain the corresponding likelihood generally, could Proteina also conduct a likelihood evaluation over the protein structures? Is it possible to study the scaling laws based on that? \n\n2. The notation of Table 1 for models with different configs is not very clear which makes it hard to read and analyze.  I also suggest the authors list the parameter size of the proposed models and baseline in an extra column. So considering the claimed fact that the proposed model is 5 times more params than previous models, the unconditional performance of M_{21M} is worse than previous methods in the sense of FPSD/ FS/diversity, for example, FrameFlow, etc. This raises a major concern of whether the general framework and formulation could provide benefits. I would like to suggest the authors elaborate more on this. \n\nI would like to consider increasing my score if the concerns are fixed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WYnJ7mSjz", "reviewer": "Reviewer_deDq", "summary": "This paper introduces Proteina, a novel flow-based model for protein backbone generation using a scalable non-equivariant transformer architecture. Proteina incorporates fold class conditioning and supports generation of diverse structures up to 800 residues long.", "questions": "1. The quality of a fold class predictor can be evaluated through classifier-guidance.\n2. How was the multiview contrastive learning pretraining conducted for GearNet architechture? Did you initialize with GearNet's existing weights or carry out full pretraining? \n3. Can you please justify the use of the exponent in the Inception score equation if you are utilizing a variant without the exponent. Additionally, the Inception score has become less popular in recent years.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Very well-written paper and very easy to follow.\n2. The authors show that large-scale non-equivariant flow models also succeed on unconditional protein structure generation.", "weaknesses": "1. The authors claim to significantly outperform all previous works; however, evidence supporting this assertion is not found in the experimental results table. Excluding unconditional models, there are no direct competitors, and comparisons can only be made with unconditional results. Even if the bold results are accepted as outperforming based on FPSD, FS, fJSD, and TM-score metrics, this model exhibits the lowest diversity.\n2. RFdiffusion, ESM3, and Genie 2 were trained on different datasets, while the comparison in Table 1 uses PDB and AFDB, the comparison should be made against their respective training datasets. FPSD, FS, fJSD, and TM-score are uninformative metrics because they lack a defined optimum. To make metrics like FPSD, fS, fJSD, and TM-score meaningful, all competitors should be trained on the same dataset for a fair comparison. Instead of aiming for lower values (fS upper), the goal should be to achieve metrics that match the training dataset. Moreover, comparisons are made with PDB and AFDB without clarifying whether these datasets were part of the training set. If these examples were not used in training, there's no reason to expect lower (fS upper) values. The metrics also lack a clear scale. \n3. There are no direct competitors for conditional generation; however, RFdiffusion is capable of conditional generation and could potentially be used for CAT. For a proper comparison with competitors, other tasks such as motif scaffolding, inpainting, and fold conditioning should be addressed. \n4. The diversity TM-score is not a particularly informative metric because it lacks a clear optimal value. Instead of simply aiming for a lower score, we should be targeting diversity that aligns with the training dataset. Additionally, only the mean is reported without any measure of deviation, reducing its informativeness. The violin plot provides a better insight, showing the distribution and indicating that there are no identical structures (no collapse mode). When performing an unconditional comparison of the method with itself, the results appear unchanged, suggesting a lack of sensitivity in the metric.\n5. The results lack statistical significance.\n6. The code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0r8D5nBrYI", "reviewer": "Reviewer_WugM", "summary": "The paper introduces a novel flow-based generative model for protein backbone design. The model leverages a non-equivariant transformer architecture, adding to the ongoing debate on the applications of SE(3)-equivariant architectures. The proposed model incorporates hierarchical fold class conditioning for controlled generation, and has been trained on a large dataset of 21 million protein structures, more than any of the previous approaches. The authors also introduce distribution-level metrics for evaluation of generative models of protein structures.", "questions": "Could you please provide the details on how the GearNet-based classifier was trained? From scratch with contrastive objective as in the original paper, or from the pretrained model, if so, how exactly?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is written exceptionally well. The authors provide a lot of auxillary information and details on nearly every aspect of the work.\n\n2. This work pushes the frontier in terms of the training data scale and parameter count for the protein structure generation models. The authors demonstrate in detail their efficient architecture that enables scaling.\n\n3. The extensive ablation studies and the details on the architecture and experiments are valuable for the community.", "weaknesses": "1. The main issue is the experimental design: in the fold class-conditional generation, Proteina is not compared to any other models, while the unconditional generation setting is invalid, since the baselines are trained on different data while being compared to PDB and AFDB. This renders the statements on \"vastly outperforming all baselines\" false. Another concern about the experimental design is self-evaluation bias: the work introduces a dataset, a model, and a set of evaluation metrics that use their own classifier without external validation. \n\n2. The proposed metrics (FPSD and fJSD) are highly correlated and redundant. Calculation of Frechet distances to PDB and AFDB datasets does not make sense until the models are trained on those datasets. Generative models should generate samples resembling their respective training data, not PDB or AFDB. Either all the baselines should be trained on the same dataset and FPSD and fJSD calculated for this dataset, or another set of metrics should be chosen to evaluate unconditional generation. When using distributional distance metrics, it is a good idea to provide a sense of scale by measuring such distance between random samples in the dataset itself.\n\n3. I agree with the Authors' critique of the existing metrics like diversity and novelty. But there is more to it. The TM-score diversity metric is not informative and lacks an optimal reference point. Providing distributions over TM-scores (e.g., violin plots) would be much more informative than the provided mean over different means over different length ranges. The TM-score expected from the generative model depends on the dataset it is trained on, so providing such value for a subset of a dataset is a good idea. Another issue with metrics is that a single threshold for cluster diversity is not enough as a 50% level cannot detect mode collapse inside the clusters. For example, when each cluster consists of identical structures memorized by the model.\n\n4. The use of a classifier is justified in the experiments that include predictions on the noised data, however, the classifier has not been trained on the noisy data which makes the claim that it can evaluate protein realism weak. If the classifier was trained on the noisy structures, it could be used for classifier guidance.\n\n5. The task scope is limited, which hinders our ability to assess the model. The Authors show unconditional generation and fold class-conditional generation. However, the comparison with other models is poorly set up in the case of unconditional generation and is not performed at all in the case of fold conditioning. The remedy could come from solving other tasks where several baselines exist, for example, motif-scaffolding/inpainting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ypeoraSUA0", "reviewer": "Reviewer_v57v", "summary": "This paper presents Proteina, a new large-scale flow-based protein backbone generator that utilizes hierarchical fold class labels for conditioning and relies on a tailored scalable transformer architecture. The model is trained on up to 21 million protein structures, a significant increase compared to previous work. Proteina achieves state-of-the-art performance on de novo protein backbone design and produces diverse and designable proteins at unprecedented length, up to 800 residues. The hierarchical conditioning offers novel control, enabling high-level secondary-structure guidance as well as low-level fold-specific generation. In addition, the paper also offers numerous practical insights that can serve as valuable references, such as LoRA fine-tuning, autoguidance, and new evaluation metrics.", "questions": "- As a method for backbone design, is the input sequence representation in the model composed entirely of a sequence made up of the same special symbol?\n- Will the newly proposed metrics, which are based on the fold classifier, create an unfair competitive advantage for Proteina, as it uses CAT as an input? Could a more general classifier be used to improve the design of these metrics?\n- For a single protein design, are all amino acids input using the same CAT label (e.g., amino acids A and B both use the label X), or is the CAT label adjusted specifically for each amino acid (e.g., amino acid A uses Xa and amino acid B uses Xb)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper demonstrates the current state-of-the-art capabilities in protein backbone design.\n- The paper addresses the longstanding issue of insufficient $\\beta$-sheets in this field by introducing the CAT lable.\n- The paper introduces new evaluation metrics that offer a fresh perspective for examining the model.\n- The paper provides valuable insights by exploring numerous practical techniques, e.g., LoRA fine-tuning and autoguidance, in the field of protein design.", "weaknesses": "- Although the article provides detailed implementation details, Proteina, as a complex system engineering project, requires open-source code and data as essential resources for reproducibility.\n- The paper does not provide a discussion on the scaling trends of the data and model, meaning we cannot determine whether the 21 million protein structure data points and 400 million model parameters are necessary conditions for achieving the current performance of Proteina.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)