Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Generation
Authors: Guillaume Huguet
James Vuckovic
Kilian FATRAS
Eric Thibodeau-Laufer
Pablo Lemos
Riashat Islam
Cheng-Hao Liu
Jarrid Rector-Brooks
Tara Akhound-Sadegh
Michael M. Bronstein
Alexander Tong
Joey Bose
Venue: NeurIPS 2024 poster (2024)
Keywords: Proteins
Flow Matching
Generative Models
Primary area: machine_learning_for_healthcare
TL;DR: We propose FoldFlow++ a new sequence conditioned protein structure generative model using flow-matching which can be finetuned for Motif Scaffolding.

Abstract: Proteins are essential for almost all biological processes and derive their diverse functions from complex $3 \rm D$ structures, which are in turn determined by their amino acid sequences. 
In this paper, we exploit the rich biological inductive bias of amino acid sequences and introduce FoldFlow++, a novel sequence-conditioned $\text{SE}(3)$-equivariant flow matching model for protein structure generation. FoldFlow++ presents substantial new architectural features over the previous FoldFlow family of models including a protein large language model to encode sequence, a new multi-modal fusion trunk that combines structure and sequence representations, and a geometric transformer based decoder. To increase 
diversity and novelty of generated samples -- crucial for de-novo drug design -- we
train FoldFlow++ at scale on a new dataset 
that is an order of magnitude 
larger than PDB datasets of prior works, containing both known proteins in PDB and high-quality synthetic structures achieved through filtering. We further demonstrate the ability to align FoldFlow++ to arbitrary rewards, e.g. increasing secondary structures diversity, by introducing a Reinforced Finetuning (ReFT) objective. We empirically observe that FoldFlow++ outperforms previous state-of-the-art protein structure-based generative models, improving over RFDiffusion in terms of unconditional generation across all metrics including designability, diversity, and novelty across all protein lengths, as well as exhibiting generalization on the task of equilibrium conformation sampling. Finally, we demonstrate that a fine-tuned FoldFlow++ makes progress on challenging conditional design tasks such as designing scaffolds for the VHH nanobody.

Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generation
Guillaume Huguet :2,3; James Vuckovic'; Kilian Fatras', Eric Thibodeau-Laufer') Pablo Lemos', Riashat Islam', Cheng-Hao Liu Jarrid RectorBrrooss
Tara Akhound-Sadegh
1.2.4 1,2.4 Michael Bronstein 1,5,6 Alexander Tongl.2.3 Avishek Joey Bose Dreamfold, 2 Université de Montréal, MMila, "MeGill University, UUniversity of Oxford, 6 Aiithyra
Abstract
Proteins are essential for almost all biological processes and derive their diverse functions from complex 3D structures, which are in turn determined by their amino acid sequences. In this paper, exploit the rich biological inductive bias of amino acid sequences and introduce FOLDFLOW- 2t, a novel siguence-conditions -equivaaaaa flow matching model for protein structure generation. FOLDFLOW-2 presents substantial new architectural features over the previous FOL.DFLOWW family of models including protein large language model to encode sequence, a new multi-modal fusion trunk that combines structure and sequence representations, and geometric transformer based decoder. To increase diversity and novelty of generated samples-cnue crucial for de-novo drug design-we we train FOLDFLOW-2 scale new dataset that is order of magnitude larger than PDB datasets of prior works, containing both known proteins in PDB and high-quality synthetic structures achieved through filtering. We further demonstrate the ability to align FOLDFLOW to arbitrary rewards, e.g. increasing secondary structures diversity, by introducing Reinforced Finetuning (ReFT) objective. We empirically observe that FOLDFLOW-2 outperforms previous state-of-the-ee protein structure- baseu generative models, improving over RFDiffusion in terms of unconditional generation across all metrics including designability, diversity, and novelty across all protein lengths, as well exhibiting generalization on the task of equilibrium conformation sampling. Finally, we demonstrate that a fine-tuned FOLDFLOW-2 makes progress on challenging conditional design tasks such as designing scaffolds for the VHH nanobody.
Introduction
Rational design novel protein structures via generative modeling holds significant promise for accelerating computational drug discovery [Chevalier et al., 2017, Ebrahimi and Samanta, 2023]. In particular, the ability to design proteins with a pre-specified functional property is arguably one the principal tools in addressing global health challenges such COVID-19 [Cao et al., 2020, Gainza et al., 2023], influenza [Strauch et al., 2017], and cancer [Silva et al., 2019]. In many instances, designing function involves the design of both the 3D geometric structure of the protein as well as its specific chemical interactions. proteins, the amino-acid sequences determine the interaction between protein backbones and side chains, which fold into distribution of protein structures. Consequently, the functional properties of protein structures can be inferred from its sequence.
"Co-first and corresponding authors: [guillaume. huguet, .j james Odreaffolddd
1 Core contributor
1 Equul advising
'Our code can be found at http:thww.cac//eeeaaaaaaaaaaaaa.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The representation of proteins plays a key aspect in any computational approach to protein engi- neering. The 3D structure of proteins can be mathematically represented on the space of rotation and translation invariant SE(3)". Several unconditional protein generative models have been developed recently to generate new protein backbones [Yim et al., 2023b, Bose et al., 2024]. While these models demonstrate the ability to design new proteins, they are insufficiently tailored for downstream drug discovery applications, where the primary challenge lies in generating proteins that are specifically tailored to interact effectively with a given target. In real-world drug design problems, one often knows the target protein (its amino acid sequence and often an experimentally verified 3D structure). In machine learning terms, the design of new proteins ("de novo" design) that can drug the given target can be framed as a conditional generation problem. This raises the following research question:
How can we leverage the structure and sequence of a target to inform de novo protein design?
Current work. In this paper, we introduce FOLDFLOW-2 a novel protein structure generative model that is additionally conditioned on protein sequences. FOL.FLL.W-22 is built on the foundations of FOLDFLOW [Bose et al., 2024] and is an SE(3)* SE(3))-invanll) iinvarintt generative model for protein backbone generation that handles multi-modal data by design. Specifically, FOLDFLOW-2 introduces several new architectural components over previous protein structure generative models that enable it to process both 3D structure and discrete sequences. These include (1) a joint structure and sequence encoder; (2) multi-modal fusion trunk that combines the representations from each modality in shared representation space; and (3) a vransformer-based geometric decoder. In contrast to prior efforts to incorporate sequences in structure- baseed generative models [Campbell et al., 2024], FOLDFLOW-2 leverages the representational power of a large pre-trained protein language model in ESM [Lin et al., 2022] enabling it to make use of the rich biological inductive bias found in sequences but at a scale far beyond ground-truth experimental 3D structures found in the Protein Data Bank (PDB).
As
model, FOLDFLOW-2 is able to tackle a suite of new tasks beyond simple unconditional generation. Specifically, our model can additionally be used for protein folding by sim- ply generating structures conditioned on sequence as well as hard, biologically motivated conditional design problems. For instance, our model can perform partial structure generation by conditioning on masked sequence, i.e., structure in-painting. This enables FOLDFLOW-2 to be better equipped than prior structure-only generative models to tackle the key challenges in de novo drug design. For example, in settings where we aim to engineer structure that binds and neutralizes desired target protein structure and sequence pair; this is precisely a structure and sequence in- painting problem.
As diversity and quantity of training samples play a crucial role in downstream generative modeling performance on conditional design tasks, we construct new large dataset- an order of magnitude larger than PDB of high-quality synthetic structures filtered from SwissProt [Jumper et al., 2021, Varadi et al., 2021]. We further investigate the impact of fine-tuning FOLDFLOW-2 using Reinforced Fine-Tuuing (ReFT), a new approach that aligns flow-matching generative models to arbitrary rewards. In the context of protein backbone generation, we apply fine-tuning to improve the properties of gen- erated backbones, such optimizing for the diversity of secondary structures, as well as improving the performance on conditional generation tasks like generating scaffolds around a target motif.
Main results. We summarize the main empirical results obtained using FOLDFLOW-2 below:
We empirically demonstrate that FOLDFLOW-2 achieves sfate-of-the art performance for uncon- ditional generation and leads to the most designable, novel, and diverse proteins. In particular,
FOLDFLOW-2 improves over RFDiffusion [Watson et al., 2023] and FoldFlow [Bose et al., 2024). We find FOLDFLOW-2 closes the gap in performance with purpose-built folding models like ESMFold and improves by factor of R 4x compared MultiFlow [Campbell al., 2024], the most comparable protein structure generative model that also leverages sequences.
We use FOLDFLOW-2
solve a biologically relevant conditional design problem in motif scaffolding. We find that a fine-tuned FOLDFLOW-2 is able to solve all 24/24 scaffolds in the benchmark dataset from Watson et al. [2023]. On challenging VHH nanobodies, it solves 9/25 refoldable motifs in comparison to 5/25 for the previous best approach RFDiffusion.
able to perform zero-shot equilibrium conformation sampling on unseen proteins in the ATLAS molecular dynamics (MD) dataset [Vander Meersche et al., 2024] based on conformation variation seen within the protein data bank. We observe that FOLDFLOW-2 is able to capture different modes of the equilibrium conformation compa- rably to ESMFlow-MD [Jing et al., 2024], a model fine-tuned on MD data, but lags behind AlphaFlow-MD [Jing et al., 2024].
We hypothesize that FOLDFLOW-2
2

---

2 Background and preliminaries
2.1 Protein backbone and sequence parametrizations
Sequence representation. Protein sequences correspond to the chain of amino acids, which for a protein of length N is identified by a discrete token a' E {L...., 20} =: A. As is customary in protein language models [Lin et al., 2022], these discrete tokens are encoded using a one-hot representation. We denote the entire amino acid sequence associated with a protein as A E Nx200
Structure representation. The 3D structure of protein backbones can be represented as rigid frames associated with each residue in amino acid sequence [Jumper et al., 2021]. Each residue, i, within a protein backbone of length N consists of idealized coordinates of their 4 heavy atoms N", , C² O' E RR, with C = (0. 0, 0). The defining property of rigid frames is that they can be viewed as elements of the special Fuclidean group SF((3) and as such each frame x = (r, 8) E SE(33) contains rotation r and translation s component. Applying a rigid transformation x' to the idealized coordinates of the heavy atoms allows us to represent the rigid frame of a given residue, [N, C. C, OJ x' o [N", C C², O*), where is the binary operator associated to the group, which for S3) is simply matrix multiplication. This leads to structure representation of the complete 3D coordinates associated with all heavy atoms of protein as the tensor X E R x4x3.
SE(3): the group of rigid motions. The special Euclidean group SE(3) contains rotations and translations in three dimensions and can be thought of in several ways. It is Lie group, i.e., dif- ferentiable manifold endowed with a group structure. SE(3) can be seen as the group of rigid frames, representing 3D rotations and translations. As Lie group, SF((3 can be uniquely identified with its Lie algebra, the tangent space at the identity element of the group. SE(3) is also a matrix Lie group, meaning that its elements can be represented with matrices. It can formally be written as the semidi- rect product of the rotation and the translation groups, SE(3) ~ SO(3) K (IR3 +). A more detailed introduction to Riemannian manifolds and Lie theory, with an emphasis on SE(3) is provided in SA.
2.2 Flow matching on the SF(3) group
As Lic groups are smooth manifolds, they can also be equipped with Riemannian metric, which can be used to define distances and geodesics on the manifold. On SE(3). a natural choice of the metric decomposes into the metrics its constituent subgroups, SO(3) and Red [Bose et al., 2024, Yim et al., 2023b]. This allows us to build independent flows on the group of rotations and translations and induce flow directly on SE(3). As flow matching on Euclidean spaces is well- [Albergo and Vanden- Vanden-Eijnden, 2023, Lipman et al., 2023, Liu et al., 2023], we restrict our focus on reviewing flows, conditional probability paths, and vector fields over the group SO(3).
Probability paths on (3)) Given two densities po: P1 E SO(3)), a probability path pt [0, 1] ((S((3)) is interpolation, parametrized by time, t, between the two densities in probability space. Without loss of generality, we may consider Po to be the target data distribution and #1 an easy-to- sample source distribution. A flow is one-parameter diffeomorphism in t, th SO(3) + SO(3).
It is the solution to the ordinary differential equation (ODE): x th (r) = u, (r),,, with initial condition ()( = r, where Ue is the deeeneepnddnt smooth vector u Jo, 1] x SO(3) + SO(3). It is said that 1h generates Pt if it induces pushforward map Pt [+o] (po).
Matching vector fields on SO(3). The framework of Riemannian flow matching [Chen and Lipman. 2024] can also accommodate Lie groups such as SO(3). Consequently, to learn a continuous normalizing flow (CNF) that pushes forward samples ro ~ Po to r1 ~ PI we must regress a parametric vector field vo E x(S)(3)) in the tangent space of the manifold to the target conditional vector field u. (re|[ro r1). for all é E [0, 1]. Conveniently, the target ut (relroo r1) is the time derivative of a point along the shortest path between To and ri-iee. the geodesic interpolant rt = expr, (1 log,, (ri)). Furthermore, for S3(3) the target conditional vector field admits a closed-form expression (r(rrlro, r1) = log,, (ru)/t as the exponential and logarithmic maps are numerically computable using the angle representation of the group elements [Bose et al., 2024]. Given these ingredients, we can formulate the flow matching objective for SO(3) as:
Lsors) (0) =
[|v0(4, r.)) - log,, (ro) |3
5013)
In eq. (1), g(ro, r1) any coupling between samples from the source and target distributions. An optimal choice is to set r(ro,T1) = T()o,T1) which is the coupling, n, that solves the Riemannian
3

---

optimal transport problem using minibatches [Bose et al.. 2024, Tong et al., 2023, Fatras et al., 2020]. Finally, the generation of samples is done by first drawing from source sample ri ~ P1 and integrating the ODE backward in time using the learned vectorfield no.
3 FOLDFLOW-2
We now present FOLDFLOW-2, our sequence- conditioned structure generative model. FOLDFLOW-2 operates on protein backbones To Po which are parametrized as N rigid frames as well as their corresponding sequence a. As protein backbones contain symmetries from SE(3), we design FOLDFLOW-2 as an SE(3)"-invarian -invariant density using flow-matching objective. We achieve translation invariance by constructing the flow the subspace SE(3)," where the center of mass of the inputs removed. Additionally, we can focus on building flows on the group of rotations SO(3)
Table 1: Overview of the conditioning capability of unconditional (0), folding (A), and inpainting (A, X) of various protein backbone generation models.
Method
D
A
(A.
Alphafoold [Jumper et al., 2021]
x
/
x
RFDiffusion [Watson al., 2023]
X
Chroma [Ingraham el al., 2023]
/
X
FrameDiff [Yim et al., 2023b]
X
FoLDFI.on oow [Bose et al., 2024]
X
FrameFlow [Yim al., 2023a]
V
Motif RFDiffusion [Watson et al., 2023]
X
V
Multiflow [Campbell et 2024]
/
FOLDFLOW-2 (Ours)
and translations R.3, for each of the N residues independently, as SE(3), can be viewed as a product manifold consisting of N copies of SE(3), The overall loss function for the model decomposes into per residue rotation and translation losses L = Lso(3) + LRa.
To 10 | (e. a) log t 8O(3)
valt, 8c, a) (Bc su)
27
L = Ex-22(0011) (e (8a (.))
+
(2)
t
where the pair 21) ~ T x1 is sampled from the optimal transport plan æ. In addition, the sequence = m corresponds zo and is masked completely, with mask m, with a probability Bern(0.5). Operationally, this means 50% of the time the model is trained unconditionally with no sequence information, i.e.. ā [0]N while the other 50% the model has access the full sequence ā = a. Optimizing the loss in eq. (2) is equivalent to maximizing the conditional log- log-likelihood of observing protein structures given their sequences log (X when the sequence not masked and maximizing the unconditional log-likelihne likelieoehood log p(X) when the sequence is fully masked. Due to the ability to mask sequences, FOLDFLOW--2 enables new modeling capabilities in comparison existing models as outlined in table 1. More precisely, FOLDFLOW-2 trained using masked sequences can perform a diverse set of tasks, outlined in table 2, beyond simple unconditional backbone generation which aids in tackling more biologically relevant problems that require conditional generation such as mimicking protein folding model and designing the 3D scaffolds around a target motif.
With the breadth of tasks T1-T3 (table 2) FOLDFLOW-2 unlocks new structural design capabilitics beyond the simple unconditional generation ability of FOLDFLOW. We next outline the architectural components of FOLDFLOW-2 in $3.1 before detailing the training procedure in $3.2, which includes key design decisions regarding the construction of our new scaled dataset of ground truth PDBs and filtered AlphaFold2 synthetic structures. We also outline the inference procedure for sampling in $B.4. We conclude by discussing various techniques to fine-tune FOLDFLOW-2, including methods based on filtering with auxiliary rewards for supervised fine-tuning $3.3 to align protein structures.
Table 2: By manipulating the input modalities, FOLDFLOW-2 able perform diverse set of con- ditional and unconditional generation tasks including biologically relevant tasks such as designing scaffolds.
Task Name
Sequence Inputs
Structure Inputs
(T))
Unconditional
Fulke-masked masked
Noise
(T2) Folding
Unmasked
Noise
(T3) In- Painting
Partially Masked
Partially Masked
3.1 FOLDFL.OW-2 Architecture
The FOLDFLOW-2 architecture is comprised of three core components: (1) Structure and sequence encoder: An encoder which encodes both structures and sequences; (2) Multi-modal fusion trunk: the trunk which combines the multi-modal representations of the encoded structure and sequences; and (3) Geometric Decoder: decoder that consumes the fused representation from the trunk and outputs generated structure. The overall architecture of FOLDFLOW-2 is depicted in fig. 1.

---

Ealer Integration Steps (during inference)
Sequince Encoder
Multi-Madal Fusion Trunk
Geometzic Decoder
Strweerr Pacedkr DPA Tinr)
Steaddaee
Combensr Modale
Trusk Blodo
Decode
[IFA Tiut
EOOLDEE PYTRETSO0: AACTERCTVV
Sequenoc (RSM)
Sequenee thr Trink Misiale
Sequenoo-To- To-Trank Module
Combiner Module
Lounoote Powing
Single LaLerr MLP
Single Project =a 4/2 & Counos
Single ager
Repossantation
Pair
auptosantation
Pair LagerNonm
Rigid
Represcutation
MLP
Poslions
Creasi
Figure 1: FOLDFLOW-2 architecture which processes sequence and structure and outputs SE(3), vectorfields.
Structure and Sequence Encoder. We leverage existing state-of-the-art architectures to encode the structure and sequence modalities separately. For structure encoding, we rely on the invariant point attention (IPA) transformer architecture [Jumper et al., 2021], which is The benefit of the IPA architecture that it highly flexible and can both consume and produce structure-i.e., N rigid frames-ant and also output single and pair representations of the input structure.
To encode amino-acid sequences, of the ESM2 sequence model [Lin et al., 2022]. Large protein language models have a strong inductive bias on atomic-level predictions of protein structures while exhibiting strong generalization properties beyond any known experimental ftructures-whine novo structure design. Moreover, the ESM2 architecture also produces single and pair representations for an encoded sequence of amino acids, which conceptually correspond to the single and pair representations from the structure encoder. Consequently. the output space of each modality prescribes a natural fusion of representations into joint single and pair latent space for given input protein.
large pre-trained protein language model: the 650M variant
we argue is highly correlated with goals of de
Multi-Modal fusion trunk. After encoding both input structure and sequence, we construct a joint representation for the single and pair representation using a "project and concatenate" combiner module with simple MIPs, see fig. 1. We LayerNorm [Ba al., 2016) throughout the architecture as it is essential to accommodate differently-scalee inputs. The joint representations further processed by series of Folding blocks [Lin al., 2023], which refines the single and pair representations via triangular self-attention updates.
Geometric decoder. To decode the joint representations of the inputs into SE(3), vector fields, we once again leverage the IPA Transformer architecture. The decoder takes as input the single, pair outputs of the trunk and the rigid representations from the structure encoder. One of our major findings is that including a skip-connection between the structure encoder and the decoder is essential for good performance the temporal information is only given the structure encoder.
Given each component, we stack 2 - blocks for the encoder, trunk, and decoder components.
3.2 Training
We train FOLDFLOW-2 by alternating between both folding and unconditional generation tasks using novel sequence-and-strrutuue flow matching procedure, described below.
Dataset construction. The generalization ability of generative models trained using maximum likelihood is determined by the quality and diversity of curated training data [Kadkhodaie et al.,
5

---

2024]. Due to the limited size of ground truth structures in the Protein Data Bank (PDB) we aim to improve training set diversity by additionally curating dataset of filtered AlphaFold2 structures from SwissProt [Jumper et al., 2021, Varadi et al., 2021]. To ensure FOI.DFL.OW F..--2 is trained on high-quality synthetic structures, we employ a set of stringent filtering techniques that remove many undesignable proteins from SwissProt. After filtering, our final dataset consists of 160 structures and constitutes approximately an 8 fold increase compared to prior works [Yim et al., 2023b, Bose et al., 2024]. Our exact layered filtering strategy for synthetic structures in SwissProt is outlined by the following steps:
Filtering low-confidence structures. We use per-residue local confidence metrics like the average pLDDT to filter out low- confcdniee structures from the initial SwissProt dataset. (Step 2) Masking low-confidence residues. Globally high-confident structures may include low-confidence residues with disordered regions that can impede training. We use a per-residue DDDDTT threshold of 70 to mask such "low-quality" residues during training. (Step 3) Filter high-confidence, low-quality structures. The nature of synthetic data means that even following steps and low-quality data persists in a curated dataset. To combat this we further filter structures by learning a light-weight structure prediction model trained on structural features predictive of protein quality.
(Step 1)
We report a detailed analysis of each step in the filtration process in $B.1 which includes examples of low-quality structures that were filtered as illustrative examples. The impact of these findings is empirically corroborated in by analyzing generated samples from FOLDFLOW-2 in $C.2.
During training, we set the fraction of synthetic samples that may be seen during an epoch to 2/3 of the epoch. This prevents the model from overfitting to the remaining noise in the synthetic data, and is also common practice when training with synthetic data [Hsu al., 2022, Lin al., 2023]. Anecdotally, we did not notice an improvement from using a smaller proportion of synthetic structures. Finally, in the FOLDFLOW-2 architecture, we keep the ESM pre-trained language model fixed during training and train all other components (encoder, trunk, and decoder) from scratch. The results presented in table 3 and $4.4 use PDB data only, as this displayed the best performance for designability scores.
3.3 Fine-Tuning FOLDFLOW-2
We explore the efficacy of fine-tuning FOLDFLOW-2 with preferential alignment. We take a super- vised fine-tuning approach [Wei et al., 2022] that uses an additional fine-tuning dataset which is filtered using pre-specified auxiliary rewards Taux to create a preferential dataset Dpref- We term this Reinforced FineTuning (ReFT) since fine-tuning in this manner can be considered aligning FOLDFLOW-2 generations to the auxiliary reward. Summarizing this in three steps: (1) We take a curated dataset of proteins with desirable metries; (2) We use Taux to score the samples from step 1 and filter them to get a subset of high-scoring samples; (3) We then improve FOLDFLOW-2 by SFT on the filtered subset. Finetuning with ReFT optimizes the following optimization objective CREFT (0),
max CREFT (0) E(zza))~Ppro [Faux(x) (1) log po (x]aa)]. Pu
Compared recent alignment methods based on reward models, as in RLHF [Bai et al., 2022], ReFT uses a filtered structure dataset to fine-tune FOLDFLOW-2. Standard RL approaches seek to fine-tune generative model-based moner-generated data and assume access to evaluating the reward function. Our approach maximizing (REFT(A) requires constructing ' Prref with auxiliary reward Taux, demonstrated by the improvement in secondary structure diversity in $4.2.
4 Experiments
We evaluate FOLDFLOW-2 on multiple protein design tasks including unconditional generation, motif scaffolding, folding, fine-tuning to improve secondary structure diversity, and equilibrium confor- mation sampling from molecular dynamics trajectories. We provide implementation details in $B.
Baselines. As our main baselines for the unconditional generation task we rely on pre-trained versions of FifmeDiff [Yim et al., 2023b], Chroma [Ingraham et al., 2023], Genie [Yeqing and Mohammed, 2023], MultiFlow [Campbell et al., 2024], and RFDiffusion which is the current gold standard [Watson et al., 2023]. In conditional generation tasks like motif scaffolding, we compare against a conditional variant of FrameFlow Yimm et al., 2023a] as well as RFDiffusion. For protein folding, we focus on comparing against ESMFold [Lin et al., 2022] and MultiFlow which also

---

FoldFlow-2
RFDiflusion
Figure 2: Uncurated designable (scRMSD < 2Å) length structures with ESMFold refolded structure from FOLDFLOW-2 and RFDifuusion colored by secondary structure assignment. FOLDFLOW-2 is significantly more diverse in terms of secondary structure composition where we see RFDiffusion generates mostly a-helices.
leverages sequence information. Lastly, for conformational sampling the principal baselines are ESMFlow and AlphaFlow [Jing al., 2024].
4.1 Unconditional protein backbone generation
We evaluate unconditional structure generation using metrics that assess the designability, novelty, and diversity of generated structures. For each method, we generate 50 proteins at lengths {100, 150, 200, 250, 300) (c.f. FOLDFLOW-2 samples in fig. 12). Designability is computed by using the self-connistency metric which compares the refolded proteins (with ProteinMPNN [Dauparas et al., 2022] and ESMFold [Lin al., 2022]) with the original the fraction of designable proteins with TM-score < 0.3 and 2.) the average maximum TM-score of designable generated proteins the training data. Finally, diversity the average pairwise TM-score designable samples averaged across lengths as well as the maximum number of clusters.
Novelty is computed using: 1.)
Table 3: Comparison of Designability (fraction with scRMSD < 2.0A), Novelty (max. TM-score to PDB and fraction proteins with averaged max. TM-score < 0.3 scRMSD < 2.0A), Diversity (avg. pairwise TM-score and MaxCluster fraction), Designability and Novelty metrics include standard error.
Designability
Novelty
Diversity
Frac. 2 (t)
Frac. TM 0.3 (t)
avg. max TM (L)
pairwise TM ())
MaxCluster (f)
RFDiffusion
0.969 1 0.023
0.116 + 0.020
0,449 i 0.012
0.256
0.172
Chroma
0.636 + 0.030
0,214 + 0.033
0.412 + 0.011
0.272
0.132
Genie
0.581 + 0.064
0.120 + 0.021
0.434 + 0.016
0,228
0,274
FiffeDiff
0.402 + 0.062
0.020 + 0.009
0.542 + 0.046
0.237
0,310
FOLDFLOW
0.820 +: 0.037
0.188 1 0.025
0,460 1 0.020
0.247
0.228
FOLDFL.OW-2
0.976 + 0.010
0.368 + 0.031
0,363 + 0.009
0.205
0.348
Results. We see that FOLDFLOW-2 outperforms all other methods on all metrics--rruialll without requiring a pretrained folding model as part of the architecture like RFDiffusion. In particular, we observe that FOLDFLOW-2 produces the most designable samples with 97.6% of samples being refolded by ESMFold to within < 2Å. We also find that FOL.FFLOW-2 novelty improves RFDiffusion by an absolute 25.2% 25.2% the fraction of designable samples with TM-score < 0.3. Furthermore, we observe 19.9% and 102.3% relative improvement in the diversity of FOLDFLOW-2 over RFDiffusion as measured by the pairwise TM-score and Max Cluster fraction. This places FOLDFLOW-2 as the current most designable, novel, and diverse protein structure generative model.
We present uncurated generated samples of FOLDFLOW-2 and RFDiffusion in fig. 2. We further visualize the distribution of secondary structures of all methods in fig. 3. We see clear indication that FOLDFLOW-2 is able to produce the most diverse secondary structures more closely matching RFDiffusion. We further observe increased
the training distribution (see fig. Be)-and improving amounts of f-sheets and coils which are particularly challenging for models like FrameDiff and FOL.FrLow that primarily generate o-helices. We also include multiple ablations architectural choices, inference annealing, and sequence conditioning in table 14.
4.2 Increasing secondary structure diversity with finetuning
We investigate ReFT based data filtering to improve diversity of secondary structures generated samples. We use diversity score based auxiliary reward for filtering, based on weighted

---

135
13
dd
175
10
00
04l
06
16
30
1:4
06
1.0
00
0.6
0.6
3.5
0-hellx
@-helix
0-hellx
(a) FOLDFLOW-2
(b) FOLDFLOW
(c) FrameDiff
250
model
Fald
C
A Saa Selge
intur
Teld feleflow (Pyn)
FoldFlowe (Diversot
150
100
642
18
30
82
0.4
0.0
5.
100
3550
200
230
300
--hellx
Length
(d) RFDiffusion
(c) Data
(1) 9-sheet Count
Figure 3: Distribution of secondary structure elements (a-helices, 8-sheets, and coils) of designable (scRMSD 2.0) proteins generated various models. FOLDFLLOW-2 generates more diverse designable backbones.
entropy on the proportions of each residue belonging to each type of secondary structure ie., alpha-helices (a), coils c, beta-sheets in the set S, that can be analytically written as Udiversity = (E.es P,w; (1 + E.es Ps pog..). Due to models producing increasing amounts of helices, we use Wo 1, We = 0.5 and W3 = 2, and take top 25% of samples according to the rdiversity- Experimental results in fig. 3f with generated samples fig. 12 demonstrate that protein at all lengths benefit from training with ReFT measured by diversity of generated samples, and produces most amount of 8-sheets, and can surpass diversity improvement already obtained by training using synthetic structures as in fig.
4.3 Folding sequences
Given that FOLDFLOW-2 is sequence conditioned, we can perform protein folding by providing valid sequence during inference. During training. FOLDFLOW-2 tries transform SE(3), noise sample into the given sequence's 3D structure. Therefore, we aim to measure the generalization properties of our model to fold unseen sequences. We evaluate folding on a test set of 268 unseen proteins from the PDB dataset. We compare the folding capabilities of FOL.DFL.w-2.. ESMFoll, and Multiflow. In table 4, we report the aligned RMSD between the predicted backbone and the ground truth backhone. We find that FOLDFLOW-2, trained for structure generation, approaches the performance of ESMFold which is purpose-buus built folding model. We contextualize this result by noting that FOLDFLOW-2 4x is better at folding than the most comparable model in MultiFlow [Campbell al., 2024] which is multi-modal flow matching model using sequences.
Table 4: Folding model evaluation on test set of 268 proteins from PDB.
Model
RMSD (L)
ESMFold
2.322 + 4.270
MultiFlow
14.995 1: 3.977
FOLDFLOW-2
3.237 + 4.145
4.4 Motif Scaffolding
In motif seaffolding, we are tasked with designing a subset of residues, termed "scaffolds", around one more subsections of ("motif") protein structure that have known biologically-importe functions through interaction with a target. This enables the design of proteins with a priori functional sites using generative models [Wang et al., 2021, Watson et al., 2023]. The motifs can be

---

small and have non-specific shapes (e.g. a helix), and hence it is important for the generative model to understand the chemical information carries on top of its geometry. We thus consider the task of motif scaffolding as an example of how our model can be fine-tuned for conditional generation tasks. We consider two datasets for evaluating motif scaffolding performance: the benchmark proposed in Watson et al. [2023] consisting of 24 single-chain motifs, and a new benchmark based on scaffolding the Complimentary Determining Regions (CDRs) of VHH nanobodies, as found in the Structural Antibody Database [Schneider et al., 2022].
Motif Scaffolding Benchmark. We use the scaffolding benchmark from Watson et al. [2023] and follow the pseudo-label fine-tuning procedure described in Yim et al. [2023b] by randomly generating motifs from proteins by training on both the motif structure and sequence. For inference, we sample the scaffold lengths for each motif and provide both the both partially masked structure and sequence to the model. We follow the same evaluation procedure used RFDiffusion (c.f. $B.6 for details). Our results in table 5 show that both FOI.FFL.OW-2 2 and RFDiffusion solve all 24/24 motifs.
CDR Scaffolding. VHH antibodies, also known nanobodies, have shown significant promise in protein design and therapeutics due to their unique properties [Muyldermans, 2021J. They are composed of single variable domain derived from camelid heavy-chain antibodies, featuring three complementarity-deenniii
regions (CDRs) that confer specificity and variability in antigen binding. As a result, creating effective scaffolds for nanobodies is challenging due to the need to maintain the designability of the CDRs and especially because any scaffolding effort must avoid altering these characteristies to preserve binding functionalityy We treat this as a conditional generative modeling problem and fix the motif atoms, and mask the scaffold sequence information. Exact training and experimental details along with additional metrics are provided in $B.6. Our results are found in table 5, where the average motif scRMSD is much higher than the average scaffold scRMSD. The result a much lower number of solved motif scaffolding.
Table Motif-scaffolding benchmarks. FrameFlow does not have public code for motif scaffolding and thus cannot he evaluated on the VHH benchmark. "+FT" indicates "with fine-tuning" *Using reported numbers with AlphaFold2 instead of ESMFoll used in our evaluation procedure; c.f. $B.6 for further discussion.
Benchmark
RFDiffusion
VIIII
Model
Solved /24
Diversity t
Motif J
Scaffold 4
Solved /25
RFDiffusion
24
0.345
3.94 + 1.54
2.40 1 0.93
5
FrameFllow (+FT)*
21
FOLDFLOW-2 (+FT)
24
0.445
2.78 +: 1.01
1.67 + 0.24
9
4.5
Zero-shot Equilibrium Conformation Sampling
We now test FOLDFLOW-2 on zero-shot equi- librium conformation sampling task. Starting from a sequence, we generate multiple con- formations of the same proteins and compare the distribution of conformations with the ones from molecular dynamic simulations. We compare FOLDFLOW-2 with AlphaFlow-MD and ESMFlow-MD; two folding models fine-tuned on a molecular dynamic dataset, and non finetuned models Figenfold [Jing et a and STR2STR [Lu et al., 2024]. In table 6, report the pairwise and global RMSD, the root mean square fluctuation (RMSF), and the 2-Wasserstein on the top two principal componentee
ESMYRuww-g HD
Figure 4: Protein conformation ensembles from the ATLAS dataset, ESMFlow-MD and FOLDFLOW-2. Protcins colored by their secondary structure with a-helices in blue, 3-sheets in red, and coils green.
components. For both RMSD and the RMSF metrics, we report the Pearson correlation between the values from the generated ensemble and those of the ground truth ensemble (the procedure is detailed in full in $B.7).
We use the same test set as in Jing et al. [2024] restricted to proteins of length at most 400 amino acids. Notably FOLDFLOW-2 performs similarly or better than the comparable model ESMFlow-MD across

---

all metrics without any fine-tuning and with significantly fewer parameters on molecular dynamics data, indicating that the base model trained only on PDB already captures similar information about protein dynamics as models given explicit access to this data. Moreover, we observed that FOI.DFL.Ww requires 4.5 less GPU hours for training and 33x less trainable parameters while allowing for 6x faster inference steps than ESMFlow-Mi as reported in table 13, improving FOL.DF..o-22' prospects as a practical base model for future work on capturing protein dynamics.
Table 6: Zero-shot performance of the hase FOL.FLLOW-2 model on the ATLAS dataset of MD trajectories compared to ESMFlow and AlphaFlow models tune on ATLAS. FOLDFLOW-2 is competitive to the comparable model ESMFlow across all metrics. r denotes Pearson's correlation coefficient. Time is per sample compare PCA
(Time / sample (s)) length 300 protein 7c45 A. Eigenfold only models Ca atom
W2-dist on the atom only.
Model
Pairwise RMSD (?)
Global RMSF r (1)
Per-tnggt RMSF (1)
PCA We-dist ())
PCA Wi-lit (Co ()
Time / sample (9)
AlgbaFlow MD
0.468 + 0.005
0.455 0.006
0,824 0.000
10.67 +: 0.29
32.6 + 0.1
ESMFlow-MD
0.293 + 0.005
0.161 0.005
0.737 1-37-000 00001
11.51 + 0.13
11.2 +.0
P
0.297 0.004
0.236 0.004
0,658 0.011
10.85 0.15
5:213 0.075
STR2STR
0.279
0.586 0.001
0.05
Eigentold
0.194 + 0.004
0.156 0.004
0.668 0.001
5,296
38.6
5 Related work
Protein design. Physics-based protein structure design yielded the first de novo proteins [Huang et al., 2016]. For example, structure-baad -bsue biophysics approaches have previously resulted in several drug candidates [Röthlisberger al., 2008, Fleishman al., 2011, Cao et al., 20i0], was followed by language models [Hie et al., 2022, Ferruz et al., 2022] and geometric deep learning [Gainza et al., 2020] for protein structure design. Recently, diffusion [Wu et al., 2024, Yim et al., 2023b, Watson et al., 2023, Ingraham et al., 2023, Wang et al., 2024, Frey et al., 2024] and flow-based models [Bose et al., 2024, Yim et al., 2023a, Jing et al., 2024] have risen to prominence. These methods employ a backbone-first approach with the notable exception of MultiFlow [Campbell al., 2024] which uses sequence perform co-generation.
RLHF and Supervised Fine- Fine-Tuning (SFT). Aligning the outputs of language models with RLHF has recently gained interest [Ouyang al., 2022, Stiennon et al., 2020, Bai et al., 2022]. These methods learn a reward model for post-training alignment to desired behavior [Mishra et al., 2022], which can prove challenging for protein design [Zhou et al., 2024]. SFT on hand-crafted data has proven to be effective in enhancing performance but requires high-quality data [Rozière et al., 2023, Yuan et al., 2023]. Filtering real data using auxiliary rewards serve substitute for steering the desired properties of the generated samples.
Conclusion
In this paper, we introduce a new sequence-conditionee protein structure generative model called FOLDFLOW-2. FOLDFLOW-2 leverages protein language model to condition Flow Matching- based protein generative models with sequences. Our model achieves state-of-the-art results on unconditional generation and generates diverse and novel proteins, especially when trained on our new dataset. Conditioning over sequences allows our model to perform novel tasks such as folding sequences and the limitations of our model, we note that it requires a competitive pre-trained language model to be sequence conditioned, which can be hard acquire. We also note that ProteinMPNN, used in evaluation pipeline, has heen trained only on PDBs. Therefore, it is possible that our models trained on new dataset generates designable proteins which are not correctly processed by ProteinMPNN.
tasks and we show its competitiveness
those tasks. Regarding
Acknowledgements
We thank Alexandre Stein, Maksym Korablyov, and the entire DreamFold team for providing vibrant workspace that enabled this research. The authors would like to acknowledge Anyscale, Amazon AWS, and Google GCP for providing computational resources for the protein experiments.
10

---

Contribution statement
Architecture design was led by G.H. Infrastructure development was led by J.V. The experiments were divided as follows: Unconditional (K.F., A.T.), diversity (E.T.L., R.I., C.L.), folding (G.H.), motif-scaffolding (G.H., J.V., E.T.L, C.L.), and equilibrium conformation (J.R.B, G.H., T.A.S.). Dataset preparation including synthetic structure filtering (J.V., P.L.,, K.F.). A.J.B. drove the writing of the paper with contributions from all other authors. A.J.B. and A.T. cosupervised the project with guidance from M.B.
References
M. Albergo and E. Vanden-Eijnden. Building normalizing flows with stochastic interpolants. International Conference Learning Representations (ICLR), 2023. (Cited page 3)
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. Advances in neural information processing systems, 2016. (Cited on page 5)
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. E. Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022. (Cited pages and 10)
H. M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, N. Shindyalov, and P. E. Bourne. The protein data bank. Nucleic Acids Research, 28(1):235-22) 242, Jan. 2000. (Cited on page 17)
A. J. Bose, T. Akhound-Sadegh, K. Fatras, G. Huguet, Rector-Brooks C.-H. Liu, A. C. Nica, M. Korablyov, M. Bronstein, and A. Tong. Se(3)-stochastic flow matching for protein backbone generation. In International Conference Learning Representations (ICLR), 2024. (Cited on pages 2, 3, 4, 6, 10, 17, 19, 20, and 21)
A. Campbell, Yim, R. Barzilay, T. Rainforth, and T. Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. International Conference on Machine Learning (ICML), 2024. (Cited on pages 2, 4, 6, 8, and 10)
L. Cao, I. Goreshnik, B. Coventry, J. B. Case, L. Miller, Kozodoy, R. Chen, L. Carter, A. C. Walls, Y.-. Park, et al. De novo design of picomolar sars-cov-2 miniprotein inhibitors. Science, 37066515):426-43 2020. (Cited pages and 10)
S. R. Carter, S. Curtis, C. Emerson, J. Gray, I. C. Haydon, A. Hebbeler, C. Qureshi, N. Randolph, A. Rives, and I.. Stuart. Community values, guiding principles, and commitments for the responsi- ble development of ai for protein design, 2024. (Cited on page 16)
R. T. Q. Chen and Y. Lipman. Flow matching on general geometries. In The Twelfth International Conference Learning Representations, 2024. (Cited on page 3)
A. Chevalier, D.-A. Silva, G. J. Rocklin, D. R. Hicks, R. Vergara, P. Murapa, S. M. Bernard, L. Zhang, K.-H. Lam, G. Yao, al. Massively parallel de novo protein design for targeted therapeutics. Nature, 550(7674):74-79, 2017. (Cited on page
J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning-basee based protein sequence design using ProteinMPNN. Science, 378(6615):49-55 2022. (Cited pages 7, 20, and 21)
S. Ebrahimi and D. Samanta. Engineering protein- baaed therapeutics through structural and chemical design. Nature Communications, 14(1):2411, 2023. (Cited on page 1)

---

K. Fatras, Y. Zine, R. Flamary, R. Gribonval, and N. Courty. Learning with minibatch wasserstein asymptotic and gradient properties. International Conference on Artificial Intelligence and Statistics (AISTATS), 2020. (Cited on pages 4 and 17)
K. Fatras, Y. Zine, S. Majewski, R. Flamary, R. Gribonval, and N. Courty. Minibatch optimal transport distances; analysis and applications. arXiv, 2021. (Cited on page 17)
N. Ferruz, S. Schmidt, and B. Höcker. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022. (Cited on page 10)
S. J. Fleishman, T. A. Whitehead, D. C. Ekiert, C. Dreyfus, J. E. Corn, E.-M. Strauch, I. A. Wilson, and D. Baker. Computational design of proteins targeting the conserved stem region of influenza hemagglutinin. Science, 332(6031) :::16 821, 2011. (Cited on page 10)
N. C. Frey, D. Berenberg, K. Zadorozhny, Kleinhenz, J. Lafrance- R. Bonneau, K. Cho, et al. Protein discovery with discrete walk-jump sampling. International Conference on Learning Representations (ICLR), 2024. (Cited on page 10)
I. Hotzel, Y. Wu, S. Ra.
Gainza, F. Sverrisson, F, Monti, E. Rodola, D. Boscaini, M. M. Bronstein, and B. E. Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-19))) 192, 2020. (Cited on page 10)
P. Gainza, S. Wehrle, A. Van hall-Beauvais, A. Marchand, A. Scheek, Z. Harteveld, S. Buckley, D. Ni, S. Tan, F. Sverrisson, et al. De novo design of protein interactions with learned surface fingerprints. Nature, pages 1-9, 2023. (Cited page 1)
B. C. Hall. Lie groups, Lie algebras, and representations. Springer, 2013. (Cited on page 16)
A. Herbert and M. Sternberg. Maxcluster: a tool for protein structure comparison and clustering, 2008. (Cited page 21)
B. Hie, S. Candido, Z. Lin, O. Kabeli, R. Rao, N. Smetanin, Sercu, and A. Rives. A high-level programming language for generative protein design. bioRxiv, pages 2022-12, 2022. (Cited on page 10)
C. Hsu, R. Verkuil, J. Liu, Z. Lin, B. Hie, T. Sercu, A. Lerer, and A. Rives. Learning inverse folding from millions of predicted structures. International conference on machine learning, 2022. (Cited on page 6)
P.-S. Huang, S. E. Boyken, and D. Baker. The coming of age of de novo protein design. Nature, 537 (7620):320-322 327, 2016. (Cited on page 10)
J. B. Ingraham, M. Baranov, Z. Costello, K. W. Barber. W. Wang, A. Ismail, V. Frappier, D. M. Lord, C. Ng-Thow-Hing, E. R. Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature, 17707989) 2023. (Cited pages 4, 6, 10, and 21)
B. Jing, E. Erives, P. Pao-Huang, G. Corso, B. Berger, and T. S. Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. In ICLR 2023-Machine Learning for Drug Discovery workshop. (Cited on page 9)
B. Jing, B. Berger, and T. Jaakkola. Alphafold meets flow matching for generating protein ensembles. International Conference on Machine Learning (ICML), 2024. (Cited pages 2, 7, 10, 23, and 24)
Jumper, R. Evans, A. Pritzel, T. Green, M. Figumnvv, O. Ronneberger, K. Tunyasuvunakool,
R. Bates, A. Židek. A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-558) 2021. (Cited on pages 2, 3, 4, 5, 6. 18, and 19)
Z. Kadkhodaie, F. Guth, E. P. Simoncelli, and S. Mallat. Generalization in diffusion models arises from gedpetry-adaptive harmonic representation. In International Conference on Learning Representations (ICLR), 2024. (Cited page 5)
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014. (Cited page 19)
12

---

Z. Lin, H. Akin, R. Rao. B. Hie, Z. Zhu, W. Lu, N. Smetanin, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, al. Language models protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022. (Cited pages 2, 3, 5, 6, 7, 19, and 20)
Z. Lin, H. Akin, R. Rao. B. Hic, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmucli, et al. Evolutionary-seale prediction of atomic-level protein structure with language model. Science, 379(6637):1123 123-663377111-1113333 2023. (Cited on pages 5, 6, and 21)
Y. Lipman, R. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. (Cited on page 3)
X. Liu, C. Gong, and Q Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. International Conference Learning Representations (ICLR), 2023. (Cited on page 3)
J. Lu, Zhong, Z. Zhang, and J. Tang. Str2str: A score-based framework for zero-shot protein conformation sampling. International Conference on Learning Representations (ICLR), 2024. (Cited on page?!
S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 3470-3487. Association for Computational Linguisties, 2022. (Cited page 10)
L. S. Mitchell and I. J. Colwell. Comparative analysis of nanobody sequence and structure data. Proteins: Structure, Function, and Bioinformatics, 86(7):697-706, 2018. (Cited on page 23)
S. Muyldermans. Applications of nanobodies. Annual review of animal biosciences, 9:401 4211, 2021. (Cited on page 9)
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS, 2022. (Cited on page 10)
C. Park and R. W. Brockett. Kinematic dexterity of robotic mechanisms. The International Journal of Robotics Research, 13(1):1-15, 1994. (Cited on page 17)
A.-A Pooladian, H. Ben- Ben-Hamu, C. Domingo- Homingg-Enrrihh, B. Amos, Y. Lipman, and R. T. Chen. Multi- sample flow matching: Straightening flows with minibatch couplings. International Conference Learning Representations (ICLR), 2023. (Cited on page 27)
D. Röthlisberger, O. Khersonsky, A. M. Wollacott, L. Jiang, J. DeChancie, Betker, L. Gallaher, E. A. Althoff, A. Zanghellini, O. Dym, et al. Kemp elimination catalysts by computational enzyme design. Nature, 553(7192)1190-199 2008. (Cited on page 10)
B. Rozière, J. Gehring. F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve. Code Ilama: Open foundation models for code. CoRR, Abs/2308.112500 2023. (Cited on page 10)
C. Schneider, M. I. Raybould, and C. M. Deane. Sabdab in the age of biotherapeutics: updates including sabdab-nano, the nanobody structure tracker. Nucleic acids research, 50(D1):D1368 D1372. 2022. (Cited on page 9)
D.-A. Silva, S. Yu, U. Y. Ulge, J. B. Spangler, K. M. Jude, C. Labāo-Almeida, L. R. Ali, A. Quijano- Rubio, M. Ruterbusch, I. Leung, et al. De novo design of potent and selective mimics of il-2 and il-15, Nature, 186- 2019. (Cited on page 1)
13

---

N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler. R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In H. Larochelle. M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020. (Cited on page 10)
E.-M. Strauch, S. M. Bernard, D. La, A. J. Bohn, P. S. Lee, C. E. Anderson, T. Nieusma, C. A. Holstein, N. K. Garcia, K. Hooper, al. Computational design of trimeric talluenzz--euuralizinngg proteins targeting the hemagglutinin receptor binding site. Nature biotechnology. 35(7):667-671, 2017. (Cited on page 1)
A. Tong, N. Malkin, G. Huguet, Y. Zhang, Rector-Brooks, K. Fatras, G. Wolf, and Y. Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint 2302.00482. 2023. (Cited on pages and 27)
Y. Vander Meersche, G. Cretin, A. Gheeraert, J.-C. Gelly, and T. Galochkina. ATLAS: protein flexibility description from atomistic molecular dynamics simulations. Nucleic Acids Research, 52 (DI):D34-D3392 11 2023. (Cited on page 23)
Y. Vander Meersche, G. Cretin, A. Gheeraert, J.-C. Gelly, and T. Galochkina. Atlas: protein flexibility description from atomistic molecular dynamics simulations. Nucleic Acids Research, 52(D1): D392- 2024. (Cited on page 2)
M. Varadi, S. Anyango, M. Deshpande, S. Nair, C. Natassia, G. Yordanova, D. Yuan, O. Stroe, G. Wood, A. Laydon, A. Zidek, T. Green, K. Tunyasuvunakool, S. Petersen, J. Jumper, E. Clancy, R. Green, A. Vora, M. Lutfi, M. Figurnov, A. Cowie, N. Hobbs, P. Kohli, G. Kleywegt, E. Birney, D. Hassabis, and S. Velankar. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 50(D1):D499-4444 11 2021. (Cited pages and 6)
C. Wang, Y. Qu, Z. Peng. Y. Wang. H. Zhu, D. Chen, and L. Cao. Proteus: exploring protein structure generation for enhanced designability and efficiency. International Conference on Machine Learning (ICML), 2024. (Cited page 10)
J. Wang. S. Lisanza, D. Juergens, D. Tischer, I. Anishchenko, M. Back, J. L. Watson, J. H. Chun, L. F. Milles, Dauparas, et Deep learning methods for designing proteins scaffolding functional sites. BioRxiv, 2021. (Cited on page 8)
J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. Ragotte, L. F. Milles, B. I. M. Wicky, N. Hanikel, J. Pellock, A. Courbet, W. Sheffler, J. Wang, P. Venkatesh, I. Sappington, S. V. Torres, A. Lauko, V. De Bortoli, E. Mathieu, S. Ovchin- nikov, R. Barzilay, T. S. Jaakkola, F. DiMaio, M. Back, and D. Baker. De novo design of protein structure and function with RFdiffusion. Nature, 1089 1100, 2023. (Cited on pages 2, 4,
10, 19, 20, 21, 22, and 23)
J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. L. Finetuned language models are zero-shot learners. International Conference on Learning Representations (ICLR), 2022. (Cited on page 6)
K. E. Wu, K. K. Yang, R. den Berg, S. Alamdari, Y. Zou, A. X. Lu, and A. P. Amini Protein structure generation via folding diffusion. Nature Communications, page 10)
2024. (Cited on
L. Yeqing and A. Mohammed. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. International Conference on Machine Learning (ICML), 2023. (Cited pages 6, 20, and 21)
J. Yim, A. Campbell, A. Y. Foong, M. Gastegger, J. Jiménez-LLnna, S. Lewis, V. G. Satorras, B. S. Veeling, R. Barzilay, T. Jaakkola, et al. Fast protein backbone generation with se (3) flow matching. arXiv preprint ar arXiv:2310.05200 2023a. (Cited pages 4, 6, 10, and 19)
14

---

J. Yim, B. L. Trippe, V. De Bortoli, E. Mathieu, A. Doucet, R. Barzilay, and T. Jaakkola. Se (3) diffusion model with application protein backbone generation. In Proceedings of the 40th International Conference on Machine Learning, pages 40001-40039, 2023b. (Cited on pages 2, 3, 4, 6, 9, 10, 19, and 21)
Yim, A. Campbell, E. Mathieu, A. Y. Foong, M. Gastegger, J. Jiménez-Lumme S. Lewis, V. G. Satorras, B. S. Veeling, F. Noć, al. Improved motif- scaffl-scafooding with se (3) flow matching. Transactions on Machine Learning Research, 2024. (Cited on page 21)
Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv, 2023. (Cited on page 10)
Y. Zhang, I. A. Hubner, A. K. Arakaki, Shakhnovich, and J. Skolnick. On the origin and highly likely completeness of single-domain protein structures. Proceedings of the National Academy of Sciences, 103(8):2605- 2610, 2006. (Cited on page 21)
X. Zhou, D. Xue, R. Chen, Zheng, Wang. and Q Gu. Antigen-specinee antibody design via direct energy-based preference optimization. arXiv, 2024. (Cited on page 10)
15

---

Broader impact
The development of generative AT for protein backbone design holds significant promise for the fields of biotechnology and medicine. By enabling the precise engineering of protein structures, these models can accelerate the discovery of novel therapeuties and vaccines, potentially leading more effective treatments for wide range of diseases. Multi-modal and conditional generative models in the space of biotechnology are seeing rapid improvements, from which we have yet to see the full impact on modern and future medicine. We also acknowledge the potential for dual use of our results but, as discussed in Carter al. [2024], the benefits of public research into computational drug discovery outweigh the potential drawbacks.
A
A short review of Riemannian geometry, Lie groups, and optimal transport
A.1 Riemannian manifolds
Informally, topological manifold, .M is topological space that is locally Euclidean (i.c. homeo- morphic to a Euclidean space). The manifold is said to be smooth or differentiable if it additionally is Cº differential for all p. An important notion is the tangent space, TzM, which is attached to every point on the manifold, E .M. The disjoint union of all the tangent spaces is called tangent bundle, X(M). If in addition, the manifold is equipped with Riemannian metric, Iz. it is said to be a Riemannian manifold. The notion of a Riemannian metric is used to define inner products on the tangent space at each point of the manifold. This means that for F. E T,M, In (2) == (f,, Fx). Similar to how inner products can be used to define key geometric properties, such as length and distance, the Riemannian metric allows us to define such notions an arbitrary Riemannian manifold: the length of a vector in the tangent space of the manifold is defined by |5| (r, x)1/2 Finally, an important property on Riemannian manifolds is geodesics, which generalizes the notion of shortest paths in Euclidean spaces. While in Euclidean space the shortest path between two points is the length of a straight line between them, on a manifold, the idea is to find the shortest smooth curve between two points, which is called a geodesic.
A.2 Lie groups
Symmetries refer to transformations of an object that preserve a certain structure. A set of continuous symmetries, paired with composition operation satisfying group axioms is a Lie group (G, o). More precisely, a group a set paired with a group operation, o G x G - G. which is associative, has an identity element and there is an inverse element for every element of the set. In addition to this group structure, a Lie group is also smooth manifold, where the group operations of multiplication, (x. v) - zy for x, G, and inversion, + - are smooth maps.
Given y E G. we can define diffcomorphism, Ly G + G defined by x + yr. known as left multiplication. Given a vector field X on the group, we say that it is left invariant, if, under this left multiplication, it left-invariant, meaning that L X, Vy E G. Note that L is the differential of left action, naturally identifying the tangent spaces, Ty - Ty.. Therefore, given the group multiplication, we can uniquely define left-invariiat vector field with values on the tangent space at the identity element of the group, Te. We can additionally equip this vector space, V. with bilinear operation known as the Lie bracket, [ V V V, that satisfies the Jacobi identity and is anticommutative. Such a vector space is called a Lie algebra. The tangent space of Lie groups form Lie algebras and are denoted with B. The elements of the Lie algebra can be mapped into group elements using invertible map called the exponential map, exp: B - G. The inverse is called the logarithmic map allowing us to go from the group elements to their corresponding elements in the Lie algebra, log G + C.
Finally, we note that the set of n x non-singular matrices forms a Lie group. The group operation is matrix multiplication and it can be seen as a smooth manifold that is an open subset of I2 This group known as the General Linear Group, GL(n). Any closed subgroup of GL(n) is known as a matrix Lie group, which are perhaps some of the most important Lie groups that are studied. In the case of these matrix Lie groups, the exponential and logarithmic maps also coincide with the matrix exponen- tial and logarithm. For a more detailed overview of this subject, we refer the reader to Hall [2013].
16

---

A.3 The Special Euclidean Group in 3 Dimensions
One of the closed subgroups (L(n) that has been studied extensively in various fields is the 3D Special Orthogonal group. SO(3). The elements of this group are 3 x 3 rotation matrices, namely (O(n) = (r GL(m): r"r J, dr)( 1}. Additionally, the translations an object in 3D space by translation vector can also be seen as a matrix Lie group by considering the translation
matrix, (1 1 1 where is an x 3 identity matrix. With the group operation being translations,
this group is also a matrix group, denoted as (RR", +).
Combining these, we can represent the rigid transformations of objects in 3D space with group that encompasses both rotations and translations. This group is known as the Special Euclidean group in 3D and is the semidirect product of the rotation and translation groups, SE(3) = SO(3) x (R3 +).
This group is also matrix group and its elements can be written as SE(3) = {
E SO(3), E {). Finally, we note that given suitable choice of metric [Park and Brockett, 1994] the inner product on SE((3 decomposes naturally into inner products on SO(3) and (IR$, +), ie. (Fi, F2) = (ti, 12)so(3) + (#1, (R², where we have denoted the elements of tangent spaces of (3) and I& with y and respectively and have omitted to so for the translation group as the tangent space coincides with the space itself.
A.4 Optimal Transport
Optimal transport (OT) focuses on finding the most efficient way to move mass or resources from one distribution another. It involves minimizing cost function c(2, y) that quantifies the expense of transporting mass from point x in one space to point in another space. This problem is framed as an optimization problem, often using the Kantorovich formulation:
/ cxx,y) dz
min yETT(..)) XXY
where I[( 2) the set of joint distributions y with marginals μ and p. The goal is find transportation plan that minimizes the total cost. Optimal transport has significant applications in probability theory, where it is used compare probability distributions. One common measure the Wasserstein distance, defined as:
( inf (TEll(a)))
Jox d(x, y) dy(sr, u)
1/P
where d(x; y) represents the distance between points and y and p 2 a parameter that defines the type of Wasserstein distance.
In machine learning, optimal transport enhances algorithms in tasks such domain adaptation, where it helps align different data distributions, and in generative modeling, where it aids in generating data samples that match given distribution. In the case of FOLDFLOW-2 and used in [Bose et al., 2024]. OT is used to sample tuples of source noisy sample and target protein (Iy, 21) to perform Flow Matching. It takes the form of using the OT plan for the joint distribution ( 21) between minibatches (Xp, X1). The OT variant performs here corresponds to what is called minibatch OT as studied in [Fatras et al., 2021, 2020].
B Experimental Details
B.1
Dataset filtering
B.1.1 PDB Structures
We use subset of PDB with resolution < 5 downloaded from the PDB [Berman et al., 2000] on July 20, 2023, We performed standard filtering to remove any proteins with 50% loops. During
17

---

preprocessing, we also removed any non-organic residues at either end of the structure. In previous works, these residues are typically kept but masked during training, however they contributed the total forward pass FL.OPs and therefore decrease training efficiency. By removing these residues during preprocessing, we are able increase the number of training examples per batch. Finally, we re-clustered the PDB dataset using mmseqs2 at the 50% sequence identity threshold to obtain 6,593 clusters during training. The PDB is made available under a CCO 1.0 Universal (CCO 1.0) Public Domain Dedication.
B.1.2 Synthetic Structures
In this section, we provide more detail on the filtering procedure used in our curation of synthetic data for training. We began with a SwissProt data dump consistent of 532,003 structures predicted by AlphaFold2 [Jumper al., 2021] accessed in February 2024. The AlphaFold2 predicted structure database is made available under a CC-BY-4.0 license for academic and commercial uses.
Global PLDDT Filtering. We first filtered out globally low-confident structures, measured by average and standard deviation of pLDDT taken across all residues in a predicted structure. Despite SwissProt already being curated set of high-confidence structures, we still found considerable variation in quality. See fig. 5a for an empirical analysis average pLDDT for random sample of 500 proteins from SwissProt. Our final filtering criteria were (avg PLDDT 85) & (std pLDDT 15) to keep only "consistently good" structures, see fig. 5b for graphical representation.
Sdl One
Kapt Aike
avg pu0o1
(a)
(b)
Figure Analysis of global PLDDT distribution sample of 500 proteins from SwissProt.
High-Co Confidence Low-Quality Filtering. Despite the global PLDDT filtering, there were still low-quality structures in the training after global pLDDT filtering. Some examples of these structures can seen fig. They are characterized by having high overall confidence and good local qualities but unrealistic global structures sub-chain interactions. Our finding that these structures easily corrupt the training data and cause a model to produce similarly "unfolded" generations.
100
Model Confidence
wmm
high (pLDDT
High (90 PLDDT 70)
PLDDT
Very low (pL 50)
A4FZT8
QOSNQ5
Figure 6: Examples of high-confidence,, low-quality structures that were filtered out of the training set. Images are from the AlphaFold Protein Structure Database https://alphats ebi ac uk/ accessed in May 2024; identifiers are the UniProt IDs of each example.
18

---

Table 7: Overview of Training Setup
Training Parameter
Value
Optimizer
ADAM Kingma and Ba [2014]
Learning Rate
0,0001
81. $2,5
0.9, 0.999, 1e-8
Effective M (max squared residues per batch) % of experimental structures per epoch
500k
33%
Minimum number of residues
60
Maximum number of residues
384
Sequence masking probability
50%
Table 8: An overview of training time. *RFDiffusion initializes from RoseTTAFold, and include that training time in the estimates, **We recall that FOLDFLOW-2 uses frozen ESM2-650M which was trained 512 GPUs for 8 days.
Model
# Steps
# GPUs
Total Time (days)
RFDiffusion [Watson al., 2023]
25k
64
8
28 3
FOLDFLOW [Bose et al., 2024]
600k
4
2.5
FOLDFLOW-2 **
500k
2
4
FOLDFLOW-2 w/o folding block
500k
2
2.5
B.2 Model architecture details
We provide details for the IPA blocks and Folding blocks. For the IPA blocks, we follow the setting
developed in [Yim al., 2023b] by adding the original IPA [Jumper al., 2021] a skip connection and transformer layer on the node representation. The IPA modules takes as inputs the single and pair representations and structure. For the structure encoder, we initialize the single and pair representations with positional and time embedding passed to MLPs. In our experiment, we used node embedding dimension of 256, an edge dimension of 128, and a hidden dimension of 256. These settings are the same for both the encoder and decoder. We have use a skip connection between the representations of the structure encoder and decoder.
We combine the single and pair representations of different modalities by projecting each with a linear layer of output dimensions 128 for the single representations and 64 for the pair representation. We then concatenate all modalities' representation to obtain single representation of dimension 128 and pair representation of dimension 256.
The Folding blocks are taken from Lin et al. [2022]. They are composed of 2 Triangular Self- Attention Blocks with single and pair head width of size 32. Finally, the pair and single representation dimensions are of 128.
The structure decoder's single and pair representations inputs are average between the refined ones from the Folding blocks and the initial ones. All other architectural details not specified here are set to the defaults of IPA or ESM, respectively.
B.3 Training Details
Hyperparameters. See table 7 for an overview the experimental setup. We train with the "length batching" scheme deseribed in Yim et al. [2023a] in which each batch consists of the same protein sampled at different times. The number of samples in batch is variable and is approximately [num_reee residues 2 /M] where M is hyperparameter in table 7. Other training details such as the loss computation the same Bose et al. [2024].
Training hardware setup. FOLDFI.OW-2 coded in PyTorch and was trained on A100 40GB NVIDIA GPUs for days. Initial tests runs were trained in a similar setting.
19

---

Generative Flow Model
WFTPHQMVKSYELIA
WWKSYELIAGWFTPHOOMYS
Inverse Folding Model
YELIABWFTPHOMMVSYELU
RMSD
AGWFTPHQMWKSYYLIAGWW
TPHOMYKSAOWFTTHOOMKK
SYELIAGWFTPHOMMKSYEL
Folding
Model
Figure 7: Schematic of designability calculation. First generative flow model used to generate protein backbone from an initial structure (possibly noise) and (optionally) a protein sequence. This is then fed to an inverse folding model (ProteinMPNN Dauparas et al. [2022]) eight times to generate eight sequences for the structure. Then all eight sequences are fed back ESMFold produce structure for each sequence. All eight structures are compared using scRMSD from the ESMFold "refolded" structure the generated structure, and the minimum is taken as the "designability" of the generated structure with at least one structure with error 2.0À being classified as designable following Watson et al. [2023].
B.4 Inference details
The inference is performed with Euler integration steps. In our experiments, we found that 50 steps gave sfate-of-the-art results. We use the Inference Annealing trick from Bose et al. [2024] multiplying the rotation vector by some time dependent scaling function i(t) where & [0. 1]. We tried different scaling parameters and as found in [Bose et al., 2024], 10t provided the best performance. In practice this means greatly speeding up the rotation components at the beginning of inference sz 1) and slowing it down at the end inference (t x 0).
B.5 Unconditional protein backbone generation
Metrics. We compute several quantity of interest measure the performance of FOLDFLOW- 2. i) We start with designability. We measure designability using the self-consistency metric with ProteinMPNN [Dauparas et al., 2022] and ESMFold [Lin et al., 2022], counting the fraction of proteins that refold (C', RMSD (scRMSD) 2.0A) over 50 proteins lengths {100, 150, 200, 250, 300). For our ablation study and sensitivity analysis in & C.2, we also provide the average self-consistency rmsd over all generated proteins.
ii) In order use generative models for drug discovery applications, we want measure how different and novel are the generated data compared to training data. We measure novelty using two metrics: 1.) the fraction generated proteins that are both designable and are dissimilar to PDB structures (quantified by template-match score to PDB, i.e., PDB-TM score) 0.3, used in Yeqing and Mohammed [2023], higher better) and 2.) for designable proteins, the average closest similarity to training data (quantified by the maximum TM score, lower is better). We note that the threshold
20

---

for similarity has been studied previously, where the average TM-score on random structure pairs is 0.3 [Zhang et al., 2006].
iii) Finally we want a model that generates diverse proteins and not just of the same type. Proteins usually gathered into different clusters during training. So for diversity, we use the number of generated clusters with a TM-score threshold of 0.5 [Herbert and Sternberg, 2008](higher is better) as well as the average pairwise TM-score of the designable generated samples averaged across lengths as our diversity metric (lower is better). Note that in certain model, designability is inversely correlated with diversity as these models can produce unrealistic (e.g. unfolded) proteins that are "diverse" because they do not align well with each other.
Baselines. On the unconditional backbone generation task compare pre- pre-trained versions of FrameDiff [Yim et al., 2023b], Chroma [Ingraham et al., 2023], Genie [Yeqing and Mohammed, 2023), FoldFlow [Bose al., 2024], and RFDiffusion [Watson al., 2023]. We use the default parameters for each model including # of Fuler steps for inference and default noise levels (0.1 for RFDiffusion and FrameDiff). We use the OT version of FoldFlow as is the most similar to our setup and achieved the highest designability.
In table 3 we use 30 Euler steps for inference to better match the diversity levels of the baseline models for more accurate comparison on these metrics. Results for the 50 step model can be seen in table 17 with slightly worse designability but improved novelty and diversity.
B.6 Motif scaffolding
Evaluation procedure. We follow the same evaluation procedure as in Watson al. [2023], Yim et al. [2023b]. In particular, to evaluate the designability of a scaffold, we use ProteinMPNN [Dauparas al., 2022] decode 8 sequences and then re-fold those sequences, fixing the motif sequence which is known priori. Given these re-folded structures, we compare three numbers:
1. Global RMSD: the overall aligned RMSD between the entire generated structure and the refolded structure.
2. Motif RSMD: the RMSD of the refolded motif residues aligned to the original motif residues.
3. Scaffold RMSD: the RMSD of the refolded scaffold residues aligned to the generated scaffold residues.
Following Watson al. [2023], a scaffold is considered "designable" if the Global RMSD AND the motif RMSD < 1 AND the scaffold RMSD < 2. A detailed breakdown of our results can be found in table 9, with some samples of designable motifs in fig. 8.
We note that the choice of folding model appears to have a nontrivial impact on this metric. In their original papers, Watson al. [2023], Yim et al. [2023b] used AlphaFold2 with no MSA and recycles to refold their structures: however ESMFold is known to be significantly more accurate when no MSAs are provided [Lin al., 2023]. Given this, we generated new samples from RFDiffusion (FrameFlow doesn't 't have public code for generating scaffolds) and re-folded them with ESMFold. The result that RFDiffusion is able solve all 24 examples; an increase of 4 vs. their reported numbers. Moreover, the proportion of solved increases relative to their reported results, suggesting that the accuracy of the folding model significantly impacts the ability to measure scaffold quality in silico.
B.6.1 Pseudo-label training
We follow similar data augmentation procedure in [Yim al., 2024, Watson et al., 2023], with the only modification of adding minimum number of contiguous residues per motif. We use the same min length and adding absolute minimum length of 2 for motif. We continue training FOLDFLLOw- for 330,000 steps on the same dataset, with a learning rate of 10e-5. This was done using 2 NVIDIA 1100 80G, for 2.85 days.
B.6.2 VHH CDR
From the Structural Antibody Database we used 615 nanobody sequences, yielding 1831 chains from the PDB as training set, and for testing used 40 sequences appearing in 106 PDB chains.
21

---

Table A detailed breakdown FOLDFLOW-2 motif scaffolding performance using ESMFold to refold all structures. All numbers are out of 100 samples.
FOLDFLOW-2
RFDiffusion Designable
Example
# Overall Valid
# Motil Valid
# Scaffold Valid
Designable
IBCF
100
100
100
100
100
2KL8
100
100
100
100
100
IPRW
100
100
98
98
91
IYCR
96
100
89
85
91
4ZYP
89
97
80
85
3XT
97
101
75
73
85
7RRX small
70
92
83
66
22
6EX7_long
83
59
91
STPN
62
95
78
57
79
6EXZ med
67
70
55
54
87
6EX7_small
63
28
6E6R long
84
98
57
51
82
IQG
54
82
76
49
80
7MRX med
59
86
67
49
22
@F6R_met
78
96
54
43
87
STRV med
86
41
80
STRV long
40
85
38
77
6E6R_stn
82
100
34
30
50
STRV small
78
47
24
53
7MRX_long
27
59
47
20
22
SYUI
14
54
48
13
8
SIUS
10
86
5WN9
12
10
4
1
4JHW
74
39
3
8
Figure 8: Samples of solved motif scaffolding problems from the benchmark of Watson al. [2023]. The motif is in red, the designed scaffold is in blue, and the refolded structure from ESMFold is in gray.
7MRX_med
4JHW
5IUS
5WN9
1BCF
2KL8
22

---

Table 10: A detailed breakdown FOLDFLOW-2 motif scaffolding performance applied refoldable VHH structures. The same comments as table 9 apply to evaluation. All numbers are out of 25 samples.
FOLDFLOW-2
RFDiffusion
Example
# Overall Valid
# Motif Valid
Scaffold Valid
# Designable
# Designable
6qtl-B
21
16
4
6qtl-G
19
18
6rpj-H
21
21
4
O
6rpj-D
23
19
6qtlFF
22
18
2
0
7epb-C
35
33
6rpj-F
3
24
21
2
0
6qtl-C
20
14
5121-B
16
6oz6-G
16
6oz6-E
0
17
6oyz-G
12
6oyz-F
15
6oyh-H
17
6oyh-G
19
6gs7-H
19
Ikxq-E
23
13
6rpj-B
21
19
7a50-C
23
18
7epb-D
22
19
2
7o31-X
14
12
7q3q-B
13
7tje-B
17
8cxr-E
14
Scxr-F
18
With both the sequence and CDRs readily available, we can build the appropriate mask for training, which used fix the motif atoms, and mask scaffold sequence information. For testing, we sampled the scaffold segment lengths based the median value of each scaffold segment, + 5. Empirically the seaffold segment lengths varied less than that amount; nanobodies are known to exhibit less variability across their framework regions, both in sequence and structure [Mitchell and Colwell, 2018J. We continue training FOLDFLOW learning rate of 10e-5. Using a single A100 80G this process takes 3.5 hours. We observed rapid increase but rapid tapering in performance using the VHH dataset; this is likely due to the lack of variability of the scaffolds themselves.
using this dataset, training for 10,000 steps with
We note the designability metric used here and in other papers shows certain limitations when applied to this dataset and this task. Using the test set sequence and structures themselves, we compute the same seR MSD scores, order to benchmark what should be, theory, the best possible performance. We observe that out of the 106 testing chains, only 25 of them are "solved" according to Watson al. [2023]'s criteria. This raises the question as to whether or not this particular criteria and setup is applicable to any motif scaffolding task. We provide the full set of results in table 12 on all 106 chains, and on subset of size 25 in table 11, with generated samples in fig. 9. In addition, the particular number of solved samples are reported in table 10.
B.7 Zero-shot Molecular Dynamics
To evaluate FOLDFLOW-2' ability to capture protein dynamics we evaluated its performance on test set of ATLAS [Vander Meersche al., 2023] molecular dynamics used by Jing et al. [2024] restricted to proteins at most 40 amino acids length. To measure performance we used the following metrics. All metrics were computed exclusively over backbone atoms.
23

---

Table 11: VHII Motif Scaffolding results on the re-foldable examples only. Reported are global, motif, and scaffold RMSD along with the number of solved tasks.
Global scRMSD
Motif scRMSD
Scaffold scRMSD
Solved (out of 25)
RFDiffusion FOLDFLOW-2 VHH
3.12
3,94+ 1.54
2.4+-00.3
2.27=00.5
2.7811.01
1.67:10.24
Test Set
0.55:+0.19
0.65++0.18
0.48+t0.19
25
Table 12: VHII Motff Scaffolding results on all samples, same numbers as table 11
Global scRMSD
Motif scRMSD
Scaffold scRMSD
Solved (out of
106)
RFDiffusion
2.86+t1.1
3.6+:1442 2.99:10.86
2.25++0.85 1.66:10.31
FOLDFLOW-2 VHH
2.3-10.47
15
Test Set
1.4911.38
2.17:11222
1.05:1..9
25
1. Pairwise RMSD To measure FOLDFLOW-2's ability to capture protein flexibility we first compute the average pairwise RMSD between every pair of conformations generated by each method. We then evaluate the average pairwise RMSD for the ground truth data and report the Pearson correlation between the average pairwise RMSD per generated protein and ground truth data.
2. Global and per-target RMSF r. To further investigate flexibility, we measure the RMSF both globally and per target and compute the Pearson correlation r to ground truth data. Global RMSF computed by, for each target calculating the backbone RMSF and taking its average, then measuring Pearson correlation between generated and ground truth samples over the sequence of averaged RMSFs. For per-target RMSF we instead compute the Pearson correlation between generated and ground truth backbone atom RMSFs and report the average taken over all targets.
3. PCA W2. Following Jing et al. [2024] we seek to measure the distributional accuracy of generated samples by evaluating the 2- Wasserstein distance using the first two principal components given by ground truth data. We use a PCA as evaluating W2 on all atom coordinates would yield inaccurate measurements due to W2 needing samples exponential in dimensionality in order to obtain reasonable estimates. We compute the PCA W22 by, for each target, computing the first two principal components of backbone atom coordinates from ground truth MD data and projecting the backbone coordinates of each method's generated samples onto these coordinates. We then compute the W2 per target and report the averaged W2 over all targets.
As the test contains 30,000 frames per protein computing test metrics using all ground truth conformations would be computationally infeasible. As such, following Jing et al. [2024] we randomly sample 300 conformations for each protein be used as the test set. table 6 reports the mean and standard deviation over 5 resamplings of these test sets. Samples were generated from FOLDFLOW-2 2 using 50 inference steps and the inference annealing trick wherein the rotation vector is multiplied by 10t.
We report in table 13 details on the resources required for training FOLDFLOW-2 compared to AlphaFlow-MD and ESMFlow-MD, We see that FOLDFLOW-2 requires an order of magnitude less time per inference step than ESMFlow-M and AlphaFlow-MD while attaining results competitive with ESMFlow-MD while using 4.5X less GPU hours for training and 33X less trainable parameters. FOLDFLOW-2 2, ESMFlow-MD, and AlphaFlow-MD were all done on NVIDIA 1000s Inference time benchmarks were done on an NVIDIA A A100, performing inference on single protein of length 300 amino acids. Finally, in appendix B.7 we provide additional generated conformational ensembles from the test set, ESMFlow, and FOLDFLOW-2.
24

---

Figure 9: Samples of scaffolds for VHHs. Motif (i.e. CDR) is in red, scaffold is in blue, and refolded structure from ESMFold is in gray.
Table 13: Molecular dynamics experiment training details.
Model
training GPU hours
parameters
trainable parameters
Inference step (sec)
AlphaFlow-MD
2224
95M
95M
3.26 = 0.01
ESMFlow-MD
872
3.5B
694M
1.12 0.01
FoLDFLoW-2
192
672M
21M
0.00
C Additional Results
C.1 Unconditional generation diversity and novelty exploration
We next provide several additional results on unconditional generation to give a better understanding of the behavior of FOLDFLOW-2 relative the baselines. In fig. 11 we can see that FOLDFLOW-2 creates many more novel proteins at all thresholds of what is considered novel as compared to
Ground truth
ESMFLow-MD
FoldFlow-2
Additional conformation generation task samples. Proteins are colored by secondary structure with a-helices in blue, J--heets in red, and loops in green.
Figure 10:
25

---

250
model
FoldFlow-2
FoldFlow
200
FrameDiff
Designable
Genie
RFDifuusion
150
100
50
0
0.0
0.2
0.4
0.6
0.8
1.0
Max TM-score PDB
Figure 11: Curve showing the number of designable proteins that least some distance away from the PDB. FOLDFLOW. has many more novel and designable proteins than baselines. We report designability fraction at TM-score 0.3 in table
previous methods. We also depict more generated samples of all lengths in fig. 12. We can see that FOLDFLLOW-2 creates more diversity of structures, especially at shorter lengths. With synthetic data or diversity fine-tuning this is expanded to all lengths 100-300.
FoldFlow2
FoldFlow-2
(+Sytth)
FoldFiow-2 Diversity
fie-tuning)
Figure 12: Designable samples from various methods. Overlayed in silver are refolded ESMFold structures. FOLDFLOW-2 exhibits significantly more diversity in secondary structure at shorter lengths than RFDiffusion with fine-tuned models able to produce diverse proteins across lengths.
C.2 blation study and sensitivity analysis
In this section, provide several ablations of our FOLDFLOW-2 method. In table 14, we provide the unconditional generation performance for the different architecture components of our FOL.DFLOW-2 method. In table 15 and table 16, we compare the performance achieved by our model FoL.DFLow-2 when we use different inference annealing values respectively unconditional generation and folding. In table 17 and in table 18, compared the performance achieved by our model FOLDFLOW-
26

---

2 when using different numbers of Euler steps at inference for respectively unconditional generation and folding.
Architecture ablation. In table 14 we seek to understand the effects of architecture and dataset on the performance across our main designability, novelty, and diversity metrics for unconditional backbone generation. Starting from FOLDFLOW-2 we first investigate the effect of replacing the Folding Block with simple MLP with FOLDFLOW-2 (- F. Block) and removing the sequence conditioning entirely (- ESM2). In this comparison we find that both the folding block and structure conditioning significantly improve the results. We find that the Folding block improves all metrics while the structure conditioning improves designability and diversity at the cost of novelty.
Dataset and Training. In table 14, we look at how adding synthetic data or stochastic flow matching affects designability, diversity, and novelty metrics. We find that both these additions actually hurt these metrics, althouug this is likely due to the change in composition of their generated structures. Overall, we find that these two models create more diverse proteins.
Number of inference steps & inference annealing scale. We also studied the influence of number of Euler steps on the generated proteins in table 17. We note that our model performs quite well at a relatively small number of steps, although performance starts to drop off under 25 steps. We attribute this to the optimal transport approach which is known to increase quality of generation especially with few inference steps Tong et al. [2023]. Pooladian al. [2023]. We find interesting tradeoff between designability and diversity in table 17 and visually in fig. 13. Specifically that more steps increases the diversity of samples the cost of designability.
We also studied the impact of the inference annealing scale factor in table 15. We see that small scaling (or none at all, corresponding to value of 1) produce highly undesignable proteins, but designability improves quickly and beyond the optimal value of 10 it is somewhat stable. We notice the opposite effect in diversity as measured by the MaxCluster metric: no time scaling yields score which is 64% larger than the same metric with scaling 10, and this trend is clearly anti-correlated with the scaling value.
Impact of synthetic augmented dataset on diversity. One of the interesting benefits of using our synthetic augmented dataset is that it increases diversity among designable generated data. Proteins have different secondary structures such as helices, beta-sheets and coil. While our model generates a lot of helices, it is important for drug discovery applications to generate other secondary structure such as beta-sheets. As shown in fig. 13, we can see that our synthetic augmented dataset leads to an increased of beta-sheets.
Impact of rotation time scaling & number of inference steps on folding. We conducted an analysis of inference parameters on FOLDFLOW-23 ability to fold proteins. In table 16 we sweep over the rotation time scaling parameter and measure its impact on folding RMSD. We see a similar trend to the unconditional case in table 15: very small scaling factors (e.g.. 2) produce worse results, but the performance improves quickly as scaling increases and remains somewhat stable in a neighbourhood of 10.
We also conducted a sweep over the number of inference steps during generation on folding RMSD) in table 18. We again see similar trend to the counterpart for unconditional generation in table 17: a small number of Euler steps near 30-50 is best, with performance degrading significantly beyond 50 steps.
11
5 5 5 OSWaa
CEW
<5
D-belx
o-helix
(a) 100 steps
(b) 50 steps
(c) 30 steps
(d) Data
Figure 13: Secondary structure elements distributions (a-helices, 8-sheets, and coils) of designable (scRMSD 2.0) proteins, along with the data's distribution. We find more steps leads more diverse designs with fewer a-helix only generations.
27

---

Table 14: Ablation study on FOLDFLOW-2 (FF-2) using: synthetic data, folding blocks, and stochastic
flow matching (SFM), We generated 250 proteins (50 of length 100, 150, 200, 250) and compared Designability (fraction with scRMSD < 2.0A), Novelty (max. TM-score to PDB and fraction of proteins with averaged max. TMscore < 0.3 and scRMSD < 2.0A). and Diversity (avg. pairwise
T'sccree and MaxCluster fraction).
Designbiliit
Novelly
Deversity
Seg.
Folding blocks
SPM
iterls
train
Fine: (1
Froe: TM 0.
mas (M)
p.wiee TM (1)
MuxCClust
card.
param.
FF-2. F Block ESME)
0.716 + 0.029
0.188
0.025
0419 = 0.012
0.240
0.228
27
17M
TT-2 (- Dlock)
0.852 1 0.025
0.148 0.023
0.438 0.010
0.227
0.271
2.1
18M
HR-2
11976
0.36N
0.363
21M
HF-2 (+Synthetic)
0.785
0.47
0.465
21M
FF-2
0.16
0.386
21M
Table 15: Comparison of Designability (fraction with scRMSD < 2.0A), Novelty (max. TM-score to PDB
fraction of proteins with averaged max. TMscore < 0.3 scRMSD 2.0A), and Diversity (avg. pairwise TMscore and MaxCluster fraction) for different inference annealing functions i(t).
Designability
Novelty
Diversity
Frac. < 2 (1)
Frac. TM < 0.3 (t)
avg. max TM (1)
pairwise TM (1)
MaxCluster (1)
1
0.104 i 0.019
0.012 + 0.007
0.427 + 0.022
0.197
0,571
0.148 + 0.023
0.040 0.012
0.403 + 0.024
0.198
0.549
5t
0.832 = 0.024
0.312 = 0.029
0.375 = 0.011
0.193
0.387
10t
0.976 = 0.010
0.368 = 0.031
0.363 = 0.009
0,205
0.348
15A
0.928
+ 0.016
0.308 0.029
0.388 + 0.010
0.199
0.358
20
0.944 + 0.015
0.304 + 0.029
0.395 + 0.010
0.199
0.347
Table 16: Speed of the integration on rotations. Integrating with a faster time for rotations compared to translation leads to more designable structures. Reporting the mean + std. on 278 test samples.
Rotation time scaling
RMSD (4)
2/
3.641 + 4,457
5/
3.257 + 4,113
10t
3.334 + 4.325
15t
3.237 = 4.145
Table 17: Comparison of Designability (fraction with scRMSD < 2.0A), Novelty (max. TM-score to PDB and fraction of proteins with averaged max. TMscore < 0.3 and scRMSD 2.0A), and Diversity (avg. pairwise TMscore and MaxCluster fraction) for different number of Euler steps inference.
Designability
Novelty
Diversity
Frac. 2
Frac. TM 0.3 (f)
avg. max TM
pairwise TM MaxCluster
15 Fuler steps
0.480 + 0.032
0.136 + 0.022
0.382 + 0.012
0.196
0.430
20 Euler steps
0.876 + 0.021
0.328 + 0.030
0.358 1 0.009
0.203
0.341
25 Euler steps
0.948 1 0.014
0.376 1 0.031
0.372 1 0.010
0.207
0.336
30 Euler steps
0.976 + 0.010
0.368 + 0.031
0.363 + 0.009
0,205
0.348
35 Fuler steps
0.980 + 0.009
0.356 + 0.030
0.370 + 0.008
0,210
0.305
40 Euler steps
0.960 0.012
0.304 = 0.029
0.382 +
0.201
0.349
45 Euler steps
+ 0.940+0.000
0.328 = 0.030
0.382 0.009
0,201
0.357
50 Euler steps
0.014
0.384
0.383 0.011
0.194
0.387
75 Euler steps
0.025
- 0,300
0.380 0.010
0.189
0.434
100 Euler steps
0.748 + 0.028
- 0.340
0.366 1 0.011
0.188
0.415
150 Euler steps
0.620 0.031
0.180 0.024
0.408
0.185
0.437
200 Euler steps
0.580 1 0.031
0.208 0.026
0.391 1 0.012
0.183
0.447
Table 18: Effect of the number of integration steps the aligned RMSD between the generated and ground truth backbone. Reporting mean + std. on 278 test samples.
# Euler steps
RMSD (L)
30 Euler steps
3.384 = 4.223
50 Euler steps
3,334 = 4.325
100 Euler steps
3.374 + 4.377
150 Euler steps
3.405
4.409
200 Euler steps
3.481 = 4.465
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are supported by extensive experiments in $4 and in the Appendix $C.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention limitations of our model and our evaluation in the conclusion $6. In $B.6.2 we also mention the difficulties of evaluating VHH with the current best folding models.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed information regarding experimental setup and model parameters is made available in the appendix. We introduce new architecture which is described graphically in fig. 1, and is described the main text with more details provided in appendix $B.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] and No
Justification: Repreducible code available under a CC-BY-NC 4.0 license at https: hwtechhw......eeeeeeeeeeeeeeeeee olloooldloo........mmmmm
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: High-level experimental details are discussed in the main text in $3.1,4 (architecture, training setup, etc) and extensive experimental details (hyperparameters, dataset details, training time, etc) are provided in $B.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
When applicable, in all tables reported the standard deviation of the scores. We provide details on the metries computation in 1, $B.7, and $B.6.
8. Experiments Compute Resources
29

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details on the training resources and initial runs in BB.3, as well as resources for finetuning in $B.7, and $B.6.1.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttpa///nerre http://s..cccpbiitcthtccsccsaaaa
Answer: [Yes]
Justification: This work uses -ublicly-availabl data and does not involve human animal participants.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The positive societal impacts of accelerated drug discovery obvious and discussed numerous places in our paper. This work represents an advance in developing tools for drug research and development, but does not have immediate impact to the broader public. This is discussed in $6.
Justification:
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work investigates protein design techniques, which do not pose a high risk of misuse.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In $B we detail the creators and owners of the datasets used in this work. We mention the license and copyright information. We intend repackage the AlphaFold2 structure database to curate structures which are useful for improved training of generative protein design models.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does release new assets at this time.
14. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
30


Reviews:
[{"review_id": "aEruH2SZ76", "reviewer": "Reviewer_PUPL", "summary": "The paper introduces FoldFlow++, a sequence-conditioned SE(3)-equivariant flow matching model designed for protein structure generation. FoldFlow++ builds upon previous FoldFlowmodels by incorporating a protein language model to encode sequences, a multi-modal fusion trunk to integrate structure and sequence representations, and a geometric transformer-based decoder. The model is trained on a large dataset of both known proteins and high-quality synthetic structures, demonstrating substantial improvements over previous state-of-the-art models in terms of designability, diversity, and novelty. FoldFlow++ also excels in conditional design tasks, such as designing scaffolds for VHH nanobodies.", "questions": "1. In section 3.1, it’s claimed that the invariant point attention encoder (IPA) is SE(3)-equivariant. Shouldn’t it be invariant?\n2. When the sequence is chosen to be masked in training, how is the mask applied? Is it applied to the input space or the embedding space (after ESM2)?\n3. The common embedding in ESM2 is a single representation. What is the pair representation in ESM2 mentioned in Section 3.1?\n4. In Section 3.1, it’s claimed that adding a skip-connection between the encoder and decoder is essential for good performance. Is there any ablation study regarding this point?\n5. In synthetic dataset processing, how is the “masking low confidence residue” handled? Is the masking applied to both structure and sequence? Does the model leave a position for the masked residue or just ignore it?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper includes a detailed ablation study examining architectural components, different flow matching schedules, and more.\n- The model surpasses previous state-of-the-art generative models in terms of designability, diversity, and the novelty of generated protein structures.\n- The authors propose a large and diverse dataset, including high-quality synthetic structures, which enhances the model's generalizability and robustness.\n- The paper explores several meaningful settings, such as Reinforced FineTuning, Motif Scaffolding, and Zero-shot Equilibrium Conformation Sampling.", "weaknesses": "1. The proposed pipeline appears to be a special case of Multi-Flow.\n2. The model architecture remains very similar to previous work (e.g., Genie, FrameFlow/FrameDiff), leaving it unclear which specific parts of the algorithm drive the observed improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and detailed feedback which gave us an opportunity to strengthen our manuscript with additional ablation and results. We are encouraged that the reviewer felt that our paper includes a “detailed ablation” study of the different architectural components which enables the model to surpass “previous state-of-the-art generative models” in terms of designability, diversity, and novelty of generated protein structures and our use of synthetic structures to improve the models “generalizability and robustness”. We further appreciate that the reviewer views our paper to explore “several meaningful settings” such as Reinforced FineTuning, Zero-Shot Equilibrium Conformation Sampling, and Motif Scaffolding—the last of which we believe is an important biologically relevant task. \n\nWe now answer the key questions raised by the reviewer, while our global response contains detailed description of new experimental findings and shared concerns.\n\n## Proposed pipeline is a special case of MultiFlow\n\nWe appreciate the reviewer's concern that FoldFlow++ follows a similar approach to MultiFlow. We respectfully disagree with this assertion for several reasons. The most important of which is MultiFlow and FoldFlow++ have key architectural differences as well as training differences that mean the pipelines are different. We outline these in detail in our global response which we encourage the reviewer to kindly read. \n\n> The model architecture remains very similar to previous work \n\nWe acknowledge the reviewers' concern that the FoldFlow++ architecture may appear similar to past work. We would like to respectfully push back against this characterization by the reviewer. While our structure encoder is indeed the same network used in FrameFlow, FrameDiff or FoldFlow, we have enriched our representations with ESM2. The encoder representation is then processed with two Trunk blocks to learn more meaningful single and pair representations, a key element to get more designable, diverse and novel proteins. Specifically, FoldFlow++ uses EvoFormer blocks from ESMFold [1]---a module that is not used in the original FoldFlow or FrameDiff architectures. Due to FoldFlow++ being a model that accepts both protein structures and sequences, it further requires intermediate multi-modal fusion blocks. We wish to highlight that neither FoldFlow, FrameFlow, nor FrameDiff can consume protein sequences as inputs and hence the architecture of FoldFlow++ is substantially different. Finally, we empirically show in Table 14 an ablation over each architectural component benefits the overall performance of FoldFlow++. In short, we found that enriching the structure embedding in the enconder with a sequence embedding leads to improved results but the key architectural inclusion of the Folding blocks have the greatest impact on final performance.\n\n> In section 3.1, it’s claimed that the invariant point attention encoder (IPA) is SE(3)-equivariant. Shouldn’t it be invariant?\n\nYes, thank you for pointing this out. We have fixed it in the manuscript.\n\n> When the sequence is chosen to be masked in training, how is the mask applied? …\n\nThe mask is applied to the input space of ESM2, which matches how ESM2 was trained using the Masked Language Modeling objective. Note by masking the sequence FoldFlow++ is simultaneously being trained for unconditional generation as well protein folding. This point may not have been sufficiently clear in the original manuscript and we will update with a more precise description of the masking process.\n\n> … What is the pair representation in ESM2 mentioned in Section 3.1?\n\nFollowing ESMFold [1], to define a pair representation from ESM2, we use the attention matrices from the protein language model. We will make this clearer in the updated manuscript. \n\n> In Section 3.1, it’s claimed that adding a skip-connection between the encoder and decoder is essential for good performance. Is there any ablation study regarding this point?\n\nThis finding is intended as an anecdote from our model development experience to help practitioners avoid some pitfalls we encountered when developing encoder-decoder flow matching models rather than an empirical fact. We didn’t conduct an ablation study of this particular choice since skip connections are extremely common in the ML literature, including flow matching. For example, the U-Net architecture commonly used in image flow matching  contains skip connections from each “contracting” block to its equivalent “expanding” block. We found something analogous to be very useful.\n\n> In synthetic dataset processing, how is the “masking low confidence residue” handled? …\n\nWe appreciate the reviewer's question to a technical point that may have lacked clarity. The low-confidence residue masking is applied directly to the final *loss*. Operationally, this means zero-ing out the loss from low-confidence residues, while not affecting those residues’ structure or sequence inputs to the model. This prevents the loss for low-confidence residues from contributing to the model’s weight update, while avoiding unexpected inputs to the model such as fragmented structures or sequences. This is a standard practice when working with predicted structures, see e.g. [2]. We will incorporate a comment regarding low confidence residue masking in our updated manuscript.\n\n## Closing comments\nWe thank the reviewer again for their valuable feedback and great questions. We hope that our rebuttal addresses their questions and concerns and we kindly ask the reviewer to consider upgrading their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise.\n\n[1] Lin et al  (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. Science\n\n[2] Ahdritz et al (2024). OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization. Nat Methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F2dbLUOIiI", "reviewer": "Reviewer_VRjh", "summary": "The paper presents a protein generative model FoldFlow++ augmented with protein language model embeddings upon FoldFlow. The model is trained with sequence and structure information to learn embedding projections in SE3 space. Experiments on unconditional generation show a favorable performance of FoldFlow++ over SOTA method RFdiffusion. FoldFlow++ has the capability to be aligned to arbitrary awards like secondary structure diversity through reinforce finetuning, as well as the capability to motif scaffolding and conformation sampling.", "questions": "1. Are the results reported for model trained with synthetic dataset or PDB dataset?\n2. Could you explain how is the loss function implemented in the code?\n3. For protein folding and inpainting test, what kind of noise is the structure input?\n4. how is the AF2 high-confidence structures further distilled?", "limitations": "Although FoldFlow++ has various types of protein generation capabilities, performance on some of the tasks like conformation sampling and protein folding is not impressive, though I believe further finetuning on more specific datasets can benefit the model on this.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n\nThe paper is an excellent piece of work implementing protein language model embedding-guided protein flow matching model. The design of the network is reasonable and novel. Training with half-time sequence masking introduces the capability of protein folding and design at the same time. Overall the model is carefully designed and shows wonderful protein generative modeling potential.\n\n**Quality**\n\nThe submission is technically sound, with much of the mathematical foundations explained in the previous FoldFlow paper. Various aspects of protein generative models are tested, e.g. unconditional sampling, protein folding, motif scaffolding, conformation sampling.\n\n**Clarity**\n\nThe paper is easy to comprehend and figures are well-designed and clear.\n\n**Significance**\n\nThis work integrates protein language model into a protein flow matching framework to make its protein modeling and design ability more versatile. It can perform various kinds of tasks in protein design and has a great potential as a foundational model for protein researcher.", "weaknesses": "1. What are the diversity of structures for motif scaffolding task? Please include some visualization of generated structures for motif scaffolding benchmarks and statistical results.\n2. Alphafold2 also has conformation sampling ability. Did you benchmark it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their enthusiastic review and positive appraisal of our work!\nWe are heartened to hear that the reviewer found our paper to be an “excellent piece of work” with the architecture being “novel” and the overall model to show “wonderful protein generative model potential”. We are also glad that the reviewer views are our work to be “technically sound” and the writing to be “easy to comprehend” with well designed figures. Finally, we are thrilled that the reviewer finds the significance of our work to have “great potential as a foundational model for protein researcher.” We now address the key clarification points raised in the review below.\n\n> What are the diversity of structures for motif scaffolding task? Please include some visualization of generated structures for motif scaffolding benchmarks and statistical results.\n\nWe acknowledge the reviewers comment regarding the diversity of structures for motif scaffolding. These additional experimental findings (along with RFDiffusion) are included in the 1pg global PDF along with a discussion in the larger global response. Summarizing these findings here, we notice that FoldFlow++ has a larger diversity of designable structures in comparison to RFDiffusion which is in line with expectation as we find that FoldFlow++ produces much more diverse secondary structures—especially for unconditional generation of short proteins as observed in Fig 3 in the main paper.\n\n\n> Alphafold2 also has conformation sampling ability. Did you benchmark it?\n\nThroughout the manuscript, we compared against methods that do not rely on multiple sequence alignment (e.g. RFdiffusion, ESMFold). While providing meaningful single and pair representations, these methods are computationally expensive as they require querying a database for each sequence. However, we do compare with finetuned AlphaFlow-MD; a model that improves on Alphafold on MD tasks (see [1]).\n\n\n> Are the results reported for model trained with synthetic dataset or PDB dataset?\n\nWe understand this point was not sufficiently clear in the main text. For fair comparison between models and methods the main table results (Tables 1-4) are based on models trained only on PDB. The utility of synthetic data to improve diversity is presented in Appendix C in Figures 4e (main text), 12 &13 (appendix), and Table 14. We note that improving diversity using synthetic data is complementary to using ReFT to finetune for diversity.\n\n\n>For protein folding and inpainting test, what kind of noise is the structure input?\n\nThe noise can be broken down into the input modality which consist of protein structures and protein sequences.\n\n- **Protein Structures:** The noise is determined by manifold which in the incase of protein structures is the manifold $SE(3)$ repeated across the N residues. Since the manifold SE(3) can be decomposed into rotations $SO(3)$ and translations $\\mathbb{R}^3$ the noise can also be decomposed as separate noise for each rotation and translation component. For rotations o SO(3) we use the Isotropic Gaussian on $SO(3)$ distribution introduced in FrameDiff (yim et. al 2023)  and FoldFlow (Bose et. al 2023) while for translations we use the familiar Gaussian distribution on $\\mathbb{R}^3$.\n\n- **Protein Sequences:** Since sequences are discrete noise corresponds to replacing an amino acid in the sequence with a specialized mask token. The percentage of mask tokens in a sequence corresponds to the amount of noise added with a fully masked sequence being pure noise and as a result useful for unconditional structure generation. Partial masking of a sequence allows FoldFlow++ to perform in-painting tasks such as motif scaffolding.\n\n\n> How is the AF2 high-confidence structures further distilled?\n\nWe thank the reviewer for raising this important technical point which may not have been sufficiently clear in the original manuscript. After filtering the AF2 structures on high-confidence, we compute per-residue masks for the loss based on the pLDDT of each residue. Finally, we use a simple feature-based model to identify the remaining high-confidence, low-quality structures (see, Figure 7 in the text for some examples). Section 3.2 in our main paper plus the additional details in Appendix B.1.2 include even more details on the precise methodology for how we filter the AF2 synthetic structures. \n\n## Closing comments\nWe hope that our responses were sufficient in clarifying all the great questions asked by the reviewer and we would love to engage further if the reviewer has any further comments. We thank the reviewer again for their time and we politely encourage the reviewer to consider updating their score if our responses in this rebuttal merit it.\n\n## References \n[1] Jing, B., Berger, B., & Jaakkola, T. (2024). AlphaFold meets flow matching for generating protein ensembles. arXiv preprint arXiv:2402.04845.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bm22CehwSy", "reviewer": "Reviewer_ffQt", "summary": "The paper proposes a new model FOLDFLOW++ for Conditional Protein Backbone Generation. It incorporates several techniques including sequence model, finetuning strategies, and high-quality synthetic structures to improve its performance on various tasks. The experimental results suggest the method achieves SOTA performance on various protein-related generation tasks.", "questions": "- Is the model framework the same as FlowFold If the sequence input is fully masked?\n- Can the model compare with the AlphaFold2/3?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and can be used in many real-world scenarios. \n\n- The proposed method gains sota performance on unconditional generation, motif scaffolding, folding, fine-tuning to improve secondary structure diversity, and equilibrium conformation sampling from molecular dynamics trajectories.\n\n- Considering the sequence embedding and ReFT is reasonable for improving the base model's performance.", "weaknesses": "- This paper has limited technical novelty. The core components are mostly proposed by previous works. \n- Fusing the well-trained sequence model(ESM) into the backbone generation method (FoldFlow) intuitively can improve the structure generation[1]. Therefore, we cannot see the insightful discussion and surprising conclusion from the paper.\n- Some baselines concerning MD may be missing: EIGENFOLD[2], STR2STR[3],CONFDIFF[4].\n\n[1] A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design\n\n[2]EigenFold: Generative Protein Structure Prediction with Diffusion Models\n\n[3]Str2Str: A Score-based Framework for Zero-shot Protein Conformation Sampling\n\n[4]Protein Conformation Generation via Force-Guided SE(3) Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback and constructive comments, which allowed us to significantly improve our submission with new experiments and results which can be found in our global response. In addition, we appreciate that the reviewer found our paper to be “well-written” and that FoldFlow++ may be employed in “many real-world scenarios”. We also thank the reviewer for agreeing that FoldFlow++ achieves “sota performance” on a variety of tasks by leveraging key algorithmic and architectural components such as ReFT and a protein language model. We now address the main concerns raised by the reviewer.\n\n> This paper has limited technical novelty… \n\nWe value the reviewers opinion on the novelty of FoldFlow++. We kindly invite the reviewer to read our global response which summarizes our main contributions as well as details the principal innovations FoldFlow++ introduces at the architectural, algorithmic, and task level.\n- Architectural novelty: FoldFlow++ is the first protein structure generative model that successfully integrates a large pre-trained protein sequence language mode in ESM at a much larger scale.\n- Algorithmic novelty: FoldFlow++ introduces Reinforced fine-tuning (ReFT) for increasing secondary structure diversity as well as a new sequence-masking-flow matching training procedure that unlocks motif-scaffolding applications.\n- Task novelty: FoldFlow++ is applied in important downstream tasks—beyond unconditional generation—with motif-scaffolding, plus introducing a new more challenging dataset on VHH since FoldFlow++ saturates the previous benchmark.\n\n> Fusing the well-trained sequence model (ESM) into the backbone generation method … we cannot see the insightful discussion and surprising conclusion from the paper.\n\nWe appreciate the reviewers concern regarding the unsurprising improvement to protein structure generation by incorporating embeddings from a protein language model. We would like to politely disagree. We note that prior to FoldFlow++ only MultiFlow and Chroma explored incorporating sequence information—not even using a pre-trained language model—to aid protein structure generation and were unable to achieve significant performance improvements (see Table 1). We argue FoldFlow++ is the first structure generative model that is able to leverage this biological inductive bias and show measurable improvement over structure-only models such as RFDiffusion, FrameFlow, and FrameDiff (see Tables 1-4).\n\nAs a result, we argue it remained an open research question to what extent sequence embeddings help structure generation as ESM2 is pre-trained on a **a much larger sequence dataset** drastically different than PDB. \n\nMoreover, unlike prior models that use sequence, FoldFlow++ uses masked training which enables us to think of **actual drug design** applications as conditional generative modeling problems. For example, given a known motif sequence where the scaffold is masked we can generate the 3D structure of the scaffold directly. More precisely, we can *train* for this task as opposed to only performing it during inference unlike MultiFlow. Finally, we note that a surprising finding of FoldFlow++ is that we did not require the usage of *pretrained folding model weights* as employed in RFDiffusion and instead by masked training FoldFlow++ learns how to perform folding (Table 4).\n\n> Some baselines concerning MD may be missing: EIGENFOLD[2], STR2STR[3],CONFDIFF[4].\n\nWe have added comparisons on MD tasks for Eigenfold and Str2Str methods in the 1pg PDF. We do not compare to ConfDiff as it does not yet have publicly available code to our knowledge, but have also added discussion of these related works in section 4.5 (conformation sampling). Overall we find FoldFlow++ slightly outperforms EigenFold (on 3/4 performance metrics) and performs comparably to Str2Str (2/4 performance metrics) as shown in Table R1 of the attached PDF. We note that FoldFlow++ is not finetuned for sampling yet performs competitively to the suggested baselines which are more purpose-built for this task. In addition, we are excited by the possibilities of combining FoldFlow++ and improved inference methods for conformation generation as developed in Str2Str and ConfDiff in future work.\n\n**Eval Protocol:**  We evaluated both Eigenfold and Str2Str using the setting in section 4.5. For both models we use the default parameters and model in the public code.\n\n> Is the model framework the same as FoldFlow If the sequence input is fully masked?\n\nThat's a great question. The model framework of FoldFlow++ **is not the same** as the original FoldFlow model even if the sequence is fully masked. We give a more detailed account of architectural differences in our global response but, in summary, FoldFlow++ uses the EvoFormer block where FoldFlow does not. It has inductive bias from sequence embeddings, which improve generation (even for unconditional generation with fully masked sequence input. However, the reviewers intuition is correct in that the loss function—i.e. flow matching over $SE(3)$—used in FoldFlow++ is the same as FoldFlow when the sequence is fully masked.\n\n> Can the model compare with the AlphaFold2/3?\n\nFoldFlow++ and AlphaFold2/3 are different classes of models that are not directly comparable aside from protein folding. On the folding task FoldFlow++ underperforms ESMFold, which itself underperforms AlphaFold2/3 as they utilize multiple sequence alignment inputs. AlphaFold2/3 is not designed for our other tasks such as unconditional structure generation. \n\n## Closing comment\n\nWe thank the reviewer again for their review and detailed comments that helped strengthen the paper. We believe we have answered to the best of our ability all the great questions raised by the reviewer. We hope our answer here and in the global response allows the reviewer to consider potentially upgrading their score if they see fit. We are also more than happy to answer any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "txnmgOjx0g", "reviewer": "Reviewer_pz4y", "summary": "The authors proposed FOLDFLOW++, which is built on top of FOLDFLOW [ICLR 2024]. It adds a joint structure and sequence representation and a transformer-based geometric decoder, enabling folding and inpainting applications.", "questions": "- I was wondering why it is not possible to calculate diversity and novelty for other tasks.\n\n- It is not clear what the main component is that helps the model improve in terms of diversity and novelty compared to FOLDFLOW on the unconditional task.\n\n- Can this be used for inverse folding as well?\n\n- I was wondering if the inpainting task is not similar to seed optimization or editing tasks. For example, LaMBO-2 [NeurIPS 2023] or GFNSeqEditor [GenBio 2023]? If yes, why one cannot compare with those kinds of models? \n\n- What are the oracles to evaluate the generated samples?\n\n\n\n--- after rebuttal ---\n\nI have read the authors' responses to my questions. They have addressed most of my concerns, and I believe the paper has merit to be accepted at NeurIPS.", "limitations": "I think the model's performance might depend on various components such as ESM-2 and the structure encoder.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The tasks the authors are attempting to solve seem very interesting and important for drug discovery.", "weaknesses": "Considering that the theoretical novelty of the paper is somewhat limited and its main contribution lies in introducing certain architectures, the experiments conducted for the new tasks (other than unconditional generation) are also somewhat limited. Please note that I am not very familiar with the topic and do not know what potential experiments could be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper. We are heartened to hear that the reviewer views that with FoldFlow++ we are tackling a problem that is “very interesting and important for drug discovery” which was our primary aim with this new state-of-the-art protein structure generative model. We next answer all the important questions raised by the reviewer while new experiments are presented in the global response to all reviewers.\n\n>  the experiments conducted for the new tasks (other than unconditional generation) are also somewhat limited. \n\nWe appreciate the reviewer's concern regarding the completeness of our experimental protocol beyond unconditional generation. We would like to politely push back against this characterization as we test FoldFlow++ on a large suite of conditional generation tasks including motif-scaffolding, protein folding, and zero-shot equilibrium conformation sampling.\n\nWe highlight that the latter two tasks are included to showcase FoldFlow++ on tasks for which it was not originally trained. For instance, FoldFlow++ is not trained on any dynamics data yet is able to sample protein conformations at the level of ESMFlow—a model built for this task.\n\nFor motif-scaffolding we found that FoldFlow++ saturates the performance of the previous benchmark and we strived to introduce an even more challenging VHH nanobody task. In our 1pg PDF, we have included a quantitative diversity measurement of our motif-scaffolding designs and provided visual samples of the distribution of generated structures conditioned on a fixed motif. We believe that including such a challenging benchmark makes our conclusions more robust for the motif-scaffolding problem—which is perhaps the most relevant task considered in this paper for computational drug design.\n\n> I was wondering why it is not possible to calculate diversity and novelty for other tasks.\n\nWe thank the reviewer for this suggestion which we implement as additional results included in the 1 pg PDF. We report diversity of generated samples for the motif-scaffolding task as well as visual generations from the conditional distribution. We find that FoldFlow++ again outperforms RFDiffusion in terms of scaffold diversity. We encourage the reviewer to kindly read our global response for more details.\n\n> It is not clear what the main component is that helps the model improve in terms of diversity and novelty…\n\nThe reviewer raises an important question. There are several distinct features in FoldFlow++ that differentiate it from the original FoldFlow in terms of improved generated structure diversity and novelty. The most important piece is the use of sequence information through a large pretrained language model ESM2. We note that ESM2 incorporates sequence information of evolutionary scale protein sequence data which is significantly larger than PDB. Furthermore, it is well established that in biology protein sequences heavily determine their structure and we expect that using this biological inductive bias aids structure generation. We note that MultiFlow further showed that using sequence information—albeit without using a large pre-trained LM—aids generated structure diversity and novelty. We conducted an ablation study on the architecture of FF++ in Table 14, showing the effect of adding sequence embeddings even for unconditional generation.\n\nWe hope that this fully clarifies the question raised by the reviewer and we have included more discussion on this aspect in the updated version of the manuscript.\n\n> Can this be used for inverse folding as well?\n\nFoldFlow++ is a sequence conditioned protein structure generative model. Consequently, it is only able to generate structure and not sequences and thus cannot perform inverse folding. \n\n> I was wondering if the inpainting task is not similar to seed optimization or editing tasks…\n\nThat's an interesting question! Inpainting in FoldFlow++ happens on the structure space, in the sense that we provide the model with partial sequence and structure and generate missing structure. The editing approaches suggested by the reviewer operate on (output) space of protein sequences—i.e. they generate amino acid sequences. Consequently, we restrict comparing FoldFlow++ against SOTA structure-based models as there is no fair way to compare to sequence-based models without performing folding/inverse-folding using a third party model which itself may introduce further error.\n\n\n> What are the oracles to evaluate the generated samples?\n\nIn-silico evaluation of structure models is an important aspect in understanding the capabilities and limitations of current SOTA structure generative models. Our evaluation pipeline closely follows that of the literature and the exact schematic, including oracles, is depicted in figure 8 of the appendix. We recall that the unconditional evaluation consists of inverse folding the generated designs to obtain protein sequences, then refolding with an oracle folding model; in our paper we use ESMFold. Amongst the metrics we compute is the RMSD between FF++ designs and ESMFold refolded ones where we define generated proteins that achieve <2 angstroms as designable. On designable proteins we further measure diversity and novelty using a function of the pairwise TM score (exact details in Appendix B.5).\n\nFor the motif scaffolding evaluation, we fix the amino acids corresponding to the motifs. In this case we also look at the RMSD between motifs, scaffolds, and the entire proteins. We will make the description of our evaluation pipeline clearer in the updated version of the manuscript. \n\n## Closing comment\n\nWe once again appreciate your time and effort in this rebuttal period. If the reviewer deems our responses detailed enough and satisfactory we encourage the reviewer to potentially consider a fresher evaluation of our paper with these responses in context and potentially upgrade their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and thorough reviews. We are glad that the reviewers found that Foldflow++ has high potential impact as a new SOTA with important applications in real-world scenarios (R VRjh, R ffQt), such as being important for drug discovery (R pz4y). We are also grateful that reviewers appreciated the clarity of presentation in that the paper is “well-written” with “well-designed and clear figures” (R VRjh, R ffQt). Finally, the reviewers agree that FoldFlow++ explores several biologically-relevant protein tasks (R PUPL, R VRjh, R ffQt) with high quality results on motif-scaffolding, protein folding, zero-shot equilibrium conformation generation, and has detailed ablation of both new architectural (e.g. ESM2 Language model) and algorithmic (e.g. Reinforced Finetuning) components. We now address the main shared concerns of the reviewers.

## Summary of new experiments and ablations 
We are grateful for the reviewers' suggestions of additional experiments to enhance our empirical results. Please see our discussion below as well as the 1pg rebuttal PDF of the new results.

**Diversity on Motif-Scaffolding (R VRjh, R pz4y)**

We computed the same clustering diversity metric for motif scaffolding as our unconditional generations (i.e, # of unique clusters in designable samples / # designable samples). We find that FoldFlow++ has excellent diversity in comparison to RFDiffusion, which in line with expectations, given FoldFlow++'s improved diversity in unconditional results section. Furthermore, in Fig R1 of the 1 pg PDF we visualize the cluster representatives for some of the motif-scaffolding examples and qualitatively observe a high degree of diversity among the samples.

**Baselines for Equilibrium conformation sampling (R ffQt)**

For our zero-shot equilibrium sampling experiments, we include baselines of Str2Str and EigenFold as suggested. As these methods are purpose-built for this task they represent strong baselines. In Table R1, we find that FoldFlow++ outperforms Str2Str and EigenFold on Pairwise RMSD and Global RMSF while is marginally worse than Eigen Fold on Per Target RMSF. We also observe that Str2Str is the best method when measuring the PCA W_2 distance. We stress that FoldFlow++ achieves these competitive results **without any task-specific** training -- combining molecular dynamics methods with FoldFlow++ is beyond the scope of this paper but remains an exciting direction of future work.

## Technical Novelty and Contributions of FoldFlow++(R PUPL, R ffQt, R pz4y)

We acknowledge the reviewers' concern that FoldFlow++ may appear similar to prior work such as the original FoldFlow and Multi-Flow. However, we believe that these similarities are only high-level and our paper makes a number of novel contributions, in particular across 3 different dimensions: model architecture, training and algorithms, and evaluation tasks.

**Architectural novelty (R PUPL)**

Firstly, we note that FoldFlow++ is a sequence-conditioned generative model which separates it from structure-only generative models such as FoldFlow or RFDiffusion. The closest comparison is Multi-Flow, however we distinguish these models by noting that FoldFlow++ is the first backbone generation model to use and demonstrate the effectiveness of a *large pre-trained protein LLM* with ESM2 while MultiFlow trains a discrete diffusion model over sequences. Consequently, FoldFlow++ requires substantially different architecture choices than MultiFlow -- or FoldFlow, RFDiffusion, FrameDiff, or similar models -- to support encoding and jointly representing the structure and sequence modalities.

**Training and algorithmic novelty**

We outline the key differences below:
Our training procedure uses masking for protein sequences during flow matching training, which is a novel approach to structure generation and allows us to perform tasks such as motif scaffolding and folding in addition to unconditional generation. In contrast, the most similar model, Multi-Flow, trains both a structural model and a sequence generative model (from scratch on PDB), and only does sequence masking at inference. As a result, the sequence component of MultiFlow is **significantly less expressive** than FoldFlow++ as a conditioning signal. Indeed in Table 4 of our paper we observe an almost **$\approx 5x$** better protein folding performance of FoldFlow++ vs. MultiFlow which suggests that masked training is a key training strategy.

Additionally, FoldFlow++ is trained on filtered synthetic data using a new and robust filtering strategy which leads to a training set size $\approx 5\times$ larger than previous structure generation models such as FoldFlow, MultiFlow, or RFDiffusion (without considering the RoseTTAFold model's pretraining). This filtering methodology represents a new advance into the use of synthetic data for training protein generation models.

Finally, we demonstrate that FoldFlow++ can be finetuned using a new Reinforced Finetuning (ReFT) strategy to align against scalar reward functions, in our case to increase secondary structure diversity, a current challenge for protein backbone generation models.

**Task novelty**

A core contribution of FoldFlow++ is demonstrating that a single model is capable of performing many protein design tasks including: unconditional generation, protein folding, motif-scaffolding, and equilibrium conformation sampling. We stress FoldFlow++ is the only current model that is validated on all tasks. For instance, FrameFlow cannot be used for folding, Multi-Flow was not evaluated on motif-scaffolding or conformation sampling. 

We also introduce a new benchmark for the highly relevant motif-scaffolding task with VHH nanobodies. We believe the saturation of previous benchmarks by RFDiffusion and FoldFlow++ necessitated a more challenging testbed for conditional generation, and demonstrate how in-domain knowledge can improve motif-scaffolding.

Meta-review: FoldFlow++ is a sequence-conditioned SE(3)-equivariant flow matching model for protein structure generation, incorporating a protein language model, a multi-modal fusion trunk, and a geometric transformer-based decoder. The model achieves state-of-the-art performance in various protein-related generation tasks, including folding and inpainting applications.

The reviewers found the paper well written and easy to read, and highlight the strong empirical results in the paper, which are SOTA in terms of designability and diversity. Three of the reviewers expressed concern with the technical novelty of the method, identifying considerable overlap with earlier methods such as MultiFlow. In their rebuttal, the authors explicitly listed the advances compared to these earlier method. While some reviewers maintained reservations about the technical novelty post-rebuttal, the performance improvements reported in the paper are likely to have impact on the community, and all reviewers ultimately recommended acceptance of the paper.
Decision: Accept (poster)