Str2Str: A Score-based Framework for Zero-shot Protein Conformation Sampling
Authors: Jiarui Lu
Bozitao Zhong
Zuobai Zhang
Jian Tang
Venue: ICLR 2024 poster (2024)
Keywords: proteins
conformational sampling
diffusion models
score-based models
generative modeling
equivariant network
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We built a score-based protein conformation sampling method via equivariant structure-to-structure translation with no reliance of simulation data

Abstract: The dynamic nature of proteins is crucial for determining their biological functions and properties, for which Monte Carlo (MC) and molecular dynamics (MD) simulations stand as predominant tools to study such phenomena. By utilizing empirically derived force fields, MC or MD simulations explore the conformational space through numerically evolving the system via Markov chain or Newtonian mechanics. However, the high-energy barrier of the force fields can hamper the exploration of both methods by the rare event, resulting in inadequately sampled ensemble without exhaustive running. Existing learning-based approaches perform direct sampling yet heavily rely on target-specific simulation data for training, which suffers from high data acquisition cost and poor generalizability. Inspired by simulated annealing, we propose Str2Str, a novel structure-to-structure translation framework capable of zero-shot conformation sampling with roto-translation equivariant property. Our method leverages an amortized denoising score matching objective trained on general crystal structures and has no reliance on simulation data during both training and inference. Experimental results across several benchmarking protein systems demonstrate that Str2Str outperforms previous state-of-the-art generative structure prediction models and can be orders of magnitude faster compared with long MD simulations.

Published as a conference paper at ICLR 2024
STR2STR: A SCORE-BASED FRAMEWORK FOR ZERO- SHOT PROTEIN CONFORMATION SAMPLING
Jiarui Lu Bozitao Zhong!? Zuobai Zhang Mila- Québec AI Institute, Université de Montréal HEE Montréal, *CIFAR Al Chair (jiarui 1u, bozitao. zhong, zuobai zhang) quebec, jian. tan..tangehec.c
Jian J
Tang!
ABSTRACT
The dynamic nature of proteins is crucial for determining their biological func- tions and properties, for which Monte Carlo (MC) and molecular dynamics (MD) simulations stand predominant tools study such phenomena. By utilizing pirically derived force fields, MC or MD simulations explore the conformational space through numerically evolving the system via Markov chain or Newtonian mechanics. However, the high-energy barrier of the force fields can hamper the ex- ploration of both methods by the rare event, resulting in inadequately sampled en- semble without exhaustive running. Existing learning- based approaches perform direct sampling yet heavily rely on target-specific simulation data for training, which suffers from high data acquisition cost and poor generalizability. Inspired by simulated anncaling. we propose STR2STR, a novel grructue-to-struucum translation framework capable of zero-shot conformation sampling with roto- translation equivariant property. Our method leverages an amortized denoising matching objective trained general crystal structures and has reliance on simulation data during both training and inference. Experimental results across several benchmarking protein systems demonstrate that STR2STR outperforms previous state-of-the-art generative structure prediction models and can be orders of magnitude faster compared long MD simulations. Our open-source imple- mentation is available at https: //tp://gtttttt com/ Iujiarui/Stz25t.
INTRODUCTION
Understanding the dynamical properties of proteins is crucial for elucidating the mechanism their biological functions and regulations. Transitions can exist in the conformational ensemble, ranging from angstrom to nanometer length, and from nanosecond to second in time. Experimental mea- surements, such crystallographic B-factors and NMR spectroscopy, can be used to probe such dynamics yet in limited spatial and temporal scale. Despite the success of structure prediction mod- els (Baek al., 2021; Jumper et al., 2021; Lin et al., 2023) which enables the study of proteins based on high-accuracy structures, the predicted ensembles often lack diversity (Chakravarty & Porter, 2022; Saldaño et al., 2022) and modeling structure-dynamies relationship remains challenge.
Traditionally, Monte Carlo (MC) and molecular dynamics (MD) are two predominant families for conformation sampling by employing empirical force field. Both of them operate by starting from an initial point and exploring the conformation space guided by the force field. MC methods sample conformations by steering a Markov chain of stochastic perturbations (eg., Gaussian noise) on the Cartesian or internal coordinates with an acceptance ratio, or Markov chain Monte Carlo (MCMC). However, the transition kernel can rapidly lose exploration efficiency with an increasing degree of freedom. On the other hand, MD simulations evolve the motion of atoms over time to gen- crate time-indexed trajectories via the Newtonian mechanics. Due to the tiny timestep, significant challenge encountered by MD simulation is the high energy-bariier, which forbids thermodynamics- favored transitions within a limited number of simulation steps. To ameliorate, enhanced sampling methods have been proposed to overcome the energy barrier and encourage simulations. For example, methods based on biased potentials, such as umbrella sampling (Torrie & Valleau, 1977) and metadynamics (Laio & Parrinello, 2002); and those inspired by simulated
exploration of MD

---

Published as a conference paper at ICLR 2024
annealing that schedule the temperature to encourage exploration, e.g., replica exchange molecular dynamics or REMD (Hansmann, 1997; Sugita & Okamoto, 1999; Swendsen & Wang, 1986).
Another increasingly appealing solution the problem is the generative modeling of protein confor- mations. Direct sampling by the neural generator is more efficient than time-consuming simulations from MC MD. Boltzmann generator (Noé et al., 2019), one of the earliest attempt, modelled the system-specific conformation distribution with normalizing flow and performed i.i.d. sampling from random noises. With reweighting, the sampled ensemble can approximate the physical Boltzmann distribution. However, learning from a specific protein system requires pre-acquired simulation data for training the sampler and can be difficult to generalize beyond the training system (Wang al., 2019), leaving the use of such methods limited. Although generative training on the across-systen conformation datasets can help, the data acquisition can be non-trivial due to lack of open-source MD) trajectories for protein systems and the computationally intensive simulations from scratch.
To address the aforementioned is- sues, we propose new framework that samples general protein confor- mations via an equivariant structure- to-structure (STR2STR) translation. Trained on general crystal structures, STR2STR has no reliance on the computationally intensive simulation data and thus performs zero-shot! conformation sampling for any un- seen protein. Specifically, we formu- late the conformation sampling task as translation problem within the conformation space of the target pro- tein. Motivated by simulated anneal- ing, the proposed translation is com- posed of stochastic perturbations fol- lowed by the score-based annealing, forming harward-backward process. As an illustration, we present the in- ference diagram of STR2STR in com- parison with three traditional meth- ods in Figure We demonstrate that the sampling process is equivariant to global -ato-translation of the protein geometry, which guarantees the inference not yielding samples trivial as rotated or translated vari- ants. For evaluation, we construct a benchmark covering various aspects for protein conformation sampling and perform a case study of protein BPTI to demonstrate the effectiveness of STR2STR. Experimental results show that our method not only significantly outperforms the previous baselines on protein conformation sampling but is also comparable to long MD simulations.
Emnre
A(r,a" =
expi ) AV 0
Ay s
Energy
m
d
V.V
A00
d=(()
Conformation space
Conformation
Monte Carlo (Metropolis)
(b) Molecular dynamics
simulation (high T)
stochistic perturbation
Energey
simulattion (low
Energy
score-based annealing
V Jog
Conformation space
Conformation space
(c) Simulated annealing
(d) Ours
Figure Illustration of traditional sampling methods with proposed STR2STR. In (d), the energy landscape is made transparent to indicate that, contrast to traditional cases, STR2STR is agnostic to and thus not relying on the energy landscape but guided by the learned score functions.
2 PRELIMINARIES
Equivariance of transformation. Equivariance of a function (mapping) indicates that applying specific transformations (for example, rotation for Euclidean space) the input or output of a func- tion should have corresponding effects on the final output value. Formally, a function F: y with equivariant property can described as:
F Fop(2) = po po F(x),
(1)
where p is some transformation which acts on the element from space X' or ).
In the context of this paper. zero-shot means having access to simulation data that helongs to the test. protein during both training and inference stage.
2

---

Published as a conference paper at ICLR 2024
Diffusion modeling Riemannian manifolds. Score-based generative models (SGMs) can be represented by a diffusion process X, E IR" defined by the Itō stochastic differential equation (SDE):
dx f(x, t)dt + g(t)dw,
(2)
with continuous time index & E [0, 7], where the f(x, t) E R" is the drift term, g(t) E R is the diffusion coefficient, and w IR" is the standard Wiener process (or Brownian motion). Then, the corresponding backward SDE that describes the dynamics from X, to Xo is (Anderson, 1982; Song al., 2020):
dx = [[(x, () g2 (1) log (x) + g())dw,
(3)
where df is negative infinitesimal timestep and w is the standard Wiener process as continuous time t flows back from T O. De Bortoli et al. (2022) proposed the corresponding forward and backward process on Riemannian manifold .M beyond Euclidean space. To steer diffusion process with validity. the drift f(x, t), Brownian motion w, and score Vx log Px(x) are elements in the tangent space M. Utilizing the exponential-logarithn map, the process can be discretized similar to the Euler-Maruyama step in Euclidean space as geodesic random walk. Several recent works realized the Riemannian diffusion for different types of geometric data. Jing al. (2022) constructed the torsional diffusion on a hypertorus T while Yim et al. (2023) developed the SE(3)" diffusion for orientation- preserrvnn rigid motions in 3D space?. 2.
Notation on protein structure. The protein conformation is represented by its Euclidean coor- dinates E R where N is the number of heavy atoms (excluding hydrogen). We adopt the backbone frame parametrization T. := [R, v, S I n) one per residue Here, R₂ E SO(3) is a 3 x rotation matrix while v; E R is translation vector for the i-th residue. Such tuple represents an Euclidean transformation for each atom æ in residue i from the local coordinate Vincal E R3 to a position in global coordinates as Eglobal T, o Flocal == R,Wlocal + U,. The global atom coordinates on the backbone, specifically [N, C , C, C$] (except for GLY which has no C can be constructed by applying the transformation induced by T, to the corresponding amino acid struc- ture with idealized bond length and angles (Jumper et al., 2021), that is X = ((T.)), where {+) := (i-1..... Il))) is brief sequence notation and (-) constructs the corresponding global coordinates. Conditioned on Xhh. the carbonyl oxygen on backbone can be parameterized by a tor- sion angle rh:a or written as XxO[O] = To (bxx): Xaa). The side chain coordinates of i-th residue can be parameterized by at most four torsion angles X; := (XI, X2, X3: X4); E [0, 2 according to the rigid groups on which these heavy atoms depend. For example, in amino acid proline (PRO), the C atom belongs to its X2-group, which further depends on X1 and X2 (see Appendix A.2 for the full rigid group definition). Given the backbone coordinates, the Euclidean coordinates of side chains can be constructed with these torsion angles, which is denoted X,c- Psc((x)): X16)- Finally, we write collectively T == (T;), R RR.. v = {v:). sb == {th and X == [x:)-
3 METHODS
Conformation sampling involves learning the probability distribution px (x) of some protein X and then drawing samples are relatively more constrained (Jing et al., 2022), the conformation data of protein is however intractable to acquire due to the complexity of protein systems. Secondly, modeling protein directly in atomic level can be difficult due to the scaling of the number of atoms: protein with merely 60 residues can contain roughly ~500 heavy atoms without considering hydrogens. To address the challenges above, we propose to approach the conformation sampling by transfer learning via translation proposal on the residue frames, which is detailed as follows: Section 3.1 formulates the modeling of probability distribution; Section 3.2 introduces the sampling framework and model architecture; Section 3.3 describes the amortized learning objectives.
Px (x). Different from organic molecules whose stable conformers
3.1 CHAIN RULE OF THE TRANSLATION DISTRIBUTION
Given an initial conformation. the goal of conformation sampling is to capture the underlying dynamics of the target protein and infer plausibly stable candidates. We represent the overall
Note that the frame T: E SE(3) is the data point. Some literature mentioned 'quivaaaaui as the function equivariance to all the (global) rotations and translations in 3D space. To avoid ambiguity, we refer to the latter roto-translation equivarionce.
3

---

Published as a conference paper at ICLR 2024
translation distribution as px (x with Xy being an initial structure of protein X. Due to the enormous degrees of freedom in the atomic structure, direct modeling and sampling from Px (x(x)) can be intractable. Based the structural hierarchy, Px (xsel X66, Xn) px [Xb6/o] X66, Xo)Px (XM Xo). The rationale behind is that: given the backbone, the corresponding side chains take relatively limited orientations and can be sampled more effi ciently. Therefore, the sampling can be performed dep-wise: firstly, backbone frames are sampled from the backbone proposal T px (Tix)), and backbone coordinates can be obtained followed by the local-to-gel -tol--obalbll construction X Toa (T)) Secondly, the torsion angles can be sampled condi- tioning on the coordinates of backbone atoms: &b Px (4/1x66. Xo) and X ~ px (X' X66, Xo). Since the torsion angles are usually treated as internal coordinates, we may assume that these conditional torsion proposals only depend on the sampled backbone itself, i.e, px (*p) X66, Xo) R px (+) Xb6) and px (xb; xo) Px (2 (X)x),,, the backbone oxygen and side chain atoms can be constructed as follows X&a[O] = 1 b6[o] (tb, Xob); X = (X xxx) and finally x = [66, Xb6(O), Xscl ~ px (x(x)).
decompose Px (x(X))
3.2 EQUIVARIANT STRUCTURE TO-STRUCTURE TRANSLATION
forward-backwant Process. To model the backbone proposal dis- tribution, we firstly consider the distribution over Riemannian man- ifold SE(3)" where length-n frame We firstly make a mild assumption that the proposal can be approxi- mated by removing the initial side chain dependeney Px (T)x))
Sequence DAYAQW...
Folding
dx
f(x, fjdt g(()dw
Perturb
sequences T populates.
X0
XT4
XT
Structure
Anneal
X2T
XT:
dx (f(x, f) 2 log)( (())d
Px P (o, to, Xo) = (T)T).
Ensemble
which forms translation prob- lem² within the space of SE(3)". Motivated by simulated annealing, we propose a general score-based forward-backwanl (FB) process* that mimics the heating and anneal- ing process. Here, the perturbing (heating) process aims to enhance the exploration while the annealing guarantees the fidelity structural characteristics) by exploitation. In practice, the FB process leverages a stochastic pertur- bation kernel and multi-scale score functions, or formally defined by the following integrals:
Figure 2: Illustration of forward- backaadd process. Given an input structure (example as Trp-cage, PDB entry: 2JOF), replicas are fed to the forward (perturb) diffusion, which in- dependently perturbs each replica until the dynamic-transition time Ts: then the reverse (anneal) process will yield the sam- pled conformations. The sequence-tor trguencc-to-ttrucuure structure task can be well solved by any existing folding module such as ESMFold.
Ts [ t)dt +(()dw]
T To +
2T's
+
(
(ggp(T=)) dr + g(r)dw},
(4)
where r = T(t) = 2T, - (T, E (O) is the change of time variable and the rest of symbols are defined similarly in Eq. (2) and (3). Here the addition operator indicates the composition of frames and updates symbolically. Intuitively, the Eq. (4) perform noise injection (forward) followed by denoising process (backward) belonging to the above diffusion process defined on the manifold of T. The bound of integration T3 is set to be strictly less than T limiting the perturbation scale not to eliminate the information of the initial condition To. Empirically, increasing To to proper extent can lead to enhanced diversity yet it may hurt exploitation by demanding more reverse steps.
Diffusion Process SE(3)a. The diffusion process (Te) (R,, vel)ueto.r] defined on manifold SE(3)" can be represented as follows, by treating SO(3) and g independently (Yim al.,
In analogy to text-to-text translation and image-to-image translation. 'Experiments in this work only involve sampling from identical input structure. However, it is natural to enforce FB sequentially as neural proposal in MCMC. We leave this for future work.
4

---

Published as a conference paper at ICLR 2024
2023):
dT: = 10,,- (0t)) + VV d o (t) (8033) BBt(ttPdw
(5)
dt
where B(t), E IR+, are diffusion noise schedules, (M) indicates the Brownian motion defined on manifold M and the projection matrix P R R removes the center of mass. The perturbation kernel P:jo (R, | (Ro) for the rotation components (Re)ee(o,T) is considered element-wise
via the isotropic Gaussian on SO(3) distribution (Leach et al., 2022; Yim et al., 2023):
[G (Ry; Ro, o²) = (Wij)) = (orjo)
(21 + 1)e (((41)0 sin((1 + () WHoo sin( Well
(6)
π
1=0
with wiroo = Axis angle R Re) is the axis-angle representation of the composed rotation matrix R Ro. On the other hand, the perturbation kernel for translation components (V=) is an Ornstein-Uhlenbeck process, also known as VP-SDE (Song et al., 2020), which induces the isotropic gaussian kernel (vo(Vvlvo) = =V(v;; Voe + ((a)de I -- Ie_ fo P(*)ds) and converges to (O, I).
Packing of Side Chains. Given the sampled frames, we can construct the atom coordinates on the backbone and then sample the side chains from px (X' Xxa). Traditionally, this has been formulated as the protein side chain packing (PSCP) task (Xu & Berger, 2006). PSCP aims to, instead of freely exploring the conformation space, finding the conformation of side chains that minimize the energy. This casts the generative modeling of px (X' xx8) into its discriminative form, i.e. regression of torsion angles. practice, we adopted the FASPR (Huang et al., 2020), an efficient open-source method that leverages the backbone-depender rotamer libraries and simulated annealing Monte Carlo searching scheme to predict the most probable side chain conformations.
Rofo-translations Equivariance. Consider the forward- backward process in Eq. (4). The SE(3)" diffusion integral in Eq. (5) only updates the local-to-globa transformations induced by the frames T, and therefore the equivariance holds due to that fact that both drift and diffusion terms in Eq. (5) are inaee-nndeen For the backward integral, the extra term in the integral is the frame- dependent score function VT, log Pr (T,). Based on the result above, the score function is equiv- ariant, the reverse diffusion as well as the whole forward-backward process are equivariant. The equivariance of packing steps naturally holds because the predicted torsion angles are naturally in- ternal coordinates and poto-translation invariant. Therefore, we can derive the following proposition:
Proposition 1 (Equivariance of STR2STR). Let x ~ px (x(x)) be the conformation sampled from the process defined in Section 3.1. If the frame score functions VT, log (TT,)) are equivariant to global roto- aat-stranslaiinns, then Xa -> x assumes aoto-translation equivariance.
The detailed proof of Proposition 1 can be found in Appendix B.
Score Network Architecture. To model the translation distributions, the score model is required to obey the equivariant property with respect to global rotations and translations. We adopted a variant of the structure module in Jumper et al. (2021) called DenoisingIPA, to predict the score and steer the backward diffusion process. In DenoisingIPA, we initialize the single embedding {s,}0 as the concatenation of the position encoding of residues and sinusoidal time embedding; the pair embedding {zij] )a is constructed from the relative positional encoding (Shaw et al., 2018). In each layer I, the single representation { and frames {T,)' are updated via the Invariant Point Attention (IPA) layer and backbone update (Algo. 22-23 in Jumper et al. (2021)), followed by the multi-head self-attention (Vaswani et al., 2017) and multiple layer perceptrons (MLP). The update of representations and frames are illustrated in Figure 3. Slightly different from the vanilla structure module, also allow the update of pair representations (2,,] ) by edge transition layers:
{ +++ MLP (Concat [{z:)] {s} @ s(])),
where @ indicates the outer product. Following Jumper et al. (2021), we leverage the single repre- sentations { from the last layer to predict angle 2b with an MLP. Because the carbonyl oxygen atoms do not affect global geometry, we treat it in discriminative manner similar to side chains.
3.3 AMORTIZED LEARNING OBJECTIVES
Amortized Score Matching Loss.
To learn the translation distribution px (T) over the mani- fold SR(3)", the conformation samples of X are required for training the score networks. However,
5

---

Published as a conference paper at ICLR 2024
backbone (n, 3x3) and (n,
buckbone (n. Soc3 and (n.
{T
Frame Update
{T.)
{s:)
single repr. (9,
Iovariant Point Attention
Multi-Head Attention
single repr. (h,Ca)
{s.)
pair (n. (.))))
Edge Transition
pair (a, (D,0,c)
-* {z,,)
Init. single rept. (g)
Figure 3: Illustration of I-th layer of DenoisingIPA, where denotes the tensor Concat and + means tensor Add operation. The multi-head attention is the transformer self-attention (Vaswani et al., 2017). The initial single representations {8,} are constructed from the positional encoding of residues and the time encoding of denoising step. Here, single representations { and back- bone frames ( are updated similar to the structure module in Jumper et al. (2021), while pair representations {2ig}/ are updated according to Eq. (7).
acquiring simulation training set suffers from high computation cost and the resulting generalization capacity is limited. To tackle this challenge, propose use the general crystal structures from Protein Data Bank (PDB) for training, which can be viewed as respective local minima in the energy landscape. In this amortized sense, it suffices to train a single score network for the inference of any unseen protein at test. The difference in the denoising score matching objective is that the data sample To are from general distribution of PDB (denoted p*) instead of a target-specific px (T):
daam = EI. A(t) To SapepppBrr ET, To [[sse(T.r t) VT, log Prjo (Pr (T.TTo))13)
(8)
where X(t) & 1/E [1, Pojo (T)ooTT) (T) is a positive loss reweighting function, and Tr ~ (Te[To) is defined by the corresponding perturbation kernel. Since the inference procedure does not require reversing from the pure random noise (when t T). the time t can be uniformly sampled over the truncated time domain [0, Tm) where 0 < Tm < T is a pre-specified hyperparameter
indicates the maximal time scale used for inference.
Auxiliary Structural Losses According to the findings in Yim et al. (2023), solely training by score matching be insufficient for learning fine-grained structural characteristies. Along with the score matching loss for the frames, we complement auxiliary structural losses including mean square error (MSE) of backbone atoms and the distogram loss as Jumper et al. (2021). MSE loss is computed over the backbone atoms (including the carbonyl oxygen) to provide supervision for prediction of ạb. Because the process roto-translation equivariant and the distogram is based on the distances which are roto-translation invariant, the structural alignment is not necessary to perform. The overall training loss can be the weighted sum of all losses: L = Lasm + aLLbackh + L (ox, B > 0). The detailed definition of auxiliary losses can be found in Appendix D.
4 EXPERIMENTS
We compare the proposed method STR2STR to several recent baselines: MSA subsam- pling (Del Alamo et al., 2022), EigenFold (Jing et al., 2023), and idpGAN (Janson et al., 2023). These basclines leverage general structure datasets for training and are claimed to be able to gen- eralize to unseen protein, which is proper for zero-shot inference. MSA subsampling (Del Alamo et al., 2022) is AF2-based protocol to sample strucuture ensemble from sequence via reduced number recycle and subsampled multiple sequence alignments (MSA); EigenFold a sequence- to-ensemble diffusion model trained on PDB for conditional generation of protein structures based on the sequence embeddings from OmegaFold (Wu et al., 2022b); idpGAN a generative adversar- sinenneeconnitionedd conformation ensembles. For sampling of
ial network (GAN) that generates STR2STR, the initial conformation for each test target set to the output of ESMFold (Lin et al., 2023). Other implementation details can be found in Appendix D.

---

Published as a conference paper at ICLR 2024
Table 1: Benchmark results of conformation sampling methods on fast folding proteins (Lindorff- Larsen et al., 201 I) with reference MD trajectories. Metrics are averaged across all protein targets for each method. Reference MD data is colored brown. The ensemble from other baselines are obtained by running their codes in the standard settings. Among these metrics, Val-Clash, Val-Bond (validity) are the higher the better (?); while JS-PwD, JS-TIC, JS-Rg (fidelity) and MAE-TM, MAE- RMSD (diversity) are the lower the better (J). The best result from generative models is bolded.
The JS and MAE are compared with full MD trajectories, whose blocks are thus colored grey
Methods
Val-Clash(ft)
Val-Bond(1)
JS-PwD(()
JS-TIC(
JS-Rg (U)
MAE-TM(L)
MAE-RMSD(L)
MSA subsampling
0.999
0.997
0.634
0.624
0.656
0.596
0.713
EigenFold
0.812
0.874
0,530
0.497
0.448
0.607
idpGAN
0.960
0.032
0:480
0,517
0.661
0.189
0.592
STR2STR(PF)
0.963
0,992
0.375
0.397
0,448
0.150
0.209
STDESTR(SDE))
0.977
0,982
0.348
0,400
0.365
0.133
0.184
Reference
1,000
0.458
0.227
0.379
Reference lus
1,000
1,000
0.317
0,394
0,303
0,206
0.339
Reference 10us
1,000
1,000
0,236
0.331
0.227
0.144
0.243
Reference 10lus
0.997
1,000
0.130
0,155
0.126
0.063
0.102
Reference Full
0.997
1,000
0.000
0.000
0.000
0.000
0.000
4.1 EVALUATION METRICS
To assess the performance of STR2STR on the zero-shot conformational sampling, we set up a benchmark based on commonly used metrics in structure design and protein dynamics research. The evaluation metrics are categorized into: (a) Validity assesses whether the sampled conforma- tions obey basic physical constraints; (b) Fidelity reflects the distributional gap between sampled ensemble and reference MD simulation (which is seen as the "ground truth"); : (c) Diversity evalu- ates the possible variety of the sampled ensemble. As for reference, we up and also benchmarked a ladder of timescales for better comparison: 100ns, lus, 10us, 100us, full (the longest simulation time of each target). These metrics are briefly defined below and detailed in Appendix E:
Validity. The validity is defined by the ratio of conformations passing the sanity check, which examines whether the sample contains any (1) steric clash or (2) broken bond. Given a conformation sample, the steric clashes are counted by checking whether the distance of each pair of C atoms is within certain threshold that is based on atomic van der wauls radius; while the Ca-Ca "bond" is considered breaking if the distance adjacent C" atoms exceed certain threshold.
Fidelity. The fidelity compares the distributional divergence between the sampled ensemble and trajecotory from reference MD simulations. We adopt the symmetric Jensen-Shannon (JS) diver- gence based on three important quantities defined for conformations: (i) pairwise distance distribu- tion (JS-PwD), (ii) the slowest two components of the time-lagged independent component analysís, or TICA (Naritomi & Fuchigami, 201 I; Pérez- Hrrez-Hernández et al., 2013) (JS-TIC) and (iii) radius of gyration distribution (JS-Rg) as in idpGAN (Janson et al.. 2023)
Diversity. The diversity can be indicated by the averaged pairwise dissimilarity scores, based on root mean square deviation (RMSD, unit: nm) and TM-score (Zhang & Skolnick, 2004). For the TM-score, we apply the inverse (i.c., 1 - TM(X, x)) to express "diversity" aligned with RMSD (the higher the more diverse). We notice that the ensemble diversity is not the higher the better and depends on the characteristics of the target system. Therefore, we report the diversity difference as the mean absolute error (MAE) compared with the reference full MD simulations on both metrics.
4.2 FAST FOLDING PROTEINS
The benchmark set consists of 12 fast-folding protein targets with up to Ims scale all-atom MD simulation trajectories as reference from Lindorff-Larsen et al. (2011). To evaluate on the metrics above, generated 1, 1,000 conformations for each target using STR2STR and other baseline models. Two different integration schemes: probability flow ("PF") and SDE are used for STR2STR. For each method, metrics evaluated independently for each target and averaged across these targets.
The benchmarking results are shown in Table 1, from which STR2STR outperforms other zero-shot sampling baselines by large margin. Note that FigenFold, also diffusion model trained on

---

Published as a conference paper at ICLR 2024
MD 100ns
MD 1us
MD 10us
MD 100us
MD Full
MSA subsampling
EigenFold
idpGAN
Str2Str (PF)
Str2Str (SDE)
Figure 4: Contact map of Trp-cage (visualized in Figure 2) of each model with MD reference.
PDB, exhibited less diversity and failed to capture the conformational dynamics when compared with STR2STR. This may be caused by the complexity of modeling distributional mapping from sequence embedding to structure: the sequence-structure relationship can be well solved by fold- ing models (Jumper et al., 2021; ; Lin al., 2023) a discriminative manner (regression), but is still challenging for conditional generative modeling. In contrast, the proposed sampling framework involves learning the structure- coctuuure-tt-suuuue to-structure within the conformation space and learns abundant dis- tributional features solely from the PDB database. Here we showcase the contact map of Trp-cage in Figure and that of all targets can be found in Appendix F.1. The sampling speed of STR2STR with MD simulation single GPU is shown in Table 2, where STR2STR exhibits significantly advanta- geous efficiency over MD simulations for a case with comparable performance. Note that in general STR2STR still underperform long MD simulations (e.g., 100us) the distributional metrics.
4.3 STRUCTURAL DYNAMICS BPTI
We conducted a case study using the protein Bovine Pancreatic Trypsin Inhibitor (BPTI). The dynamic characteristics of BPTI have been well studied with 1.01ms-long MD simulation in Shaw et al. (2010), based on which five ki- netic clusters have been revealed. To better demonstrate the performance of STR2STR, present the TICA plots Hérréz-Hernándzz et al., 2013) for the sampled conformations from each method. Specifically, the conformation coordi- nates are reduced to the first TICA dimen- sions, which indicates the slowest two compo- nents and can embody the meta-stable states with distinction. The TICA parameters are fit using the the reference full MD trajectories. As shown in Figure 5, where the kinetic clusters are colored red, STR2STR successfully captured four clusters similar to 100us simulations with small variation and outperform the rest of baselines.
Table 2: Profiling of sampling speed between
explicit solvent MD simulation for 100us and STR2STR(PF) with comparable fidelity metrics on the example target WW domain.
MD 100us
STR2STR
JS-PwD (L)
0.399
0.379
JS-TIC (J)
0.438 JS-Rg (1)
0.458
0.406
0.402
Time 160 GPU days 510 GPU secs
RELATED WORK
Protein Backbone Design. parallel research interest emerging recently focuses on the protein backbone structure design based on deep generative models. Early attempts include ProtDiff (Trippe et al., 2022), which generates novel C Cn-only backbones; protein structure-sequence co-generation based structural constraints (Shi et al., 2022); and diffusion models tailored for antibody de- sign (Luo et al., 2022). Diff complements these by applying diffusion to the dihedral an- gles of backbones. Chroma (Ingraham al., 2022) designs novel protein backbones with several conditional inputs including natural language and comprehensively evaluates the programmability. Meanwhile, RFDiffusion (Watson al., 2022) pushed the diffusion-based protein design the
8

---

Published as a conference paper at ICLR 2024
MD 100ns
MDD Ius
MID 10us
MD 100us
MD Full
12
14
71
67 Str2Str (PF)
MSA subsampling
EigenFold
idpGAN
Str2Str (SDE)
123
325
Figure 5: Visualization of TICA plots for BPTI conformations sampled by each model with MD references. The kinetic clusters are colored red. In each subfigure, totally 1,000 samples were scattered in the 2D space. Note that most of the points are outside the target region for idpGAN.
perimental side and validated the effectiveness of generative modeling for this task. More advanced methods including Genie (Lin & AlQuraishi, 2023) and FrameDiff (Yim et al., 2023) have been pro- posed very recently that leverages the invariant point attention modules to enhance model capacity.
Learning from Simulation Data. Due the inefficiency of classical simulations for protein dy- namics, several works attempted to perform efficient sampling learn neural force fields from protein-specific simulation data. Boltzmann generators (Noé et al., 2019) were developed to gener- ate equilibrium samples using normalizing flows (Dinh et al., 2014; Rezende & Mohamed, 2015) trained on simulation data or energy. CGNets (Wang et al., 2019) proposed to learn coarse gearse-grained (CG) force fields in supervised learning manner. Köhler et al. (2023) improved this by comple- menting density estimation and sampling right before force-matching, thus not relying on ground- truth forces in simulation data. Arts et al. (2023) proposed train diffusion model conformations from equilibrium distribution of a specific protein, and leveraged learned score functions as force field for simulation i.i.d. sampler. Wang al. (2022) attempted recover the REMD ensembles of small peptide by training denoising diffusion on trajectories. However, these models suffer from the transferability problem (Wang et al., 2019) and cannot be generalized to unseen proteins. Klein et al. (2023) improves by modeling transition of a large timestep in MD simulation using normal- izing flow, which achieves good performance, yet only for very small peptides (only 2-4 AA). Our STR2STR is distinguished from these methods by performing zero-shot conformation sampling for unseen protein without any simulation data, and has more promising use in practice.
6 CONCLUSION
In this paper, we presented STR2STR, score-based structure-to-struucuuee translation framework for zero-shot protein conformation sampling. Motivated by simulated annealing. STR2STR tactfully combines both exploration and exploitation into a forward-backwan process based on the denoising diffusion for protein frames. STR2STR was trained solely on crystal structures from the Protein Data Bank (PDB) and has no dependency on any simulation data during training or inference. Experimen- tal results on several MD benchmarking systems demonstrate that STR2STR can effectively sample a diverse ensemble from the input structure in zero-shot manner. Limitations and potential future (1) The isotropic perturbation kernels could be biased towards more efficient subspace based on some collective variables. (2) Since STR2STR samples all-atom conformation, it can be plugged into atom-level MD simulations by incorporating physical- based force fields and perform enhanced sampling.
directions STR2STR encompass:
(3) The pre-trained STR2STR can be further fine-tuned by simulation data from specific systems to improve the sampling quality in a few-shot manner or towards the unbiased sampling from Boltzmann distribution.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
For reproducibility, we provide the detailed implementation details and training procedures in Ap- pendix D. To describe the proposed bawaadd-ackwwrd sampling process, pseudo-code snippet is shown in Algorithm 2. The construction procedure of atom coordinates are discussed in Appendix A. The definition of evaluation metries are listed in Appendix E. The source code of this work is available at https: /ttp::////ghtbtaa com
ACKNOWLEDGMENTS
We thank Zhaocheng Zhu and Sophie Xhonneux for helpful feedback as well as anonymous review- for their constructive suggestion and comments. This project is supported by Twitter, Intel, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program. Samsung Electronics Co.. Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, an NRC Collaborative R&D Project (AI4D-CORE-06) well as the IVADO Fundamental Research Project grant PRF-2019-33831139777
REFERENCES
Cameron Abrams and Giovanni Bussi. Enhanced sampling in molecular dynamics using metady- namics, replica-exchange, and temperature- aeppeaattre-acceleaatioo... Entropy, 16(1):163-199, 2013.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Marloes Arts, Victor Garcia Satorras, Chin-Wei Huang, Daniel Zuegner, Marco Federici, Cecilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg. Two for one: Diffusion models and force fields for coarse-grand boase-grrained molecular dynamics. arXiv preprint arXh:22302.00600, 2023.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876. 2021.
Inigo Herrin-Hernandez, Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron LM Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering pre- dicted structures at the scale of the known protein universe. Nature, 622(7983):637-665, 2023.
Devlina Chakravarty and Lauren L Porter. Alphafold2 fails predict protein fold switching. Protein Science, (16))33333, 2022.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling. Advances in Neural Information Processing Systems, 35:2406-2422. 2022.
Djurre H de Jong, Gurpreet Singh, WF Drew Bennett, Clement Arnarez, Tsjerk A Wassenaar, Lars V Schafer, Xavier Periole, D Peter Tieleman, and Siewert J Marrink. Improved parameters for the martini coarse-grained protein force field. Journal of chemical theory and computation, 9(1): 687-697, 2013.
Diego Del Alamo, Davide Sala, Hassane S Mchaourab, and Jens Meiler. Sampling alternative conformational states transporters and receptors with alphafold2. Elife, 11:775751, 2022. 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516. 2014.
Peter Eastman, Jason Swails, John Chodera, Robert T McGibbon, Yutong Zhao, Kyle A Beauchamp. Lee-Ping Wang, Andrew C. Simmonett, Matthew P Harrigan, Chaya D Stern, et al. Openmm 7: Rapid development of high performance algorithms for molecular dynamics. PLoS computational biology, 13(7)xe1005659, 2017.
RA Engh and R Huber. Structure quality and target parameters. 2012.
10

---

Published as a conference paper at ICLR 2024
Philippe Ferrara, Joannis Apostolakis, and Amedeo Caflisch. Evaluation of fast implicit solvent model for molecular dynamics simulations. Proteins: Structure, Function, and Bioinformatics, 46 46(1))24-333 2002.
Kristen A Fichthorn and W Hh Weinberg. Theoretical foundations of dynamical monte carlo simu- lations. The Journal of chemical physics, 95(2):1000-109, 1991.
Mert Gur, Elhan Taka, Sema Zeynep Yilmaz, Ceren Kiline, Umut Aktas, and Mert Goleuk. Con- formational transition of sars-cov-2 spike glycoprotein between its closed and open states. The Journal of chemical physics, 153(7), 2020.
UIrich HE Hansmann. Parallel tempering algorithm for conformational studies of biological molecules. Chemical Physics Letters, 281(13)::40-150, 1997.
Nana Heilmann, Moritz Wolf, Mariana Kozlowska, Elaheh Sedghamiz, Julia Setzler, Martin Brieg, and Wolfgang Wenzel. Sampling of the conformational landscape of small proteins with monte carlo methods. Scientific reports, 10(1):18211, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Moritz Hoffmaann, Martin Scherer, Tim Hempel, Andreas Mardt, Brian de Silva, Brooke E Hu- sic, Stefan Klus, Hao Wu, Nathan Kutz, Steven L. Brunton, et Deeptime: a python library for machine learning dynamical models from time series data. Machine Learning: Science and Technology, 3(1):015009, 2021.
Shinya Honda, Toshihiko Akiba, Yusuke S Kato, Yoshito Sawada, Masakazu Sekijima, Miyuki Ishimura, Ayako Ooishi, Hideki Watanabe, Takayuki Odahara, and Kazuaki Harata. Crystal structure of a ten-amino acid protein. Journal of the American Chemical Society, 130(46):15377- 15331, 2008
Xiaoqiang Huang, Robin Pearce, and Yang Zhang. Faspr: an open-source tool for fast and accurate protein side-chain packing. Bioinformatics, 36(12):3758-3755, 2020.
John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, et al. Illuminating protein space with programmable generative model. bioRxiv, pp. 2022- 12, 2022.
Giacomo Janson, Gilberto Valdes-Garcia, Lim Heo, and Michael Feig. Direct generation of protein conformational ensembles via machine learning. Nature Communications, 14(1):774, 2023.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional dif- fusion for molecular conformer generation. Advances in Neural Information Processing Systems, 35:24240 24225533- 2022.
Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.021993 2023.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- 589, 2021.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of hydrogen-bonded and geometrical features. Biopolymers: Original Research Biomolecules, 22(1)):5577-263 1983.
Horst Kessler. Conformation and biological activity of cyclic peptides. Angewandte Chemie Inter- national Edition English, 21 277::1122-5,, 1982.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 442266980, 2014.
11

---

Published as a conference paper at ICLR 2024
Leon Klein, Andrew YK Foong, Tor Erlend Fjelde, Bruno Mlodozeniec, Mare Brockschmidt, Sebas- tian Nowozin, Frank Noé, and Ryota Tomioka. Timewarp: Transferable acceleration molecular dynamics by learning time-coarsened dynamics. arXiv preprint
2023.
Jonas Köhler, Yaoyi Chen, Andreas Krämer, Cecilia Clementi, and Frank Noé. Flow-matching: Efficient coarse-graining of molecular dynamics without forces. Journal of Chemical Theory and Computation, 19(3):942-955 -222 2023.
Alessandro Laio and Michele Parrinello. Escaping free-energy minima. Proceedings of the national academy of sciences, 99(20): ((120)::2522-22555 2002.
Adam Leach, Sebastian M Schmon, Matteo T Degiacomi, and Chris G Willcocks. Denoising diffu- sion probabilistic models on so (3) for rotational alignment. In ICLR 2022 Workshop on Geomet- rical and Topological Representation Learning. 2022.
Marie-Paule Lefranc, Christelle Pommié, Manuel Ruiz, Véronique Giudicelli, Elodie Foulquier, Lisa Truong. Valérie T'houveiinContee, and Gérard Lefranc. Imgt unique numbering for im- munoglobulin and cell receptor variable domains and ig superfamily y-like domains. Develop- mental & Comparative Immunology, 27(1):55-77, 2003.
Faming Liang and Wing Hung Wong. Evolutionary monte carlo for protein folding simulations. The Journal of Chemical Physics, 115(7)3334-3330, 2001.
Yeqing Lin and Mohammed AlQuraishi. Generating novel, designable, and diverse protein struc- tures by equivariantly diffusing oriented residue clouds. arXiv preprint arXiv:2301.12488; 2023.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level (1116663)::123-1111 2023.
protein structure with language model. Science,
Kresten Lindorff- Stefano Piana, Ron O Dror, and David Shaw. How fast-folding proteins fold. Science, 534(6055):517-52,) 2011.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specifi antibody design and optimization with diffusion generative models. bioRxiv, pp. 2022-07, 2022.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679-682. 2022.
Yusuke Naritomi and Sotaro Fuchigami. Slow dynamics in protein fluctuations revealed by time- structure based independent component analysis: the case of domain motions. The Journal of chemical physics, 134(6). 2011.
Paul S Nerenberg and Teresa Head-Gordon. New developments in force fields for biomolecular simulations. Current opinion in structural biology, 49: 19:129-138, 2018.
Frank Noé. Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of mony-body systems with deep learning. Science, 365(6457) 147, 2019.
Eric Paquet, Herna L Viktor, et al. Molecular dynamics, monte carlo simulations, and langevin dynamics: computational review. BioMed research international, 2015, 2015.
Guillermo férez-Hernández, Fabian Paul, Toni Giorgino, Gianni De Fabritiis, and Frank Noé. Iden- tification of slow molecular order parameters for markov model construction. The Journal of chemical physics, 139(1), 2013.
Srinivas Ramachandran, Pradeep Kota, Feng Ding, and Nikolay V Dokholyan. Automated mini- mization of steric clashes in protein structures. Proteins: Structure, Function, and Bioinformatics, 79(1):261-270, 2011.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- fional conference machine learning, pp. 1530-1538. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2024
Tadeo Saldaño, Nahuel Escobedo, Julia Marchetti, Diego Javier Zea, Juan Mae Donagh, Ana Julia Velez Rueda, Eduardo Gonik, Agustina García Melani, Julieta Novomisky Nechcoff, Martín N Salas, et al. Impact of protein conformational diversity alphafold predictions. Bioinformatics, 38(10):2742-2744, 2022.
David E Shaw, Paul Maragakis, Kresten Lindorff-Larsen, Stefano Piana, Ron O Dror, Michael P Eastwood, Joseph A Bank, John M Jumper, John K Salmon, Yibing Shan, et al. Atomic-level characterization of the structural dynamics of proteins. Science, 33066002):341-336 2010.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions, 2018.
Chence Shi, Chuanrui Wang, Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and struc- ture co-design with equivariant translation. arXiv preprint arXiv:2210.08761, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(11):1026 2017.
Yuji Sugita and Yuko Okamoto. Replica-exchange molecular dynamics method for protein folding. Chemical physics letters, 314(1-2):141-151, 1999.
Robert H Swendsen and Jian-Sheng Wang. Replica monte carlo simulation of spin-glasses. Physical
review letters, 57(21):2607, 1986.
Glenn Torrie and John Valleau. Nonphysical sampling distributions monte carlo free-energy estimation: Umbrella sampling, Journal of Computational Physics, 23(2):187-199, 1977.
Brian Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206. 04.004111 19119, 2022.
Bodhi Vani, Akashnathan Aranganathan, Dedi Wang, and Pratyush Tiwary. Alphafold2-rave:
From sequence boltzmann ranking. Journal of Chemical Theory and Computation, 2023.
Kenno Vanommeslaeghe, Elizabeth Hatcher, Chayan Acharya, Sibsankar Kundu, Shijun Zhong, Jihyun Shim, Eva Darian, Olgun Guvench, P Lopes, Igor Vorobyov, et al. Charmm general force field: force field for drug-like molecules compatible with the charmm all-atom additive biological force fields. Journal of computational chemistry, 31(4):671-690, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jiang Wang, Simon Olsson, Christoph Wehmeyer, Adrià Pérez, Nicholas E Charron, Gianni De Fab- ritiis, Frank Noé, and Cecilia Clementi. Machine learning of coarse-grained molecular dynamics force fields. ACS central science, 5(5):755-767, 2019.
Junmei Wang, Romain M Wolf, James W Caldwell, Peter A Kollman, and David A Case. Devel- opment and testing of a general amber force field. Journal of computational chemistry, 25(9): 1157-1174, 2004.
Yihang Wang, Lukas Herron, and Pratyush Tiwary. From data noise data for mixing physics across temperatures with generative artificial intelligence. Proceedings of the National Academy of Sciences, 119(22)e2206666119 2022.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv, pp. 2022-12, 2022.
13

---

Published as a conference paper at ICLR 2024
Kevin E Wu, Kevin K Yang. Rianne van den Berg, James Y Zou, Alex X Lu, and Ava P Amini. Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.15611 2022a.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022b.
Jinbo Xu and Bonnie Berger. Fast and accurate algorithms for protein side-chain packing. Journal of the ACM (JACM), 53(4):533-55) 2006.
Jason Yim, Brian L Trippe. Valentin De Bortoli, Emile Mathicu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint 203022230. 2023.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004.
14

---

Published as a conference paper at ICLR 2024
A CONSTRUCTION OF RESIDUES COORDINATES
A.1 TRANSFORMATION BETWEEN FRAMES AND EUCLIDEAN COORDINATES
As discussed in Section 2, the atom coordinates on protein backbone are parameterized by the frames and can be constructed by applying the transformation induced frames to the local coordinates with the idealized bond lengths and angles. The local coordinates sets, depending on the amino acid type. were used in Jumper al. (2021) and originally introduced in Engh Huber (2012). For example, the corresponding coordinates for amino acid ALA are listed as below, with the set centered with respect to Ca as origin:
x/ (-0.225, 1.363, 0.000)
(0.000, 0.000, 0.000)
-0.000,
x 0
(-0.529, 0.7774,
x (0.627, 0.000)
The local-to-global procedure for any atom coordinates can be performed by applying the transfor- mation induced by backbone frame 7 frame (firrame frame) to the local coordinates:
frame
pframe
frame
Eglobal =
Elocal
Doccal + v
where Attrae E SO(3) is 3 x 3 rotation matrix while frame R is a translation vector.
As in Jumper et al. (2021), the backbone oxygen is additionally parameterized by the torsion angle vi, which is based on the axis-rotation of C*-C single bond. The oxygen coordinates be obtained by applying torsion transformation followed by the local-to-global transformation similar to other backbone atoms (for i-th residue example):
0 x global = Tframe T Focal
(10)
where the Tframe := (R, v) indicates the local-to-globee transformation defined in Section while T (R², v²) the additional transformation induced by vb. Let [sin( cos(v)) as its sin-cos representation, the corresponding transformation can be write in:
Rd
sin(1) cos(1) (b))
cos(1) sin (b))
v C
(1I)
Lo
To construct the frames from coordinates of protein structures in PDB, we adopt the Gram-Schmidt process in Jumper et al. (2021) from the coordinates of N-C*-C for each residue, which is:
Algorithm Frame from three points (Algo 21, Jumper et al. (2021)
1: Require: 11, Ø2 Ø3 as global coordinates of atoms N, CA, C.
2: in Ø3 -22
3: U2 .11
.22
4: e1
u1 m1/l|ua]
5: ug (c, 4s
[e-er(ee wall
6: E3 E1 e2
7: R CONCAT(ei, e2, e3)
8: x²
9: return R,
15

---

Published as a conference paper at ICLR 2024
MD Full
Str2Str (PF)
Str2Str (SDE)
Angle [radians)
1.
Angle (radians)
(radians)
Angle (radians]
Figure S1: The Ramachandran plot of d th angles for the protein Chignolin example. STR2STR(PF and SDE) are compared with the snapshots of the reference full MD simulation.
A.2 RIGID GROUPS OF AMINO ACIDS
According the representation used in the main text, here provide the detailed rigid group definition to construct the corresponding Euclidean atom coordinates. The notations in this table are aligned with the settings in Jumper al. (2021).
Table S1: Specification of all the rigid groups for each residue type.
AA type
Backhone
X1
X²
X3
X4
ALA
N,Ca, N,C*.C,C* C.
N,C*,0,0*
C
N
N N Cc
ASN
N,C C,
N 0
ASP
N, C C²
O 0
CYS
N. CºC C.
ST
GLN
N,Cº,C
C7
C
N on
GLU
N,C*,C,C*
CY
C
0
GLY
N.C..C
HIS
C,C²,
ca N91 cC. .
ILE
N. N.C².C.0*
Ca1
LEU
N, C. C²
C
of G
LYS
N,C C,C",C,C*
CY
C6
C
NO
MET
N, N,C",C,C²
S4
PHE
N. C.
C C gdl or2 CF
N.C*.C.C C.C
C4
SER
N, C. C,C,
o
THR
N. C C
TRP
N, C* C
C
C+1 (2 2*2 grd N qu2 CC2 cyca
TYR
N. CO. C. Ca
CY
Gd1 C2 gdd a O",
VAL
C. C, Cd
A.3 CHIRALITY
To examine whether the STR2STR can generate correct chirality with respect to the input protein, we showcase the Ramachandran plots for torsion angles in conformation from STR2STR (PF and SDE) and MD trajectory, where we select as example the smallest peptide Chignolin (length=10) in fast folding targets. The results are shown in Figure $1. In this case, there are three clusters in the o - &b within the 2D space [- T and STR2STR has basically covered three of them. The correct capturing of chirality comes from the backbone construction procedure in the same manner as AlphaFold2 (Jumper al., 2021).
B EQUIVARIANT STRUCTUURE- TRANSLATION
To better model particle system such as protein conformation, equivariance with respect to spatial transformations (eg. rotation 90" along some axis) should be injected as inductive bias for better generalization. For protein conformations, chirality important to convey structure- structure-function rela- tionship and should be conserved during the mapping. Therefore, consider only global translation
16

---

Published as a conference paper at ICLR 2024
and rotation elements (or roto- noto-translation operations), excluding the reflection which change the chirality. Especially, the conformations exist 3D space and thus realize the translation vector R and rotation matrix R I33x3 Under such constraints, the function (mapping) is said to be SE(()-equivariint The pseudo-code for the forward-backward process shown in Algorithm 2 for better illustration.
Algorithm Forward-backwwr process of protein frames
1: Require: input frames T, scheduled perturbation scales (Ts, 1i= score network sø; the mini- mum of diffusion time > O, noise scale factor Ç >>> O.
2: ST 0 // initialize result set
3: for k = 1=1to K do
4:
T(0) T II initialize state
5:
{f1...,,aal ...,, t Discretize((e, Tx.)) || discretize time domain T T -c)M
6:
7:
I1 forward perturbation
(T1)) ...., 1: (
8:
9:
for i -- 1=110 t0 n do
(R,, t T,!))
R R((M) PPss (R (tarr
RRi), (tM)
(u (1 (1)) 11 forward diffusion
11:
T(Um)) (R(tas) ((tac))
12:
T(ts) + (t(t
Tn (
13:
14:
backward annealing
15:
for j = M
do
(I (
I (((++))) (()))))
16:
(9) (441)
...., (()+1)) - Sa (TU))) Il estimated scores
17:
18:
for i 1 to do
(R,, v2) t (b)
19:
(sf 8)) f- 8 ((++1)
20:
disentangle scores
21:
2n (0,1)
22:
2R ~ R.
I| tangent space of rotation
23:
of - P P s/ + +SVF2, translation update w/ removing center of mass
24:
I2 + of + +VVT2R/ // rotation update
25:
T: (1) + exp, T ()-1) [R
// exponential map
T(t) < (T(+)) ...., )
26:
27:
ST ST U [T
28:
return ST
Invariant Backbone Frame Update. The update rule of frames follows the implementation of Algo. 23 in Jumper al. (2021) which can be viewed as composition between original frame and frame update induced by the single representation 8;. The roto-translation equivariant property of frame update (Algorithm 3) is straightforward: the resulting updated frames from each layer are simply a (right) composition of the update Truddate which is induced by the invariant single rep- resentation {s.} and thus cquivariant. In each layer, the IPA only takes invariant features of the input frames update the single representation s;, which is, as being named, invariant to global lat-ttranslaiions on the atom coordinates (backbone frames). As remark, the mentioned addi- tion operations ("+"") for frames (such as in Eq. (4)) symbolically denotes the frame update as the composition of two frames T, + T2 Ti T2 (R, 01) (R2, 22) = [R, R2, R U2 t,).
Proposition 1. Proof. Without loss of generality, consider the residue with atoms coordinates No.; and backbone frame To,; where subscript 0 indicates the input. The corresponding frame T; = (R,, v;) is sampled via the the perturbation kernel (drift, diffusion coefficients) defined in Eq. has no dependency on the input frames and thus the update of forward is invariant to global roto-translation. Therefore, the
(FB) process in Eq. 4. For the forward component,
17

---

Published as a conference paper at ICLR 2024
Algorithm 3 Backbone frame update
1: Require: input frame T:()) updated single representation (1+1) of i-th residue for layer I,
2: di, Ci, di, i2 < Linear(s 8 ((+1)) //b,, Ct, d; E R, 1 E IR3
3: (as; bi, Ci, d3) (1,b, Co d (1,
// normalize quaternion
4: Rg Quat.to.rotmate
bi, Ci di)
// convert normalized quaternion to rotation matrix
5: Tupdate + (Rs, vz)
6: (((+1)) f- fT; o T Tuuddate
7: return (1+1)
composition of input To,; and the forward update is equivariant; the other hand, the backward component is similar to that of forward, with the only difference on the drift term as the score function. Since the score functions are assumed be equivariant to global roto-translation, the backward process also assumes equivariant property. By definition of equivariance, we can extract the global roto-translation transformation Tglobal from the output frames by:
FBB o To,s) Tgoobat FB.((T.))
(12)
where FBr(.) indicates the forward-backwer process for frames defined in Eq. 4. With equivariant transformed frames T; = [R; v;). the atom coordinates constructed from local-to-global transfor- mation also be equivariant. Valglobll R let FBz(-) be the corresponding FB FB-operator
global coordinates instead of frames:
F (Tglobal @gglobl)
F (Tolobal (To, [local)]
(definition of global coordinates)
F (Tylobal o To,;) o Doocll
(associative property of transformation) (equivariance frames)
Tglobut o FBB (To,s) Viocal
Tolobat o FB, (To, i Dlocal)
(definition of operator)
Tglobal FB2 (x gllobal);
(13)
which indicates equivariant property for any atom coordinates Fgllbbl backbone. Plus the invari- ant nature of torsion angles as internal coordinates, the rest of coordinates is thus equivariant with all backbone coordinates being equivariant.
C ABLATION STUDIES
C.1 TRAINING WITH DIFFERENT COIL CUTOFF
The sampling performance by learning from the amortized objective in Section 3.3 relies on the structural characteristics of the training dataset. Here, we studied how different ratio of structural gions in the training structures would affect the benchmarking performance. In specific, we screened out the structures in the training set according to the ratio of coil (no secondary structure presents). To construct the training set, we totally set three coil ladders r.: 0.5, 0.75, 1.0, where, for example, Te = 0.5 indicates the maximal number of residues that forms the coil region is less than or equal to 0.5 * L (L is the number of residues of such protein structure). To annotate the secondary structure to records
PDB, we utilized the DSSP algorithm (Kabsch & Sander, 1983). Results are shown in Table S2. For the result in the main text, the 0.5 is adopted for generally better structural validity. However, we found that an intermediate cutoff with re 0.75 can have better performance for distributional matching.
C.2 ALTERNATIVE NOISE DURING INFERENCE
The forward- barrrdd-bwwrrddd process proposed in Section 3.2 allows in-distribution perturbed samples x during inference by using the same perturbation kernel as training. As an ablation, we alternatively experimented the inference by simply replaced the Gaussian noise parameterized by (p, during
18

---

Published as a conference paper at ICLR 2024
Table S2: Benchmark results of fast folding proteins (Lindorff- et al., 2011) for different filtering ratio of coil used for training. The best result for each metric is bolded.
Setting
Val-Clasb(f)
Val-Bond(")
JS-PwD(J)
JS-TIC(J)
JS-Rg
MAE/TM(J
MAE-RMSIX()
0.5 (PF)
0,963
0.992
0.375
0.397
0.448
0.150
0.209
To 0.75 (PF)
0.953
0.978
0.352
0.373
0.429
0.137
0.202
Te 1.0 (PF)
0.927
0.919
0.360
0.393
0.471
0.141
0.201
Ye = 0.5 (SDE)
0.977
0.982
0.348
0,400
0.365
0.133
0.184
Te (SDF)
0.205
1.0 (SDE)
0.931
0.909
0.332
0,394
0,389
0.143
0.204
forward perturbation with the corresponding uniform noise within the compact of [и-3а, 3a, +u+ 3d), where is mean and a is the standard deviation of Gaussian. Although the support of such uniform distribution quite similar to the original Gaussian (3o rule: the span above has covered 99.73% area), the distributional characteristics of them can be quite different. As shown in Table S3, the use of uniform perturbation kernel does significantly change the performance STR2STR. This result aligns with the previous claim that forward perturbation mainly does exploration and the backward annealing executes exploitation in the forward-backwart backward process. Different exploration scheme induced by the random noise can be studied future works.
Table S3: Benchmark results of fast folding proteins (Lindorff-Larsee al., 2011) by using different perturbation distribution during inference. The best result for each metric is bolded.
Setting
Val-Clash(T)
Val-Bond(T)
JS-PwD(J)
JS-TIC(L)
JS-Rg (L)
MAE-TM(.)
MAE-RMSD(L)
Goussion (PF)
0.963
0.992
0.375
0.397
0.448
0.150
0,209
Gaussian (SDE)
0,977
0.982
0.348
0,400
0,365
0.133
0.184
3r Uniform (PF)
0.962
0.992
0.360
0.397
0.413
0.156
0,231
3o Uniform (SDF)
0.976
0.983
0.343
0.395
0.356
0.141
0.192
C.3 THE CHOICE OF FOLDING MODELS
To sampling from the conformation space, STR2STR requires an initial structure input to provide necessary information of the target protein. Thanks the accurate performance of single struc- ture folding models (Jumper et al., 2021; Lin et al., 2023), this a tractable assumption and no simulation data is introduced. To shown the effect of different folding module on STR2STR, benchmarked both structure prediction models: single sequence-based ESMFold and MSA-based model AlphaFold2 and results are shown in Table S4. Both models are executed under default settings. Note that the structure prediction accuracy of ESMFold can be worse than AlphaFold2 with lower PIDDT. The results demonstrate that the sampling of STR2STR is basically robust the choice of folding models. In our work, we mainly used ESMFold to obtain the initial point because its inference only depends on a single input sequence, which fast and aligned with other baselines.
Table S4: Benchmark results fast folding proteins (Lindorff-Larse et al., 2011) by using different folding modules initialize conformation for sampling. The best result for each metric is bolded.
ESMFoold Val-Clash(1)
Setting
Val. B.BB(B)))))
JS PwD())
TSC
Ry (L)
MAB MMBLTM(L)
MAE RMSD())
0.963
0.992
0.375
0.150
0.209
ESMFold (SDE)
0.977
0.984
0.348
0.400
0.365
0.133
0.184
AlphaFold2 (PF)
0.964
0,365
0.148
0.229
AlphaFold2 (SDE)
0.977
0.981
0.336
0.376
0.351
0.133
0.201
C.4 ABLATING THE ORIENTATION MODELING
To illuminate whether the modeling of orientations (say rotation matrix) of frame necessary, we conduct an ablation study by canceling out the rotation modeling and only modeling the C Corre- spondingly, the ablated model is trained w/o rotation loss. To craft the rotations for IPA input during inference, we leverages two strategies that either use: (1) input rotations or (2) random rotations for
19

---

Published as a conference paper at ICLR 2024
(a) w/ Rot
(b) w/o Rot-I
(c) w/o Rot-R
Figure S2: Visualization of Trp-cage samples of STR2STR for rotation ablation, with rotation modeled; (b) no rotation modeled but using input rotations; and (c) no rotation modeled but using random rotations. Across three subfigures, the input (folded) conformation colored golden, on which the samples of each setting are aligned and superposed.
each backbone frame. As shown in Table S5, the model w/o rotation loss experiences significant performance drop. To illustrate such gap, we take for example the Trp-cage and visualize the struc- tures in Figure $2. It demonstrates that: (1) when using the input rotation, the model almost loses diversity but only generate conformations close to the input; (2) when using random rotations, the model fails capture the local secondary structures but generates random coil. implies orientation modeling can be necessary for the current STR2STR framework.
Table S5: Benchmark results of fast folding proteins (Lindorff-Larsen et al., 2011) by ablating orientation modeling. Two strategies are used to construct the (pseudo) rotations during inference: (I) w/o Rot-/ that uses the input rotations; and (2) w/o Rot-R that adopts randomly sampled rotations. The best result for each metric is bolded.
Setting
Val-Clash(()
Val-Bond(")
JS-P&D(1)
JS-TIC(U)
JS-Rg (U)
MAE-TM(J)
MAE-RMSDXL)
Rot (PF)
0.963
0.375
0.397
0,448
0.150
0.209
w/o (PF)
1.000
1,000
0.675
0.619
0.612
0.641
0.748
(PF)
Rot (SDE)
0.977
0.982
0.348
0.400
0.365
0.133
0,184
wfo Rot-l (SDE)
1.000
1,000
0.639
0,589
0.5951
0.193
0.486
Rot (SDF)
0.001
0.105
0.427
0.442
0.585
0.612
0.735
C.5 ADJUSTING NOISE SCALE FOR SDE
Similar to the temperature factor for sampling from categorical distributions, the sampling can af- fected by simply re-scaling the scale of noise during backward process. Several previous works (In- graham et al., 2022; Watson et al., 2022: Yim et al., 2023) applied such strategies to the task of backbone design generate backbones with controlled diversity and designability. For the zero- shot conformation sampling, we also investigated the effect of different noise scale on the ensemble metrics. As shown in Table S6, we found that although lower noise scale can improve the sam- pling validity, adjusting (increase or decrease from 1.0) noise scale can hurt the sampling fidelity for SDE. Improving both the fidelity and diversity STR2STr by scheduling the noise scales during backwards can be an interesting future work.
D IMPLEMENTATION DETAILS
Definition of Auxiliary Losses. The auxiliary losses are composed of backbone mean square error (MSE) Learkh and distogram loss Laist- Firstly, the backbone MSE loss for structure with
20

---

Published as a conference paper at ICLR 2024
MD Full
MSA subsampling
EigenFold
idpGAN
Str2Str (PF)
Str2Str (SDE)
BBA
VIIIO
Trp-espe
BOL
23D
Chigaolia
WWW domain
Lambiaa
NTL9
Protela
Protein B
Homesdoomain
Figure S3: Contact map for each target protein in the fast folding benchmarks. Samples from reference MD full, baselines as well as STR2STR are shown.
21

---

Published as a conference paper at ICLR 2024
Table S6: Benchmark results of fast folding proteins (Lindorff-Larsen et al., 2011) for different scale of noise during backward diffusion for SDE. The best result for each metric is bolded.
Setting
Val-Clash(t)
Val-Bond(t)
JS-PwD(J)
JS-TIC(L)
JS-Ry
MAE-TM()
MAE-RMSDK.)
Noise scale 0.1
0,985
0.992
0.399
0,419
0.420
0.156
0.214
Noise scale 0.25
0.985
0.992
0.393
0.419
0.414
0.152
0.207
Noise scale 0.5
0.984
0.990
0.382
0.409
0.403
0.147
0.204
Noise scale = 0.75
0.985
0.986
0.367
0.404
0.386
0.142
0.186
Noise scale
0.981
0.184
Noise scale 2.0
0.668
0.722
0.371
0.457
0.457
0,186
0.474
Noise scale 5.0
0.434
0.000
0.585
0.587
0.798
0.285
3.043
residues in training set is defined as:
Dbackb
Σ
|| -
(14)
4n 1=1 RE(N,CU,CO0)
where ét is the predicted coordinates for the atom of residue i by applying the local-to-global coordinate construction by the predicted frames, and the summation is iterated over the residues and the types of backbone atoms shared among all types of amino acids, say
i = 1, 1,2,.... (N, C, 0}. For the distogram loss, let d' be the atomic distance between atom i and 12 of residue and j respectively, the loss is a penalty to the distance error of local contacting neigh- bors within 6Å:
Eii=1 E EE,.EENN,O...OO/ I1 - o.j 121 < 6A, t2)
Ldist
(15)
E 1( <6,, #12)
As indicated in Section 3.3, the overall training loss is L Ldsm + aeLaackb + 3 L (a, B 0)-
Here, we use a = 3 = 0.25 to combine auxiliary loss with the denoising score matching loss. Following Yim al. (2023), the auxiliary loss is only applied for diffusion steps close
0.
which is specifically when / < T/4.
Score Predictions. The Denoising will finally output an updated frame T from the noisy input T at time t. According to the SE(3)" diffusion (Yim et al., 2023), the corresponding score constructed from such output for the i-th residue frame T7 = (R), ot) is:
R log
& RT)
ow Pa Pe R(), of - 2000
so (T,
(16)
e ((())
where w = Axis_angle R R2) and Prjo R) is defined in Eq. 6. log ( is the logarithm mapping
defined by log R
9 (RT Rt) with + 2 cos 0 trace( (R) (Leach al., 2022).
2sin
Training Diffusion Modules. For the diffusion on translation components E R3 we use the linear beta schedule of VPSDE (Song al., 2020) with 8min 0.1 and 8max 20, say 3(t) 8min + (t/T)) - Puin) ); for the rotation diffusion, we use the logarithm sigma schedule of VESDE as (1) log(t "m (T t) e with Omin 0.1 and dmax 1.5. the score matching objective, the maximal time scale is selected to be Tm = T = 1.0. To optimize the parameters of DenoisingIPA, the Adam optimizer (Kingma & Ba, 2014) used with Ir 10-4 and B1 0.9, B2 0.999. The training time is approximately ~0 GPU days the NVIDIA Tesla V100- SXM2-32GB GPU.
Training Data. To curate the training data for STR2STR, we collected the mmCIF structures from the Protein Data Bank (PDB) on June 9th, 2023 in single chain (monomer) with experimental reso- lution better than 5Å. Then we screened out those records with length L > 512 or L < 10, which prevents memory overflow a single GPU while removing too short peptides. To avoid possible data leakage, the PDB records in the training set with the same sequence identity as test proteins removed. After filtering, there are totally 26,459 monomeric structures in the training set.
22

---

Published as a conference paper at ICLR 2024
Parameter
Number
Single channel Cs
256
Pair channel C=
128
Initial skip channel Cakip
64
Number of heads Whedd
Number of query-key points fig
8
Number of value points ne
12
Number of MHA heads aM
4
Number of MHA layers LAHA
Number of IPA layers LIPA
Table S7: Hyperparameters of DenoisingIPA module.
Hyperparameters of DenoisingIPA. The definition of hyperparameters for DenoisingIPA net- works is shown in Table S7.
Test Targets. The fast folding targets with the corresponding PDB structures listed as fol- lows (Lindorff-Larsen et al., 201 1): Chignolin (No PDB entry, reported in the supplementary of Honda al. (2008)), Trp-cage (PDB entry 2JOF), BBA (PDB entry IFME), Villin (PDB entry 2F4K), WW domain (PDB entry 2F21), NTL9 (PDB entry NTL9), BBL (PDB entry 2WXC), Pro- tein (PDB entry IPRB), Homeodomain (PDB entry 2P6J), Protein G (PDB entry IMIO), a3D (PDB entry 2A3D) and Lambda-reppressor (PDB entry 1LMB). The protein BPTI has the corre- sponding PDB deposit with PDB id 5PTI. The reference MD trajectories for evaluation were ob- tained by sending requests the authors of Lindorff-Larsen et al. (2011) and Shaw al. (2010).
Inference Stage. During inference, we discretized the time domain (e, 1] with 1,000 timesteps to reverse diffusion process via Langevin Dynamics. In practice, minimum diffusion time 0.01 instead of zero used for numerical stability. To obtain initial conformation from folding models, utilized both ESMFold (Lin al., 2023) and AlphaFold2 (Jumper et al., 2021) in Appendix C.3 to obtain the initial structure for conformation sampling of STR2STR from a sequence input. For each test target, we ran the inference procedure of pretrained ESMFold-v] using the default setting and ran the ColabFold (Mirdita et al., 2022) implementation of AlphaFold2 with accelerated MSA search. Among predictions, the model with the highest confidence score (pIDDT) is adopted to initialize the structure-to ttruuuuuurure translation. Since different perturbation scale, i.e. transition times Ta can lead to different scale of exploration, we applied linear schedule of Ta between [Lmin mmax)] For fast folding proteins, mmin 0.25, fmax 0.7 and the stride is 0.05. For the structural protein BPTI, Imi 0.10, 'max 0. and the stride 0.05 are used; moreover, we anneal the noise scale to be 0.1 for the SDE sampling for BPTI. The resulting ensemble of STR2STR is obtained by merging the sampled conformations from each perturbation scale and then evaluated.
Baselines. The conformations sampled from MSA subsampling (Del Alamo et al., 2022) involves limited recycle and depth-reduced multiple sequence alignments (MSA) during AlphaFold2 (AF2) inference. Specifically. JacKHMMER and HHBlits including UniRef90,, MGnify, and BFD, following AF2's original pipeline (Jumper et al., 2021) instead of using MMSeqs2 (Steinegger & Söding, 2017) as in Del Alamo al. (2022). The number of recycles was set to one. Only a small portion of the found MSA was used as input to compute the MSA representation fed Evoformer. To diversify the generated ensemble, the max_ext ra a and max_msa.ce.m aallsaccluss clusterss were set to be 16 and respectively (Del Alamo et al., 2022). To sample five AF2 pTM models (checkpoints) were used for inference and the conformations were joined together; For FigenFold, we executed the inference loading the pretrained model weights with default hyperparameter configurations, say a 1.0 and 3 = 3.0; Conformation samples from idpGAN were obtained by running their official released code with the default configuration for each evaluation target sequence.
used for searching MSA from datahases
conformational landscape more exhaustively,
Potential Energy Evaluation. In order to evaluate the reweighted ensemble sampled by STR2STR with probability flow, we calculate the potential energy by a force field. Specifically, we perform the energy minimization procedure for each conformation with the side chains packed. After the
23

---

Published as a conference paper at ICLR 2024
protonation (add corresponding hydrogens to each heavy atom), the all-atom conformation is min- imized by the Amber-ff14SB force fields using the OpenMM package (Eastman et al., 2017) with implicit solvent GBn2. Similar to Jumper al. (2021), the harmonic restraints are independently applied to each heavy atom (non-hydrogen) that keep the whole conformation similar to the input, with spring constant being 10 kcal/mol--*" The maximal step of minimization is set to be unlimited with tolerance 2.39 kcal/mol. After convergence, the minimized potential energy is reduced to be unit-less by dividing the tempering coefficient k which is approximately 0.5918 mol/kcal for energy unit in kcal/mol.
E DETAILS OF EVALUATION METRICS
In this section, we elaborate the definition of the evaluation metries introduced the experiments.
Validity The Validity will examine each conformation with respect to (1) clash, and (2) breaking bond. Firstly, the non-clash validity (Val-Clash) is defined as the ratio of clash-free conformations. It is calculated as the number of conformations that do not contain steric clashes divided by the number of all evaluating examples. Sterie clash determined by whether two contacting atoms is too close each other. For an example conformational ensemble (x(1) J. 1=1 we have: Val-
Clash(( (x) = 1.0- 인 E ji s.t. (
Xx x= < d), where x C E IR'S indicates
the Cd* coordinaee of j's residue in conformation sample x(9) In practice, we choose 8 according to the van der Waals radius of C minus an allowable overlap 8d. or formally 5 = 2 x 1.7
da (unit: A). The default value of 6 is set to 0.4, which a reasonable value when examining protein-protein interactions (Ramachandran et al., 201 1). Secondly, the bonding validity (Val-Bond) is defined as the ratio of conformations that maintain the distance between adjacent C' atoms within certain threshold. The rationale behind is that adjacent C are "bonded" by the chemical bonds of Ca C - N - C and should not be too distant from each other. Since there is no general rule defining the valid "bonding" distance for CO-Co, defined by the maximum adjacent C*-C" distance among the reference MD (full) trajectories of the target protein. By definition, for target protein P. we have Val-Bond( [x P) = 1.0 1
adopt statistical threshold which is
N E 1(3 j, s.t. [x j X s+1 > oyohd) where o is the P-specific threshold defined above, and other symbols are similar to above.
(i) (i)
Fidelity The fidelity of a set of conformations is evaluated by measuring the distributional similar- ity between the reference ensemble and the distribution of generated ensemble, similar to the Fréchet inception distance (FID) (Heusel et al., 2017) for evaluating the synthetic images. We adopted the Jensen-Shannon (JS) divergence due to its symmetric property that has penalty to the model distri- bution for lack of ground truth coverage and biased coverage. To cater for baseline models, only C -atamoms are consider to calculate the divergence metries. Inspired by Janson et al. (2023) and Arts et al. (2023), we adopt three important roto-translation invariant features that can loyally reflect the ensemble characteristics: (1) Pairwise distance. Following the setting of Arts et al. (2023), the pairwise distance is computed by enumerating all pairs of atoms with an offset three. To transform the continuous values into distribution, histograms categorized pairwise distribution over which the JS divergence is calculated. For each channel of histogram, a pseudo-count value of 10-6 was used for zero frequencies slightly smooth the distribution. (2) The slowest two components of the time-lagged independent component analysis (TICA) (Naritomi & Fuchigami, 2011:
built with Nbi. = 50 bins represent the
al., 2013). We firstly calculate and flatten the pairwise distances for each protein conformation in the ensemble to be evaluated. We then fit the projection of TICA based on the reference full MD trajectories for each target using the Deeptime Library (Hoffmann et al., 2021). The first and second coordinates are selected after applying the TICA dimension reduction to each ensemble. Note that the samples from neural models are biased with respect to Boltzmann ensemble and time-invariant, and we thus project them uniformly to the TICA directions of the reference MD trajectory for visualize purpose. Histograms are built for both components similar to the pairwise distance above. (3) Radius of gyration, which measures the root mean square distance of each atoms relative to the center of mass. Similar histogram treatment is applied same above.
24

---

Published as a conference paper at ICLR 2024
MD Full
MSA subsampling
ElgenFold
IópGAN
Str2Str (PF)
Str2Str (SDE)
BBA
Vilto
Trp-cage
BBL
430
Chignolin
WW domain
Lambda
NTL9
622
Protein G
Protein B
Homeodomaio
Figure S4: TICA plots for each target protein in the fast folding benchmarks.
25

---

Published as a conference paper at ICLR 2024
Diversity The diversity of the ensemble of interest can be derived from any structural similarity score by enumerating and averaging the pairwise scores for that ensemble. Here we adopt two most commonly used scoring functions: root mean square deviation (RMSD) and TM-score (Zhang & Skolnick, 2004). RMSD reflects the deviation degree in length (here we use nanometer (nm) as unit) and is unnormalized. TM-score, tural similarity between two input structures, ranging from O to 1 and unit-free. During evaluation, both scores are calculated using the officially released binary from (Zhang & Skolnick, 2004) in- dependently for each fast folding target. After that, we use the mean absolute error (MAE) of both diversity metries between the reference full MD trajectory and each model and finally report the av- erage across all targets. The rationale is that the ensemble diversity is not the higher the better, which instead depends on the structural characteristics of different proteins. Specifically, some proteins can show rigid structure, of which a typical example is the cyclic peptides (Kessler, 1982). In this case, the motions in the protein dynamics are limited, and thus embody small ensemble diversity. On the other hand, other proteins show more dynamic behaviors like distinguishable different states. They can have relatively larger diversity, which can be loyally reflected in the long MD simulations. For example, the SARS-Cov-2 spike protein was found to have transitions between its open and closed states (Gur et al., 2020). Therefore, we align the ensemble diversity with MD references rather than evaluating as the higher the better.
the contrary, is normalized score evaluate the struc-
F EXTENDED EXPERIMENTAL RESULTS
F.1 FAST FOLDING PROTEINS
Here we visualized the contact map and TICA plots for each target from the fast folding proteins, which are shown in Figure S3 and Figure S4, respectively. Following Arts et al. (2023), we use d = 10A as the criterion discriminate contact pair of atoms, and take the logarithm of frequency to color the contact map. STR2STR outperformed other neural sampling baselines by better capturing the metastable states reflected in TICA and contacting dynamics contact map. By comparing STR2STR and reference MD full, we found that STR2STR basically explored the coarse-grained contact patterns yet the fine-graine ine-grained characteristics of the sampled structures can be further improved as the TICA plots are not perfectly matched. One limitation of STR2STR, due to learning simply from crystal structure, is that for some cases, it failed to well explore the transient states (colored in shallow green). These points, with low probability or high free energy, are explored by MD simulations via climbing the energy barriers and do not present during the sampling of STR2STR.
F.2 CONFORMATION INPAINTING OF NANOBODY
In this study, we investigated the inpainting capability of our model a nanobody derived from Llama glama (PDB entry: 1G9E). The dynamic nature of CDR loops in nanobodies underpins their binding capacity
specific targets. To sample the conformation CDR loops, we initialized from one of the PDB structures, froze the nanobody frame and only perform STR2STR translations for the three CDR loops according to with IMGT numbering (Lefranc et al., 2003). To enhance the loop modeling, the sampling is based on the = 1.0 training checkpoint and SDE with other configurations setting as default. As illustrated in S5, the loop structures produced by STR2STR are comparable to the native configurations seen in NMR structures, which demonstrates the potential of STR2STR to perform conformation inpainting.
F.3 VISUALIZE CONFORMATIONS OF VARYING T3
In this section, we showcase by visualizing the protein conformations from different Ts in Figure S6. To enhance the distinction, we color the protein ribhons according to their secondary structure assignment: helix is colored red, beta sheet is colored blue and coil (loop) is colored brown. As shown in Figure S6, increased T3s renders sampled conformations to possess distinguishable differ- ences while keeping certain characteristics of the initial structure. Additionally, we show the moving TM-diversity of the sampled ensembles along the increasing T3 in Figure S7, from which we find that the overall diversity plateau after around T3 0.8 for both PF and SDE.
To better illustrate the effect of T3 on sampling, we compare the sampled conformations of WW domain between STR2STR and the reference MD simulation in the two-dimension TICA space. As
26

---

Published as a conference paper at ICLR 2024
Figure S5: Visualization of nanobody conformation inpainting by STR2STR (SDE). On the left, the structure ensemble is derived from the NMR structures (PDB entry: 1G9E), On the right, we present the inpainted structures of CDR loops to 3.
Table S8: Pearson correlations between sampled diversity and apo-holo diversity, measured TM- score residue flexibility as in Jing al. (2023). Results of EigenFold is obtained from the original paper. The mean/median correlations are reported for per-target residue flexibility.
Method
TM
Residue ficxibility (global)
Residue ficxibility (per-target)
EigenFold
0.12
0.13
0.41/0.4
STR2STR (PF)
0.21
0.18
0.33/0.31
STR2STR(SDE)
0.11
0.13
0.33/0.34
shown in Figure S8, we plot the scatter of 100 samples from STR2STR in both T3 = 0.3 (relatively small perturbation) and 0.7 (relatively large perturbation). comparison with that reference sample the distant mode and exhibit some unfolded patterns. MD. It demonstrates that STR2STR fine-granne fine-grained structural deviation which not reflected the However, one can still differentiate the -iwdeennsional TICA space, which may explain the performance gap between JS-PwD and JS-TIC in Table 1.
F.4 APO/HOLO DIVERSITY
In this section, we evaluate the performance STR2STR on the Apo/holo dataset (Jing et al., 2023), which contains 90 pairs of apo/holo records in PDB. Following Jing et al. (2023), sample structures per apo/holo pair using STR2STR and evaluate them using the same script as in EigenFold. As shown in Table S8, STR2STR with probability flow (PF) shows better correlation with the ground truth diversity, yet worse on the per-target correlations. The performance inconsistency between EigenFold and STR2STR is due that the per-target correlation can be affected by the varying residue lengths.
Moreover, following Jing et al. (2023), scatterplot the TM-scores of each apo/holo target for both EigenFold and STR2STR. In specific, we investigate (i) the ability to capture both apo/holo using the ensemble TM-score (TMens) and (ii) the diversity of the sampled conformations with the diversity TM-score (TM,,r) As shown in Figure 89, the EigenFold generates better apo/holo structures with overall higher ensemble TM-scores (TMens). This merit can come from the OmegaFold embeddings on which the FigenFold is conditioned, while the baward-backkwad process of STR2STR alone is not sufficient to recover accurate fine-grained structures. On the other hand, from the lower plots, we found that STR2STR (PF) has a apparently better correlation (0.21 v.s. 0.12, in Table S8) between predicted diversity TMvar and ground-truth diversity TMapo/hntos which indicates the STR2STR aligns better with the underlying dynamic characteristics. Inspired by this finding, one promising future work can focus on jointly modeling accurate fine- Ene-grained structures like FigenFold/AF2, while capturing diversity well 2TR2STR.
27

---

Published as a conference paper at ICLR 2024
Increased perturbation scale To
BBA
Villin
Trp-cage
BBL
a3D
Chignolin
WW domain
Lambda
NTL9
Protein G
Protein B
Homeodomain
Figure S6: Visualize of protein conformation sampled by STR2STR under different T3 (from left to
right: 0.3, 0.4, 0.5, 0.6, 0.7) for the fast folding targets. Helical residues are colored red, beta sheets colored blue and coils are colored brown.
28

---

Published as a conference paper at ICLR 2024
0.8
Probeality flow SDE
TM div (mean per target)
0.3
0.2
6.1
6.0
0.2
0.4
0.6
0.8
1.0
T d.delta
Figure S7: Moving TM-diversity of the sampled ensemble along increasing Tg. The TM-diversity each Ta averaged across all per-target TM-diversity for STR2STR in PF or SDE settings.
D.
MO
tC11
tIC1
Figure S8: The illustrative diagram for sampling WW domain conformations. (left) STR2STR under different perturbation Ts, being compared with (right) long MD simulations. The grey contour is the kernel density estimate (KDE) plot based on the whole reference MD trajectories.
29

---

Published as a conference paper at ICLR 2024
Foge
Str2Str (PK)
(SDE)
0.4
0.
0.6
0.4-
0.0
0.6
0.5
0.5
0.2
0.4 spoteed
pponsio
apoiosi
1.0
0.2
0.
0.0
0.2
0.-
0.0
0.2
044
0.6
0.5 1.0 0.5 0.2
0.4 0,5 0.5 1.0 0.5
0.2
0.4
0.2 1.0
Figure S9: The scatterplot of the TM-score on (upper) TM versus TM TMapo/holo- and (lower) TMvar versus hhaaoo for EigenFold and STR2STR, following Jing et al. (2023). Note that for the lower plots, the regression line is shown with confidence interval. We found that EigenFold generate structures with higher quality and thus achieve better ensemble TM-score (TMens), by conditioning on the OmegaFold embedding; while STR2STR has better predicted diversity (TMvar) correlation with the ground truth diversity.
F.5 TILTING TOWARDS BOLTZMANN DISTRIBUTION
The data-driven diffusion sampler STR2STR may not guarantee the generated samples from the Boltzmann distribution, but instead, an ensemble independently sampled conformations. To tilt the generated samples to obey this physical assumption, ples from probability flows with the learned score. By approximately replacing the conformation log-likelihood log px (x) by the amortized translation distribution pa over frames with learned pa- rameters 0:
compute the likelihood of sam-
(x AV(T) log pe TT) log ((To)
log Pr, (T To) - /
1 To (2(() (T,, C,
AV (T) log
(17)
where AV(T) : log]odee(((((rr (T)) is the change of volume for reconstructing Euclidean co- ordinate from frames and Jr... is the corresponding Jacobian matrix; V the divergence operator, with the prior likelihood log (TT,) C' being constant for a fixed ensemble. We can then reweight the conformation ensemble by calculating the importance weights:
(xx) x exp |- (u(x) + log (x)))
(18)
where u(-): R3N + R is some all-atom force field for proteins with its unit reduced by 1/kgT where kg is the Boltzmann constant and T the simulation temperature. The generation with reweighting can approximately achieve unbiased sampling from Boltzmann distribution (Noé al., 2019).
Table 89: Distributional divergences on fast folding proteins (Lindorff-Larse) et al., 2011) for prob- ability flow STR2STR w/ and w/o reweighting.
Setting
JS-PwD
IS-TIC
JS-Rg
Str2Str (PF)
0.375
0.397
0.448
Str2Str (PF, reweighted)
0.487
0.461
0.489
However, in practice, we found that the reweighting for zero-shot conformation sampling using STR2STR cannot improve the distributional metrics shown in Table $9. The reason may lie in
30

---

Published as a conference paper at ICLR 2024
the fact that, unlike previous Boltzmann generators (Noé al., 2019; Jing al., 2022), STR2STR generates conformation hypothesis by repurposing the translation distribution pe (T in transfer learning manner, and receives energy supervision during training of STR2STR to induce close matching to equilibrium distribution. Moreover, the all-atom force field can be sensitive to the position of side-chain atoms, which are however predicted by imperfect external module, and thus make the estimator in Eq. (18) suffer from large variance. Still, the fidelity metrics (JS-*) used in the benchmark can also reflected (penalized) the the potential bias between the model distribution and underlying Boltzmann distribution represented by MD trajectories. The improvement of STR2STR to accommodate energy- based training for acore-based STR2STR a promising future work.
G EXTENDED DISCUSSION
To better contextualize the proposed STR2STR, we review much broader range of related and relevant literature as follows. In Section G.I, the traditional molecular dynamics and Monte Carlo methods are overviewed. In Section G.2, we discussed in detail the STR2STR with related methods that predicting or generating the protein structures.
G.1 MOLECULAR DYNAMICS AND MONTE CARLO METHODS
In this section, we overview the related works of Molecular dynamics (MD) and Monte Carlo (MC) methods for protein conformation sampling. As described in previous sections, MD evolyes the Newtonian equation along time explore the conformation space guided by some force field, such as Amber (Wang et al., 2004) and CHARMM (Vanommeslaeghe et al., 2010). Although MD is simple and considered effective, the simulation can be computational intractable to study natural dynamic behaviors such as folding (Lindorff-Larsen et al., 2011). To accelerate, several strategies sacrifice accuracy trade for speed, such as using implicit solvent (Ferrara al., 2002) or coarse- graining (de Jong et al., 2013). Enhanced sampling methods have been proposed to enforce the ex- ploration of MD simulations (Abrams& & Bussi, 2013) and make long timescale accessible with mild computation. To name a few, the umbrella sampling (Torrie & Valleau, 1977). metadynamics (Laio & Parrinello, 2002) and replica exchange molecular dynamics (REMD) (Hansmann, 1997; Sugita & Okamoto, 1999; Swendsen & Wang, 1986). However, none of these methods can totally replace canonical MD by providing completely comparable thermodynamics and kinetic characteristics.
MC simulation typically steers Markov chain Monte Carlo (MCMC) trajectory to explore the protein conformation space (Fichthorn & Weinberg, 1991). Due to lack of temporal information, MC methods only yield samples thermodynamic ensemble and cannot reveal insights into kinetics directly (Paquet et al., 2015). Nevertheless in Liang & Wong (2001), the authors showed that large- scale kinetics can be reconstructed from the MC-sampled ensembles. Since the MC moves not driven by the gradient (force) field, typically only a few degrees of freedom can be altered in a single move, which can be hindered from using explicit solvents (Nerenberg & Head-Gordon, 2018) and scaling up to large systems. In Heilmann et al. (2020), the authors showcased the MC simulations on (Trp-cage, Villin and WW domain) out of 12 fast folding proteins in comparison with the MD counterpart (Lindorff-Larser et al., 2011). But the result is less convincing than MD since the folding temperatures are overestimated. A most recent work (Klein et al., 2023) leverage the normalizing flow to build a neural MC sampler by training on MD simulation data. However, such model only built for very short peptides (2-4 amino acids) and the generalization capacity still questionable. Currently, MC falls out of the mainstream for studying thermodynamics and kinetics for malccllm systems such as protein.
G.2 PREDICTION AND GENERATION FOR PROTEIN GEOMETRY
In this paper, we have proposed the STR2STR for zero-shot conformational sampling which lever- ages the bidirectional diffusion dynamics in SGMs. This method shares many aspects with structure prediction and generation methods. Structure prediction models such as Jumper et al. (2021); Baek et al. (2021); Lin et al. (2023) aim to recover the ground protein folding state by learning on crystal structures in PDB and exhibit highly accurate accuracy (Jumper al., 2021). The EigenFold (Jing et al., 2023) formulated the structure prediction task as conditional generative modeling, and as- sumes there exist multiple stable conformations (conformers) for single amino-acid sequence.
31

---

Published as a conference paper at ICLR 2024
However, one of the biggest problem is the lack of ground truth multi-state structure data that be- longs to a single sequence for training the conditional diffusion model. Methods for protein back- bone design, described in Section 5, aim to learn the unconditional distribution of PDB structures and generate de novo backhone or scaffold structures. These backbone generative models (Ingra- ham et al., 2022; Trippe al., 2022; Watson et al., 2022; Wu al., 2022a; Yim et al., 2023) model the protein structure landscape and successfully yield protein-like decoys with good "designability". None of these methods discover the potential of such models for conformation sampling. Several recent works (Del Alamo et al., 2022; Vani et al., 2023; modified MSA imput of AlphaFold2 diversify the predicted samples but this protocol still exhibits limited structural deviation since the training objective of AF2 is deterministic.
et al., 2023) adopted
Our proposed 2TR2STR can have closed relation to and well based on above research directions. Firstly, to transfer the modeled structure distribution to sampling proposal, STR2STR relies on folding module such as ESMFold (Lin et al., 2023) that provides starting point. Secondly, both EigenFold and STR2STR leverage the PDB database for training with a denoising score match- ing (DSM) objective and aim to sample multiple structures. The difference is that, the EigenFold modeled mapping from the sequence-encoded latent space (say OmegaFold (Wu et al., 2022b) embeddings (s;, (8,,2))), ). to the structure space. In contrast, the STR2STR delineated solely the struc- ture space and sampled conformations in amortized manner. The performance gap between them can be because Figenfold handles a more challenging conditional generative modeling task, where the embedding condition (s,, 2ig) far complicated than typical scalar class label. Without enough data for each condition in PDB, the resulting diversity (exploration) is thus greatly limited. In contrast, STR2STR learns simply the structure space where the conformations can be represen- tation as data-like points, and thus perform better. Lastly, the amortized learning objective in Eq. (8) coincides with the one used in the (unconditional) generative modeling of backbone structures. inference-time plug-and-
Intuitively, one can view the STR2STR tive models by applying the forward-bace -awkwrdd dynamics. Our results demonstrate that learning on purely structure space can be effectively transferred to specific conformation sampling.
of backbone genera-
The core assumption behind the proposed framework STR2STR that the dynamic patterns among conformations can be partially shared with the patterns between different structures in the PDB. That is to say, the common inter-protein structural characteristics in the training set can be distilled by DSM in the score network, which is "unlocked" during sampling to infer the inter-conformation deviation within the test system. Our work opens up a broad research direction that leverages the generative modeling on structure alone tackle the protein conformation sampling task.
32


Reviews:
[{"review_id": "vnnmTdgJ5S", "reviewer": "Reviewer_BVKt", "summary": "This paper introduces a technique for sampling equilibrium distributions of proteins, eliminating the dependency on costly Molecular Dynamic simulations. The suggested technique utilizes the ESMFold protein embeddings and trains an equivariant denoising diffusion model using samples from the Protein Data Bank, predominantly featuring single folded states (i.e., absolute minima of the equilibrium distribution). During the testing phase, the model diffuses to a specified noise level, partially erasing the protein structure but not entirely, and then it denoises it again, resulting in a different protein conformation. This translation from one structure to another is employed to sample different conformations, starting from the folded structure. The method's effectiveness is assessed using various metrics such as validity (defined as the non-clash) fidelity (defined as the JS between reference and samples TICA distributions) and Diversity. The method seems to outperform previous works.", "questions": "1)\nThe core part of the proposal of this method is described in 3.2 (forward-backward Dynamcis) where I think a more elaborate explanation could be done here.\n\nFor example, when sampling a conformation T ~ p(T | x_0), is x_0 consistently the initiating folded structure, or could it be a T derived from a preceding sampling step?  This is not clear to me from the text. I imagine that if setting it always to the folded structure it would bias the distribution to the minima. Could the authors provide a more precise description of this in the method section?\n\n2)\nIn section 3.1, could the authors provide more details, or cite relevant literature, explaining how the side chain atoms are derived from the backbone atoms?\n\n3)\nIn the following sentence \"Empirically, increasing T_delta leads to enhanced diversity yet it may hurt exploitation by demanding more reverse steps\".\n\nIs this true for any T_delta value? I would suspect that if T_delta goes to a large enough amount of noise (reaching the gaussian distribution), the result would be equivalent sampling from the trained PDB distribution, resulting again in sampling from the folded minima instead of a diverse equilibrium dataset.\n\n4) In section 3.2 (Score Network architecture), the authors indicate that there have been minor modifications to IPA to include pair representations with edge layers. Could the authors provide more context as to why this is necessary and not arbitrary?\n\n5) In Appenix B, Algorithm 2\nHow is the algorithm returning x_0? Based on the paper wasn't x_0 the starting point?\n\n6) Better interpretation on why the method works:\nGiven that the model is optimized on the PDB dataset, we would expect it to only learn the PDB landscape. However when degenerating the process with the proposed approach it actually learns to generate samples close to the equilibrium distribution. Because it never had access to a Force Field and only to the folded state it is quite surprising it is able to \"make up\" that information. Could it be this is only possible because of the ESMFold embeddings? I.e. do these embeddings contain information about the protein landscape beyond the folded structure, and then are you recovering that information present in the ESMFold embeddings with the proposed method? It would be interesting to know the authors interpretation about this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation and goals of this work are very relevant. Being able to sample protein equilibrium distributions without need of computing expensive molecular dynamics can have a high impact in the sampling community.\n- The metrics used in the paper to assess the quality of the equilibrium distributions show the proposed method outperforms previous works.\n- The paper contains many metrics assessing different aspects of the generated distributions.", "weaknesses": "1)\nA more thorough review of prior studies could be beneficial. For instance, EigenFold (Jin et al, 2023) is a diffusion model trained solely on PDB samples with the same aim to generalize to protein distributions. It would be highly useful for readers to have a more detailed comparison between the proposed method and EigenFold, specially considering the apparent superior performance of the proposed method.\n\n1.1 What is novel in the proposed method w.r.t. EigenFold?\n1.2 Is the performance gap between EigenFold and the proposed method attributed to a difference in the conceptual approach or is it due to a more technical element such as the use of ESMFold in place of OmegaFold embeddings?\n\nI think answering these questions can really benefit future works when trying to spot the key aspects of the model without need to dig into the codebase.\n\n\n\n2) \nThe validity metrics analyze the non-clash ratio, but it would also be as relevant to examine the distributions of bonds and ensure no bonds are breaking when categorizing a sample as valid. Have the authors conducted this analysis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tobho058JE", "reviewer": "Reviewer_dLpk", "summary": "The authors propose STR2STR, a structure-to-structure framework to perform conditional conformation sampling of protein structures. The goal of conformation sampling is to sample new stable, energetically favorable structures from an initial protein structure. DIirecting atomistic modeling of protein structure is often intractable due to the size and high degrees of the system. Rather than sampling the entire structure at once, the authors reformulate the task into multiple conditional sampling tasks. The backbone is generated conditioned on a ground truth conformation. The backbone and ground truth conformation are used to then generate the side chains and carbonyl oxygen on the backbone. The authors present an updated implementation of the invariant point attention from AlphaFold and show that their confirmation sampling process is SE(3) equivariant. Compared to recent deep learning methods, STR2STR has comparable validity while improving fidelity and diversity of generated conformations.", "questions": "* “We notice that the ensemble diversity is not the higher the better and depends on the characteristics of the target system”\nCould you please elaborate more on this?\n* Is there a reason run times are provided only for a single target protein rather than all 12 (other than the high cost of sampling)?\n* Any particular reason why 100ns MD simulation runtimes are not compared with STR2STR runtimes?\n    * According to Table 1, smaller trajectories sometimes outperform the proposed model. Having all the data allows the reader to have a more complete understanding of the capabilities of the proposed model", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors present a novel representation decomposition for protein structures to enable structure-to-structure translation using a diffusion model.\n* The amortized learning objective which uses only pre-confromed data and does not require new simulated sets is significant and can reduce the cost of training future models.\n* The improved diversity and fidelity of the model while not relying on simulated conformations is also significant. Improvements over EigenFold, a similar diffusion model is interesting.", "weaknesses": "* The benchmarks set is small and make it difficult to judge the results provided. \n* Some presentation issues (refer to clarifications)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lbTmuAeAW", "reviewer": "Reviewer_Fe1f", "summary": "The authors present Str2Str, a score model for protein conformation sampling. The model is trained on crystal structures from the PDB and could generate diverse conformations for unseen protein systems. Unlike previous mothods (e.g., EigenFold), Str2Str is sequence-agnostic. The model learns protein conformation distribution without knowing the amino acid residue sequence. Through the proposed forward-backward dynamics, which is similar in vein to simulated annealing, the model is able to effectively explore different potential energy minima without suffering from rare event sampling problems.\n\nI find this manuscript well-written and provides promising results towards solving the protein conformation sampling challenge. Please see some of my main concerns below.", "questions": "- What is the difference between the proposed Str2Str pipeline and applying forward-backward dynamics using a pretrained FrameDiff?\n\n- Is it possible to provide an additional ablation study on a few fast-folding proteins, where only alpha-Carbon coordinates are modeled? Maybe you can just turn off the rotation loss. I am curious whether frames are essential for accurate protein conformation sampling.\n\n- From Appendix >> Inference Stage on page 22, the ensemble structures is obtained by merging samples from each perturbation scale, in total 1,000 conformations for fast-folding proteins. It is surprising that metrics such as JS-PwD/MAE-TM remains so small even though $T_{\\delta}$ can be as large as 0.7. I expect that many samples are quite different from the starting conformation, especially for large $T_{\\delta}$. Could you please explain this behavior?\n\n- Since the model is sequence-agnostic, the model should sample many \"outlier\" conformations, which is not accessible from the given protein sequence. In Fig. S5, are most of the shown structures valid conformations for each protein system?\n\n- Page 18, I do not understand why reweighting could not help improve relevant distributional metrics. Does that mean the model likelihood estimation is not accurate, or model diversity originates from model uncertainty? Quick check, if we manually make $\\log p_{X}$ a constant value across all samples, would it improve the metrics?\n\n- Would you mind showing model performance on the apo/holo dataset for a fair comparison with EigenFold?\n\n- Could you please (1) provide the source code in supplementary materials; (2) make the benchmark data open-access for reproducibility?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The major contribution of this work is the proposed forward-backward dynamics using a structure-based score model trained on the PDB dataset. From Fig. 5 and Fig. S3, it is clear that Str2Str outperforms other existing protein conformation sampling methods.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQuOUhh1HA", "reviewer": "Reviewer_6bVL", "summary": "This paper aims to capture the dynamic protein structures and obtain various protein conformations. Instead of the resource-demanding MD and MC methods, the authors take inspiration from simulated annealing and propose a structure-to-structure translation framework for zero-shot conformation sampling. Roto-translation equivariance is appropriately guaranteed. Experiments on the 12 proteins in a newly established benchmark confirm the validity, fidelity, and diversity of the produced conformations. Ablation analysis and case study are provided for readers to have a deeper understanding of this paper. The problem under study is significant and this proposed idea is interesting.", "questions": "- What is the total number (or dimension) of variables T, R, v, X? Does this have something to do with the number of residues in each protein?\n- The score matching objective is the supervision signal. That is, the model is required to approximate the distribution of atom positions/angles of a target protein. I'm wondering how the proposed method encourages diversity of the sampled conformations. \n- Please properly use the term dynamics. Not sure if the authors properly interpret this term. To the best of my understanding, the concept of dynamics should have something to do with time and is usually used to indicate a temporally changing variable. Here in this paper, I do not see the conformation of a target protein changes with time. Instead, this paper cares more about different candidates of stable conformations. The proposed forward-backward process seems like a dynamics but essentially is a process of heating-then-simulated-annealing, which is different from the dynamics of protein conformation. Please properly modify some statements, especially those regarding dynamics.\n- How similar are the 12 proteins in the benchmark to the training protein data? Need to ensure zero-shot by quantifying the dissimilarity of train/test proteins at sequence and structure levels.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strucutre-to-structure translation seems a novel way to sample various conformations of a target protein. It requires less computational resources than MD and MC. \n- The experiments are described in details, which is very helpful for the readers the understand the proposed method.\n- The paper is well-written and easy to follow. More explanation about the formulas would further increase readability.", "weaknesses": "- Although the overall structure-to-structure translation for protein conformation sampling is novel, there may not be adequate points to support the whole paper. The forward-backward process is most similar to a diffusion process, where the forward process adds noise (or heating) to data distribution and the backward process denoise (or annealing). The learning objective is similar to DenoisingIPA, except the minor edge translation layer.\n- Proper discussion of MC and MD-based related studies for protein conformation sampling could give readers a more thorough picture of where this paper is located.\n- Also, I'd like to see MC and MD-based baselines in the experiments. Though they are slow, I'm wondering how they perform in terms of validity, fidelity and diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)