Diffusion Probabilistic Modeling of Protein Backbones in 3D for the motif-scaffolding problem
Authors: Brian L. Trippe
Jason Yim
Doug Tischer
David Baker
Tamara Broderick
Regina Barzilay
Tommi S. Jaakkola
Venue: ICLR 2023 poster (2023)
Keywords: Diffusion Models
Sequential Monte Carlo
Protein Design
Geometric Deep Learning

Abstract: Construction of a scaffold structure that supports a desired motif, conferring protein function, shows promise for the design of vaccines and enzymes. But a general solution to this motif-scaffolding problem remains open. Current machine-learning techniques for scaffold design are either limited to unrealistically small scaffolds (up to length 20) or struggle to produce multiple diverse scaffolds. We propose to learn a distribution over diverse and longer protein backbone structures via an E(3)-equivariant graph neural network. We develop SMCDiff to efficiently sample scaffolds from this distribution conditioned on a given motif; our algorithm is the first to theoretically guarantee conditional samples from a diffusion model in the large-compute limit. We evaluate our designed backbones by how well they align with AlphaFold2-predicted structures. We show that our method can (1) sample scaffolds up to 80 residues and (2) achieve structurally diverse scaffolds for a fixed motif.

Published as a conference paper at ICLR 2023
DIFFUSION PROBABILISTIC MODELING OF PROTEIN BACKBONES IN 3D FOR THE MOTIE-SCAFFOLDING
PROBLEM
Brian Trippe* otrippe@mit.eem edu
Jason Yim*t jyimêmit.cmm edu
Doug Tischer :
David Baker
dtischer&uw.cm edu
dabaker@uw edu:
Tamara Broderickt tbroderick@mit. .dd
Regina Barzilay?
Tommi Jaakkola
regina@csail mitttetee edu
commi@csal mit..... edu
ABSTRACT
Construction of a scaffold structure that supports a desired motif, conferring protein function, shows promise for the design of vaccines and enzymes. But general solution to this scaffolian problem remains open. Current machine-learning techniques for scaffold design either limited to unrealistically small scaffolds (up to length 20) or struggle to produce multiple diverse scaffolds. We propose to learn distribution over diverse and longer protein backbone structures via E(3)- equivariant graph neural network. We develop SMCDi ff to efficiently sample scaf- folds from this distribution conditioned on given motif; our algorithm is the first to theoretically guarantee conditional samples from a diffusion model in the large- compute limit. We evaluate our designed backbones by how well they align with AlphaFold2
peeeeeoldd--rediteedd predicted structures. We show that our method can (1) sample scaffolds up to 80 residues and (2) achieve structurally diverse scaffolds for fixed motif.!
INTRODUCTION
central task in protein design is creation stable scaffold to support target motif. Here, motifs are structural protein fragments imparting biological function while scaffolds stabilize the motif's structure. Vaccines and enzymes have already been designed by solving certain instances of this motif-scaffolding problem (Procko et al., 2014; Correia et al., 2014; Jiang et al., 2008; Siegel et al., 2010). However, successful solutions to this problem in the past have necessitated substantial expert involvement and laborious trial and error. Machine learning (ML) offers the hope to automate, and better direct this search. But existing ML approaches face one of two major roadblocks. First, many methods do not build scaffolds longer than about 20 residues. For many motif sizes of interest, the resulting proteins would be smaller than the shortest commonly-studied simple protein folds (35-40 residues) (Gelman & Gruebele, 2014). Second, while other methods may generate longer scaffolds using stochastic search techniques, they require hours of computation to generate a single plausible scaffold (Wang al., 2022; Anishchenko al., 2021: Tischer al., 2020). Moreover, when plausible scaffold is found, it remains to be experimentally validated. Therefore, it is desirable to of scaffolds exhibiting diverse sequences and structural
return not just single scaffold but rather variation to increase the likelihood of success in practice.
In the present work, we demonstrate the promise of a particular generative modeling approach within MI for efficiently returning a diverse set of motif-supporting scaffolds. Generative models have been shown to capture a distribution over diverse protein structures (Lin et al., 2021). But it is not clear how to handle conditioning (on the motif) using these approaches. Diffusson probabilistic models (DPMs) offer a potential alternative; not only do they provide a more straightforward path to handling conditioning, but they have also enjoyed success generating small-molecules in 3D (Hoogeboom
"Contributed equally to this work.
I "assaacuustts Institute of Technology University of Washington
Code: https: http::////gttub.bc.////beeee//et
www.oodwioffsooooooooooooooooooo ff_srcDiff

---

Published as a conference paper at ICLR 2023
ProtDiff: backbone generative model
Self-consistency evaluation
Forward diffusion: q(xx0) (x((-))
Sampled
x
x
Backboue
Reverse diffusion: p x
Sequence Design
Generated
MFAELKAKF
SMCDiff: conditional sampling procedure
Sequence
FLEIGDRDAARNALRK
Forward diffuse molif: g(x (x!(-))
x(
Structure Prediction
SMC estimate: Polx
1x
Calculate:
Predicted
(1)TM-score
Structure
(aCTM)
(2) Motif RMSD
Figure 1: Overview of the conditional generative modeling approach the motif-scaffolding problem. We train our new protein backbone diffusion model, ProtDi ff, to generate realistic protein backbone structures. Next, run SMCDiff, our conditional sampling algorithm, with ProtDiff to generate scaffolds (colored in red) conditioned on the motif (colored in blue). For self-consistency evaluation, we use pretrained exed-bbckbooe sequence-design model (ProteinMP. et al., 2022)) to generate the scaffold sequence from sampled backbone. We then input the sequence structure prediction model, in our case A AlphaFo o d2 (AF2) (Jumper et al., 2021), 1), to generate the full protein structure from the generated sequence. We compare the backhone of the predicted structure with the original backbone structure using TM-score (Xu Zhang, 2010) and root-mean- square-distance (RMSD) for the motif.
(Dauparas
al., 2022). Extending DPMs to protein structures, though, is non-trivial; since proteins are larger than small molecules, modeling proteins requires handling the sequential ordering of residues and long-range interactions. Finally, while existing models often generate distance matrices (Anand & Huang, 2018; Lin al., 2021), instead focus generating full set of 3DD coordinates, which should improve designability practice. Our resulting model, P ProtDiff ff, similar to concurrent work on E(3)-cquivariant diffusion models for molecules (Hoogeboom al., 2022), but with modifications specific to protein structure. Moreover, we develop novel motif-scaffolding procedure based on Sequential Monte Carlo, SMCDiff, that repurposes an unconditionally trained DPM for conditional sampling. We prove that if DPM matches the data distribution, SMCDi ff guaranteed provide exact conditional samples in compue limit; this property contrasts with previous methods (Song et al., 2021; Zhou et al., 2021). which we show introduce non-trivial approximation error that impedes performance. Our final motif-scaffolding generative framework, then, has two steps (Fig. 1): first train ProtDiff to learn distribution over protein backbones, and then we use SMCDiFf with ProtDiff to inpaint arbitrary motifs.
Ours is the first machine- learning method to construct scaffolds longer than 20 residues around motifs we build up to 80 residues scaffolds on test case. Beyond our progress on the sofif-scaffoldin problem, provide the following technical contributions: (1) introduce protein- backbbe generative model in 3D with the ability to generate backbone samples that structurally agree with alppaaoll d2 predictions, and (2) develop novel conditional sampling algorithm for inpainting.
2

---

Published as a conference paper at ICLR 2023
1.1 RELATED WORK
Motif-scaffolding. Past approaches have sought to scaffold a motif with native or prespecified protein fragments, but are limited finding a suitable match in the Protein Data Bank (PDB) and cannot adapt the scaffold to compensate for slight structural mismatches (Cao et al., 2022; Silva et al., 2016; Yang et al., 2021; Sesterhenn et al., 2020; Linsky et al., 2020). More recently Wang et al. (2022) used pre-trained protein structure prediction networks to recapitulate native scaffolds, but this method failed to generate scaffolds longer than 20 residues and can output only a single candidate scaffold rather than a diverse set. By contrast, our goal is to sample diverse, long scaffolds.
Diffusion models for molecule generation. Several concurrent works have extended equivariant diffusion models to molecule generation. Anand & Achim (2022) extended diffusion models for generation of protein backbone frames and sequences conditioned on secondary structure adjacency matrices. Similarly, Luo et al. (2022) focused on CDR-loop generation using diffusion models conditioned on non-CDR regions of the antihody-antigen. Our method does not require conditioning work
and is applicable to general proteins. Lee & Kim (2022) approach the same problem but build diffusion models over 2D distances matrices that requires post pest-prcessiig to produce 3D structures through Rosetta minimization. We demonstrate capability of diffusion models directly model 3D coordinates of proteins. Hoogeboom et al. (2022) developed an equivariant diffusion model (EDM) for generating small molecules in 3D. However, because EDM does not enforce a spatial ordering of the atoms that compose small molecules, we describe in Section 5, it does not learn coherent chain structure as needed in proteins.
Inpainting and conditional sampling in diffusion models. Point- Voxel Diffusion (PVD) (Zhou al., 2021) is 3D diffusion model for generating shapes from the ShapeNet dataset. Though trained to generate shapes unconditionally, PVD completes (or impaints) full shapes when partial point cloud is fixed during inference. For general diffusion models, Song et al. (2021) proposed an alternative inpainting approach and remarked that this approach produces approximate conditional samples. However, these methods do not provide theoretical guarantees, and when we compare them to SMCDi ff, we find that their approximation error impedes performance when applied motif- scaffolding. Saharia et al. (2021) developed an inpainting diffusion model by training a diffusion model to denoise randomly generated masked regions while unmasked regions were unperturbed. However, their approach requires detailed data augmentation strategy that does not exist for proteins.
We describe additional related work on protein generative models in Appendix A.
PRELIMINARIES
2.1
THE MOTIF-SCAHFOLDING PROBLEM
protein can be represented by its amino acid sequence and backbone structure. Let A be the set of 20 genetically-encodee amino acids. We denote the sequence of an N-residue protein by E A and its C-a backbone coordinates in 3D by x (xi...., X N,3 We describe a protein as having a fixed structure that is function of its sequence, so we may write x(s). We divide the N residues into the functional motif M and the scaffold S, such that MUS S = (1. .N. The goal is to identify, given the motif structure X.M sequences whose structure recapitulates the motif to high precision ((s) s XM- Appendix B discusses several caveats of this simplified framing (e.g. our assumption of static structures).
2.2 DIFFUSION PROBABILISTIC MODELS
Our approach the motif-scaffoldie problem builds on denoising diffusion probabilistic models (DPMs) (Soht-Dickstein et al., 2015). We follow the conventions and notation set by Ho et al. (2020), which we review here. DPMs are class of generative models based on reversible, discrete-time diffusion process. The forward process starts with sample (0) from unknown data distribution q. with density denoted by ()), and iteratively adds noise at each step I. By the last step, T. the distribution of xT) is indistinguishable from an isotropic Gaussian: xT) (x(((T); 0,
Specifically, we choose variance schedule (11), B(2) ..... ((T) and define the transition distribution step /. as ((x(() (((-1)) (x(1), V ((-1) ()
3

---

Published as a conference paper at ICLR 2023
DPMs approximate q with second distribution pe by learning the transition distribution of the reverse process at each i, px( x( We follow the conventions set by Ho et al. (2020)
in our parameterization and choice of objective. In particular, we take po (x((x11) xx())
gl VI-a0) co ex(x ())
(xx((=11; (ox(()))))))()))) px((x0) 8 and a(t) II a11), We implement cx(x("), t) as neural network. For training,
(1,,(1))),80))))))))))) (()) with po(x("), () -vao Vatu
a00)
marginally sample x(t) ~ g(x(1) 1)) from the forward process as x(()) (a( + - e
by stochastic optimization (Ho et al., 2020, Algorithm 1). To generate samples from po (x(0)), we simulate the reverse process. That is, we sample noise for time T x(T) (0, I), and then for each T L....,0, 0, we
and minimize the objective T ET= E g x0)) [[le ca (x("),
+)] (x((",)))'
simulate progressively "de-noised" samples as x px(x(t) (x(+1)).
PROTDIFF: A DIFFUSION MODEL OF PROTEIN BACKBONES IN 3D
Implementation of diffusion probabilistic models requires choosing an architecture for the neural network <x(x(1), 1) introduced abstractly in Section 2.2. In this section we describe ProtDi f f, which corresponds to the choice of (x(x)),, t) as translation and rotation equivariant graph neural network tailored to modeling protein backbones. We leave architectural and input encoding details to Appendix C.
The challenge of modeling points in 3D. The properties and functions of proteins are dictated by the relative geometry of their residues, and are invariant to the coordinate system chosen to encode them. Recent work neural network modeling of 3D data has found, both theoretically and empirically, that neural networks constrained to satisfy geometric invariances can provide inductive biases that improve generalization and training efficiency (Batzner et al., 2022). Motivated by this observation, we parameterize cg by an equivariant graph neural network (EGNN) (Satorras et al., 2021), which in 3D is equivariant to transformations in the Euclidean group. Xu et al. (2022) proved that if ca is equivariant to a group then po is invariant to the same group.
Tailoring EGNN to protein backbones. We now describe our EGNN implementation, which tailor to protein backbones and DPMs through the choice of edge and node features. To model every pairwise residue interaction, we represent backbones by a fully connected graph. Each node in the graph is indexed by n = I..... N, and corresponds to residue. We associate each node with coordinates X, R and D features hn E R... For each pair of nodes n, n' we define an edge and associate it with edge features. We construct our EGNN by stacking L equivariant graph convolutional layers (EGCL). , Each layer takes node coordinates and features as input, and outputs updated coordinates and features with the first layer taking initial values (x, h). We write the output of EGNN after I. layers as x = EGNN (x, ht). In the context of diffusion models, we predict the noise at time t with the following parameterization:
co c (9) = x ((), X -EGNN (9), ,A())--
We now describe our choice of node and edge features. Our choice is motivated by the linear chain structure of protein backbones; residues close in sequence are necessarily close in 3D space. To allow this chain constraint to be learned more easily, we fix an ordering of nodes in the graph to correspond to sequence order. We include as edge features positional offsets as done in Ingraham et al. (2019), which we represent using sinusoidal positional encoding features (Vaswani al., 2017). For node features, we similarly use sinusoidal encoding of sequence position well as of the diffusion time step é following Kingma et al. (2021). We additionally process the time encoding to be orthogonal to the positional encoding.
SMCD IFF: CONDITIONAL SAMPLING IN DIFFUSION MODELS BY PARTICLE FILTERING
The second stage of our generative modeling approach to the motif-scaffolding problem is to sample scaffolds (0) from (x( x (0)). Section 4.1 discusses the intractability of sampling from
4

---

Published as a conference paper at ICLR 2023
pa (
(0)
(0)
M exactly and the limitations of simple approximation introduced by (Song al.,
2021). In Section 4.2, we then frame computation of pa (Xg (xM) as sequential Monte Carlo (SMC) problem (Doucet et al., 2001) and approximate with particle filtering algorithm (Algorithm 1).
4.1 THE CHALLENGE OF CONDITIONAL SAMPLING AND THE ERROR OF THE REPLACEMENT METHOD
The conditional distributions of DPM are defined implicitly through the steps of the reverse process. We may write the conditional density explicitly as
T-1 II po(x! dx
(0) px xxx) x po(xs xxx) po (x) = po (T)
(0)
However, the high-dimensional integral on the right-hand side above is intractable (both analytically and numerically) compute.
To overcome this intractability, we build on the work of Song et al. (2021), who introduced practical algorithm that generates approximate conditional samples. This strategy is to (1) for- ward diffuse the conditioning variable to obtain (1:2)
Algorithm 1 SMCDiff: Particle filtering for con- ditionally sampling from unconditional diffusion models
1: Input: x XM (motif), K (# particles)
(0)
2: // Forward diffuse motif
XM
(1:2")
x M),, and then (2) for each
3: x XM q(x) ((x(1-) lxx)
(1:7)
t, sample Xg ~ po (xg) Xx .g x)+ We
(t)
(e)
(++1)
4:
call this approach the replacement method (fol- lowing Ho et al. (2022)) and make it explicit in Appendix Algorithm/ 2. However, in Proposi- tion D.1 we show that the replacement method introduces irreducible approximation error that cannot be eliminated by making po more expres- sive. Additionally, although this approximation error is not analytically tractable in general, we show in Corollary D.2 the dependence of this error the covariance of 10 and (0) in the (0)) is bivariate Gaussian.
5: // Reverse diffuse particles
(T) i.id. po(x(2)) 7: for = =T...., do
6: Vk: x
8:
1/ Replace motif
Ak, k + [x xSA
9:
10:
11:
// Re-weight based on x XMM
Vk, w (+))
(x po
vk, w + W / Σ5 x= Wk
(t)
13:
case that q(x10) M
(0)
14:
x ~ Resamp 1 e (10 1:K: x(:x)
15:
16:
I/ Propose next step
4.2 CONDITIONAL SAMPLING
17: Ak, X (t-1) indep.
(x(x x(())
IS A SEQUENTIAL MONTE CARLO PROBLEM
18:
end for
We next frame approximation of q(xs x XM) as a sequential Monte Carlo problem that we may solve by particle filtering. Intuitively, par-
(0) (0)
(0) 19: Return X5,1:K
ticle filtering addresses limitation of the replacement method: the failure at each time i to look beyond the current step to the less-noised motif X
(t-1) when sampling x5) ~ px
(t) I (+)))). Our
key insight is that because po (t-1) x())) provides mechanism to assess the likelihood of x (t-1)
we can prioritize noised scaffolds that are more consistent with the motif. Particle filtering leverages
this mechanism to provide a sequence of discrete approximations to each (x
(t)
(t-1:7)) 1:T)) that
s M
look ahead by this extra step. Finally, at f we have an approximation to po (x (0)
x
(0TT)
M
Then,
using Proposition 4.1 below, we can obtain an approximate sample from q(x5 (0) (0)
5M) ). This framing
permits the application of standard particle filtering algorithms (Doucet et al., 2001). Algorithm
summarizes an implementation of this procedure that uses residual resampling (Doucet & Johansen, 2009) to mitigate the collapse of the sequential approximations into point masses. SMCD i ff provides tunable trade-off between computational cost and statistical accuracy through the choice of the number of particles K. In our next proposition we make this trade-off explicit.
Proposition 4.1. Suppose that po exactly matches the forward diffusion process such that for every x(t+1), (x!! x((+1))
q(x(1) x(t+1) and consider any motif ( Let XS.K be particle
5

---

Published as a conference paper at ICLR 2023
chosen at random from the output of Algorithm with K particles. Then xs,K converges in distribution to q(x's (0) ( as K goes to infinity.
The significance of Proposition 4.1 is that it guarantees Algorithm can provide arbitrarily accu- rate conditional samples provided an accurate diffusion model and large enough compute budget (determined by the number of particles). To our knowledge, SMCDi ff the first algorithm for asympiotically exact conditionally sampling from unconditional DPMs. Our proof of the proposition, which we leave to Appendix D, is obtained from an application of standard asymptotics for particle filtering (Chopin & Papaspiliopoulos, 2020, Proposition 1.4)..
5 EXPERIMENTS
We empirically demonstrate the ability of our method to scaffold motifs and sample protein backbone structures. We describe our procedure for evaluating backhone designs in Section 5.1. We demonstrate the promise of our method for the motif-scaffolding problem in Section 5.2. And we investigate our method's s strengths and weaknesses via experiments unconditional sampling in Section 5.3. We train a single instance of P rotDiff ff and use it across all of our experiments. For simplicity, we limited our training data to single chain proteins taken from PDB that are no longer than 128 residues. See Appendix F for training details.
Baselines. As mentioned in Section 1.1, Wang al. (2022) is the only prior machine learning work to address the no stable implementation was available at the time of writing. The most closely related method for unconditional sampling with available software is trDesign (Anishchenko et al., 2021), but this method does not allow specification a motif. The ML method most similar to P ProtDi is the concurrently developed equivariant diffusion model (EDM) proposed by Hoogeboom al. (2022). Like P ProtDiff, EDM uses a densely connected EGNN architecture but without sequence-disted disttaece edge features. Consequently, it does not impose any sequence order, and therefore does not yield way to relate generated coordinates to a backbone chain.
problem. We do not compare against this as baseline because
5.1 In silico EVALUATION OF DESIGNED BACKBONES
While experimental validation via X- X-ray crystallography remains the gold standard for evaluating computationally designed proteins, recent work (Wang et al., 2022; Lin et al., 2021) has proposed to leverage highly accurate protein structure prediction neural networks as an in silico proxy for true structure. More specifically, Wang et al. (2022) jointly design protein sequence and structure, and validate by comparing the design and A Al phaaalo d2 (AF2) (Jumper et al., 2021) predicted structures. Here, our goal is to assess the quality of scaffolds generated independent of specific sequence, so we treat fixed backbone sequence design as a downstream step as in Lin et al. (2021).
Our evaluation with AF2 is as follows. For each generated scaffold we use C-a only version of ProteinMPNG (Dauparas et al., 2022) with temperature of 0.1 to sample 8 amino acid sequences likely to fold to the same backbone structure. We then run AF2 with the released CASP14! weights and 15 recycling iterations. We do not include a multiple sequence alignment as an input to AF2. Our choice of utilizing F Protiii i nMP NI and AF2 (without MSAs) is motivated by their empirical success in various de novo protein design tasks and the ability to recapitulate native proteins (Dauparas al., 2022: Bennett et al., 2022). To assess unconditionally sampled scaffolds, we then evaluate the agreement of our backbone sample with the AF2 predicted structures using the maximum TM- score (Zhang & Skolnick, 2005) across all generated sequences which we refer to as ScTM, for self-consistency TM-score. To assess whether prospective scaffolds generated support motif, we compute the root mean squared distances (RMSD) of the desired and predicted motif coordinates after alignment and refer this metric as the motif RMSD. Appendix Algorithm 4 outlines the exact steps.
Because TM-score > 0.5 indicates that two structures have the same fold (Zhang & Skolnick, 2005), we say that a backbone designable if SCTM > 0.5. The ability for AF2 to reproduce the same backbone from an independently designed sequence is evidence a sequence can be found for the
'Biannual protein folding competition where AF2 achieved first place. Weights available under Apache License 2.0 license.

---

Published as a conference paper at ICLR 2023
A.
B.
5trv
C.
6exz
Sampling methed
Scaffold
Motif
Morth RMSD (A)
fixed
SMCDIf (Ours) replacement
2
RMSD
RMSD
30-39
40-49
50-59
60-63
70-79
80-89
90-99
10-19
20-29
30-39
40-49
50.59
Staffold size (4 residues)
Figure 2: Motif-scaffolding case studies. (A) Example of two scaffold structures generated around a segment of 5trv. Orange: desired input motif, Grey: Alphafold-predicted structure of two scaffolds, with the motif highlighted (purple). Both scaffolds were sampled using SMCDi i ff with s sCTM > 0.5. (B,C) Motif RMSD for Strv and 6exz test cases, its dependence scaffold size, and comparison of SMCDi E f to two naive inpainting methods (f (fixes xed, repl acement)).
starting structure. To verify this is a reasonable cutoff, we analyzed ScTM over our training set and found 87% to be designable.
5.2 MOTIF-SCAFFOLDDIN VIA CONDITIONAL SAMPLING
We evaluated our scaff-scaffolding approach (combining SMCDi ff and ProtDi ff) on motifs extracted from existing proteins in the PDB and found that our approach can generate long and diverse scaffolds that support these motifs. We chose to first evaluate on motifs extracted from proteins present in the training set because we knew that least one stabilizing scaffold exists. We considered 2 examples taken from the PDB with IDs 6exz and Strv, which are 69 and 118 residues long, respectively. We chose these examples due to their high secondary structure composition while being representative of the shortest and longest lengths seen during training. For each structure, we chose 15-25 residue helical segment as the motif (see Appendix H for details). The remainder of each protein is one possible supporting scaffold. We sought to assess if we could recover this and other scaffolds with the same size and motif placement.
Based on prior work (Wang et al., 2022), we expected that building larger scaffolds around a motif would be more challenging than building smaller seaffolds. To assess this length dependence, we expanded the segment of used the motif when running SMCDi ff by including additional residues on each side. In each case, though, we compute the motif RMSD over the minimal motif. In Figure 2B, we present motif-scaffolding performance and its dependence the longer of the two test proteins. For the 5trv test case, the lower quartile of the motif RMSD for SMCD i E is below 1 for scaffolds up to 80 residues. Since 1 is atomic-level resolution, we conclude that our approach can succeed in this length range.
scaffold size for Strv,
Figure 2A provides a visualization of our method's 's capacity to generate long and diverse scaffolds. The figure depicts two dissimilar scaffolds of lengths 34 and 54 produced by SMCDi ff with 64 particles. Both scaffolds are designable and agree with AF2 (scTM> > 0.5). Diversity is particularly evident in the different orderings of secondary structures.
Figure 2B compares SMCDi ff to two naive inpainting methods, fixed and replacement. In fixed, the motif is fixed for every timestep i, and the reverse diffusion applied only to the scaffold (as done by Zhou et al. (2021)); replacement is the method described in Section 4. In contrast to SMCDi ff. these baselines fail to generate successful scaffolds longer than 50 residues on 5trv, as determined by the location of their lower quartiles.
We next applied these three inpainting methods to harder targets order to measure generalization to out-of difdirrtiibtion and more difficult motifs comprising of dis-contiguous regions and loops. We consider a motif obtained from the respiratory syncytial virus (RSV) protein and calcium binding EF-hand motif, both of which are not in the training dataset. RSV is known to be difficult due to its composition of helical, loop, and sheet segments, while EF-hand is a dis-contiguous loop motif found in calcium binding protein. More details about both motifs can be found in Wang et al. (2022); there the authors report the only known successful scaffold of these motifs but they attain it with computationally intensive hallucination approach. We found that our method failed to generate
7

---

Published as a conference paper at ICLR 2023
A.
(50 as) Long (?1-120 aa)
B.
handec helix
C.
No left-handed helix
Hiighest cMac to trainina set
1.0
6c59
0.8
Noenalize
0.4
0.2
0.4
0.6
1.0
0.4
0.6 Highest SCTM
0.8 1.0
Highest SCTM
Figure 3: Protein backbone samples from ProtDiff. (A) Density plot of SCTM for different length categories (50-70, 70-128). The dashed line at scTM = 0.5 indicates the threshold of "designability" points to the right are considered "designable" (see text). (B) Scatter plot of scTM and the highest TM-score of each sample to all of PDB. Points represented (invalid) left- oeft-handed helix. Dashed lines indicate thresholds s cTM = 0.5. (C) Example of designable backbone sample (rainbow) with ScTM 0.5 (boxed in red in panel B) to its closest PDB example (6c59, grey) with a TM-score 0.54.
grey "x" are detected to contain an
scaffolds predicted recapitulate the motif (Appendíx H); however, SMCDiff provided smaller median motif RMSDs than the other two inpainting methods.
Compute cost. The computation of SMCDiff ff with 64 particles is approximately 2 minutes per inde- pendent sample, while alternative methods f xed and rep acement can produce 64 independent samples in the same time. By contrast, the hallucination approach of Wang al. (2022) involves running a Markov chain for thousands of steps, and has runtime on the order of hours for a single sample (Anishchenko al., 2021).
5.3 UNCONDITIONAL SAMPLING
We next investigate the origins of the diversity seen in Figure by analyzing the diversity and designability of ProtDif ff samples without conditioning on motif.
We first check that ProtDiff produces designable backbones. To do this, we generated 10 backbone samples for each length between 50 and 128 and then calculated scTM for each sample. In Fig. 3A, we find that 1.8% of samples have SCTM 0.5. However, the majority of backbones do not pass this threshold. We also observe designability has strong dependence on length since expect that longer proteins are harder to model in 3D and design sequences for. We separated the lengths below 128 residues into two categories and refer to them short (50-70) and long (70-128), Our results in Figure 3A indicate 17% of designs in the short category are designable vs. 9% in the long category. In Figure 13, present structural clustering of these designable backbones; we find that these backbones exhibit diverse topologes.
We next sought evaluate the ability of ProtDiff to generalize beyond the training set and produce novel backbones. In Figure 3B each point represents backbone sample from ProtDi ff. The horizontal coordinate of point is the ScTM, and the vertical coordinate the minimum TM-score across the training set. We found strong positive correlation between ScTM and this minimum TM-score, indicating that many of the most designable backbones generated by ProtDiff were result of training set memorization. However, if the model were only memorizing the training set, we would see TM-scores consistently near 1.0; the range of scores in Figure 3B indicate this not the case and the model is introducing degree of variability. Figure 3C gives example of backbone with SCTM > 0.5 that appears to be novel. Its closest match the PDB has TM-score = 0.54.
Fig. 3B illustrates limitation of our method: many of our sampled backbones are not designable. One contributing factor is that ProtDiff does not handle chirality. Hence ProtDiff generates backbones with the wrong handedness, which cannot be realized by any sequence. Fig. 3B shows that 45% of all backbone samples had at least one incorrect, left-handed helix. Of these, most have SCTM < 0.5. We describe calculating left-handed helices in Appendix G.
8

---

Published as a conference paper at ICLR 2023
0.00
0.60
0.70
0.78
0.84
0.91
1.00
0.00
0.11
0.17
a
Figure 4: Interpolations between ProtDiff samples demonstrating the diversity of backbones cap- tured. Top: 64-residue example. Bottom: 56-residue example. ProtDiff samples are determined by the Gaussian noise across all steps, (0:2),
Fig. 4 illustrates an interpolation between two samples, showing how ProtDiff's outputs change as function of the noise used to generate them. To generate these interpolations, we pick two backbone samples that result in different folds. For independent samples generated with noise (0:T) and g(0:3) we interpolate with noise set to Vive + v1 (o((0:2) for between O and 1. The depicted values of a are chosen to highlight transition points with full interpolations included in Appendix H.3. future direction is to exploit the latent structure of ProtDiff f to control backbone topology.
DISCUSSION
The motif-scaffolding problem has applications ranging from medicine to material science (King al., 2012), but remains unsolved for many functional motif'. We have created the first generative modeling approach to motif-scatie motif-scaffolding by developing P ProtDiff, diffusion probabilistic model of protein backbones, and SMCDiff, a procedure for generating scaffolds conditioned on motif. Although our experiments were limited to a small set of proteins, our results demonstrate that our procedure is the first capable of generating diverse scaffolds longer than 20 residues with computation time reliably on the order of minutes or less. Our work demonstrates the potential of machine learning methods be applied in realistic protein design settings.
General conditional sampling. SMCDi ff is applicable to generic DPMs and is not limited to only proteins and -caff-scaffolding. While we not make claims SMCDiff outperforming fate--of-the-art conditional diffusion models other tasks such as image generation, a clear advantage of SMCDiF ff over the replacement method on a toy task of inpainting MNIST images in Appendix I. Extending SMCDifE outside of motif-scaffolding is outside the scope of the present work, but the advantages of a single model for both unconditional and conditional generation warrants additional research.
demonstrate
Modeling limitations. Our present results do not indicate our procedure can generalize to motifs that are not present in the training set. We believe improvements protein modeling could provide better inductive biases for generalization. ProtDi ff, based on EGNN, is reflection equivariant since it only sees pairwise distances between 3D C-a coordinates. Additionally, ProtDiff does not explicitly model primary sequence or side-chains. Hoogeboom et al. (2022) demonstrate the benefits modeling sequence information in small molecules; joint modeling sequence and structure in single model could improve the designability of protein scaffolds and backbones as well.
Data limitations. We remarked our training set is small due to filtering based on length and oligometry (using only monomeric proteins). Scaling up to longer proteins opens up thousands more examples from the PDB, but in preliminary experiments has proven challenging. Lastly, further development and comparison of methods for motif scaffolding will benefit from standard evaluation benchmarks. Developing benchmark proved to be difficult since motifs are not labeled in protein databases. It will be important to gather motifs of biological importance in order to guide ML method development towards real-world applications. Because no such benchmarks exist, developing them is valuable direction for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The authors thank Octavian-Bugen Ganea, Hannes Stärk, Wenxian Shi, Felix Faltings, Jeremy Wohlwend, Nitan Shalon, Gabriele Corso, Sean Murphy, Wengong Jin, Bowen Jing, Renato Berlinghieri, John Yang, and Jue Wang for helpful discussion and feedback. We thank Justas Dauparas for access to an early version of P Prote eetMMNNN We dedicate this work in memory of Octavian-Fugen Ganea who initiated the project by connecting all the authors.
BLT and JY were supported in part by an NSF-GRFP. BLT and TB were supported in part by NSF grant 2029016 and an ONR Early Career Grant. JY, RB, and TJ acknowledge support from NSF Expeditions grant (award 1918839; Collaborative Research: Understanding the World Through Code), Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, the Abdul Latif' Jameel Clinie for Machine Learning in Health, the DTRA Discovery of Medical Countermeasures Against New and Emerging (DOMANE) threats program, the DARPA Accelerated Molecular Discovery program and the Sanofi Computational Antibody Design grant. DT and DB were supported with funds provided by a gift from Microsoft. DB was additionally supported by the Audacious Project the Institute for Protein Design, the Open Philanthropy Project Improving Protein Design Fund, an Alfred P. Sloan Foundation Matter-to-Life Program Grant (G-2021-1689)) and the Howard Hughes Medical Institute.
REFERENCES
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv: 2022.
Namrata Anand and Possu Huang. Generative modeling for protein structures. In Advances in Neural Information Processing Systems, volume 31, 2018.
Christian B Anfinsen. Principles that govern the folding of protein chains. Science, 181(4096): 223-230, 1973.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang, Asim K Bera, Frank DiMaio, Lauren Carter, Cameron M Chow, Gactano T Montelione, and David Baker. De novo protein design by deep network hallucination. Nature, 600(7889):547-552 2021.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kombluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-vquivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communicationss 13, 2022.
Nathaniel Bennett, Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Po Peng, Justas Dauparas, Minkyung Back, Lance Stewart, Frank DiMaio, Steven De Munck, Savvas N. Savvides, and David Baker. Improving de novo protein binder design with deep learning. bioRxiv, 2022.
Longxing Cao. Brian Coventry, Inna Goreshnik, Buwei Huang, Joon Sung Park, Kevin M Jude, Iva Marković, Rameshwar U Kadam, Koen H G Verschueren, Kenneth Verstraete, Scott Thomas Rus- sell Walsh, Nathaniel Bennett, Ashish Phal, Aerin Yang, Lisa Kozodoy, Michelle De Lora Picton, Lauren Miller, Eva-Maria Strauch, Nicholas D) DeBouver, Allison Pires, Asim K Bera, Samer Halabiya, Bradley Hammerson, Wei Yang. Steffen Bernard, Lance Stewart, Ian A Wilson, Hannele mhohola-Bakese Joseph Schlessinger, Sangwon Lee, Savvas N Savvides, K Christopher Garcia, and David Baker. Design of protein binding proteins from target structure alone. Nature, 605(7910):551 2022.
Nicolas Chopin. Central limit theorem for sequential Monte Carlo methods and its application to Bayesian inference. The Annals of Statistics, 3236))2355-2111, 2004.
Nicolas Chopin and Omiros Papaspiliopoulos. An Introduction to Sequential Monte Carlo. Springer, 2020.
Bruno E Correia, John T Bates, Rebecca J Loomis, Gretchen Baneyx, Chris Carrico, Joseph G Jardine, Peter Rupert, Colin Correnti, Oleksandr Kalyuzhniy, Vinayak Vittal, Mary J Connell, Eric
10

---

Published as a conference paper at ICLR 2023
Stevens, Alexandria Schroeter, Man Chen, Skye Macpherson, Andreia M Serra, Yumiko Adachi, Margaret A Holmes, Yuxing Li, Rachel E Klevit, Barney Graham, Richard Wyatt, David Baker, Roland K Strong, James E Crowe, Jr. Philip R Jobnson, and William R Schief. Proof of principle for epitope-focused vaccine design. Nature, 507(7491):201-22 2014.
Dauparas, I. Anishchenko, N. Bennett, H. Bai, J. Ragotte. L. F. Milles, B. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. Y. Leung, T. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, Kang, B. Sankaran, A. K. Bera, N. King, and D. Baker. Robust deep-learning based protein sequence design using ProteinMPNN. Science, 378(6615):49-66 2022.
Arnaud Doucet and Adam M Johansen. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of Nonlinear Filtering, 12(656-704):3, 2009.
Arnaud Doucet, Nando De Freitas, and Neil James Gordon. Sequential Monte Carlo Methods in Practice, volume 1. Springer, 2001.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nature Communications, 12(4348), 2022.
Sarel J Fleishman, Andrew Leaver-Fay. Jacob E Corn, Eva-Maria Strauch, Sagar D Khare, Nobuyasu Koga, Justin Ashworth, Paul Murphy, Florian Richter, Gordon Lemmon, Jens Meiler, and David Baker. Rosettascripts: a scripting language interface to the Rosetta macromolecular modeling suite. PLOS ONE, 6(6)ee20161, 2011.
James S Fraser, Henry van den Bedem, Avi J Samelson. P Therese Lang, James M Holton, Nathaniel Echols, and Tom Alber. Accessing protein conformational ensembles using room-temperaturr X-ray crystallography. Proceedings of the National Academy of Sciences, 108(39):1224--12522 2011.
Hannah Gelman and Martin Gruebele. Fast protein folding kinetics. Quarterly Reviews of Biophysics, 47(2):95 142, 2014.
Alex Herbert and MJE Sterbberg. MaxCluster: tool for protein structure comparison and clustering. 2008.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Deep Generative Models for Highly Structured Data Workshop, ICLR, volume 10, 2022.
Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D). In International Conference on Machine Learning, 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International Conference on Machine Learning. 2022.
Bin Huang, Yang Xu, Xiuhong Hu,, Yongrui Liu, Shanhui Liao, Jiahai Zhang, Chengdong Huang, Jingjun Hong, Quan Chen, and Haiyan Liu. A backbone-centred energy function of neural networks for protein design. Nature, 602(7897):523-522
2022.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In Advances in Neural Information Processing Systems, volume 32, 2019.
Lin Jiang, Eric Althoff, Fernando R Clemente, Lindsey Doyle, Daniela Rothlisberger, Alexandre Zanghellini, Jasmine L Gallaher, Jamie L Betker, Fujie Tanaka, Carlos F Barbas III, Donald Hilvert, Kendal Houk, Barry L. Stoddard, and David Baker. De novo computational design of retro-aldol enzymes. Science, 31 319(5868) 2008

---

Published as a conference paper at ICLR 2023
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Bergham- mer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(787))5583 589, 2021.
Neil P King, William Sheffler, Michael R Sawaya, Breanna S Vollmar, John P Sumida, Ingemar André, Tamir Gonen, Todd O Yeates, and David Baker. Computational design of self assembingg protein nanomaterials with atomic level accuracy. Science, 336(6085):1111-11777
2012.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In
Advances in Neural Information Processing Systems, volume 34, 2021.
Jin Sub Lee and Philip M Kim. ProteinSGM: Score-based generative modeling for de novo protein design. bioRxiv, 2022.
Zeming Lin, Tom Sercu, Yann LeCun, and Alexander Rives. Deep generative models create new and diverse protein structures. In Machine Learning for Structural Biology Workshop, NeurlPS, 2021.
Thomas W Linsky, Renan Vergara, Nuria Codina, Jorgen W Nelson, Matthew J Walker, Wen Su, Christopher O Barnes, Tien- Ying Hsiang, Katharina Esser-Nobis, Kevin Yu, Z. Beau Reneer, Yixuan J Hou, Tanu Priya, Masaya Mitsumoto, Avery Pong, Uland Lau, Marsha I. Mason, Jerry Chen, Alex Chen, Tania Berrocal, Hong Peng, Nicole S Clairmont, Javier Castellanos, Yu-Ru Lin, Anna Josephson-Day Ralph Baric, Deborah H Fuller, Carl D Walkey, Ted M Ross, Ryan Swanson, Pamela J Bjorkman, Michael Gale, Luis M Stanca-MMeja Hui-Ling Yen, and Daniel- Adriano Silva. De novo design of potent and resilient hACE2 decoys to neutralize SARS-CoV-2. Science, 370(6521): 1200882)))22.-.: 1214-1214 2020.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specific antibody design and optimization with diffusion-based generative models. In Advances Neural Information Processing Systems, 2022.
Matt McPartlon, Ben Lai, and Jinbo Xu. deep SE())-equivariant model for learning inverse protein folding. bioRxiv, 2022.
Erik Procko, Geoffrey Berguig, Betty W Shen, Yifan Song, Shani Frayo, Anthony J Conver- tine, Daciana Margineantu, Garrett Booth, Bruno E Correia, Yuanhua Cheng, William R Schief, David M Hockenbery, Oliver W Press, Barry L Stoddard, Patrick S Stayton, and David Baker. A computationally designed inhibitor of an Epstein- Barr viral BCL-2 protein induces apoptosis in infected cells. Cell, 157(7):1644-1666 2014.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho. Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. arXiv preprint arXiv:2/ 1.05826, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E(n) equivariant graph neural networks. In International Conference on Machine Learning, 2021.
Fabian Sesterhenn, Che Yang, Jaume Bonet, Johannes T Cramer, Xiaolin Wen, Yimeng Wang, Chi-I Chiang, Luciano A Abriata, Iga Kucharska, Giacomo Castoro, Sabrina S Vollers, Marie Galloux, Elie Dheilly, Stéphane Rosset, Patricia Corthésy, Sandrine Georgeon, Mélanie Villard, Charles- Adrien Richard, Delphyne Descamps. Teresa Delgado. Elisa Oricchio. Marie-Anne Rameix Welti, Vicente Más, Sean Ervin, Jean- François Eléouêt, Sabine Riffault, John Bates, Jean- PhilPpilippe Julien, Yuxing Li, Theodore Jardetzky, Thomas Krey, and Bruno E Correia. De novo protein design enables the precise induction of Rn--nuutra antibodics. Science 368(6492), 2020.
Justin B Siegel, Alexandre Zanghellini, Helena M Lovick, Gert Kiss, Abigail R Lambert, Jennifer L StClair, Jasmine L Gallaher, Donald Hilvert, Michael H Gelb, Barry L Stoddard, Kendall N Houk, Forrest E Michael, and David Baker. Computational design of an enzyme catalyst for a stereoselective bimolecular Aldes reaction. Science, 329 2010.
12

---

Published as a conference paper at ICLR 2023
Daniel-Adriano Silva, Bruno E Correia, and Erik Procko. Motif-driven design of Protein-protenn interfaces. Computational Design of Ligand Binding Proteins, pp. 285-304. 2016.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021.
Doug Tischer, Sidney Lisanza, Jue Wang. Runze Dong, Ivan Anishchenko, Lukas F Milles, Sergey Ovchinnikov, and David Baker. Design of proteins presenting discontinuous functional sites using deep learning. bioRxiv, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jue Wang, Sidney Lisanza, David Juergens, Doug Tischer, Joseph L.. Watson, Karla M. Castro, Robert Ragotte, Amijai Saragovi, Lukas F. Milles, Minkyung Back, Ivan Anishchenko, Wei Yang. Derrick R. Hicks, Mare Expòsit, Thomas Schlichthaerle, Jung-Ho Chun, Justas Dauparas, Nathaniel Bennett, Basile I. M. Wicky, Andrew Muenks, Frank DiMaio, Bruno Correia, Sergey Ovchinnikov, and David Baker. Scaffolding protein functional sites using deep learning. Science, 377(6604):387-394, 2022.
Jiaxiang Wu, Shitong Luo, Tao Shen, Haidong Lan, Sheng Wang, and Junzhou Huang. EBM- Fold: fulle-differentiable protein folding powered by energy-based models. arXiv preprint arXiv:210S.00771, 2021.
Peng Xiong, Xiuhong Hu, Bin Huang, Jiahai Zhang, Quan Chen, and Haiyan Liu, Increasing the efficiency and accuracy of the ABACUS protein sequence design method. Bioinformatics, 36(1): 136-144, 2020.
Jinrui Xu and Yang Zhang. How significant is a protein structure similarity with TM-score= 0.5?
Bioinformatics, 26(7):889-895, 2010.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation. In International Conference Learning
Representations, 2022.
Che Yang. Fabian Sesterhenn, Jaume Bonet, Eva A van Aalen, Leo Scheller, Luciano A Abriata, Johannes T Cramer, Xiaolin Wen, Stéphane Rosset, Sandrine Georgeon, Theodore Jardetzky, Thomas Krey, Martin Fussenegger, Maarten Merkx, and Bruno E. Correia. Bottom-up de novo design of functional proteins with complex structural features. Nature Chemical Biology, 17(4): 492-500, 2021.
Yang Zhang and Jeffrey Skolnick. TM-align: a protein structure alignment algorithm based on the TM-score. Nucleic Acids Research, 33(7):2322-239, 2005.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. In International Conference Computer Vision, pp. 5826-5835, 2021.
13

---

Published as a conference paper at ICLR 2023
Part I
Appendix
Table of Contents
A Additional related work
14
B Problem assumptions and modeling heuristics
14
C Additional ProtDiff details
15
D Conditional sampling: SMCDiFE details and supplementary proofs
16
D.1 The replacement method and its emorr
17
D.2 SMCDi fi details and verification proof of Proposition 4.1
18
D.3 Proofs and lemmas
19
E Detecting chirality
22
F Training details
23
Additional metric details
23
Additional experimental results
24
Additional motif-scaffolding results
24
H.2 Qualitative analysis of scTM different ranges ................................
26
Additional latent interpolation results
28
H.4 Structural clustering
29
Applicability SMCDiff beyond proteins: MNIST inpainting
29
A ADDITIONAL RELATED WORK
We cover additional related work on generative models of proteins sequence and structure, beyond the discussion in Section 1.1. Following the success of deep language models, Ferruz et al. (2022) developed protein sequence models to generate new proteins, but these models do not allow specification of structural motifs. Another class of methods, referred to as fixed backbone sequence design (Fleishman et al., 201 I; Ingraham et al., 2019; Xiong et al., 2020; McPartlon et al., 2022; Hsu et al., 2022), attempts to solve the problem of identifying sequence that folds into any given designable backbone structure.
the present work, we utilize a particular sequence design method, Proteinkh NN (Dauparas et al., 2022), but in principle any other fixed-backbone sequence design method could be used in its place. Anand Huang (2018); Lin et al. (2021); Wu et al. (2021) propose generative adversarial networks, variational autoencoders, and energy based models, respectively, distance matrices, but these approaches (1) do not generate backbones compatible with a specified motif and (2) rely on unwieldy optimization step translate the distance matrix into backbone coordinates. Other authors use neural net (Tischer et al., 2020; Anishchenko et al., 2021; Wang et al., 2022; Huang et al., 2022; Wu al., 2021), but require computationally challenging conformational landscape exploration.
B PROBLEM ASSUMPTIONS AND MODELING HEURISTICS
The formulation of the motif scaffoldff problem presented in Section 2.1 makes several simplifying assumptions, and our modeling approach relies on several heuristics. We deseribe these assump- tions and heuristics in what follows, and comment how they might be addressed by further
14

---

Published as a conference paper at ICLR 2023
methodological developments. But first describe illustrative example of an instance of motif scaffolding.
Protein sequence-structurr relationship. Generally speaking. protein's sequence encodes an ensemble of conformations, populated to different degrees at biological temperatures. Anfinsen's hypothesis states that the ground state conformation is thermodynamically accessible (Anfinsen, 1973), providing a mapping from sequence to a unique (ground state) structure. In practice, the ground state structures make up the vast majority of experimentally determined protein conformations, as over 95% of structures in the Protein Data Bank (PDB) are collected at cryogenic temperatures (Fraser al., 2011). Thus we simplify our problem by saying that a sequence uniquely maps to a static structure (i.e. the ground state structure). However, violations of this assumption arise in some PDB structures as result of (1) of context specific determinants of structure such as post-raanslatoonl modifications and environmental factors including pH, binding partners, and salts, as well as (2) thermodynamic inaccessibility of the ground state.
Motif sequence and side-chains. As stated in Section 2. we assume we may represent a functional motif by the coordinates its C-a atoms. However, the biochemical functions proteins depend not only on backbone structure, but also on side-chains. For example, the activity of many enzymes is imparted by triplets of residues, known as catalytic triads, whose ability to catalyze reactions depends on the spatial organization of side-chain atoms. Our problem statement and subsequent evaluation scheme are agnostic to the amino acid identity of motif residues, let alone side-chain positioning. A more complete representation of motif would include the side-chain identities (i.e. the amino acid sequence) and side-chain atom coordinates.
Scaffold length and motif placement. We have additionally assumed that the size of scaffolds and the indices of motif residues within the backbone chain, .M, are known a priori. However, in practice satisfactory scaffolds could have different lengths and different motif placements, and typically is not known a priori what lengths and placements will be best. Previous works have addressed this challenge through brute foree by sampling multiple lengths and placements, and relied on post-hoc filtering to identify the most promising scaffolds (Wang et al., 2022; Yang et al., 2021). Subsequent work on MI. methods could potentially generalize beyond this assumption efficiently sample appropriate scaffold lengths and motif placements.
Sequence and side-chain modeling. ProtDiff models only the backbone coordinates and leaves sequence design to subsequent stage, for which we have used Prote ProteinMPNN nMPNN NN. A more complete representation a proteins could include both sequence and structure (where structure can be divided separately
into the backbone and side-chain atom coordinates). To model sequence, we rely trained neural network, Prote: NN, but this is not ideal. Unless ProtDi f produces perfect backbones, one would expect the backbone samples of ProtDi ff present substantial domain shift when used as input for P Protes PooteinMi iiMMPNN.
3D backbone representation. In this work, we represent protein structure using the C-& coor- dinates of every residue along the backbone. However, this representation is coarse-grained and ignores additional backbone atomic coordinates, namely the backbone carbon and nitrogen atoms. Dauparas et al. (2022) observed additionally modeling the heavy atoms of the backbone nitrogen and carbon atoms along with the C-8 of every residue (to capture side-chain information) improved performance (by sequence recovery) for fixed-backbon sequence design. We hypothesize modeling additional coordinates of every residue would also improve designability performance of Prota rotDi Dii ff. Constraining ProtDiff to place the remaining atoms in the correct orientation could help enforce correct chirality and mitigate chain breaks.
C ADDITIONAL PROTDIFF DETAILS
As reminder from Section 3, each node in the graph is indexed by I,,..,N and corresponds to residue with coordinates X,, E R and node features h, E R., For each pair of nodes n, n' we define an edge and associate with edge features Own' E R.. Our neural network predict ep is an instance of EGNN composed of multiple EGCL layers We recount details of EGCL and then discuss construction of edge and node features, ann' and hn.
15

---

Published as a conference paper at ICLR 2023
Equivariant graph convolution layers (EGCL). Each layer 1..... defines update as (x,h) = EGCL/x ",rp"",h'11
[| where for each node n and N1, = On( mm)) for dnnt: ann')
x = =xx=
1 + Σ Wnn' Ox (x 111
x X Vdnw
Σ (h M " 1 ,nnn' ann)) and dwnn = ||x² 1 x 112.
m
6es Oh, and Øx are fully connected neural networks, and y is small positive constant included for numerical stability. The first EGCL layer takes in initial node embeddings, ht while edge embeddings, ann's are kept fixed throughout.
Initial node and edge embeddings. Each edge between two residues indexed in the sequence by (n,, is featurized with D features obtained through a sinusoidal encoding of its relative offset:
(n 1)
where ((x,k))) k.) = sin (x Ve.T/NN T/N2k/D)
k mod 2 = 0
Onn' : ppnn n', D)
cos (z T/ //N2 7/N2((-1//))
A mod 2 1.
For node features, we similarly use sinusoidal encoding of sequence position as well as of the diffusion time step f
cp(n, 1)
(t, 1) + R
hn (t)= =
(nnn D)
(ot D)
where R is a Dx orthogonal matrix chosen uniformly at random. Intuitively, applying R transforms the time encoding be orthogonal the positional encoding.
Coordinate scaling While protein structures are typically parameterized in Angstroms, we transform the input protein coordinates to be in nanometers rather by dividing by 10. This scaling brings the backbones to spatial scale similar to the reference distribution at which the forward noising process stationary, unit variance isotropic Gaussian. Importantly, the distribution of the final step T is indistinguishable from an isotropic Gaussian (Supplementary Fig. 5.)
Empirical distribution
cov(X
0.40
x67
0.35
0.8
N1O. 1)
0.6
Density
0.4
0.20
0.15
x50
x
x²
x5
0.00
Coordinate value
Figure 5: Distribution of (T) after centering and scaling (0) to nanometers.
D CONDITIONAL SAMPLING: SMCDIFF DETAILS AND SUPPLEMENTARY PROOFS
We here provide additional details related to SMCDiff and the replacement method described in Section 4. Details of the replacement method (Song al., 2021) and analysis of its emror are in
16

---

Published as a conference paper at ICLR 2023
Algorithm 2 Replacement method for approx-
imate conditional sampling
Input: x (motif)
2: // Forward diffuse motif (1:4)
(1:1)
g ((x
3: x XM
4:
5: // Reverse diffuse scaffold
6: x(T) (x(x(T))
7: for t 1==1 _T...., 1....o0. do
8:
// Replace with forward diffused motif
9:
x(1) | x
10:
11:
1/ Propose next step
x(t11) ~ po (x((t=11) x()))
12:
13: end for
14: Return x5S (1:T))
Appendix D.I. Appendix D.2 provides details of our sampling method, SMCDiff, including (1) proof of Proposition 4.1 and (2) details of the residual resampling step. We leave technical proofs and lemmas to Appendix D.3.
Notation. In the following, we require notation that is more precise than in previous sections. For each 0, ...... T, we let (-) and P-(-) denote the density functions of x()) according to the forward process and to our neural network approximation of the reverse process, respectively. We denote densities restricted to the motif and scaffold with subscripts M and S. For example, we here write
PM.((x (0) whereas we wrote (x (9) in the main text. We write (random) conditional densities as (t-1) x XM (t-1)) and write the (deterministic) conditional density for observation x x T.M as
a. XM
(t-1) Z.M).
An object of interest will be the Kullback-Leibler (KL) divergence. We write 2((r)
KL (-))))(())) = f (x)) log ps(a) dr, where log(-) is the natural (base e) logarithm. We will also encounter the expected KL between conditional densities, which we will write as EKL (- xx(-))))lp(( ( x((-D) == I (=- (q(( xxt-1) = x)]lp( x(+-1) = x²) dz, where the outer expectation is taken with respect to the unconditional density associated with first argument of EKT.[-II.]
D.1 THE REPLACEMENT METHOD AND ERROR
The replacement method proposed by Song al. (2021) for the task of inpainting in the context of score-based generative models. Work (Ho et al., 2022) concurrent with the present paper applied the replacement method to DPMs. Although Song al. (2021) notes that this approach can be understood as approximate conditional sampling, they provide no discussion of approximation error. We here show that the replacement method introduces irreducible error that inherent to the forward process. Algorithm 2 provides an explicit description of the replacement method.
The first return of Algorithm 2, x is used putative inpainting solution approximate conditional sample. But Algorithm additionally returns subsequent time steps, (1:T), We denote the approximation over all steps implied by the generative procedure in Algorithm by P (:
(0) XM Z.M) and compare it to the exact conditional, 9-:1(- xX 2,M).. We here consider error in KL divergence because it permits an analytically tractable and transparent analysis. We additionally consider the idealized scenario where Po:r(.) perfectly captures the reverse process. Under this condition, the forward KI. takes a surprisingly simple form.
17

---

Published as a conference paper at ICLR 2023
Proposition D.1. Suppose that pu:r (.) exactly matches the forward diffusion process such that for every 2, Pc(- [x(++) x) =c((- (+++1) z). Then for any motif IM,
KT. (1:(( XM = (m)llpi.T
Reple xxxx> = =xxi)]
EKL
(9,a(. x((+1))
(0) .M =.Millas.a /
Proposition D.1 reveals that the replacement method introduces approximation error that is intrinsic to the forward process and cannot be eliminated by making Po:r (-) more expressive. Although the individual terms in the right hand side of Equation (2) are not analytically tractable in general, in the following corollary we show that this approximation error can be non-trivial by considering a special case. For this following example, we depart from the earlier assumption that x is in 3D, and consider scalar valued X and Xs.
Corollary D.2. Suppose [x Mi X5 ( is bivariate normal distributed with mean zero, unit variance, and covariance p. Further suppose that qs,e(- x (i: Va@xgg a) and 98,41((; N(: V 3(141)xg (1) g(++)) as in Section 2, where ((t-1) and (1) are between O and 1.
(1)
X5
Then
EKL (gs.e(
(t+1) XSS
2 xxlllass x ( 2 -
1 (
a0 (²))
We note two takeaways of Corollary D.2. First, as we might intuitively expect, this error can be large when significant correlation in the target distribution present. Second, we see that the approximation error can be larger at earlier time steps, when (+) closer to
D.2 SMCDIFF DETAILS AND VERIFICATION PROOF OF PROPOSITION 4.1
The idea behind the Of procedure in Algorithm 1 is to break sampling of x (0) into three stages:
gs.o( x
(1:T) 1. Draw x M 4M,1T( xx))
(1T)) qs.IT( XM (0:T)).
2. Draw X
3. Draw X gs XM ,xs
(0:T) (1:T))
If all three steps were performed exactly, by the law of total probability x in step (3) would (marginally) be an exact sample from qs,o(- xx))). As such, SMCDif2 ff aims to perform step (1) and approximate steps (2) and (3). Step (1) corresponds to forward diffusing the motif lines 2-3 and is exact because we diffuse according to q-
Step (3) corresponds to line 17 in the last iteration (when = 1). Specifically, to sample from (. X .x8
(U:2), (1:T) we make three observations.
(i) The Markov structure of the forward process XM xx"). (ii) By the assumption that the forward
implies that q-.0(- XM .xS (1:T)) = 95.0(- x
(0:T)
(0:1) xx ())). and approximated reverse process agree, we have gs,o(- | x XM x()) ps.ol- XM
(iii) Finally, because pe(- | x(++1) factorizes across M and S for each f, ps,o( (0:1) (1)
ps.o(- x XM.X x5
(1)) As a result, under the assumptions of the proposition, we may sample from (0:7)
(1)
(.
XM x (1TT)), and perform step (3) exactly well.
Step (2) is the only non-trivial step, and cannot be performed exactly. The challenge is that although
the reverse process approximation, Ps..:r(: (- x (0:T') is well-defined, computing it explicitly involves
an intractable, high-dimensional integral.
The sequential Monte Carlo approach of SMCDiFE, then, is to circumvent this intractability by constructing a sequence of approximations. For each T,T - I,..., ....,,,, 1, we approximate ps.c(: XM ((-1:1) (and thereby (: x((-1:T)) with K weighted atoms (the particles). We denote these
18

---

Published as a conference paper at ICLR 2023
Algorithm 3 Residual Resample
1: Input: wi:K (weights), X1:K (particles)
2: Vk, (Ck, [ik) ) + (LK wk Kwk
K Kwv))
3: XC x1,...,,X]
,K....,Xx Xx
4: R t K Ek=
k= C Multinomial (ri:k: R)
5: [1,,.... ixl
6: XR f [Xi.................. Xin]
7: X concat(xx, Xc)
8: Return x
0(::x5,)), where each approximations (which are implicit in Algorithm 1) by ()(() := Ek (i))
w and Stk are as in Algorithm 1, and 8(;;x) denotes a Dirac mass at x. In particular, (() is an approximation to ps.. xM²)) Proving the proposition amounts to showing that in the limit as K goes to infinity, each (-) converges weakly to Ps,1(- x (Q:T)) which by assumption is equal (- x XM (0rT)) This weak convergence follows from standard asymptotics for particle filters (Chopin & Papaspiliopoulos, 2020, Proposition 11,4), which we make explicit in Lemma D. As (1)
result, if we perform step (3) with x3 P (-), then this lemma implies that x converges in
(0)
distribution to qs o (xx {xM), since (i) gs gs,o(xs X xs
(0)
(1) is continuous in xg and (ii) X is
independent of x conditional on x())
Recall that to show the proposition, it was to sufficient to show that K converged weakly to (- x XM (0:T)); ); this implied that the K particle returned by Algorithm would then converge in distribution 9s,o(- x)))) which, by the law of total probability, implied that they marginally converge to qs,o(- xx) However, while the particles return by Algorithm I may be treated as exchangeable, they are not independent, because they depend on shared randomness in x (1:T), To obtain approximate samples that are independent, it is necessary to run Algorithm multiple times.
Residual resampling. Line of Algorithm I indicates a Resample step. In particle filtering, resampling steps (or branching mechanisms (Doucet et al., 2001, Chapter 2)) filter out particles with very small weights, and replace them with additional copies of particles with large weights. Notably, the resampling step is the only point of departure of Algorithm 1 from the replacement method: without resampling, the algorithms behave identically. While a variety of possible branching mechanisms exist, we use residual resampling (Algorithm 3) in our implementation for its simplicity.
D.3 PROOFS AND LEMMAS
Particle filtering lemma with technical conditions
(1) Lemma D.1. Consider Pll)) = -k ūgō(=; (1)) where wy and x' are as constructed in Algorithm 1. Assume the conditions of Proposition 4.1. Then P(.) converges weakly to ps. x (0:T)
as K goes infinity. That is, for any Borel measurable A, limg-x P.P (A) PA | (0:T) x M
)dr.
Proof. The proof of the lemma follows from application of standard asymptotics for particle filtering (Chopin & Papaspiliopoulos, 2020, Proposition 11.4). In particular, to apply Proposition 11.4 we use the formalism of Kay (FK) models, following the notation of (Chopin & Papaspiliopoulos, 2020, Chapter 5). Though typically (and in (Chopin & Papaspiliopoulos, 2020) FK models are defined via sequence of approximations at increasing time steps, we consider decreasing time steps because we are approximating the reverse time process, We take the initial distribution
as Mir (x
PS.T(x (x (I') the transition kernel as Mx ((+1)
,x5 xx())
(x.
xl++))), and
19

---

Published as a conference paper at ICLR 2023
the potential functions (x5x p correspond to
(8-1) xx()).
The sequence of FK models, Qt, then
Qx(x5 ( = L, "M (x(()) (T
II Mi(x)*) 1
, (xx))
for each I, where L, is a normalizing constant.
By substituting in choices of M, and Gt, we rewrite and simplify Q
Q1(xs (x)))) L, (x5 (T) (T-1) ()) (pM..s-)MM..MMMMM
II
ps, (x |x pM
xx(1)
T=
L. (x (T)
P )p (x((1 |x(1))
(x.((((11) xxM
(1T) a ps.er(xs xx (t-1:))).
(I:T)
where lines and 4 drop multiplicative constants that do not depend on X5 (t:T), From the above derivation, that each (x() (t)) (x. XM ( and in particular that (x('))
Ps. (x
XM (0:7)) ). As such, the desired convergence in the statement of the lemma is equivalent to that P ((]) converges to Q1.
Chopin & Papaspiliopoulos (2020, Proposition 11.4) provide this result for the generic particle filtering algorithm (see Chopin Papaspiliopoulos (2020, Algorithm 10.1), which is written in the FK model form described above). More specifically, Proposition 11.4 proves almost sure convergence of all Borel measurable functions of p((t) which implies the desired weak convergence.
Although the proof provided in Chopin & Papaspiliopoulos (2020) is restricted to the simpler, but higher variance, case where the resampling step uses multinomial resampling. the authors note that Chopin (2004) proves it holds in the case of residual resampling (which we use in our experiments) as well.
Replacement method error lemmas and proofs We here provide proof's of Proposition D.I and Corollary D.2.
Proof of Proposition D.1:
Proof. The result obtains from recognizing where the replacement method approximation agrees with the forward process, using conditional independences in both processes, and applying the chain rule for KI. divergences. We make this explicit in the derivation below, with comments explaining
20

---

Published as a conference paper at ICLR 2023
the transition to the following line.
KL ( xM = zmillP1 Repl( xx = mmns
= =ʃ 1ur (1:11 (
(x:((((1:T)
10 = I.M)
xxX = zm) log
dz (1:2)
PIT (x(1::))
Repl
XM -MM
// By the chain rule of probability.
(Ir (1:21
x (0) FM
- OuT (x(
(1:))
xX mmm log
2M.11T
Pi.
(aMM
(1:2))
x M
(0)
a.M
log (T (1:T)
x (0:T)
(0:T))
M
Repl (1:T)
(0:T)
()))
PT
x XM
IM
// By the agreement of q and PRopl on the motif, and the chain rule of probability.
- -f 1 -far((afm)) (1:1))
XM = xmm)
(0)
log gs.r(15 (T)
x M
(0:T) z 2((77)
PT (5's XMM
Repl (T)
(0:T)
x (1:7))
T-1
Σ log (5. (t)
(1+1)
F. (1+1) (0:1) x M
(0:T))
x5
M
Ps,t
Repl
(a's
X5 = J.S x M
((+1)
(1+1) (U:T)
= x ()
// Because (.) = PS.T Repl()
(-) =N(;;, I) and the assumption that pe matches q.
= =/ / (1:T)
T-1
(5.)(=))) x((+1) = (1+1)) x (0) = x
XM = xMM
Σ
log
dx (1:T)
(5,0(15)) (t) |x(++1) = x((+1))
==1
T-1
EKL gs,e(. |x(++1) x =x (l |x(((t))
t=1
Proof of Corollary D.2:
The proof of the corollary relies of on lemma on the variances of the two relevant conditional distributions. We state this lemma, whose proof is at the end of the section, before continuing. For notational simplicity, we drop the scripts and annotations on (t)) and g(++1), and instead write a and B3 respectively.
Lemma D.2. Suppose x XM* X5 and x are distributed as in Corollary D.2. Then Var[x (t)
(0) (t)
(++1)
(++1)] 3 and Var(x' xs
(++1) [0)] s B(1 3p)
x
Now provide proof of Corollary D.2.
Proof. First recall that
1
033
of
+ (Wi - 122 1)
KL [N(H,,
o3)]
log
+
2
o
o
1
log
-
2
21

---

Published as a conference paper at ICLR 2023
and observe that this lower bound is monotonically decreasing in of for of S oï. Therefore
EKT.
Xs
(1+1)
x 10)
(.)lls..e 1xxx5 x
emo (0)
(x
(1+11) 1x |x>i
KI [ss.el.
(t+1) (1+1) (0)
as XM = (x(+t) 1g =
x5
de
M
(0) (1+1)
/ (MMo=MMass (1+1) xx1a
KI. [WO,, Varix xs = =xg (1+1) (00) = (M) Var(x! x (1+1) = ==s =
((+1)
de de ((+ (e+1)
M
2 KL [V(0, 3(1 -Bp
3]]
3
8(1 -Bp -1)
log
B(1 -Bp²aa)
83
1 (log(1- - -3p²a) + 3ppa)
where the second inequality follows from Lemma D.2, and the monotonicity of the KI. in of-
Proof of Lemma D.2:
Proof. That Varlx (t)
x5 1 follows immediately from that [x]] ,
is marginally
bivariate normal distributed with covariance /I V1-B. B.
The upper bound on Var (t)
x
(1+1),
xx0i is trickier. Observer that [x,',
X5
(1+1)
X
bivariate
Gaussian and that
(0) x
1 -a Pa)
31- 1 p²a))
Var
M
3(1 -11110 31-301-pm)
1 ++3p - ppax
As such, the conditional variance may be computed closed form as Var(x (1) (1 (1 (1-))
((+1) (0)
X5 .xM
/²a)/(1 p²a + 8p²a)) But since (1 (i-paa/(1-p))))
3p²a)= 2 11 11-(8p3))((1-pp) and therefore (1- p²a) (1- (11pppppp p²a 3p²a) < (8p²a)) (1-
we can write
Var(x (t)
(1+1 xs
x = - -aa- -Paa (1 (1- Pa + 3p
3
1 1- -²a
Bi²a
$ (0) (1- p²a))
1 p²a
3(1 2a) + (11 8)8paa
3311 3p²a).
E DETECTING CHIRALITY
Section 6 noted the limitation of ProtDiff that it can generate left-handed helices (which do not stably occur in natural proteins). Figure 6 presents two such examples. We additionally note that, as in Figure 6 Left, model samples include multiple helices with differing chirality.
22

---

Published as a conference paper at ICLR 2023
Figure 6: Two examples of protein backbone samples with incorrect left handed helices.
F TRAINING DETAILS
ProtDiff uses equivariant graph convolutional layers (EGCL) with 256 dimensions for node and edge embeddings. The training data was restricted to single chain proteins (monomers) found in PDB and lengths in the range [40, 128]. We additionally filtered out PDB with >5Á atomic resolution. This amounted 4269 training examples. Training was performed using the Adam optimizer with e=le44, $1 0.9, and Bn = 0.999. We trained for 1,000,000 steps using batch size 16. We used a single Nvidia A100 GPU for approximately 24 hours. We implemented all models in PyTorch. We used the same linear noise schedule as Ho et al. (2020) where 3n- 0.0001, 3T 0.02, and T 1024. We did not perform hyperparameter tuning.
hyperparameters Learning
G ADDITIONAL METRIC DETAILS
Self-consistency algorithm. Section 5.1 described our self-consistency metrics for evaluating the designability backbones generated with ProtDiff. Algorithm 4 makes explicit the procedure use for computing these metries.
Algorithm Self-consistency calculation
Input: x E R N.3
1: for E .I,...,88 8
2: 8; f- ProteinMPMM)))
3: X; + AF2(ss,
4: end for
5: sc_tm <- maxiel.... (Mscore(kx
Output: sc_tm
Using dihedral angles to calculate helix chirality. Natural proteins are chiral molecules that contain only right-handed alpha helices. However, because the underlying EGNN our model is equivariant to reflection, it can produce samples with left-handed helices. While examining model samples, we additionally observed samples with both left and right- hight-handed helices (Figure 6), though in theory the EGNN should be able to detect and avoid the chiral mismatch. Left- handed helices are fundamentally invalid geometries in proteins and represent a trivial failure mode when calculating the self-consistency and other metrics. Samples with mixture of left and right-
23

---

Published as a conference paper at ICLR 2023
handed helices are especially problematic because they cannot be corrected simply by reflecting the coordinates. As such, it is important to identify and separate samples with mixed chirality.
To detect chiralty, compute the dihedral angle between four consecutive C-a atoms chiral metric to distinguish between the two helix chiralities. Algorithmically. for every C-a i, we calculate the dihedral between C-a 1, 1+1, 1+2, and 1+3. C-a with dihedral angles between 0.6 and 1.2 radians are classified as right-handed helices, and angles between -1.2 and -0.6 are classified left-handed helices, with everything else classified as non-helical. Because C-a atoms in native helices tend to form contiguous stretches longer than one residue in the primary sequence, helical stretches less than one amino acid were removed. This filtering meant to help avoid accidentally counting the occasional isolated backbone geometry that falls into helical bin as a true helix. Finally, for all C-a atoms that are still categorized as part of helix, the associated i i+1, 1+2 and C-a atoms are also counted as part of that helix.
H ADDITIONAL EXPERIMENTAL RESULTS
In this section, we describe additional results to complement the main text. We provide a description of the motif targets in Section 4, along with results of scaffolding failure case Appendix H.I. To understand the qualitative outcomes of scTM, we present additional results of backbone designs, their AF2 prediction, and most closely related PDB parent chain for different thresholds of ScTM in Appendix H.2. We provide additional examples of latent interpolations in Appendix H.3. Finally, Appendix H.4 presents a structural clustering unconditional backbone samples; this result provides further evidence of ProtDiff Dif''s ability to generate diverse backbone structures.
6exz
5trv
Figure 7: Structures used for motif-scaffolding test cases. Native structures (grey) and their motif's (orange) that were used for the motif-scaffolding task are shown.
H.1 ADDITIONAL. MOTIF-SCAFFOL.DE RESULTS
We here provide additional details of the motif-scaffoldingg experiments described in Section 5. Table 1 specifies the total lengths, motif sizes, and motif indices of our test cases. In Figure we depict the structures of the native proteins (6exz and 5trv) from which the motifs examined quantitatively in the main text were extracted. Figure analyzes commonly observed failure modes of P rotDiff backbone samples involving chain breaks, steric clashes, and incorrect chirality.
Figure presents quantitative results on harder inpainting target. this case, the motif is defined residues 163-181 of chain A of respiratory syncytial virus (RSV) protein (PDB ID: 5tpn). We
24

---

Published as a conference paper at ICLR 2023
attempted to scaffold this motif into 62 residue protein, with the motif residues 42-68 22-62. We chose this placement because previous work (Wang et al., 2022) identified a promising candidate scaffold with this motif placement. In contrast the cases described in the main text, for which a suitable scaffold exists in the training set, SMCD i ff and the other inpainting methods failed to identify scaffolds that recapitulated this motif to within a motif RMSD Å.
Table I: Motif-scaffolding test case additional details.
Origin/ Protein
Total length
Motif size (residue range)
6exz.
72
15 (30-44)
5trv
118
(42 62)
RSV (PDB-ID: Stpn)
62
19 (1)-34)
EF EF- hand (PDB-ID:
53
5 (0-4), 13 (31-43)
IPRW)
A.
B.
Figure 8: Failure modes in ProtDiff backbone samples. (A) Backbone clashes and chain breaks. The C-a atoms can be spaced further than the typical 3.8 between neighbors, resulting in chain break (dashed lines). Additionally, backbone segments can be too close to each other, resulting in obvious overlaps and clashes. (B) Backbones with mixture of left (circled in red) and right (circled in green) handed helices. These chirality emrors cannot be corrected simply by mirroring the sampled backbone.
25

---

Published as a conference paper at ICLR 2023
RSV
fixed
SMCDiff (Ours)
replacement
Motif RMSD (A)
4
20-29
30-39
40-44
Scaffold Size (# residues)
EF-hand
fixed
4.0
SMCDiff (Ours)
replacement
3.5
Matif RMSD (A)
3.0
2.5
1.5
10-19
20-29
30-39
Scaffold size (# residues)
Figure 9: Additional inpainting results more challenging motif extracted from the respiratory syncytial virus (RSV) and EH-hand motif. The three inpainting methods are evaluated as described in Section 5.
H.2 QUALITATIVE ANALYSIS OF SCTM DIFFERENT RANGES
In this section, we give intuition for backbone designs and AF2 predictions associated with different values of s scTM to aid the interpretation of the ScTM results provided in Section 5. Figure 10 examines a possible categorization of scTM three ranges. The first two rows correspond to backbone designs that achieve s scTM 0.9. We see the backbone designs in the first column closely match the AF2 prediction in the second column. closely related PDB example can be found when doing similarity search of the highest PDB chain with the highest TM-score to the AF2 prediction. We showed in Figure 3B that SCIM > 0.9 is indicative close structural match being found PDB.
The middle two rows correspond to designs that achieve SCTM 0.5. These are examples of backbone designs on the edge of what we deemed as designable (sCTM > 0.5). In these cases, the AF2 prediction shares the same coarse shape the backbone design but possibly with different secondary -ttuudtr- ordering and composition. the length 69 example, we see the closest PDB chain has T'M-soore of only 0.65 to the AF2 prediction but roughly the secondary-structure
26

---

Published as a conference paper at ICLR 2023
ordering the backbone design. The length 100 sample is similar of AF2 producing roughly similar shape to the backbone design, but has no matching monomer in PDB.
The final category of ScTM 0.25 reflects failure cases when SCTM is low. The AF2 predictions in this case have many disordered regions and bear little structural similarity with the original backbone design. Similar PDB chains not found. We expect that improved generative models of protein backbones would not produce any samples this category.
Backbone design
AF2 prediction
PDB parent
ScTM > 0.9
Length 55
acTM-0991
PDB ID=lqkx. TMscore=0095
Length
aCTM-0.91
ID= TMscore=0.99
SCTM 0.5
Length
aCIM-00
ID-7c78,
Length 100
PDB ID-4h3, TMccore-0.56
SCTM < 0.25
Length
ECTT=00255
Length 120
SCTM-024
Figure 10: Qualitative analysis of unconditional backbone samples from ProtDiff. The first column displays backbone designs from ProtDiff f and their sequence lengths. The second column displays the highest scTM scoring AF2 predictions from the Protein NMPNN sequences of the corresponding backbone design to the AF2 prediction in the second column with the PDB ID and TM-score written below. The third column is blank for the last two rows since no PDB match could found. See Appendix H.2 for discussion.
first column. The third column displays the closest PDB chain
27

---

Published as a conference paper at ICLR 2023
H.3 ADDITIONAL LATENT INTERPOLATION RESULTS
We here provide additional latent interpolations. Figures 11 and 12 depict interpolations for between model samples for lengths 89 and 63, respectively.
2 - 0.000
0.057
a- 0.133
0.200
0.267
0.333
0.400
0.467
=
0.533
=
0.600
0.667
a= 0.733
0.800
0.857
0.933
1.000
Figure 11: Latent interpolation of length 89 backbone sample from 0 1.
0.000
0.067
0.133
0.200
0.267
D.333
0.430
0. 0.467
0.533
0.500
0.657
0.733
0.800
D
0.933
1.000
Figure 12: Latent interpolation of length 63 backhone sample from 0 to 1.
28

---

Published as a conference paper at ICLR 2023
H.4 STRUCTURAL CLUSTERING
All 92 samples with scTM 0.5 were compared and clustered using MaxCluster Herbert & Sternberg (2008). Structures were compared in sequence independent manner, using the TM-score of the maximal subset of paired residues. They were subsequently clustered using hierarchical clustering with average linkage, 1 TM-score as the distance metric and TM-score threshold of 0.5 (Figure 13 A).
-1.0
-0.0
Figure 13: Clustering of self-consistent ProtDiff samples. The distance matrix is 1 TM-score between pairs of samples, and ranges from 0 (exact matach) to 1 (no match). Dendrograms hicrarchical clustering using the average distance metric. Designs on the right are cluster centroids. Gray lines connect larger clusters with more than member to its centroid, while the remaining designs are from a random selection of the remaining single-sample clusters. Protein backbones are colored from blue the N-terminus to red at the C-terminus.
from
I APPLICABILITY OF SMCDIFF BEYOND PROTEINS: MNIST INPAINTING
by applying to inpainting on the MNIST digits dataset. We compare SMCDi ff with the replacement method on the task of sampling the remaining half of MNIST digits. We first train DDPM with
Our goal in this section is to study the applicability of SMCDiff beyond
31 = 10 10-4 Br = 0.2, T 1000 using a small 8-layer CNN on MNIST with batch size 128 and ADAM optimizer for 100 epochs until it is able generate reasonable MNIST samples (Figure 14). We then selected 3 random MNIST images and occluded the right half. The left half would then serve the conditioning information to the diffusion model (Figure 15).
29

---

Published as a conference paper at ICLR 2023
7
25
4
4
4
5
0
7
6
7 17
C
7
4
70
7
1
7
7
Figure 14: Unconditional MNIST
Figure 13:
Full MNIST images and their occluded halves used for inpainting experiments.
samples.
For each occluded image, we fixed single forward trajectory and sampled 16 images from each method: replacement method and SMCDiff with 16 or 64 particles (K). Results are shown Fig. 16. We observe the replacement method can sometimes produce coherent samples continuation of the conditioning information, but more often it attempts to produce incoherent digits. SMCDiff on the other hand tends to produce digits that compliment the conditioning information. For more difficult occlusions, such as 5 and O, SMCD ff can still fail although increasing the number of particles (K 64) tends produce samples that are more visually coherent.
It is important to note SMCDiff has additional computation overhead based on the number of particles. be more expensive than replacement method but result higher quality samples. Investigating SMCDi ff in more difficult datasets with improved architectures is a direction of future research.
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
1
7
7
7
7
7
7
7
7
7
7
7
7
7
3
7
7
7
7
7
7
7
7
B
V
3 K
4
5
5
5 3
5
5
F
D
5
T
3
5
5
5
5
5
5
5
8
5
5
34 5
R 5
B
E
5
55
9
5
4
5
3
5
5
3
5
9
5
5
5
5
6
6
O
0
0
0
O
C
0
6
Conditioning
Replacement
SMCDiff (K=16)
SMCDiff (K=64)
Figure 16: MNIST inpainting results for replacement and SMCD i ff. See text for explanation.
30


Reviews:
[{"review_id": "nDSeZTt1TXk", "reviewer": "Reviewer_eV4U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a E(3)-equivariant diffusion probabilistic model for both unconditional and conditional sampling of protein backbones, which can be used for the motif-scaffolding problem. Particularly, the conditional sampling is implemented as a particle filtering algorithm, where scaffolds that are more consistent with the motif are prioritized. Authors demonstrate that the proposed method generated valid scaffolds for the given motif, as verified by AlphaFold2’s predicted structures. Experiments on unconditional sampling indicate that this model captures the underlying distribution of protein backbones.", "summary_of_the_review": "The DDPM-based conditional sampling formulation of motif-scaffolding problem is novel. Authors propose ProtDiff and SMCDiff for both unconditional and conditional sampling of protein backbones. The validness of some assumptions (Cons 1 & 2) may need further discussion.", "strength_and_weaknesses": "Pros:\n1. The formulation of motif-scaffolding problem as a conditional sampling process is novel, and authors propose a single diffusion model that can be used for both unconditional and conditional sampling to realize such motif-based scaffold design.\n2. The re-weighting scheme in the particle filtering algorithm is based on the likelihood assessment of the diffusion model, which can better guide the conditional sampling process towards the promising direction.\n3. Clear discussion on the modeling and data limitation of the proposed method.\n\nCons:\n1. During the training of ProtDiff, the diffusion model of protein backbones, 3D coordinates are represented in nanometers, rather than Angstrom, so that the spatial scale is similar to an isotropic Gaussian with unit variance. I am not sure whether such scaling is sufficient to ensure that x^{T} at the last step is indistinguishable with an isotropic Gaussian, which is one of the fundamental assumptions in DDPM.\n2. In Section 4.2, authors state that the proposed SMCDiff can provide arbitrary accuracy for conditional sampling, as guaranteed by Proposition 4.1. However, such proposition is based on the assumption that p_theta **exactly** matches the forward diffusion process, which might be too strong and even unrealistic.\n3. In Section 5.1, authors mention that when prediction protein structure with AF2, no multiple sequence alignment (MSA) is used. However, as shown in several single-sequence PLM-based protein structure prediction papers, AF2 w/o MSA input works inferior to such PLM-based structure prediction, possible due to the gap between training and inference inputs. Authors may need to consider feed MSA into AF2 for more reliable predictions, or directly use PLM-based methods (e.g., ESM-Fold, OmegaFold and HelixFold-Single) for in silico evaluation.\n4. In Section 6, authors state that only using 3D backbone coordinates as inputs to ProtDiff is the main cause that ProtDiff is reflection symmetric and may generate left-handed helices. However, I think the EGCL-based network architecture may also be one of root causes (even the biggest one), since EGCL only takes pairwise distance as spatial information, which is reflection invariant. In other words, ProtDiff cannot distinguish one structure and its mirrored counterpart by design, and this won’t be resolved even full-atom 3D coordinates are taken as inputs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dUgOrxVlMX", "reviewer": "Reviewer_EPhs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied protein generation problem, and specifically, motif-scaffolding conditional protein design problem. This paper first proposed a general protein 3D structure generation framework, which can generate high-quality protein backbones in 3D. Then the author proposes a new sampling algorithm for inpainting problem, which means part of the data is masked while the model aims to recover the missing part. Analytical experiments demonstrate the effectiveness of the proposed method for both unconditional and conditional protein design.", "summary_of_the_review": "The paper aims to solve the protein motif-scaffolding problem. The author mainly proposed ProtDiff and SMCDiff methods. ProtDiff is for protein generation in 3D, which is similar to previous EDM model for molecule generation. SMCDiff is a new proposed conditional sampling method. Experiments are well designed and sound, showing the model can generate reasonable and realistic structures.\n\nQuestions:\n1. I think from the description of SMCDiff sampling, it is a general sampling algorithm that is not only for proteins. Then, whether you can also show the effectiveness of SMCDiff sampling for general data? I understand running experiments for large-scale images costs too much time, but maybe you can test it on some synthetic data?\n2. Besides, since this is a conditional generation task, I'm a little confused why you only define the problem as an \"inpainting\" problem instead of a conditional generation problem. In other words, for this conditional generation task, you actually don't need to learn the joint distribution and just do conditional sampling. Maybe I misunderstand the available dataset or your tackled scenario, but in my mind in the learning procedure, you can also just learn a conditional model, which may make more sense.\n3. By the way, for the provided PDF on my PC, the hyper references (e.g., citations) don't work correctly. Clicking on the reference will always be directed to the first page, making it difficult for me to check related work or equations. Please double-check this.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well organized. The domain-specific motif-scaffolding problem is clearly explained, and the method is easy to follow.\n2. The paper nicely provides technical contributions from both biology and ML perspectives.\n3. The paper conducts careful experiments to demonstrate the effectiveness of the proposed method.\n\nWeakness:\n\n1. The topic is a little bit narrow, which is even a subarea of general protein generation. Therefore, the paper may not be attractive to ML conferences.\n2. The protein generation framework (ProtDiff) itself is similar to existing EDM models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JhgeQHfpfD", "reviewer": "Reviewer_BXA4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors translate diffusion models to the protein structure domain to design new protein scaffolds and motifs.\n", "summary_of_the_review": "While I think it is an interesting approach and great area of exploration, more work needs to be done to justify it's utility.", "strength_and_weaknesses": "The diffusion approach seems interesting and a promising way to explore sequence and structure space.\n\nSection 5.1 - Two works are cited to justify using AlphaFold2 as the gold standard of accuracy of those structures. But for de novo scaffold and motif design, as you propose here, has this been validated? To what extent? Shouldn’t this be done first to make sure you can trust those predictions? Minimally, how accurate is it without a multiple sequence alignment?\n\nSection 5.2 - Why PDBs 6exz and 5trv?\n\nIn Figure 3A, it looks like the vast majority of scTMs is around 0.25, which is random. If you are using this software during inference, how do you know that your scTM is > 0.5, so it is “designable? \n\nIn Figure 4, it looks like chains are able to pass through one another in space during sampling. This does not happen for proteins in vitro or in vivo. Does your model allow for chains to pass through one another?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9O2s2HPA0on", "reviewer": "Reviewer_KTe1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For many protein design applications, the user specifies a motif, and then needs to build a scaffold of additional amino acids that will act to stabilize the motif. This paper proposes an interesting technique for generating scaffolds conditional on motifs. First, a 3D diffusion model of protein structures is fit. Then, conditional 'inpainting' is performed to sample the 3D configuration of the scaffold given the motif using a particle filtering technique. Then, an fixed structure -> sequence model from prior work is used to produce a protein sequence. Given the accuracy of AlphaFold, the overall system can be evaluated without performing prohibitively expensive wet-lab experiments: simply check whether the AlphaFold-predicted structure of the output sequence contains the desired motif.  ", "summary_of_the_review": "Overall, I really like this paper. I think the method is really interesting and it is presented well. However, there are clear concerns around the experiments, as they mostly focus on a couple of specific case studies and there are clear shortcomings of the current method (e.g. ability to capture chirality). Below are some questions mostly to clarify why more comprehensive experiments would be challenging.\n\n=Scale of experiments=\nI don't fully understand why sec 5.2 is only on a couple of example proteins. What is the limiting factor? Is it that we don't know good motifs to scaffold? That other natural targets for scaffolding have protein sequences that are too long? Were you compute bound? \n\n=Length of proteins=\nI understand that scalability was a key concern, and thus you trained on sequences of length < 128. Was this a computational or modeling issue? Do you have to actually model every single amino acid in order to capture the gist of the 3d structure? I wonder if the diffusion model could be defined over a coarser-grained version of the structure.\n\n=Types of motifs=\nFig 2a doesn't seem that challenging of a motif. Is it just a single helix? Is the motif consecutive in the primary sequence? For enzymes, for example, I'd imagine that the motif (the active site) would correspond to non-consecutive residues.\n\n=Reliance on ProteinMPNN=\nSurely there are realizable structures that ProteinMPNN can't produce a sequence for. It would be good to know if this is a bottleneck in your system. Do you know what the self-consistency rate of ProteinMPNN is in general? What fraction of PDB structures are such that structure -> sequence -> predicted_structure has high similarity between structure and predicted_structure?\n\n=Minor comments=\nIt would be helpful in the intro to provide a bit more rationale for why prior work only considered 20 residues. It was hard for me to understand why the current work doesn't scale.\n\n\"A protein’s structure is a function of its sequence, so we may write x(s).\" You should acknowledge at some point in the paper that the notion of a single static structure for a sequence is fraught.\n\nI appreciate the negative result at the bottom of page 7!\n", "strength_and_weaknesses": "=Strengths=\nThe paper is very well grounded in current literature. \nThe paper addresses a high-impact application.\nThe paper provides a clean use of sequential Monte Carlo (SMC) that will be of independent interest to the community.\n\n=Weaknesses=\nUsing SMC for inpainting with diffusion models is a generic idea that could have high impact, for example, to computer graphics research. However, I worry that that target audience won't read this paper because it is quite focused on proteins. If the paper was more methods oriented 'hey, we have a generic SMC method for diffusion models…\", it could perhaps have higher impact.\n\nThe experiments are careful and well-thought out, but mostly focus on a couple of small-scale case studies. I don't quite understand why the analysis of the conditional sampling (aka motif scaffolding) couldn't be done at a larger scale (see below). This will hurt the impact of this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster