Protein structure generation via folding diffusion
Authors: Kevin Eric Wu
Kevin K Yang
Rianne van den Berg
James Zou
Alex Xijie Lu
Ava P Amini
Venue: Submitted to ICLR 2023 (2023)
Keywords: Generative modeling of protein backbone structures
structural biology
diffusion
diffusion modeling
generative modeling
proteins
internal coordinates

Abstract: The ability to computationally generate novel yet physically foldable protein structures could lead to new biological discoveries and new treatments targeting yet incurable diseases. Despite recent advances in protein structure prediction, directly generating diverse, novel protein structures from neural networks remains difficult. In this work, we present a new diffusion-based generative model that designs protein backbone structures via a procedure that mirrors the native folding process. We describe protein backbone structure as a series of consecutive angles capturing the relative orientation of the constituent amino acid residues, and generate new structures by denoising from a random, unfolded state towards a stable folded structure. Not only does this mirror how proteins biologically twist into energetically favorable conformations, the inherent shift and rotational invariance of this representation crucially alleviates the need for complex equivariant networks. We train a denoising diffusion probabilistic model with a simple transformer backbone and demonstrate that our resulting model unconditionally generates highly realistic protein structures with complexity and structural patterns akin to those of naturally-occurring proteins. As a useful resource, we release the first open-source codebase and trained models for protein structure diffusion.

Under review as a conference paper at ICLR 2023
PROTEIN STRUCTURE GENERATION VIA FOLDING DIF- FUSION
Anonymous authors Paper under double-blind review
ABSTRACT
The ability computationally generate novel yet physically foldable protein structures could lead to new biological discoveries and new treatments targeting yet incurable diseases. Despite recent advances in protein structure prediction, di- rectly generating diverse, novel protein structures from neural networks remains difficult. In this work, we present a new diffusion-based generative model that de- signs protein backbone structures via a procedure that mirrors the native folding process. We describe protein backbone structure as series of consecutive angles capturing the relative orientation of the constituent amino acid residues, and gen- erate new structures by denoising from a random, unfolded state towards a stable folded structure. Not only does this mirror how proteins biologically twist into ergetically favorable conformations, the inherent shift and rotational invariance this representation crucially alleviates the need for complex equivariant networks. We train a denoising diffusion probabilistic model with simple transformer back- bone and demonstrate that our resulting model unconditionally generates highly realistic protein structures with complexity and structural patterns akin to those of proteins. As useful resource, we release the first open-source codebase and trained models for protein structure diffusion.
INTRODUCTION
Proteins are critical for life, playing a role in almost every biological process, from relaying signals across neurons (Zhou al. 2017) recognizing microscopic invaders and subsequently activating the immune response (Mariuzza et al. 1987), from producing energy for cells (Bonora et al. 2012) to transporting molecules along cellular highways (Dominguez & Holmes] 2011). Misbehaving pro- teins, on the other hand, cause some of the most challenging ailments in human healthcare, includ- ing Alzheimer's disease, Parkinson's disease, Huntington's disease, and cystic fibrosis (Chaudhuri & Paul 2006). Due to their ability to perform complex functions with high specificity, proteins have been extensively studied as a therapeutic medium (Leader al. 2008 Kamionka, 2011 Dim- itrov 2012) and constitute rapidly growing segment of approved therapies (H Tobin et al.) 2014). Thus, the ability to computationally generate novel yet physically foldable protein structures could open the door to discovering novel ways harness cellular pathways and eventually lead to new treatments targeting yet incurable diseases.
Many works have tackled the problem of computationally generating new protein structures, but have generally run into challenges with creating diverse yet realistic folds. Traditional approaches typically apply heuristics assemble fragments of experimentally profiled proteins into structures (Schenkelberg & Bystroff [0166] Holm & Sander 199T). This approach is limited by the boundaries of expert knowledge and available data, More recently, deep generative models have been proposed. However, due to the incredibly complex structure of proteins, these commonly do not directly gen- erate protein structures, but rather constraints (such as pairwise distance between residues) that are heavily posteprocessed to obtain structures (Anand et al. 2019 Lee & Kim, 2022). Not only does this add complexity to the design pipeline, but noise in these predicted constraints can also be com- pounded during post-processing, resulting in unrealistic structures that is, if the constraints are at all satisfiable to begin with. Other generative models rely on complex equivariant network ar- chitectures loss functions to learn generate 3D point cloud that describes protein structure (Anand & Achim 2022 Trippe et al., 2022) Luo et al., [022] Eguchi et al.) (2022.. Such equivariant architectures ensure that the probability density from which the protein structures are sampled is

---

Under review as a conference paper at ICLR 2023
invariant under translation and rotation. However, translation- and rotation- equiiiriiiiiv architectures are often also symmetric under reflection, leading to violations of fundamental structural proper- ties of proteins like chirality (Trippe et al. 22022). Intuitively, this point cloud formulation is also quite detached from how proteins biologically fold - by twisting to adopt energetically favorable configurations (Sali et al. 1994 Englander et al. 2007).
Inspired by the in vivo protein folding process, we introduce generative model that acts the inter-residue angles in protein backbones instead of on Cartesian atom coordinates (Figure I This treats each residue as independent reference frame, thus shifting the equivariance requirements from the neural network to the coordinate system itself. A similar angular representation has been used in some protein structure prediction works (Gao et al.] [017]] AlQuraishi 2019) Chowdhury iet al. 2022). For generation, we use denoising diffusion probabilistic model (diffusion model, for brevity) (Ho al. 2020 Sohl-Diccsstein al.) 2015) with a vanilla transformer parameterization without any equivariance constraints. Diffusion models train a neural network to start from noise and iteratively "denoise" to generate data samples. Such models have been highly successful in a wide range of data modalities from images (Saharia et al.) 2022 Rombach et al. 2022) to audio (Rouard & Hadjeres 2021| Kong et al., 2021| ,, and are easier to train with better modal coverage than methods like generative adversarial networks (GANs) (Dhariwal & Nichol 2021 Nichol & Dhariwal] (02T) We present a suite of validations to quantitatively demonstrate that unconditional sampling from our model directly generates realistic protein backbones from recapitulating the natural distribution of protein inter-residue angles, to producing overall structures with appropriate arrangements of multiple structural building block motifs. We show that our generated backbones diverse and designable, and are thus biologically plausible protein structures. Our work demonstrates the power of biologically-inspired problem formulations and represents an important step towards accelerating the development of new proteins and protein- protein-based therapies.
2
RELATED WORK
2.1 GENERATING NEW PROTEIN STRUCTURES
Many generative deep learning architectures have been applied to the task of generating novel pro- tein structures. Annad et al. (2019) train GAN to sample pairwise distance matrices that describe protein backbone arrangements. However, these pairwise distance matrices must corrected, re- fined, and converted into realizable backbones via two independent post-processing steps, the Al- ternating Direction Method of Multipliers (ADMM) and Rosetta. Crucially, inconsistencies in these predicted constraints can render them unsatisfiable or lead significant errors when reconstructing the final protein structure. Sabban & Markovsky (2020) use a long short-term memory (LSTM) GAN to generate (ò, マ/) dihedral angles. However, their network only generates helices and relies on downstream post-processing to filter, refine, and fold structures, partly due to the fact that these two dihedrals do not sufficiently specify backbone structure. [Eguchi et al. (0022) propose vari- ational auto-encoder with equivariant losses to generate protein backbones in 3D space. However, their work only targets immunoglobulin proteins and also requires refinement through Rosetta. Non- deep learning methods have also been explored: (Schenkelberg & Bystroff (2016) apply heuristics to ensembles of similar sequences to perturb known protein structures, while Holm & Sander (199T) use database search to find and assemble existing protein fragments that might fit new scaffold structure. These approaches' reliance on known proteins and hand-engineered heuristics limit them to relatively small deviations from naturally-occurring proteins.
2.1.1 DIFFUSION MODELS FOR PROTEIN STRUCTURE GENERATION
Several recent works have proposed extending diffusion models towards generating protein struc- tures. These predominantly perform diffusion on the 3D Cartesian coordinates of the residues them- selves. For example, Triipee et al. (2022) use an Equu-equva graph neural network to model the coordinates of protein residues. |Anand & Achim (022) adopt a hybrid approach where they train an equivariant transformer with invariant point attention (Jumper et al. 2202]); this model generates the 3D coordinates of Ca atoms, the amino acid sequence, and the angles defining the orientation of side chains. Another recent work by Luo al. (222) performs diffusion for generating anti- body fragments' structure and sequence by modeling 3D coordinates using an equivariant neural network. Note that these prior works all use some form of equívariance translation, rotation,
2

---

Under review as a conference paper at ICLR 2023
Pe Pa( (Tt-1
xt)
TT
It
Jt-1
TO
g
xt--1)
TT
R
FO
N
Ca
8
8
8
Ca
N
Corresponding structure
Corresponding structure
Figure 1: We perform diffusion six angles illustrated in the schematic in the bottom center (also defined in Table []. Three of these are dihedral torsion angles (orange), and three are bond angles (green). We start with an experimentally observed backbone described by angles zop and iteratively add Gaussian noise via the forward noising process until the angles are indistinguishable from wrapped Gaussian at ZT. We use these examples learn the "reverse" denoising process Pę-
and/or reflection due to their formulation of diffusion on Cartesian coordinates. Another method, ProteinSGM (Lee Kim 2022), implements score-based diffusion model (Song et al. (020) that generates image-like square matrices describing pairwise angles and distances between all residues in an amino acid chain. However, this set of values is highly over-constrained, and must be used as of input constraints for Rosetta's 's folding algorithm (Yang et al. 2020), which in turn produces the final folded output. This is a similar approach to Anand et al. (2019). and likewise subject to the aforementioned concerns regarding complexity, satisfiability, and cleanliness of predicted con- straints. Our work instead uses minimal set of angles required to specify a protein backbone, and thus directly generates structures without relying additional methods for refinement. Un- fortunately, none of these prior works have publicly-availate code, model weights, or generated examples the time of this writing. Thus, our ability to perform direct comparisons is limited.
2.2 DIFFUSION MODELS FOR SMALL MOLECULES
A related line of work focuses creating and modeling small molecules, typically in the context of drug design, using similar generative approaches. These small molccules average 44 atoms in size (Jing et al. 2022). Compared proteins, which average several hundred residues and thousands of atoms (Tiessen et al. 2012), the relatively small size of small molecules makes them easier to model. The E(3) Equivariant Diffusion Model (Hoogeboom al.) (022) uses equivariant transformer to design small molecules by diffusing their coordinates in Euclidean space. Other works have explored torsional diffusion, i.e., modelling the angles that specify small molecule, to sample from the space of energetically favorable molecular conformations (Jing et al., 2022). This work still requires an SE(3) -qquivrraaa model as the input to their model is 3D point cloud. In contrast, our problem formulation allows us to work entirely in terms of relative angles.
3 METHOD
3.1 SIMPLIFIED FRAMING OF PROTEIN BACKBONES USING INTERNAL ANGLES
Proteins are variable-length chains of amino acid residues. There are 20 canonical amino acids, all of which share the same three-atom N C. C' backhone, but have varying side chains attached to the C, atom (typically denoted R, see illustration in Figure [Tp. These residues assemble to form polymer chains typically hundreds of residues long (Tiessen al. 2012). These chains of amino acids fold into 3D structures, taking on a shape that largely determines the protein's 's functions. These folded structures can be described on four levels: primary structure, which simply captures the linear sequence of amino acids; secondary structure, which describes the local arrangement of amino acids and includes structural motifs like a-helices and B-sheets; tertiary structure, which describes the full
3

---

Under review as a conference paper at ICLR 2023
Table 1: Internal angles used to specify protein backbone structure. Some of these involve multiple residues, indicated via index subscripts. These are illustrated in Figure
Angle
Description
Dihedral torsion about N; - Ca; Cj - No+1
w
Dihedral torsion about Ca; - C; Ni41 Ca.+1
Dihedral torsion about C; - Ni+1 Cait! C.+1
01
Bond angle about N; - Ca., C₁
02
Bond angle about Ca; C. N4+1
0g Bond angle about C₁ Ni+1 Caf+1
spatial arrangement of all residues; and quaternary structure, which describes how multiple different amino acid chains come together to form larger complexes (Sun et al. 2004).
We propose a simplified framing of protein backbones that follows the biological intuition of protein folding while removing the need for complex equivariant networks. Rather than viewing protein backbone of length N amino acids as a cloud of 3D coordinates (i.e., E N33 if modeling only Co atoms, or x E 3R x3 for a full backbone) as prior works have done, we view it as sequence of six internal, consecutive angles I E [-T,, (N)x)) That is, each vector of six angles describes the relative position all backbone atoms the next residue given the position of the current residue. These six angles are defined precisely in Table [ and illustrated in Figure These internal angles can be easily computed using trigonometry, and converted back to 3D Cartesian coordinates by iteratively adding atoms to the protein backbone as described in Parsons et al. (2005), fixing bond distances to average lengths (Appendix [A.]] Figure(ST), Despite building the structure from fixed point, this formulation does not appear to overly accumulate errors (Appendix A.2 Figures S2 S3
1)x66
This internal angle formulation has several key advantages. Most importantly, since each residue forms its own independent reference frame, there is need to use equivariant neural network. No matter how the protein is rotated or shifted, the angles specifying the next residue given the current residue never changes. This allows ture; in fact, we demonstrate that our model fails when substituting our shift- and rotation-invariant internal angle representation with Cartesian coordinates, keeping all other design choices identi- cal (Appendix [.I] Figure SS). This internal angle formulation also closely mimics how proteins actually fold by twisting into more energetically stable conformations.
simple transformer as the backbone architec-
3.2 DENOISING DIFFUSION PROBABILISTIC MODELS
Denoising diffusion probabilistic models (or diffusion models, for short) leverage Markov process ((x,) 1;-1) to corrupt data sample xig over T discrete timesteps until indistinguishable from noise at A diffusion model P: ( | xx) parameterized by Ç is trained to reverse this forward noising process, "denoising" pure noise towards samples that appear drawn from the native data distribution Bihl-Dickstein al. 2015). Diffusion models were first shown achieve good gener- ative performance by Ho et al. (2020); we adapt this framework for generating protein backbones, introducing necessary modifications to work with periodic angular values.
We modify the standard Markov forward noising process that adds noise at each discrete timestep t to sample from wrapped normal instead of a standard normal (Jing al. (022):
|x|
VI B + k
q(0; 1r-1) = Nwrapped (Tr; / 11,1 x Σ exp
2,37
where B, E (), are set by a variance schedule. We use the cosine variance schedule (Nichol & Dhariwal 2021 with T 1000 timesteps:
it 0.999
m(1) f(O)
1/T [1/T
8 =clip
f(1)
04-1
1+
4

---

Under review as a conference paper at ICLR 2023
where 8 10-3 is small constant for numerical stability. We train model for Pg pg( (24)) with the simplified loss proposed by Ho et al. (2020). using a neural network nng t) that predicts the noise ~ (0, I) present
given timestep (rather than the demoised mean values themselves). To handle the periodic nature of angular values, we introduce a function to "wrap" values within the We use to wrap smooth L.1 loss (Girshick) 2015) Lw. which behaves like L1 loss when error is high, and like an L2 loss when error is low; we the transition between these two regimes 3r, 0.1 While this loss is not as well-motivated as torsionll losses proposed by [ing et al. 2022), we find that it achieves strong empirical results.
range (-,,,); w((a) ((x: T) mod 2x)
dw w (e nng (w (Vorso + v1 (1)())) .t))
0.5 0.
Lu
if [dw < BL
ddwl 0.5,3
otherwise
During training, timesteps are sampled uniformly t ~ UO,, T). We normalize all angles in the training set to be zero mean by subtracting their element-wise angular mean μe; validation and test sets are shifted by this same offset.
Figure [ iillustrates this overall training process, including our previously described internal angle framing. The internal angles describing the folded chain Zig are corrupted until they become indis- tinguishable from random angles, which results a disordered mass of residues at ær; we sample points along this diffusion process to train our model nng. Once trained, the reverse process of sam- pling from Pe also requires modifications to account for the periodic nature of angles, as described
in Algorithm The variance of this reverse process is given by oe -- .3.
T-ai
Algorithm Sampling from Pe with FoldingDiff
I: zT ~ w I))
Sample from a wrapped Gaussian
2: for =T...., do
3: = -00 7) ift 1 else 2 == =
4: Tt-1 = w Jai
(It 1-ay nnn (n) + 012)
Wrap sampled values about [- T)
5: end for
6: return w(zo + p1)
> Un-shift generated values by original mean shift
This sampling process can be intuitively described as refining internal angles from an unfolded state towards a folded state. As this is akin to how proteins fold in vivo, we name our method FoldingDiff.
3.3 MODELING AND DATASET
For our reverse (denoising) model Pe (z,,,) t). we adopt a vanilla bidirectional transformer architecture Vaswwnn al. 2017) with relative positional embeddings (Shaw et al.) 2018). Our six-dimensional
input is linearly upscaled to the model's embedding dimension (d = 384). To incorporate the timestep we generate random Fourier feature embeddings (Tancik al. 2020) done [Sog ct al. (2020) and add these embeddings to each upscaled input. To convert the transformer's final per-position representations
our six outputs, we apply regression head consisting of a densely connected layer, followed by GELU activation (Hendrycks & Gimpel| 2016). layer normalization, and finally fully connected layer outputting our six values. We train this network with the AdamW optimizer [Loshchilov & Hutter] 2019) over 10,000 epochs, with learning rate that linearly scales from
10 over 1,000 epochs, and back to 0 over the final 9,000 epochs. Validation loss appears to plateau after se 1,400 epochs; additional training does not improve validation loss, but appears to lead to a poorer diversity of generated structures. We thus take a model checkpoint at 1,488 epochs for all subsequent analyses.
We train our model on the CATH dataset, which provides a "de-duplicated" set of protein structural folds spanning wide range of functions where two chains share more than 40% sequence identity over 60% overlap (Silltooe et al.) 2013). We exclude any chains with fewer than 40 residues. Chains longer than 128 residues are randomly cropped to 128-residue window at each epoch. A random
5

---

Under review as a conference paper at ICLR 2023
80/10/10 wwaining/valiiaatinnnn/ split yields 24,316 training backbones, 3,039 validation backbones, and 3,040 test backbones.
4 EXPERIMENTS
4.1 GENERATING PROTHIN INTERNAL ANGLES
After training our model, we check that able recapitulate the correct marginal distributions of dihedral and bond angles in proteins. We unconditionally generate 10 backbone chains each for ev- ery length E [50, 128), generating total of 780 backbones as was done 'Trippe et al. (2022). We plot the distributions of all six angles, aggregated across these 780 structures, and compare each dis- tribution to that of experimental test set structures less than 128 residues in length (Figures S9
We observe that, across all angles, the generated distribution almost exactly recapitulates the test distribution. This true both for angles that are nearly Gaussian with low variance (w, 81, 82 02) as well as for angles with highly complex, the - -//n boundary (w) are correctly handled as well. Compared to similar plots generated from other protein diffusion methods (e.g., Figure in Anand & Achim (022), reproduced with permis- sion in Figure S1O), we qualitatively observe that our method produces much tighter distribution that more closely matches the natural distribution of bond angles.
distributions (ø, v)). Angles that wrap about
@ distribution, KL=0.0412
y ddistribution, KL=0.0480
distribution, KL=0.0318
Noraaize frequency
2.5
1.0
1.0
0.
0.5
0.2
22
-1
-2
-1
-3
-2
0: distribution, KL-0.0259
8y distribution, KL-0.0636
0g distribution, KL=0.0312
Normalized frequency
1.0
1.5
2.5
0.5
1.0
1.5
2.5
3.0
0.0
1.0
2.0
3.0
Angle
Angle (rad)
Angle (rad)
Figure 2: Comparison of the distributions of angular values in held-out test set and generated samples. Top row shows dihedral angles (torsional angles involving 4 atoms), and bottom row shows bond angles (involving atoms). KL divergence is calculated between D Px (sampled (test). Figure 59 shows the cumulative distribution function (CDF) corresponding to these histograms.
However, looking individual distributions of angles alone does not capture the fact that these angles are not independently distributed, but rather exhibit significant correlations. A Ramachandran plot, which shows the frequency of co-occurrence between the dihedrals (é, v)), ) is commonly used to illustrate these correlations between angles (Ramachandran & Sasisekharan 1968). Figure shows the Ramachandran plot for (experimental) test chains with fewer than 128 residues, as well that for our 780 generated structures. The Ramachandran plot for natural structures (Figure 3a) contains helices, left-handed helices, and B sheets. All three of these regions are recapitulated in our generated structures (Figure 3bb) In other words, Folding backbones. Furthermore, we see that our model correctly learns that right-handed helices much more common than left-handed a helices (Cintas 2002). Prior works that use equivariant networks, such Trippe et al. (002)) cannot differentiate between these two types of helices due to network equivariance to reflection. This concretely demonstrates that our internal angle formulation leads to improved handling of chirality (i.e., the asymmetric nature of proteins) in generated backbones.
three major concentrated regions corresponding right-handed
able to generate three major secondary structure elements in protein

---

Under review as a conference paper at ICLR 2023
(a) Ramachandran plot, set P sheet
(b) Ramachandran plot, generated backbones
helix,
(radians)
y (radians)
a helix, RH
(radians)
(radians)
Figure 3: Ramachandran plots comparing the (ò, 1b) dihedral angles for test set (3a) and generated protein backbones (3b). Each major region of this plot indicates a different secondary structure ele- ment, as indicated in panel Ba All three main structural elements are recapitulated in our generated backbones, along with some less common angle combinations. Lines are artifacts of null values, and appear shifted due zero centering/uncentringg
4.2 ANALYZING GENERATED STRUCTURES
have shown that our model generates realistic distributions of angles and that our generated joint distributions capture secondary structure elements. We now demonstrate that the overall structures specified by these angles are biologically reasonable. Recall that natural protein structures contain multiple secondary structure elements. We use P-SEA (Labesse al. 1997) count the number of secondary structure elements in each test-set backbone of fewer than 128 residues, and generate 2D histogram describing the frequency of o co-occurrence counts in Figure 4a Figure 4b repeats this analysis for our generated structures, which frequently contain multiple secondary structure elements just as naturally-occurrin proteins do. FoldingDiff thus appears to generate rich structural information, with consistent performance across multiple generation replicates (Figure (ST1). This nontrivial task an autoregressive transformer, for example, collapses a failure mode of endlessly generating helices (Appendix C.2
Figures S5 S6
(a) Secondary structure co-occurrence, test
(b) Secondary structure co-occurrence, generated
0.08
0.05
0.07
0.07
Number of 8 sheets
Number of sheets
0.06
0.05
4-
0.04
0.04
0.00
0.09
0.00
0.00
Figure 4: 2D histograms describing co-occurrence of secondary structures in test backbones (4a and generated backbones (46). Axes indicate the number of secondary structure present in a chain; color indicates the frequency of specific combination of secondary structure elements. Our generated structures mirror real structures with multiple a helices, multiple sheets, and mixture both. See Figure STT for additional generation replicates.

---

Under review as a conference paper at ICLR 2023
Beyond demonstrating that FoldingDiff's generated backbones contain reasonable structural motifs, it is also important to show that they are designable - meaning that we can find a sequence of amino acids that can fold into designed backbone structure. After all, novel protein structure is not useful if we cannot physically realize it. Previous works evaluate this in silico by predicting possible amino acids that fold into a generated backhone and checking whether the predicted structure for these sequences matches the original backbone (Trippe et al. 2022] Lee & Kim [002] Appendix B). Following this general procedure. for a generated structure s, we use the ProteinMPNN inverse folding model (Dauparas et al., 2022) to generate 8 different amino acid sequences, as it yields
improved performance compared ESM-IF1 (Hsu et al. 20022 Appendix [C.3 Tables [S] S2). We then use OmegaFold (Wu et al. 2022) to predict the 3D structures 81, $1,.... 88 corresponding to each of these sequences. We use TMalign (Zhang & Skolnick, 2005), which evaluates structural similarity between backbones, to score each of these 8 structures against the original structure s. The maximum score m TMallgn(s, (s, 8,) is the self-consistency TM (scTM) score. A scTM score of > 0.5 is considered to be in the same fold, and thus is "designable."
ScTM scores, 780 generated protein backbones
training similarity
120
night des guabe
length
0.9
High Syaaiinnnng sim buity
100
kmg (= as)
0.8-
0.7
Medkum des grabiiity Simmiarit 30 traveng
Count
0.6
60-
0.5
20
length short (s 70 aal
long (> ax)
0.2
0.3
0.5
0.6
0.7
0.8
0.9
0.3
0.4
0.5
0.6
0.7
0.5
0.9
Self-coreistency scsccoe (scTM)
Maximum training
(a) Backbone designability by length
(b) Designability compared to training set similarity
Figure 5: Of our 780 generated backbones, ranging in length from 50-128 residues, 177 are des- ignable (scTM 0.5) using ProteinMPNN and OmegaFold. Shorter structures of 70 amino acids or fewer tend to have higher scTM scores than longer structures (5a). Generated backbones that are more similar to training examples (greater maximum training TM score) tend have better designability (5b). The three structures indicated by arrows are illustrated in Figure S12
With this procedure, we find that 177 of our 780 structures, or 22.7%, are designable with an scTM score 2 0.5 (Figure without any refinement or relaxation. This designability is highly consistent across different generation runs (Table ST]. and is also consistent when substituting AlphaFold2 without MSAs @Jmpeer al. 2021 in place of OmegaFold (163/78 780 designable with AlphaFold2). Trippe et al. (2022) use an identical scTM pipeline using ProteinMPNN and AlphaFold2, and report significantly lower proportion of designable structures (92/ 780 designable, p 10-5 Chi-square test). Compared to this prior work, FoldingDiff improves upon designability of both short sequences (up to 70 residues, 76/210 designable compared 36/210, p long sequences (beyond 70 residues, 87 37/570 designable compared to 56/570. p 5.6 x 10 Chi-square test). While ProteinSGM (ESM-IF1 for inverse folding, AlphaFold2 for fold prediction) reports an even higher designability proportion of 50.5%. this value is not directly comparable, as ProteinSGM generates constraints that are subsequently folded using Rosetta; therefore. their des- ignability does not directly reflect their generative process. The authors themselves note that Rosetta "post-prood poost-processing" significantly improves the viability of their structures. To further contextualize scMM scores, evaluate naive method that samples from the empirical distribution of dihe- dral angles. This baseline produces no designable structures whatsoever (Appendix (C3) Figures SS7 (8). Conversely, evaluate experimental structures to establish upper bound for designability; 87% natural structures have an se' TcTM 2 0.5 (Appendix C,3 Figure (S7).
10 Chi-square test) and
We additionally evaluate the similarity of each generated backbone to any training backbone by taking the maximum TM score across the entire training set. The maximum training TM-score is significantly correlated with scT' score (Spearman's 0.78, p 7.9 10-165 Figure 55b. indi-
8

---

Under review as a conference paper at ICLR 2023
cating that structures more similar to the training set tend to be more designable. However, this does not suggest that we are merely memorizing the training set; doing so would result in a distribution of training TM scores near 1.0, which is not what observe. ProteinSGM reports distribution of training set I'Mscores much closer to 1.0; this suggests a greater degree of memorization and may indicate that their high designability ratio is partially driven by memorization.
Selected examples of our generated backbones and corresponding OmegaFold predictions of various lengths are visualized using PyMOL (Schrödinger, LLC 2015) in Figure 6 Interestingly, find that of our 177 designable backbones, only 16 contain & sheets as annotated by P-SEA. Conversely, of our 603 backbones with scTM < 0.5, 533 contain 3 sheets. This suggests that generated structures with 8 sheets may be less designable (p < 1.0 x 10 Chi-square test). We additionally cluster our designable backbones and observe large diversity of structures (Figure (S4)) comparable to that of natural structures (Figure (T5).. This suggests that our model is not simply generating small variants on a handful of core structures, which prior works appear to do (Figure (16).
Genernted (ours)
mamn
OmegaFold Fold
m
59 residues, scTM 0,59
residues, scTM 0.51
107 residues, scTM 0.53 111 residues, scTM = 0.55
Figure 6: Selected generated protein backbones varying length that are approximately designable (scTM R 0.5). Top row shows directly generated backbones; bottom row shows OmegaFood predicted structure for residues inferred by ProteinMPNN to produce our generated backbone. Struc- tures contain both helices (coils, columns 1-4) and /3 sheets (ribbons, columns and 2), and each appears meaningfully different from its most similar training example (Figure S13).
5 CONCLUSION
In this work, we present novel parameterization of protein backbone structures that allows for simplified generative modeling. By considering each residue to be own reference frame, we describe a protein using the resulting relative internal angle representation. We show that vanilla transformer can then used to build a diffusion model that generates high-quality, biologically plausible, diverse protein structures. These generated backbones respect protein chirality and exhibit high designability.
While demonstrate promising results with model, there are several limitations Though formulating a protein as a series of angles enables use of simpler models without equivari- ance mechanisms, this framing allows for errors early in the chain to significantly alter the overall generated structure a sort of "lever arm effect." Additionally, some generated structures exhibit collisions where the generated structure crosses through itself. Future work could explore methods to avoid these pitfalls using coometrically-infomm (2022). Our generated structures are still of relatively short lengths compared natural proteins which typically have several hundred residues; future work could extend towards longer structures, potentially incorporating additional losses or inputs that help "checkpoint" the structure and reduce accumulation of error. We also do not handle multi-chain complexes ligand interactions, and are only able to generate static structures that do not capture the dynamic nature proteins. Future work could incorporate amino acid sequence generation in parallel with structure generation, along with guided generation using functional or domain annotations. In summary, our work provides an important step in using biologically-inspinn problem formulations for generative protein design.
work.
architectures such as those used in Wu et al.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammed AlQuraishi. End-to-end differentiable learning of protein structure. Cell systems, 8(4): 292-301, 2019.
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant de- noising diffusion probabilistic models. arXiv preprint arXiv:2205. 2022.
Namrata Anand, Raphael Eguchi, and Po-Ssu Huang. Fully differentiable full-atom protein back- bone generation. DGS@ ICLR, 2019.
Massimo Bonora, Simone Patergnani, Alessandro Rimessi, Flena De Marchi, Jan M Suski, Angela Bononi, Carlotta Giorgi, Saverio Marchi, Sonia Missiroli, Federica Poletti, et al. ATP synthesis and storage. Purinergic Signalling, 8(3):343-357, 2012.
Tapan K Chaudhuri and Subhankar Paul. Protein-misfolding diseases and chaperone-based thera- peutic approaches. The FEBS Journal, 273(7):1331-1349 2006.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M Church, et al. Single- sequence protein structure prediction using language model and deep learning. Nature Biotech- nology, pp. 1-7, 2022.
Pedro Cintas. Chirality of living systems: helping hand from crystals and oligopeptides. Ange- wandte Chemie International Edition, 41 139-1114,, 2002.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- 2022.
based protein sequence design using proteinmpnn. Science, 378(6615):49-35
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Ad- vances in Neural Information Processing Systems, 34:8780-8794, 2021.
Dimiter S Dimitrov. Therapeutic proteins. Therapeutic Proteins, pp. 1-26, 2012.
Roberto Dominguez and Kenneth C Holmes. Actin structure and function. Annual Review of Bio- physics, 40: 40:169, 2011.
Raphael R Eguchi, Christian A Choe, and Po-Ssu Huang. Ig-vae: Generative modeling of protein structure by direct 3d coordinate generation. PLoS computational biology, 1006)211 2022.
S Walter Englander, Leland Mayne, and Mallela MG Krishna. Protein folding and misfolding: mechanism and principles. Quarterly Reviews of Biophysics, 40(4):1-41, 2007.
Yujuan Gao, Sheng Wang, Minghua Deng, and Jinbo Xu. Real-value and confidence prediction of protein backbone dihedral angles through hybrid method of clustering and deep learning. arXiv preprint /7/2.007 2017.
Ross Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1448, 2015.
Peter H Tobin, David H Richards, Randolph A Callender, and Corey J Wilson. Protein engineering: new frontier for biological therapeutics. Current Drug Metabolism, 15(7):743-756, 2014.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). 16Xiv:1606.088 00006.0466...... 2016.
arXiv preprint
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Liisa Holm and Chris Sander. Database algorithm for generating protein backbone and side-chain co-ordinates from a Ca trace: application to model building and detection of co-ordinate errors. Journal of Molecular Biology, 218(1):183-194 1991.
10

---

Under review as a conference paper at ICLR 2023
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3D. In International Conference on Machine Learning, pp- 8867-8887. PMLR, 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. In International Con- ference on Machine Learning, pp. 8946-899 8970. PMLR, 2022.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffu- sion for molecular conformer generation. arXiv preprini arXiv:2206.011 01722. 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with Alphafoold. Nature, 596(7873) 2021.
Mariusz Kamionka. Engineering of therapeutic proteins production in Escherichia coli. Current Pharmaceutical Biotechnology, 12(2):268-277 2011.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Gilles Labesse, Colloc' h, Joöl Pothier, and J-P Mornon. P-SEA: a new efficient assignment of secondary structure from Ca trace of proteins. Bioinformatics, 13(3):291-295, 1997.
Benjamin Leader, Quentin J Baca, and David E Golan. Protein therapeutics: summary and phar- macological classification. Nature reviews Drug discovery, 7(1):21-39, 2008.
Jin Sub Lee and Philip M. Kim. ProteinSGM: Score- based generative modeling for de novo protein design. bioRxiv, 2022. doi: 10.100120022001133449900.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specife antibody design and optimization with diffusion-based generative models for protein structures. bioRxiv, 2022. doi: 10.1101/202.00..1.44999
RA Mariuzza, SEV Phillips, and RJ Poljak. The structural basis of antigen-antibode recognition. Annual Review of Biophysics and Biophysical Chemistry, 16(1):139-155) 159, 1987.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov. and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature Methods, pp. 1-4, 2022.
Alexander Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In
International Conference Machine Learning, pp. 8162-8171. PMLR, 2021.
Jerod Parsons, J Bradley Holmes, J Maurice Rojas, Jerry Tsai, and Charlie EM Strauss. Practi- cal conversion from torsion space to cartesian space for in silico protein synthesis. Journal of Computational Chemistry, 26 163:1066-6 2005.
Zhao Qin, Andrea Fabre, and Markus J Buchler. Structure and mechanism maximum stability of isolated alpha-helical protein domains at a critical length scale. The European Physical Journal E, 36(5):1-12, 2013.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
GN Ramachandran and V Sasisekharan. Conformation of polypeptides and proteins. Advances in Protein Chemistry, 23:283-437, 1968.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.

---

Under review as a conference paper at ICLR 2023
Simon Rouard and Gaëtan Hadjeres. CRASH: Raw audio score-based generative modeling for controllable high-resolution drum sound synthesis. arXiv preprint arXiv:2166.0431,, 2021.
Sari Sabban and Mikhail Markovsky. RamaNet: Computational de novo helical protein backbone design using long short-term memory generative neural network. bioRxiv, pp. 67 671552, 2020.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
Andrej Šali, Eugene Shakhnovich, and Martin Karplus. How does a protein fold. Nature, 369(6477); 251, 1994.
Christian D Schenkelberg and Christopher Bystroff. Protein backbone ensemble generation explores the local structural space of unseen natural homologs. Bioinformatics, 32(10):1454-1461 2016.
Schrödinger, LLC. The PyMOL molecular graphies system, version 1.8. November 2015.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprint
2018.
Ian Sillitoe, Tony E Lewis, Alison Cuff, Sayoni Das, Paul Ashford, Natalie L Dawson, Nicholas Furnham, Roman A Laskowski, David Lee, Jonathan G Lees, et al. CATH: comprehensive structural and functional annotations for genome sequences. Nucleic Acids Research, 43(D1):
D376-D381, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dicksteif Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.133:5 2020.
Peter Sun, Christine E Foster, and Jeffrey C Boyington. Overview of protein structural and functional folds. Current Protocols in Protein Science, 35(1):17-1, 2004.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil,,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Process- ing Systems, 33:7537-7547, 2020.
Martha M Teeter. Water structure of a hydrophobic protein atomic resolution: Pentagon rings of water molecules in crystals of crambin. Proceedings of the National Academy of Sciences, 81 (19):6014-6018, 1984.
Axel Tiessen, Paulino Pérez- Prrri-Rod,,,ez, and Luis José Delaye-Arredonde Mathematical modeling and comparison of protein size distribution in different plant, animal, fungal and microbial species reveals a negative correlation between protein size and protein number, thus providing insight into the evolution of proteomes. BMC Research Notes, 5(1):1-23, 2012.
Brian Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3D for the motif- scaffolding problem. arXiv preprint arXiv:2206.041 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Informa- tion Processing Systems, 30, 2017.
Ruidong Wu, Fan Ding, Rui Wang. Rui Shen, Xiwen Zhang, Shitong Luo. Chenpeng Su, Zuo- fan Wu, Qi Xie. Bonnie Berger, Jianzhu Ma, and Jian Peng. High-resolution de novo structure 27..002......1.000000
prediction from primary sequence. bioRxiv, 2022. doi: 10. 10.101/2022.7.21.500000
12

---

Under review as a conference paper at ICLR 2023
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceed- ings of the National Academy of Sciences, 17(3):1496-150: 2020.
Yang Zhang and Jeffrey Skolnick. TM-align: a protein structure alignment algorithm based on the TM-score. Nucleic Acids Research, 33(7):2302-209, 2005.
Qiangjun Zhou, Peng Zhou, Austin L Wang, Dick Wu, Minglei Zhao, Thomas C Südhof, and Axel T Brunger. The primed snare-complexin 44807568):400-455, 2017.
pere--ooppeiii-sssaptttaggana synaptotagmin complex for neuronal exocytosis. Nature,
13

---

Under review as a conference paper at ICLR 2023
A INTERNAL ANGLE FORMULATION OF PROTEIN BACKBONES
A.1 CHOICE ANGLES FOR REPRESENTATION
A protein backbone structure can be fully specified by total of 9 values per residue: 3 bond dis- tances, bond angles, and dihedral torsional angles. The three bond angles and dihedrals are described in Table D and the three bond distances correspond to N; + Ca., Ca, - C2. and - N1+1 where denotes residue index. These values enable protein backbone to be loss- lessly converted from Cartesian internal angle representation, and vice versa. To determine which subset of values to use to formulate proteins in our model, we take a set of experimentally profiled proteins and translate their coordinates from Cartesian internal angles and distances and back, measuring the TM score between the initial and reconstructed structures. When excluding an angle distance, fix all corresponding values the mean. The reconstruction TM scores of various combinations of values is illustrated in Figure ST Of these 9 values, the three bond distances are the least important for reliably reconstructing structure from Cartesian coordinates the inter-residue representation and back; they can usually be replaced with constant average values without much impact on the recovered structure. In comparison, removing even two bond angles with relatively little variance (02, 03) results in large loss in reconstruction TM score (third bar). Removing all bond angles and retaining only dihedrals (ø, v, w) results only about half of proteins being able to be reconstructed (last bar). Thus, we model the three dihedrals and the three bond angles (second bar in Figure [S]; this simplifies our prediction problem to use only periodic angular values (instead of mixture angular and real values) without substantial loss in the accuracy of described structures. Future work might include additional modeling of these real-valued bond distances.
Reconstruction accuracy, combinations of angles/distances
1.0
TM-score of reconstruution
0.8
0.6
0.4
dihedrats. angise, and distances
ditedrals and anples
dhedralss, e, and distances
diiedrais, distances
dihedrals only
dthedrals
Figure S1: Various combinations of angles and distances and their ability to faithfully reconstruct protein backbones. A TM-score of 0.5 (dashed grey line) indicates the minimum similarity for two structures to be considered to have the same general shape. Error bars represent standard deviation in reconstruction TM scores. Using all bond angles, dihedral angles, and bond distances perfectly reconstructs Cartesian coordinates from internal angles (first column). The second column corre- sponds to the formulation used in the main text, where model the 3 dihedrals and bond angles, but keep the 3 bond distances fixed to average values. Other columns fix even more values to their respective means and result in reconstruction TM scores that are too low to be reliably useful.
14

---

Under review as a conference paper at ICLR 2023
One detail when converting between N-residue set of Cartesian coordinates between consecutive residues that the latter representation does not capture the first residue's information (as there is prior residue orient against). To solve this, Cartesian coordinates, using the N specified angles to
of N 11 angles
fixed set of
coordinates to "seed" all generation build out from this fixed point. For all generations, this fixed point extracted from the coordinates of the N - Ca - C' atoms in the first residue on the N-terminus of the PDB structure ICRN (Teeter 1984..
Doing so does not result in any meaningful reconstruction error natural structures.
A.2 EFFECT OF LENGTH STRUCTURE RECONSTRUCTION
One of the primary concerns of using an angl-bassed formulation is that small errors might propagate across many residues to culminate in large difference in overall structure. To try and quantify the effect of this, we evaluate the "lossiness" of the representation itself, and the ability of the model to learn long-range angle dependencies.
We start by evaluating the accuracy of our representation itself over different structure lengths. To do this, we sample 5000 structures of varying length from the CATH dataset. For each, we compare the original 3D coordinate representation Ir and the 3D coordinates obtained after converting the structure to angles and back coordinates &e using the TM score algorithm, i.e., TMscore (x), #t)- We find that longer structures exhibit greater TM score divergences when converted through representation (Figure S2). However, even at our maximum considered structure length of 128 residues, structures still retain reconstruction TMscore 0.9, which is well above the accepted threshold 0.5 denoting the same fold. This indicates that while our representation itself is slightly "lossy", the losses do not change the overall fold. Even when considering longer structures up 512 residues in length, the reconstructed structures still share a TMscore similarity much greater than 0.5 (graph not shown), which suggests that our method could scale up to larger structures.
Structure reconstruction with dihedrals and angles
1.00
0.95
TM-score of reconstruction
0.90
0.85
0.80
0.75
40
60
80
100
120
Structure length
Figure S2: Faithfulness of reconstruction (via TMscore, y-axis) when using the dihedrals and angles described in Table and keeping bond distances fixed to average values, evaluated across 5000 structures of varying length (x-axis). We observe significant negative correlation between length and reconstruction TM score (Spearman's correlation -0.18, 7.2 10 10-39).
Next, evaluate our model's ability to successfully reconstruct sequences of varying length. For each structure in our held-out test set (n = 3040), we add t = 750 timesteps of noise to that structure's 's angles (recall that significant amount noise to corrupt the structure, while retaining a hint the target true structure weak "guide" to what the model should ultimately denoise towards. We then apply trained
total of T 1000 timesteps during training). This adds
15

---

Under review as a conference paper at ICLR 2023
model to these mostly -oised examples, rumning them for the requisite 750 iterations to fully denoise and reconstruct the angles. Afterwards, we assess reconstruction accuracy by taking the TMscore between the structure specified by the reconstructed angles, and the true structure specified by the ground truth angles. Comparing structures specified by reconstructed and true angles isolates the ef- fects of model error, irrespective of any minor lossiness induced by the representation itself, which we investigated previously (Figure S2). Figure [S3 illlstraas the relationship between test set struc- ture length and this reconstruction TM score. We find no significant correlation between length and reconstruction TM score (Spearman's 's correlation = 0.0024, = 0.89). FoldingDiff accurately reconstructs structures with a TM score of greater than 0.95 for 98% of test set, and achieves an average test reconstruction TM score of 0.988.
Test structure (n=3040) reconstruction, add 750/1000 noise timesteps
1.00
TM Cccorerecoostraiee coords, truth)
0.98
0.96
0.92
0.88
0.84
80
100
120
Structure length
Figure S3: Test set structure reconstruction accuracy after injecting 750 timesteps noise into an- gles. This is evaluated by taking the TM score between the structured specified by the reconstructed angles, and the structure specified by the ground truth angles. The x-axis denotes length of the test set structure, and each bar denotes the distribution of TM scores within that length.
All in all, these results indicate that (1) representation itself is indeed lossier with longer struc- tures, but not to a degree that would sigmificantly impact the overall structures, and (2) our model is capable of learning robust relationships between angles regardless of structure length.
B ADDITIONAL NOTES ON SELF-CONSISTENCY TM SCORE
Our scTM evaluation pipeline is similar previous evaluations done by Trippe al. (2022) and Lee & Kim (022), with the primary difference that we use OmegaFold (Wu et al. 2022) instead of AlphaFold (Jumper et al. 202I). OmegaFold is designed without reliance on multiple sequence alignments (MSAs), and pertorms similarly to AlphaFold while generalizing better to orphan pro- teins that may not have such evolutionary neighbors (Wu et al. 2022). Furthermore, given that prior works use AlphaFold without MSA information in their evaluation pipelines, OmegaFold appears to be a more appropriate method for scTM evaluation.
OmegaFold is using default parameters (and release weights). We also AlphaFold without MSA input for benchmarking against Trippe et al. (2022). We provide a single sequence reformatted to mimic "MSA" to the colabfo tool (Mirdita al. 2022) with 15 recycling iterations. While the full AlphFold model runs models and picks the best prediction, we use singular model (mode 11) to reduce runtime.
Trippe et al. (2022) use ProteinMPNN (Dupparas et al. 2022 for inverse folding and generate
candidate sequences per structure, whereas Lee & Kim (2022) ESM-IF1 (Hsu al. 2022)
16

---

Under review as a conference paper at ICLR 2023
and generate 10 candidate sequences for each structure. We performed self-consistency TM score evaluation for both these methods, generating 8 candidate sequences using author-recommenddd temperature values (T' 1.0 for ESM-IFI, T 0.1 for ProteinMPNN), We use OmegaFold to fold all amino acid sequences for this comparison. We found that ProteinMPNN in C. mode (i.e., alpha-carbon mode) consistently yields much stronger scT' values (Tables SST S2p ; we thus adopt ProteinMPNN for our primary results. While generating more candidate sequences leads to a higher scTM score (as there are more chances to encounter successfully folded sequence), we conserva- tively choose to run 8 samples to be directly comparable to Trippe et al.] (2022). We also use the same generation strategy as TTripee et al. (2022). generating 10 structures for each structure length I E [50, 128) - thus the only difference in our scTM analyses is the generated structures themselves.
C ABLATIONS AND BASELINES
C.I SUBSTITUTING INTERNAL ANGLE FORMULATION FOR CARTESIAN COORDINATES
We perform an "ablation" of our internal angle representation by replacing our framing of proteins as a series of inter-residue internal angles with a simple Cartesian representation of C, coordinates E RNN Notably, this Cartesian representation is no longer rotation or shift invariant. We train a denoising diffusion model with this Cartesian representation, using the same variance schedule, transformer backbone architecture, and loss function, but sampling from standard Gaussian and with all usages of our wrapping function w removed. This represents the same modelling approach as our main diffusion model, with only internal angle formulation removed.
To evaluate the quality of this Cartesian- based diffusion model's 's generated structures, we calculate the pairwise distances between all C, atoms in its generated structures and compare these with dis- tance matrices calculated for real proteins and for our internal angle diffusion model's generations. For real protein, this produces pattern that reveals close proximity between pairwise residues where the protein is folded inwards to produce compact, coherent structure (Figure (S4a). How- ever, similarly visualizing the C. pairwise distances in the Cartesian model's generated structures yields no significant proximity or patterns between any residues (Figure (S4b). This suggests that the ablated Cartesian model cannot learn to generate meaningful structure, and instead generates a nondescript point cloud. Our internal angle model, on the other hand, produces visualization that is very similar to that of real proteins (FiguresS44c) Simply put, our model's performance drastically degrades when change only how inputs are represented. This demonstrates the importance and effectiveness of our internal angle formulation.
C.2 AUTOREGRESSIVE BASELINE FOR ANGLE GENERATION
As a baseline method for our generative diffusion model, we also implemented an autoregressive (AR) transformer fAR that predicts the next set of six angles in a backbone structure (i.e., the same angles used by FoldingDiff, described in Figure and Table I given all prior angles.
Architecturally, this model consists of the same transformer backbone as used in FoldingDiff com- bined with the same regression head converting por-token embeddings to angle outputs, though is trained using absolute positional embeddings rather than relative embeddings as this improved vali- dation loss. The total length of the sequence is encoded using random Fourier feature embeddings, similarly to how time was encoded in FoldingDiff, and this embedding is similarly added to each position in the sequence of angles. The model is trained to predict the i-th set of six angles given all prior angles, using masking to hide the i-th angle and onwards. We use the same wrapped smooth LI loss as our main FoldingDiff model to handle the fact that these angle predictions exist in the where superscripts indicate positional index- ing. This approach is conceptually similar to causal language modeling (Radford et al. 2019), with the important difference that the inputs and outputs are continuous values, rather than (probabilities over) discrete tokens.
range [- T); specifically: Th (x ( [A.(
This model is trained using the same data set and data splits as our main FoldingDiff model with the same preprocessing and normalization. We train fAR using the AdamW optimizer with weight decay set to 0.01. We use a batch size of 256 over 10,000 epochs, linearly scaling the learning rate from 0 to 5 x 10-5 over the first 1 1,000 epochs, and back to O over the remaining 9,000 epochs.
17

---

Under review as a conference paper at ICLR 2023
(a) C. pairwise distances, real structure
60
5
25
Residue index
10
45
50
70
75
20
95
100
SERRRRRSSS 33 2353 33338R
Residue index
(b) Ca pairwise distances, Cartesian model
(c) Co pairwise distances, FoldingDiff (ours)
50
Residve index
Distance
distance
55
65
Residue
75
85
20
20
100
100
110
110
120
120
RARA A.A489 493444493
50-
300
105
100
110
Residue index
Residue index
Figure S4: Pairwise distances between all Ca atoms in various protein backbone structures, all of similar length. All panels the color scale. S4a illustrates set of distances for real protein structure; note the visual patterns that correspond to various secondary structures and potential con- tacts and interactions between residues. (S4b shows these distances for structure generated by ablated model that replaces our proposed internal angle representation with Cartesian coordinates, which results in no coherent structural generation. For comparison, FoldingDiff model produces structures that compactly fold to create many potential contacts, just as real proteins do (S4c).
To generate structures from SAR, we "seed" the autoregressive model with sets angles taken from the corresponding first 4 angle in randomly-chosen naturally occurring protein structure. This serves as a random, but biologically realistic, "prompt" for the model to begin generation. We then supply a fixed length and repeatedly run fAR to obtain the next i-th set of angles, appending each prediction to the existing - values in order to predict the + set of angles. We repeat this until we reach our desired structure length.
We use the above procedure to generate 10 structures for each structure length 1 with different of seed angles. Examples of generated structures of varying lengths are illustrated in Figure S5 All structures generated by this autoregressive approach consist of one singular helix. This confirmed by running P-SEA on the generated structures (Figure S6a). Besides being an obvious sign of modal collapse, these extremely long helices also not commonly observed in nature (Qin et al. 2013).
[50. 128) each
Such behavior where autoregressive models generate single looped pattern endlessly has been observed in language models as well, where it is often circumvented using a temperature parameter to inject randomness. In continuous regime, try to do something similar by adding small amounts of noise to each angle during generation, but are unable to meaningfully deviate from the aforementioned modal collapse to endless helices.
18

---

Under review as a conference paper at ICLR 2023
50 residues
80 residues
89 residues
115 residues
Figure S5: Structures generated using an autoregressive (AR) baseline approach trained on the same angle-based formulation we propose. This approach predicts the next set of angles given all prior angles, and can be thus used to iteratively generate structures in an autoregressive fashion (see Ap- pendix C.2 for additional details). However, the structures generated this way are all straight helices, regardless of initial "prompt" angles (see Figure (6a). This complete lack diversity and meaningful complexity indicates that while this AR model can produce technically "correct" struc- tures, it cannot be used for generative modeling to any meaningful capacity. Figure analogously illustrates FoldingDiff's generations, which structurally much more diverse.
SCTM scores, 780 generated protein backbones
length long (> aa)
Number B sheets
6-
60
Prequency
4-
60
0.2
0.0
0.55
Number
helices
Sell-consistency TM score (sCTM)
(a) AR baseline, sccondary structure co-occurrence
(b) AR bascline, scTM designability
Figure S6: Secondary structure elements (a) and designability (b) for structures generated by the autoregressive baseline described in Appendix C.2 We observe that using P-SEA to annotate these generated structures detects exclusively singular helices, and 3 sheets (S6ap. This quantifies the observations in Figure S55 that the AR model has "collapsed" into repeatedly generating these coils. We find that these helices exhibit greater designability via scTM scores (computed using ProteinMPNN and OmegaFold) (S6b), with 693 of the 780 structures having scTM 2 0.5, though this increase is not meaningful due to the utter lack of diversity in generated sequences.
We evaluate the AR model's generated structures' scTM designability using ProteinMPNN (CA- only mode) and OmegaFold. We find that 693 of the 780 generated structures are designable, leading to an overall designability ratio of 0.89. (Figure S6b). Importantly however, this gain in designability meaningless, singular endless coils are biologically neither useful nor novel.
C.3 BASELINES CONTEXTUALIZING SCTM SCORES
To contextualize FoldingDiff's scTM scores (Figure 5a), we implement a naive angle generation baseline. We take our test dataset, and concatenate all examples into a matrix of E [- TT) Nx6 where Ñ denotes the total number of angle sets in our test dataset, aggregating across all individual chains. To generate backbone structure of length I, we simply sample indices from (0, N). This creates chain that perfectly matches the natural distribution of protein internal angles, while also perfectly reproducing the pairwise correlations, i.e., of dihedrals in Ramachandran plot, but
19

---

Under review as a conference paper at ICLR 2023
eritically loses the correct ordering of these angles. We randomly generate 780 such structures (10 samples for each integer value of E [50, 128)). This is the same distribution of lengths as the gen- erated set in our main analysis. For each of these, we perform scTM evaluation with ProteinMPNN (CA-only mode) and OmegaFold. The distribution of scT' scores for these randonly-sampled structures compared to that of FoldingDiff's generated backhones is shown in Figure [S7] We ob- serve that this random protein generation method produces significantly poorer scTM scores than FoldingDiff (p- 1.6 10-121, 121,, Mann-Whitne test). In fact, not single structure generated this way is designable. This suggests that our model is not simply learning the overall distribution of angles, but is learning orderings and arrangements of angles that comprise folded protein structures.
SCTM scores, naive baseline vs. generated structures
Randomly sampled angles
Generated
12
Test set (natural) structures
Normalized frequency
0.2
0.4
0.6
0.8
Self-consistency TM (scTM) scores
Figure S7: Distribution of scTM scores for our generated structures (orange), compared to scTM scores for structures created by randomly shufiling naturally-occurrin internal angles (blue). The randomly sampled angles result in designable structures, despite perfectly capturing the overall distribution and pairwise relations between angles. This suggests our method correctly learns the spatial ordering of angles that folds valid structure. We additionally take set of 780 experimental structures and pass them through our scTM pipeline to evaluate the fragility of this pipeline itself. We find that 87% of natural proteins (green) designable; this forms "soft" upper bound for what would be achievable by sampling from the true data distribution.
We take the structures specified by these randomly sampled angles and analyze them for secondary structures using P-SEA, S8, and represents a completely different distribution of secondary structures compared to natural structures, or that of FoldingDiff's generations. This further suggests that this random angle baseline cannot generate natural- aapurai-ap structures. is clear that scTM scores and secondary structure analyses cannot be satisfied using this trivial solution.
did for Figure 4 The resulting 2D histogram is illustrated in Figure
20

---

Under review as a conference paper at ICLR 2023
1.0
0.8
Number of B sheets
6
0.6
Frequency
0.4
3
0.2
1
0.0
1
2
8
9
Number of helices
Figure S8: Secondary structures present in structures generated by randomly shuffling naturally- occurring angles, as described Appendix C33 This random baseline produces a few hits to B sheets by chance, but notably lacks the helices present in both natural structures and FoldingDiff's generations (Figures S11
21

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL SUPPLEMENTARY FIGURES AND TABLES
$ CDF
CDF
CDF
Cumulative frequency
Sampeed
6--
58
088
-2
6, CDF
CDF
6, CDF
Cunulativv frequency
2.-
5.-
5.0-
5.8
0.
0.5
3.5
2.0
2.5
5.0
0.2
1.0
355 Angle (rad)
2.2
3.0
5.
0.5
1.
2.2
2.0
Angle trac]
Angle (rad)
Figure S9: Comparison of the cumulative distribution functions (CDF) of angular values in test set and in generated samples. Top row shows dihedral angles (torsional angles involving 4 atoms), and bottom row shows bond angles (involving 3 atoms). Figure shows the histogram distributions corresponding to these CDFs.
CA-CA distance
Ci-Nit bond
CA-C-N.nn angle
0.8
0.8
100
3.5
0.6-
(dlg)
0.4
Native
0.2
-100
0.0
0.0
0.0
-100
Distance (A)
Bond length (A)
Angle (rad)
(degg)
reproduced with permission for ease of ref- erence. Illustrated Ca. - C. No+1 bond angle (third plot from the left) corresponds to 02 in our formulation. Sampled angles in the work of Anand & Achim (202) exhibit much larger spread than the natural distribution of angles, whereas our work matches much more tightly (Figures
Figure S10: Figure 1B from Anand & Achim 2022
S9
22

---

Under review as a conference paper at ICLR 2023
Table S1: Self-consistency TM scores (using ProteinMPNN and OmegaFold) across replicates. Each run generates 10 different structures for each length / [50, 128), resulting in 780 total generated structures, and started from a different random seed, using the same pre-trained model in the primary text and Appendix (. Self-consistency TM scores are computed using ProteinMPNN and OmegaFold, as described the primary text. Short structures are defined as having 70 residues or fewer (n 210); long structures are defined having more than 70 residues (n 570). Values from our main text are reproduced in the last row for ease of reference. Each generation run pro- duces consistently high number of designable structures, and also contains realistic mixture of secondary structure elements (Figure SIT..
Seed
Designable
Designable, short (n 210)
Designable, long (n 570)
173
101
2
154
75
79
3
185
90
95
4
182
86
96
5
187
76
111
7344 (main text)
177
97
Table S2: Self-consistency TM scores calculated using ESM-IF1 to perform inverse folding, rather than ProteinMPNN, These analyze the same FoldingDiff generations as Table ST] and are likewise folded with OmegaFold. ESM-IFI consistently results in much lower scTM scores.
Seed
Designable
Designable, short (n 210)
Designable, long 570)
117
61
56
2
105
64
41
3
122
76
46
4
115
72
43
5
126
67
59
7344 (main text)
111
57
54
A0l
4000
Ase
5.0
4.00
Numre
Numeer 4
4.00
Numse 4
4.05
Ale
4.00
A00
4.40
4.00
Are
E.00
6/0
6.09
Namber
Number
(a) Generations, seed
(b) Generations, seed 2
(c) Generations, seed 3
hurberof sinckes
(d) Generations, seed
(e) Generations, seed
we use P-SEA to annotate secondary structures. Each run's generations contain mixture of helices (x-axis) and 3 sheets (y-axis) that is comparable to natural structures (Figure 4a). FoldingDiff generates reasonable, complex structures consistently.
Figure S11: For each of the 5 replicates shown Table S1
23

---

Under review as a conference paper at ICLR 2023
nmu
Generated (ours)
Max training TM
me smm
Omegal old
111 residues max training 1M 0.93
residues max training 1M 0.47 scTM 0.51
residues
max training TM 0.32
Sc 0.90
scTM = 0.21
Figure S12: Generated structures representing the full range of designability (scTM) and training similarity scores. The top row indicates the original generated structure, the middle row shows the training structure with the highest TM score, and the bottom by OmegaFold based on residues predicted to produce our generated structure by ProteinMPNN. The first column shows structure with high designability and high training similarity. The second column shows structure with designability and training similarity close to 0.5. The third column shows a generated structure that very different from any training chain, but is also not designable.
indicates the structure predicted
Max triiiing TM Generated (ours)
OmegaFold
w
59 residues max training TM = 0.45 scTM = 0.59
61 residues max training TM 0.45
107 residues
111 residues
max training TM 0.71
max training TM 0.59
scTM 0.51
scTM 0.53
scTM 0.55
Figure S13: Structures from Figure jillustrate with the training example with the highest TM score (most similar). Figure rows are arranged as in Fig Our generated structures are visually quite different compared to the best training set match in almost every example, our generated structure contains completely different arrangement of secondary structure elements than the closest training structure, indicating that they may be more distinct than TM scores alone might suggest.
24

---

Under review as a conference paper at ICLR 2023
1.0
0.6
0.4
0.0
Figure S14: Clustering of our 177 "designable" generated backbones with scTM scores > 0.5. We use the average distance metric to perform hierarchical clustering on the pairwise distance matrix d(z, v) 1- LL-MMccore(,, y). Dark values corresponding to 0 (or conversely, a TM score of 1) indicate (nearly) identical structures. While there are some loosely related groups of structures, we do not observe clearly delineated groups. This indicates that the designable backbones generate are diverse and represent wide range of potential structures. For comparison to naturally-occurring structures and prior works, see Figures S15 and S16
25

---

Under review as a conference paper at ICLR 2023
0.8
0.6
0.2
Figure S15: To provide additional context for the degree of diversity that reasonable to expect from our generated sequences, similarly perform clustering set of 177 randomly chosen naturally-occurrig CATH structures between 50 and 128 residues length. We find that natural sequences tend to cluster much more similarly our designable sequences (Figure SS14) compared to that of prior works (Figure
26

---

Under review as a conference paper at ICLR 2023
-1.0
-0.0
Figure S16: Figure from Trippe et al. (002 These authors similarly cluster unconditionally generated backbones with scT' 2 0.5 using TMscore(z, y) distance metric. Compared to our identical evaluation, illustrated in Figure S14 we notice that this clustering has a few dark blocks of nearly distance, or a TM score of nearly T. This suggests that among these designable backbones, many actually minor variants of structure; actuality, though this work claims to produce 92 designable structures, there seem to be fewer unique structures in this set due many being near-duplicates.
reproduced with permission for ease of reference.
27


Reviews:
[{"review_id": "qNA5Jf13lAD", "reviewer": "Reviewer_84e9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a denoising diffusion probabilistic model for generating protein backbone structures. To overcome the complication of equivariance when working with 3D structures, the approach directly models the angles that describe the orientation of residues relative to their neighbors. Results show that the model generates diverse structures that recapitulate the test distribution to a large extent", "summary_of_the_review": "The paper introduces a novel parametrization of the protein structure generation problem by modeling angles instead of coordinates. Results convincingly show that it has some advantages over directly generating coordinates using equivariant model architectures.", "strength_and_weaknesses": "Strengths\n- Novel approach for modeling 3D structure\n- Convincing performance\n\nWeaknesses\n- Generated proteins are very short (not a problem within the claimed scope, the benchmarking is done well, but will be interesting to see how it scales - with increasing length, there could be an increase in collisions?)\n\nQuestions\n- Does the presented approach really mirror native folding, as claimed in the abstract? I.e. when tracking the diffusion process through time does it recapitulate a folding process? I understand that the angle parametrization is closer to the native process, but that doesn't necessarily mean them model will learn to follow this process when denoising (clashes etc.) \n- Could the authors elaborate on why to use ESM-IF1 over alternatives such as ProteinMPNN, and why to generate exactly 8 sequences?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "hZ7rOKBNgo7", "reviewer": "Reviewer_DFe6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors show that featurizing a protein via angles allows generation of protein structures via diffusion. They provide a technique for diffusing in angular space, and show that it's able to generate interesting looking protein structures. The authors provide many qualitative metrics to compare their model with recent structure generation techniques. The introduced technique seems to improve on angular characteristics of the generated proteins, though it does not seem to generate compact and interesting backbones, resulting in mostly alpha helices.", "summary_of_the_review": "The work itself is reasonably solid, though the generations do not seem signifcantly better than existing works. Additionally, the paper does not solve the problem that structure generation works are hard to compare, which makes it difficult to know the impact of this work. The generated structures are underwhelming.", "strength_and_weaknesses": "Strengths:\n- Many qualitative measure of model performance, ranging from rama plots to scTM scores\n- Diffusing on angles is relatively well motivated, and interesting.\n\nWeaknesses:\n- Like many other works on protein structure generation, this paper does not solve the problem that good metrics are hard. Therefore, it's hard to compare how much better this model does compared to previous works. My suggestion is to produce a set of comprehensive samples, and compare them to other techniques that way.\n- Many metrics selected are claims about overall statistics, not individual proteins. The few samples shown are underwelming\n- The scTM score provided is underwhelming as well, having most scores significantly lower than 0.5. For well generated proteins, I would expect >0.8 to be reasonable. RMSD might be a more interpretable metric to display also - for which we're looking for sub 3 A self consistency.\n- The model seems unable to generate beta sheets, which require the model to better understand the global structure of the protein.\n- Why does the generated samples not show any secondary structure through pymol? That usually indicates to me that the proteins generated have significant deficiencies in stereochemical properties.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LHKsBKignh", "reviewer": "Reviewer_Rmta", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new diffusion-based generative model that designs protein backbone structures via a procedure that mirrors the native folding process. By considering each residue to be its own reference frame, it describes protein backbone structure as a series of consecutive angles capturing the relative orientation of the constituent amino acid residues instead of Cartesian atom coordinates. A vanilla transformer is used to build a diffusion model to generate new protein structures by denoising from a random, unfolded state towards a stable folded structure. Some experiments indicate resulting model generates lifelike protein, better-respecting protein chirality.", "summary_of_the_review": "This paper presents a novel parameterization of protein backbone structures that allows for simplified generative modeling. It trains a denoising diffusion probabilistic model with a simple transformer back-bone and demonstrates resulting model unconditionally generates real-life protein structures.\n", "strength_and_weaknesses": "1. This paper proposes a simplified framing of protein backbones . Unlike viewing a protein backbone of amino acids as a cloud of 3D coordinates, authors view it as a sequence of six internal, consecutive angles. The independence of reference frame of each residue leads to no need to use an equivariant neural network. No matter how the protein is rotated or shifted, the angle of the next residue given the current residue never changes.\n2. As mentioned before, no requirements of the use of equivariant networks leads to the possibility of using simple transformer as the backbone architecture, thus the model directly generates structures without relying on additional methods for refinement. Generated backbones exhibit greater designability compared to prior works that use equivariance assumptions.\n3. It presents a suite of validations to quantitatively demonstrate that unconditional sampling from proposed model directly generates realistic protein backbones – from recapitulating the natural distribution of protein inter-residue angles, to producing overall structures with appropriate arrangements of multiple structural building block motifs.\n4. Abundant numerical experiments are supplied to demonstrate that generated backbones contain reasonable structural motifs and they are designable.\n\n**Questions:**\n1. The effectiveness of the proposed model has been shown via solid experiments. The potential advantages can be known from an explanation of the paper, but there seems to be less compared with other models in terms of performance and prediction results, could you please provide more valid evidence?\n2. Although formulating a protein as a series of angles enables to use simpler models without equivariance mechanisms, this framing allows accumulated errors to significantly alter the overall structure of a generated structure. Could you further analyze the impact of this cumulative error compared to other noise errors based on Cartesian coordinate systems on the prediction results? Does additional refinement similar to the original approach help improve reliability?\n3. The generated structures are still simple compared to natural proteins which typically have several hundred residues and it is of static structures. Could you please further explain the possibility and operability of the proposed model in terms of further extensions?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3bF7Sd7LW4", "reviewer": "Reviewer_t1c7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a diffusion based approach for unconditional generation of realistic protein structures. It is claimed that the generated structures are similar to naturally-occuring proteins in complexity and structural patterns. ", "summary_of_the_review": "The main novelty of this work lies in the protein structure representation, and the introduction of this generative modeling task. Both novelties are located in an application field. The significance of the work is limited because of the lacking experimental part, with only a single task approached, absence of baselines and naive methods, and insufficient metrics. These same things can also be listed as technical problems, some of which are easier to overcome, e.g. adding baselines and other methods, while others are tricky, e.g. to come up with a good metric.", "strength_and_weaknesses": "[+] The representation of the protein backbone as a sequence of angles should be a usefule representation for protein structures\n[+] Introduces a new generative learning problem\n[+] Using diffusion models to generate foldable proteins is a reasonable approach\n[-] Complete absence of informative baselines and other compared methods\n[-] Metrics are performed without repetitions and error bars\n[-] The metrics can easily be tricked and are insufficient to show that this method works\n[-] Low designability of backbones containing beta sheets.\n[-] Although there is novelty concerning the framing of the protein backbone as angles, no truly new methods regarding model architecture were established.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject