Energy-based models for atomic-resolution protein conformations
Authors: Yilun Du
Joshua Meier
Jerry Ma
Rob Fergus
Alexander Rives
Venue:  (2020)
Keywords: energy-based model
transformer
energy function
protein conformation

Abstract: We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model’s outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.

Published as conference paper ICLR 2020
ENERGY-BASED MODELS FOR ATOMIC-RESOLUTION PROTEIN CONFORMATIONS
Yilun Du
Joshua Meier
Jerry Ma
Massachusetts Institute of Technology Cambridge, MA vilundu@mit.ccm edu
Facebook AI Research New York, NY jmeier fb. weieerffb.cmm com
Facebook Al Research Menlo Park, CA ma 1 fb. com
Rob Fergus
Alexander Rives New York University New York, NY arivesêcs nyu. edu
Facebook AI Research New York University
New York, NY robferguse fb. com
ABSTRACT
We propose an energy-based model (EBM) of protein conformations that operates atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close that of the Rosetta energy function, sfte-of-the-art method widely used in protein structure prediction and design. An investigation of the model's ' outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.
INTRODUCTION
Methods for the rational design of proteins make use of complex energy functions that approximate the physical forces that determine protein conformations (Cornell et al.] 1995| Jorgensen et al. 1996| MacKerell al. 1998) incorporating knowledge about statistical patterns in databases of protein crystal structures (Boas & Harbury 2007). The physical approximations and knowledge-derived features that are included protein design energy functions have been developed over decades, building results from large community of researchers (Alford et al. (017).
In this wo investigate learning energy function for protein conformations directly from protein crystal structure data. To this end, we propose an energy-based model using the Transformer architecture (Vaswani ct al. 2017). that accepts as inputs sets of atoms and computes an energy for their configuration. Our work is a logical extension of statistical potential methods (Tanaka & Scheraga 1976] Sippl 1990; Lazaridis & Karplus 2000) that fit energetic terms from data, which, in combination with physically motivated force
'Wokk performed during an internship at Facebook
Data and code for experiments are available httpa///attub..ccnacccboktrrrret
protein-ebm

---

Published as a conference paper at ICLR 2020
fields. have contributed the feasibility of de novo design of protein structures and functions (Kuhlman et al. 2003
Ambroggio & Kuhlman 2006 Jiang et al. 2008 King et al. 2014).
To date, energy functions for protein design have incorporated extensive feature engineering, encoding knowledge of physical and biochemical principles (Boas & Harbury 2007] Alford et al.) (017). Learning from data can circumvent the process of developing knowledge-basee potential functions by automatically discovering features that contribute to the protein's s energy, including terms that are unknown or are difficult to express with rules or simple functions. Since energy functions are additive, terms learned by neural Energy-basess models can be naturally composed with those derived from physical knowledge.
principle, neural networks have the ability to identify and represent non-additive higher order dependencies that might uncover features such as hydrogen bonding networks. Such features have been shown to have
important roles in protein structure and function (Guo & Salahub 1998 Redzic & Bowler 2005 Livesay
ct al. 2008), and are important in protein design (Boyken et al. 2006)). Incorporation of higher order terms has been an active research area for energy function design (Maguire et al. 2018).
Evaluations of molecular energy functions have used as measure of fidelity, the ability to identify native side chain configurations (rotamers) from crystal structures where the ground-truth configuration has been masked out (Jacobson et al. 2002. Bower al. 1997p. Leaver-Fay al. (2013) introduced a of benchmarks for the Rosetta energy function that includes the task of rotamer recovery. In the benchmark, the ground-truie -ruth configuration of the side chain masked and rotamers (possible configurations of the side chain) are sampled and evaluated within the surrounding molecular context (the rest of the atoms in the protein structure not belonging the side chain). The energy function is scored by comparing the lowest-energy rotamer (as determined by the energy function) against the rotamer that was observed in the empirically -eetrrenl crystal structure.
This work takes an initial step toward fully learning an itomic-resolution energy function from data. Prediction of native rotamers from their context within a protein is a restricted problem setting for exploring how neural networks might be used to learn atomicrreoolttin energy function for protein design. We compare the model to the Rosetta energy function, as detailed in Leaver-Fay et al. (2013). and find that on the rotamer recovery task, deep learning-basee models obtain results approaching the performance of Rosetta. We investigate the outputs and representations of the model toward understanding its representation of molecular energies and exploring relationships to physical properties of proteins.
Our results open for future work the more general problem settings of combinatorial side chain optimization for fixed backbone (Tuffery et al. 1991 Holm & Sander 1992) and the inverse folding problem (Pabo) 1983 - the recovery of native sequences for a fixed backbone which has also been used in benchmarking and development of molecular energy functions for protein design (Leaver-Fay et al. 2013).
2 BACKGROUND
Protein conformation Proteins are lincar polymers composed of an alphabet of twenty canonical amino acids (residues), each of which shares common backbone moiety responsible for formation of the linear polymeric backbone chain, and differing side chain moiety with biochemical properties that vary from amino acid to amino acid. The energetic interplay of tight packing of side chains within the core of the protein and exposure of polar residues at the surface drives folding of proteins into stable molecular conformations
(Richardson & Richardson 1989 Dill 1990
The conformation of protein can be described through two interchangeable coordinate systems. Each atom has a set of spatial coordinates, which up to an arbitrary rotation and translation of all coordinates describes a unique conformation. In the internal coordinate system, the conformation is described by sequence of rigid-body motions from each atom to the next, structured as a kinematic tree. The major degrees of freedom in protein conformation the dihedral rotations (Richardson & Richardson 1989), about the backbone
2

---

Published as a conference paper ICLR 2020
bonds termed phi ()) and psi (+)) angles, and the dihedral rotations about the side chain bonds termed chi (x) angles.
Within folded proteins, the side chains of amino acids preferentially adopt configurations that are determined by their molecular structure. A relatively small number of configurations separated by high energetic barriers are accessible to each side chain (Janin al. 1978). These configurations are called rotamers. Rosetta and other protein design methods, rotamers are commonly represented by libraries that estimate probability distribution over side chain configurations, conditioned on the backbone á and to torsion angles. We use the Dunbrack library (Shapovalov & Dunbrack Jr 201T) for rotamer configurations.
Energy-based models A variety of methods have been proposed for learning distributions of high- dimensional data, e.g. generative adversarial networks (Goodfellow et al.) 2014) and variational autoen- coders (Kingma& & Welling 20133. In this work, adopt energy- -ased models (EBMs) (Dayan et al.. [9955]
Hinton & Salakhutdinov 2006;] LeCun et al. (006). This is motivated by their simplicity and scalability, as well as their compelling results other domains, such image generation (Du & Mordatch] 2019).
In EBMs, a scalar parametric energy function Fg (I)) is fit to the data, with 0 set through a learning procedure such that the energy is low in regions around x and high elsewhere. The energy function maps to probability density using the Boltzmann distribution: pa (z) = - (xp(-Eg(r))/Z(0)) where Z exp()Li(()) dr denotes the partition function.
EBMs are typically trained using the maximum likellhood method (ML), in which @ is adjusted to minimize (L(pD(z) (po(()) (z)), the Kullback- divergence between the data and the model distribution. This corresponds to maximizing the log- likeliikooh of the data under the model:
LML (0)
Ex-pp [log (+)] = Ee-po [Ba (z) log [()]
Following Carreira- & Hinton (2005), the gradient of this objective can be written as:
VoLML Epiopo Veli Ex -po Volis()))
Intuitively, this gradient decreases the energy of samples from the data distribution x+ and increases the energy of samples drawn from the model x Sampling from pa can be done in a variety of ways, such as Markov chain Monte Carlo or Gibbs sampling (Hinton & Salakhutdinov 2006), possibly accelerated using Langevin dynamies (Du & Mordatch, 2019). Our method uses simpler scheme to approximate VoLML. detailed in Seection 3.4]
3 METHOD
Our goal is to score molecular configurations of the protein side chains given fixed target backbone structure. We define an architecture for an energy-based model and describe its training procedure.
3.1
MODEL
The model calculates scalar functions, fA), of size-k subsets, A, of atoms within protein.
Selection of atom subsets In our experiments, we choose A to nearest-neighbor sets around the residues of the protein and set l = 64. For a given residue, we construct A to be the k atoms that are nearest to the residue's 's beta carbon.
Atom input representations Each atom in A is described by its 3D Cartesian coordinates and categorical features: (i) the identity of the atom (N, C, O, S); (ii) ordinal label of the atom in the side chain (i.e. which specific carbon, nitrogen, etc. atom it is in the side chain) and (iii) the amino acid type (which of the 20 types of amino acids the atom belongs to). The coordinates normalized to have zero mean across the k
3

---

Published as conference paper ICLR 2020
fo(A)
Conitext atoms
Emberdding laver
MLP (2 kyys)
Position (Xy,2)
(172d)
Amino Acid
(28d)
Max poo1
Atom (N,C,0,5)
(28d)
atomss
Ordinal
(28d)
Attention
Transformer layer
Crystalized Protein
Atom features (256d)
Transformer Model
Figure 1: Overview of the model. The model takes as input set of atoms, A. consisting of the rotamer be predicted (shown in green) and surrounding atoms (shown in dark grey). The Cartesian coordinates and embeddings is processed by Transformer blocks, and the
attributes of each atom are embedded. The final hidden representations are pooled over the atoms to produce vector. The vector is passed through two-layer MLP to output a scalar energy value, (A)).
atoms. Each categorical feature is embedded into 28 dimensions, and the spatial coordinates are projected into 172 dimensions) which are then concatenated into 256-dimensional atom representation. The parameters for the input embeddings and projections of spatial information are learned via training. During training, a random rotation is applied to the coordinates in order to encourage rotational invariance of the model. For visualizations, a fixed number of random rotations (100) is applied and the results are averaged.
Architecture In our proposed approach, (e) takes the form of Transformer model (Vaswani et al. 2017) that processes set atom representations. The self-attention layers allow each atom to attend the representations of other atoms in the set, modeling the energy of the molecular configuration non-linear interaction of single, pairwise, and higher-order interactions between the atoms. The final hidden representations of the Transformer pooled across the atoms produce single vector, which finally passed to a two-layer multilayer perceptron (MLP) that produces the scalar output the model. Figure illustrates the model.
For all experiments, we use a 6-layer Transformer with embedding dimension of 256 (split over 8 attention heads) and feed-forward dimension of 1024. The final MLP contains 256 hidden units. The models are trained without dropout. Layer normalization Ba et al.
2016 is applied before the attention blocks.
3.2 PARAMETERIZATION OF PROTEIN CONFORMATIONS
The structure of a protein can be represented by two parameterizations: (1) absolute Cartesian coordinates of the set of atoms, and (2) internal coordinates of the atoms encoded and displacements relative to each atom's 's reference frame. Out-of plane rotations are parameterized by angles which are the primary degrees of freedom in the rotamer configurations. The coordinate systems are interchangeable.
t--plane/outoo-plaae rotations
"hhe high dimensionality of the spatial projection important ensure high weighting the spatial coordinates, which proved necessary for the model to train reliably.
4

---

Ea(2,c) Published as a conference paper ICLR 2020
3.3 USAGE AS AN ENERGY FUNCTION
We specify our energy function Eg c) to take input set composed of two parts: (1) the atoms belonging a rotamer be predicted, I; and (2) the atoms the surrounding molecular context, C. The energy function is defined as follows:
Eo( (o(r
where A(x,c) c) is the set of embeddings from ls atoms nearest the rotamer's beta carbon.
3.4 TRAINING AND LOSS FUNCTIONS
all experiments, the energy function trained to learn the conditional distribution of the rotamer given its context by approximately maximizing the log likelihood of the data.
[()) = - E c) log Zo(c)
To estimate the partition function, we note that:
log Zø(c) = log
E-Eo(a.0) dx = log log(BB(z(e) g(x c)
for some importance sampler q(zlc). Furthermore, if we assume ((xr(c) is uniformly distributed on supported configurations, we obtain simplified maximum likelihood objective given by
(()) = Eo, c) log(B (el
for some context dependent importance sampler q (x) c). We choose sampler ((zcc) to be empirically collected rotamer library (Shapovalov & Dunbrack Jr, 2011) conditioned on the amino acid identity and the backhone and v angles. We write the importance sampler interchangeable with the angular coordinates in the rotamer library. The library consists of lists of means and standard deviations of possible x angles for each 10 degree interval for both Φ and yb We sample rotamers uniformly from this library, given by a continuous ø and vh by sampling from weighted mixture of Gaussians of X angles at each of the four surrounding bins, with weights given by distance the bins via bilinear interpolation. Every candidate rotamer at each bin is assigned uniform probability. To ensure our context dependent importance sampler effectively samples high likelihood areas in the model, we further add the real context as a sample from g(ælc).
function of atomic coordinates which are
Training setup Models were trained for 180 thousand parameter updates using 32 NVIDIA V100 GPUs, a batch size of 16,384, and the Adam optimizer (a = 2 10 ,1 = 0.99, B3 0.999). We evaluated training progress using held-out 5% subset of the training data as a validation set.
4 EXPERIMENTS
4.1 DATASETS
We constructed a curated dataset of high-resolution PDB structures using the CullPDB database, with the following criteria: resolution finer than 1.8 A: sequence identity less than 90%; and R value less than 0.25 defined in Wang & R. L. Dunbrack (2003). To test the model rotamer recovery, structures from Leaver-Fay Fay et al. (2013). To prevent training on structures that are similar to those in the test set, ran BLAST on sequences derived from the PDB structures and removed all train structures with more than 25% sequence identity to sequences the test dataset. Ultimately, our train dataset consisted of 12,473 structures and our test dataset consisted of 129 structures.
the test set of
5

---

Published as a conference paper ICLR 2020
Model
Avg
Buried
Surface
Rosetta score12 (rotamer-trials)
72.2 (72.6)
Rosetta 5 (rotamer-trials)
73.6
Atom Transformer
70.4
87.0
58.3
Atom Transformer (ensemble)
71.5
89.2
59.9
Table 1: Rotamer recovery of energy functions under the discrete rotamer sampling method detailed in Section Parentheses denote value reported by Leaver-Fay al. (2013).
4.2.J
4.2
BASELINES
We compare three baseline neural network architectures: fully-connected network, the architecture for embedding sets in the set2set paper (Vinyals et al., 2015); and a graph neural network (Veličković et al.) 2017). All models have around 10 million parameters. Details of the baseline architectures are given in Appendix A.1.2
Results are also compared Rosetta. We ran Rosetta using score 12 and and ref15 energy functions using the rotamer trials and rtmin protocols with default settings.
4.2.1 EVALUATION
For the comparison of the model to Rosetta in Table we reimplement the sampling scheme that Rosetta uses for rotamer trials evaluation. We take discrete samples from the rotamer library, with bilinear interpolation of the mean and standard deviations using the four grid points surrounding the backbone & and zo angles for the residue. We take discrete samples of the rotamers at μ., except that for buried residues we sample X1 and X2 at y and +t a as was done in Leaver-Fay et al. (013).. We define buried residues to have 24 Cs neighbors within 10Å of the residue's 's C3 (Ca for glycine residues). For buried positions we accumulate rotamers up to 95%. We score rotamer as recovered
98% of the distribution, and for other positions the accumulation correctly all angles are within 20 of the ground-truth residue.
We also use continuous sampling scheme which approximates the empirical conditional distribution of the rotamers using mixture of Gaussians with means and standard deviations computed by bilinear interpolation above. Instead of sampling discretely, the component rotamers are sampled with the probabilities given by the library, and a sample is generated with the corresponding mean and standard deviation. This is the same sampling scheme used train models, but with component rotamers now weighted by probability as opposed to uniform sampling.
4.3 ROTAMER RECOVERY RESULTS
Table directly compares our EBM model (which we refer to as the Atom Transformer) with two versions of the Rosetta energy function. We run Rosetta on the set of 152 proteins from the benchmark of Leaver--Fay jet al. (2013). We also include published performance on the same test set from Leaver-Fay et al. (2013). As discussed above, comparable sampling strategies are used to evaluate the models, enabling a fair comparison the energy functions. We find that a single model evaluated on the benchmark performs slightly worse than both versions of the Rosetta energy function. An ensemble of 10 models improves the results.
Table 2 evaluates the performance of the energy function under alternative sampling strategies with the goal of optimizing recovery rates. We indicate performance of the Rosetta energy function recovery rates using the rtmin protocol for continuous minimization. We evaluate the learned energy function with the continuous sampling from mixture of Gaussians conditioned the d/t settings of the backbone angles as detailed
6

---

Published as conference paper ICLR 2020
Model
Avg
Buried
Surface
Fully-connected
39.1
54.4
30.0
Set2set
43.2
60.3
31.7
GraphNet
69.0
94.3
54.2
Atom Transformer
73.1
91.1
58.3
Atom Transformer (ensemble)
74.1
91.2
59.5
Rosetta score 12 (rt-min)
75.4 (74.2)
Rosetta ref2015 (rt-min) 76.4
Table 2: Rotamer recovery of energy functions under continuous optimization schemes. Rosetta continuous optimization is performed with the rtmin protocol. Parentheses denote value reported by (2013).
Leaver-Fay et al.
Amino Acid
K
M
S
V
Atom Transformer
37.2
31.7
53.0
93.3
82.6
79.0
96.5
94.0
Rosetta score12
26.7
31.7
49.6
85.4
87.5
72.5
92.6
94.3
Amino Acid
N
D
E
W
F
Y
Atom Transformer
67.4
76.0
40.8
49.8
65.5
83.5
80.3
77.6
Rosetta score
56.8
60.4
30.7
33.6
55.0
85.0
85.4
82.9
Table 3: Comparison of rotamer recovery rates by amino acid between Rosetta and the ensembled energy- based model under discrete rotamer sampling. The model appears to perform well on polar amino acids glutamine, serine, asparagine, and threonine, while Rosetta performs better larger amino acids phenylala- nine, tyrosine, and tryptophan and the common amino acid, leucine. The numbers reported for Rosetta are from Leaver-Fay et al. (2013).
above. We find that with ensembling the model performance is close to that the Rosetta energy functions. We also compare to three baselines for embedding sets with similar numbers of parameters to the Atom Transformer model and find that they have weaker performance.
Buried residues of the protein. In comparison, surface residues are more free to vary. Therefore we also report performance separately both categories. We find that the ensembled Atom Transformer has 91.2% rotamer recovery rate for buried residues, compared to 59.5% 5% for surface residues.
constrained in their configurations by tight packing of the side chains within the core
Table 3 reports recovery rates by residue comparing the Rosetta 1core12 results reported in Leaver- Fay et al. (2013) to the Atom Transformer model using the Rosetta discrete sampling method. The Atom Transformer model appears to perform well smaller rotameric amino acids well as polar amino acids such glutamate/aspartate while Rosetta performs better on larger amino acids like phenylalanine and tryptophan and more common ones like leucine.
4.4 VISUALIZING ENERGIES
this section, we visualize and understand how the Atom Transformer models the energy rotamers in their native contexts. We explore the response of the model to perturbations in the configuration of side chains away from their native state. We retrieve all protein structures in the test set and individually perturb rotameric x angles across the unit circle, plotting results in Figures 2 and 4
7

---

Published as conference paper ICLR 2020
15
10-
lcu
03
169
hs
140
twlative
0.
120
100
pr
upp
Surface
50
455
-190 -30
90
180
0.2: 1T
Figure 2: The energy function models distinct be- havior between core and surface residues. Core residues are
Figure 3: There is relation between the residue size and the depth of the energy well, with larger amino acids (e.g. Trp, Phe, Thr, Lys) having steeper wells.
sensitive to perturbations away from the native state in the torsion angle. On average, residues closer the core have steeper energy well.
Core/Surface Energies Figure shows that steeper response to variations away from the native state observed for residues in the core of the protein (having 24 contacting side chains) than for residues on the surface (=16), consistent with the observation that buried side chains are tightly packed (Richardson Richardson 1989).
Rotameric Energies Figure shows a relation between the residue size and the depth of the energy well, with larger amino acids having steeper wells (more sensitive to perturbations). Furthermore Figure that the model learns the symmetries of amino acids. We find that responses perturbations of the X2 angle for the residues Tyr, Asp, and Phe are symmetric about X2. 180% periodicity is observed, contrast to the non-symmetric residues.
Embeddings of Atom Sets Building the observation of relation between the depth of the residue and its response to perturbation from the native state, we ask whether core and surface residues are clustered within the representations of the model. To visualize the final hidden representation of the molecular contexts within a protein, we compute the final vector embedding for the 64 atom context around the carbon-5 atom (or for glycine, the carbon-ar atom) for each residue. We find that projection of these representations by 1-SNE (Maaten & Hinton 2008) into dimensions shows a clear clustering between representations of core residues and surface residues. representative example is shown
Figure 5
Saliency Map The 10-residue protease-binding loop in a chymotrypsin inhibitor from barley seeds highly structured due the presence of backbone- backbone and bockkone-sidechaan hydrogen bonds in the same residue (Das) 2011). To visualize the dependence of the energy function on individual atoms, we compute the energy of the 64 atom con- text centered around the backbone carbonyl oxygen of residue 39 (isoleucine) PDB: 2C12 (McPhalen & James 1987) and derive the gradients with respect to the input atoms. Figure 60verlay the gradient magnitudes the structure, indicating the model at- tends to both sidechain and backbone atoms, which participate in hydrogen bonds.
Tyr
oH
Asp
OH
-62
Phe
OH
380
Figure Note the periodicity for the amino acids Tyr, Asp. and Phe with terminal symmetry about X2.
8

---

Published as conference paper ICLR 2020
Core
Figure 5: Left: 3-dimensional representation of CemG reducing oxidoreductase (PDB ID IKNG; Edeling ct al. 2002). protein from the test set. Atoms colored dark blue (buried), orange (exposed), neither (not colored). Right: t-SNE (Maaten & Hinton 2008) projection EBM hidden representation when focused the alpha carbon atom for each residue in the hidden representation. In the embedding space, buried and surface residues are distinguished.
Figure 6: The model's saliency map applied to the test protein, serine proteinase inhibitor (PDB ID: 2C12: McPhalen & James (1987)) The atom context centered on the carbonyl oxygen of residue 39 (isoleucine). Atoms in the context are labeled red with color saturation proportional to gradient magnitude (interaction strength). Hydrogen bonds with the carbonyl oxygen are shown by dotted lines.
5 RELATED WORK
Energy functions have been widely used in the modeling of protein conformations and the design of protein sequences and structures (Boas Harbury 2007). Rosetta, for example, uses combination physi- cally motivated terms and knowledge-based potentials (Alford et al. 2017) to model proteins and other macromolecules.
Leaver-Fay et al. (2013) proposed optimizing the feature weights and parameters of the terms of energy function for protein design; however their method used physical features designed with expert knowledge and data analysis. Our work draws their development of rigorous benchmarks for energy functions, but contrast automatically learns complex features from data.
Neural networks have also been explored for protein folding. Xu 2018 developed deep residual network that predicts the pairwise distances between residues in the protein structure from evolutionary covariation information. (Senior et al. (2088) used evolutionary covariation to predict pairwise distance distributions,

---

Published as a conference paper ICLR 2020
using maximization of the probability of the backbone structure with respect to the predicted distance distribution to fold the protein. Ingraham et al. (2018) proposed learning an energy function for protein folding by backpropagating through differentiable simulator. /AlQuraishi (2019) investigated predicting protein structure from sequence without using co-evolution.
Deep learning has shown practical utility in the related field of small molecule chemistry. Gilmer et al. (2017) achieved state-of-the-arr performance suite of molecular property benchmarks. Similarly, Feinberg et al. (2018) achieved state-of-the-ard performance on predicting the binding affinity between proteins and small molccules using graph convolutional networks, Mansimov et al. (2019) used graph neural network to learn an energy function for small molecules. In contrast to our work, these methods operate over small molecular graphs and were not applied large macromolecules, like proteins.
In parallel, recent work proposes that generative models pre-trained on protein sequences can transfer knowledge to downstream supervised tasks (Bepler & Berger 2019; Alley et al.) 22019; Yang al. 2019
Rives et al. 2019). These methods have also been explored for protein design (Wang et al. 2018).
Generative models of protein structures have also been proposed for generating protein backbones (Anand & Huang 2018) and for the inverse protein folding problem (Ingraham et al. 2019).
6 DISCUSSION
In this work we explore the possibility of learning an energy function of protein conformations atomic resolution. We develop and evaluate the method in the benchmark problem setting of recovering protein side chain conformations from their native context, finding that a learned energy function nears the performance in this restricted domain to energy functions that have been developed through years of research into approximation of the physical forces guiding protein conformation and engineering of statistical terms.
The method developed here models sets of atoms and can discover and represent the energetic contribution of high order dependencies within its inputs. We find that learning an energy function from the data of protein crystal structures automatically discovers features relevant to computing molecular energies; and we observe that the model responds its inputs in ways that are consistent with an intuitive understanding of protein conformation and energy.
Generative biology proposes that the design principles used by nature can be automatically learned from data and can be harnessed to generate and design new biological molecules and systems (Rives et al.) 2019). High-fidelity generative modeling for proteins, operating at the level of structures and sequences, can enable generative protein design. To create new proteins outside the space of those discovered by nature, it is necessary to use design principles that generalize to all proteins. [Huang et al. (2016) have argued that since the physical principles that govern protein conformation apply to all proteins, encoding knowledge of these physical and biochemical principles into an energy function will make possible to design de novo new protein structures and functions that have not appeared before in nature.
Learning features from data with generative methods is a possible direction for realizing this goal to enable design in the space of sequences not visited by evolution. The generalization of neural energy functions to harder problem settings used in the protein design community, e.g. combinatorial side chain optimiza- tion (Tuffery et al., [191 Holm & Sander, 1992), and fnverig-folding (Pabo) 1983). is direction for future work. The methods explored here have the potential for extension into these settings.
ACKNOWLEDGMENTE
We thank Kyunghyun Cho, Siddharth Goyal, Andrew Leaver-Fay, and Yann LeCun for helpful discussions. Alexander Rives was supported by NSF Grant #1 #1339362.
10

---

Published as conference paper ICLR 2020
REFERENCES
Rebecca Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov. P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6))3031-3044 2017.
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-only deep representation learning. bioRxiv, 2019. doi: 10.1 589333. /ttpa///////////////////////////
Mohammed AlQuraishi. End-to-end differentiable learning of protein structure. Cell Systems, 8(4):292 URL htLL:ttp:///
301.e3, 2019. ISSN 2405-4712. doi: http:://ooooorr//./...oooo...... sclencedirect cttee/ctccc.eeeeeeeeeeeeeeeeeee http:///ww........o.ooooooo////a
Xavier Ambroggio and Brian Kuhlman. Computational design of a single amino acid sequence that can switch between two distinct protein folds. Journal of the American Chemical Society, 128(4): 116,, 2006.
Namrata Anand and Possu Huang. Generative modeling for protein structures. In ICLR, 2018.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv e-prints, art.
arXiv:1601 004500, Jul 2016.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
F Edward Boas and Pehr B Harbury. Potential energy functions for protein design. Current opinion in structural biology, 17(2):199-204, 2007.
Michael J Bower, Fred E Cohen, and Roland L. Dunbrack Jr. Prediction of protein side-chain rotamers from backbone-dependent rotamer library: a new homology modeling tool. Journal of molecular biology, 267 (5):1268-1282. 1997.
Scott E Boyken, Zibo Chen, Benjamin Groves, Robert A Langan, Gustav Oberdorfer, Alex Ford, Jason M Gilmore, Chunfu Xu, Frank DiMaio, Jose Henrique Pereira, al. De novo design of protein homo- oligomers with modular hydrogen-bond network-mediated specificity. Science, 352(6286):680-687,, 2016.
Miguel A Carreira-Perpinal and Geoffrey E Hinton. On contrastive divergence learning. In Aistats, volume 10, pp. 33-40. Citeseer, 2005.
Wendy D Cornell, Piotr Cieplak, Christopher I Bayly, Ian R Gould, Kenneth M Merz, David M Ferguson, David C Spellmeyer, Thomas Fox, James W Caldwell, and Peter A Kollman. A second generation force field for the simulation of proteins, nucleic acids, and organic molecules. Journal of the American Chemical 1995.
Society, 117(19):5179-511
R. Das. Four small puzzles that Rosetta doesn't 't solve. PLoS ONE, 6(5)ee2044,, 2011.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard Zemel. The helmholtz machine. Neural computation, 7(5):889-904, 1995.
Ken A Dill. Dominant forces in protein folding. Biochemistry, 99(31):1133-7555 1990.
11

---

Published as conference paper ICLR 2020
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv 1903.08689, 2019.
Melissa A Edeling, Luke W Guddat, Renata A Fabianek, Linda Thöny-Meyer, and Jennifer L Martin. Structure of cemg/dsbe at 1.14 à resolution: high- high-fidelity reducing activity in an indiscriminately oxidizing environment. Structure, 10(7):973-99) 2002.
Evan N. Feinberg, Debnil Sur, Zhenqin Wu, Brooke Husic, Huanghao Mai, Yang Li, Saisai Sun, Jianyi Yang. Bharath Ramsundar, and Vijay S. Pande. Potentialnet for molecular property prediction. ACS
Central Science, 4())4:15111100000 Nov 2018. ISSN 2374-7943. doi: 10. htt.:esseeenssi.s.00000000 URL
https: //o0///dooi.o .rrg/ /tttps://doi..rg111..111 1tt.1/0c///ccccnscccccccccc 8b00507
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL 1a.. .hmml
pp.
http://proceed/ wttp:///www......../ee..o//oo///
mww.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL http::/paperss httpssccc/ppprr 5423
pdf
Hong Guo and Dennis Salahub, Cooperative hydrogen bonding and enzyme catalysis. Angewandle Chemie International Edition, 37(21):2985-2999 1998.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 5134 2006.
Lisa Holm and Chris Sander. Fast and simple monte carlo algorithm for side chain optimization in proteins: application to model building by homology. Proteins: Structure, Function, and Bioinformatics, 14(2): 213-223, 1992.
Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537 (7620):320, 2016.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with differentiable simulator. 2018.
John Ingraham, Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. 2019.
Matthew Jacobson, George A Kaminski, Richard A Friesner, and Chaya S Rapp. Force field validation using protein side chain prediction. The Journal of Physical Chemistry B, 106(44):11633-1166) 2002.
Joel Janin, Shoshanna Wodak, Michael Levitt, and Bernard Maigret. Conformation of amino acid side-chains in proteins. Journal of molecular biology, 125 1978.
Lin Jiang, Eric A Althoff, Fernando R Clemente, Lindsey Doyle, Daniela Röthlisherger, Alexandre Zanghellini, Jasmine L Gallaher, Jamie L Betker, Fujie Tanaka, Carlos F Barbas, et al. De novo computa- tional design of retro-aldol enzymes. science, 11977868))1887113999 2008.
12

---

Published as conference paper ICLR 2020
William L Jorgensen, David S Maxwell, and Julian Tirado-Rives. Development and testing of the opls all-atom force field on conformational energetics and properties of organic liquids. Journal of the American Chemical Society, 118(45): 1100:-11222212211 1996.
Neil King, Jacob B Bale, William Sheffler, Dan E. McNamara, Shane Gonen, Tamir Gonen, Todd O Yeates, and David Baker. Accurate design of co-assembling multi-component protein nanomaterials. Nature, 510 (7503):103, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Brian Kuhlman, Gautam Dantas, Gregory C Ireton, Gabriele Varani, Barry L. Stoddard, and David Baker. Design of novel globular protein fold with atomic-level aceuracy. science, 302(5649) 2003.
Themis Lazaridis and Martin Karplus. Effective energy functions for protein structure prediction. Current opinion in structural biology. 10(2):199-145, 2000.
Andrew Leaver-Fayy Fay, Matthew O' Meara, Mike Tyka, Ron Jacak, Yifan Song, Elizabeth H Kellogg, James Thompson, Ian W Davis, Roland A Pache, Sergey Lyskov, et al. Scientific benchmarks for guiding macromolecular energy function improvement. In Methods in enzymology, volume 523, pp. 109-143. Elsevier, 2013.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Dennis R Livesay, Dang Huynh, Sargis Dallakyan, and Donald J Jacobs. Hydrogen bond networks determine emergent mechanical and thermodynamic properties across protein family, Chemistry Central Journal, 2(1):17, 2008.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov): 2008.
Alex MacKerell Jr, Donald Bashford, MLDR Bellott, Roland Leslie Dunbrack Jr, Jeffrey Evanseck, Martin J Field, Stefan Fischer, Jiali Gao. H Guo, Sookhee Ha, et al. All-atom empirical potential for molecular modeling and dynamics studies of proteins. The journal of physical chemistry B, 102(18): 3586-3616, 1998.
Jack B Maguire, Scott E Boyken, David Baker, and Brian Kuhlman. Rapid sampling hydrogen bond networks for computational protein design. Journal of chemical theory and computation, 14(5):2751-2760, 2018.
Elman Mansimov, Omar Mahmood, Seokho Kang. and Kyunghyun Cho. Molecular geometry prediction using a deep generative graph neural network. arXiv preprint arXiv:1904.00 190.0003444 2019.
CA McPhalen and MNG James. Crystal and molecular structure of the serine proteinase inhibitor ci-2 from barley seeds. Biochemistry, (6(1):261-269 1987.
Carl Pabo. Molecular technology: designing proteins and peptides. Nature, 351(7897) 1983.
Jasmina S Redzic and Bruce E Bowler. Role of hydrogen bond networks and dynamics in positive and negative cooperative stabilization of protein. Biochemistry, 44(8):2900- 2908,, 2005.
Jane Richardson and David C Richardson. Principles and patterns of protein conformation. In Prediction of protein structure and the principles of protein conformation, pp. 1-98. Springer, 1989.
13

---

Published as conference paper ICLR 2020
Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma. and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, 2019. doi: 10.1101/622803, URL htthht:p///wwi wttp:///wwwwwoxo..oooog.....
cotteenn//aally/2015////////// 622803
Andrew Senior, John Jumper, and Demis Hassabis. AlphaFold: Using Al for scientific discovery, 12 2018.
com
Maxim V Shapovalov and Roland I. Dunbrack Jr. A smoothed backbone-dependdntt rotamer library for proteins derived from adaptive kernel density estimates and regressions. Structure, 19(6):8448558 2011.
Manfred Sippl. Calculation of conformational ensembles from potentials of mena force: an approach the knowledge-baad baaed prediction of local structures in globular proteins. Journal of molecular biology, 213(4); 859-883, 1990.
Seiji Tanaka and Harold Scheraga. Medium-and long-range interaction parameters between amino acids for predicting threc-dimensional structures of proteins. Macromolecules, 9(6)9955 950, 1976.
P Tuffery, C Etchebest, Serge Hazout, and R Lavery. A new approach to the rapid determination of protein side chain conformations. Journal of Biomolecular structure and dynamics, 8(6): 1667-129,,, 1991.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06333 2015.
G. Wang and Jr. R. L. Dunbrack. Pisces: protein sequence culling server. Bioinformatics, 19:1589 1591, 2003.
Jingxue Wang, Huali Cao, John Z. H. Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific Reports, 8(1):6349, 2018. ISSN 2045-2322, doi: 10.1088844159--018-24777777 [ttp::t:s/// wtt.://rr......1./....111 10111//1118---112222278-88 010-5988-18-2276--11
Jinbo Xu. Distance-based protein folding powered by deep learning. arXiv preprint arXiv:1811.03448) 2018.
Kevin K. Yang. Zachary Wu, and Frances H. Arnold. Machine-leaming-guns directed evolution for protein
engineering. Nature Methods, 16(8):687-694 2019, ISSN 1548-7105. doi:
httpttts:: //dp:s::////....// /
14

---

Published as conference paper ICLR 2020
A.1 APPENDIX
A.1.1 PSEUDOCODE FOR THE TRAINING ALGORITHM
Pseudocode for the training algorithm is given below Algorithm
Algorithm 1 Training Procedure for the EBM
Input: Rotamer library (<zlc), Training set of proteins D for Protein d; of D do
Sample random amino acid from d:
R dj
Set positive sample to 64 nearest neighbor atoms of carbon beta of R
NN6a R))
Generate N negative samples from the rotamer library
c + -(<x(ct))))) )
Compute loss of model (logsumexp across all negative samples)
Lmd = EEcc=; 0) + logsumexp( E(c=; ()) - -E(co 0)), -.,
Minimization step of Lmnt using Adam optimizer
6-0 0 V Lmm
end for
A.1.2 MODEL ARCHITECTURE
Architectural descriptions are provided below for each the three neural network baselines as well as for the Atom Transformer. For Set2set models, use 6 processing steps of computation. For graph networks, we add residual connections between each layer.
Embed Each Atom 256 Dim
Embed Each Atom to 256 Dim
Flatten
Dense 1024
Dense + 1024
Repeat (6x):
1024 + 1024
LSTM 2048
1024 + 1024
Attention 2048 + 128
ResBlock down 256
End Repeat
Global Mean Pooling
Dense - 1024
Dense
1024 - 1
(a) Fully Connected Model
(b) Set2Set Model (6 Permutation Invariant Blocks)
Figure A1: Architectures for Fully Connected and Set2Set Baselines
15

---

Published as a conference paper ICLR 2020
Embed Each Atom to 512 Dim
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Graph Attention Layer
Global Average Pooling
dense +
Figure A2: Graph Network Model Architecture (9 Graph Attention Blocks)
Embed Each Atom to 256 Dim
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Transformer Encoder Block (8 heads, feedforward dim 1024, 256 encoder dim)
Global Max Pooling
dense
Figure A3: Atom Transformer Model (6 Transformer Encoder Blocks)
16


Reviews:
[{"review_id": "Skgupl9TYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an Energy-Based-Model (EBM) for scoring the possible configurations of amino acid side chain conformations in protein structures with known amino acid backbone structure. The energy of the side-chain conformation (the chi-angle) for a given amino acid in the structure is calculated as a function of a local neighbourhood of atoms (A), where each atom is embedded into a 256d vector using its cartesian coordinates, atom identity, atom side-chain position and amino acid identity. The model is trained using approximate likelihood where the model samples are generated using precalculated table (from literature) of possible Chi angles conformations conditioned on the back-bone amino acid identity and back-bone angles. The results seem comprehensive comparing the transformer based energy function parameterization with two sensible baselines as well as the Rosetta energy function  which is the de facto standard tool for these types of calculations. Using rotamer recovery accuracy as the benchmark measure the empirical results are close to performance as the Rosetta energy model however always slightly worse. Further visualizations of the energy levels for different Chi angles seems to support that the learned energy function captures well known characteristics of the rotamer configuration energy landscape.\n\nScore:\nOverall I think that the paper is solid and tackles an interesting problem of learning energy functions for physical systems from data. The experimental results are comprehensive and mostly supports the claims made in the paper. My main (slightly minor) concern about the paper is the obsession with discarding years of learned knowledge about handcrafted energy functions with fully learned functions. It seems to me that combining domain knowledge with a learned model should close the last small gap (and presumably surpass) the performance of e.g Rosetta. Combining I think the paper should be accepted at the ICLR.\n\nComment/Questions:\n\nMotivation\nQ1.1) Paper motivation: The paper several times seems to suggest that energy functions derived from physical knowledge are problematic (e.g. abstract). For many physical systems I don’t think this is true and would like the author's comments on why learned energy functions are preferable - arguably they can only capture properties present in the data and not any prior knowledge?  \n\nQ1.2) Given that the paper is motivated by fully learning an energy function from data without injecting any prior knowledge i would like the authors comment on the construction of the q(x|c) distribution. This is based on essentially a contingency table between Phi/Psi/Chi angles and to me seems like injecting prior knowledge about physically possible rotamer configurations into the learned procedure? \n\nMethod / Experimental Results:\nQ2.1) With respect to the primary results in table 1) and table 2). The authors claim comparable results to the Rosetta Energy function (page 5. Sec 4.3, page 9, sec 6). However the experimental results are always slightly worse than the Rosetta energy function and I (strongly) suggest that the authors rephrase those statements to reflect that.\n\nQ2.2 With Respect to Table 3) Firstly, Why are the only results for 16 amino acids in the table ? Secondly, just naively counting the Atom Transformer are better than Rosetta in 11 of 16 amino acids - this seems slightly at odds with the main result where the Atom Transformer is performing slightly worse than Rosetta?\n\nClarity\nQ3): The notation in section 3 and 4 is slightly confusing. Especially I think q(x|c) is slightly misleading since it, (to my understanding) is a conditional distribution over Chi, conditioned on Psi/Phi/AminoAcid and not rotamer (x) and surrounding molecular context (c). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxY2dsAFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper\n\nThe authors propose a predictive model based on the energy based model that uses a Transformer architecture for the energy function. It accepts as input an atom and its neighboring atoms and computes an energy for their configuration. The input features include representations of physical properties (atom identity, atom location within a side chain and amino-acid type) and spatial coordinates (x, y, z). A set of 64 atoms closest to the beta carbon of the target residue are selected and each is projected to a 256-dimensional vector. The predictive model computes an energy for the configuration of these 64 atoms surrounding a residue under investigation. The model is reported to achieve a slightly worse but comparable performance to the Rosetta energy function, the state-of-the-art method widely used in protein structure prediction and design. The authors investigate model’s outputs and hidden representations and conclude that it captures physicochemical properties relevant to the protein energy in general.\n\nStrengths\n+ A very interesting contribution to structural biology with a non-trivial application of deep learning.\n+ The study is accompanied with structural biology interpretation of the designed energy predictor. Usually similar studies do not attempt and are limited with prediction rates and mathematical analysis.\n+ Appropriate background is provided for non-experts in protein structural biology. The paper is clear and well written.\n+ An adequate literature review is provided relative to the problem. Bird-eye view is given for future direction with justified optimism.\n\nWeaknesses:\n- The study claims to provide an energy predictor in general while training is performed on rotamer recovery of a single amino acid. These claims should be downplayed and what was shown at most is that this predictor can be applied to predict energy for a restricted problem of the rotamer recovery task. \n- A reader gets a wrong impression at the beginning of the manuscript that the study tries to solve general and classical rotamer prediction for an entire protein. It becomes clear only at the very end that the study does not try to resolve all rotamer conformations in a protein, it can only predict one rotamer at the time given the atoms surrounding the target residue are correct. This should be explained in the beginning that the study does not attempt combinatorial side chain optimization for a fixed backbone;\n- In-depth description of the neural network is required at least in supplemental materials, so that the study could be replicated. Ideally, a working application and training protocol code would be available.\n- No techniques against overtraining are discussed. How was the model validated?\n\nSmall issues:\n* function f_theta(A) is used before it is introduced. \n* noun is missing: using our trained with deep learning.\n* articles missing: that vary from amino acid to amino acid.\n* KL abbreviation not explained\n* MCMC abbreviation not explained\n* wrong artile:  that processes the set of atom representations.\n* misprint: resolution fine r than 1.8\n* wrong word, has to be “lower”: sequence identity greater ˚\n* everyday word: break out\n* abbreviation not explained: t-SNE\n* misprint case: Similarly, In contrast to our work\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg4BP-N5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an energy based model (learned using the transformer architecture) to learn protein conformation from data describing protein crystal structure (rather than one based on knowledge of physical and biochemical principles). To test the efficacy of their method, the authors perform the task of identifying rotamers (native side-chain configurations) from crystal structures, and compare their method to the Rosetta energy function which is the current state-of-the-art. \n\n+ Presents an important and emerging application of neural networks\n+ Relatively clearly written, e.g. giving good background on protein conformation and related things to a machine learning audience\n\n-The baselines that are compared to are set2set and Rosetta. Are there newer baselines the authors can compare to? For example, the related works section discusses some recent works.\n-I am having a hard time interpreting the numbers in the experimental results (Table 1 and Table 2). I understand that they are rotamer recovery rates and that the max would be 100%. But I can’t tell what amount of difference is significant. \n-Figure 1 should have a caption clarifying notation etc. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a data-driven approach to learning atomic-resolution energy functions. Experiment results show that the proposed energy function is similar to the state-of-art method (Rosetta) based on physical principles and engineered features. 

The paper addresses an interesting and challenging problem. The results are very promising. It is a good showcase of how ML can be applied to solve an important application problem. 

For the final version, we suggest that the authors can tune down some claims in the paper to fairly reflect the contribution of the work. 
Decision: Accept (Spotlight)