Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes an energy based model (learned using the transformer architecture) to learn protein conformation from data describing protein crystal structure (rather than one based on knowledge of physical and biochemical principles). To test the efficacy of their method, the authors perform the task of identifying rotamers (native side-chain configurations) from crystal structures, and compare their method to the Rosetta energy function which is the current state-of-the-art. 

+ Presents an important and emerging application of neural networks
+ Relatively clearly written, e.g. giving good background on protein conformation and related things to a machine learning audience

-The baselines that are compared to are set2set and Rosetta. Are there newer baselines the authors can compare to? For example, the related works section discusses some recent works.
-I am having a hard time interpreting the numbers in the experimental results (Table 1 and Table 2). I understand that they are rotamer recovery rates and that the max would be 100%. But I can’t tell what amount of difference is significant. 
-Figure 1 should have a caption clarifying notation etc. 

review_title: Official Blind Review #2477

final_score: rating: 8: Accept
review_content: Summary of the paper

The authors propose a predictive model based on the energy based model that uses a Transformer architecture for the energy function. It accepts as input an atom and its neighboring atoms and computes an energy for their configuration. The input features include representations of physical properties (atom identity, atom location within a side chain and amino-acid type) and spatial coordinates (x, y, z). A set of 64 atoms closest to the beta carbon of the target residue are selected and each is projected to a 256-dimensional vector. The predictive model computes an energy for the configuration of these 64 atoms surrounding a residue under investigation. The model is reported to achieve a slightly worse but comparable performance to the Rosetta energy function, the state-of-the-art method widely used in protein structure prediction and design. The authors investigate model’s outputs and hidden representations and conclude that it captures physicochemical properties relevant to the protein energy in general.

Strengths
+ A very interesting contribution to structural biology with a non-trivial application of deep learning.
+ The study is accompanied with structural biology interpretation of the designed energy predictor. Usually similar studies do not attempt and are limited with prediction rates and mathematical analysis.
+ Appropriate background is provided for non-experts in protein structural biology. The paper is clear and well written.
+ An adequate literature review is provided relative to the problem. Bird-eye view is given for future direction with justified optimism.

Weaknesses:
- The study claims to provide an energy predictor in general while training is performed on rotamer recovery of a single amino acid. These claims should be downplayed and what was shown at most is that this predictor can be applied to predict energy for a restricted problem of the rotamer recovery task. 
- A reader gets a wrong impression at the beginning of the manuscript that the study tries to solve general and classical rotamer prediction for an entire protein. It becomes clear only at the very end that the study does not try to resolve all rotamer conformations in a protein, it can only predict one rotamer at the time given the atoms surrounding the target residue are correct. This should be explained in the beginning that the study does not attempt combinatorial side chain optimization for a fixed backbone;
- In-depth description of the neural network is required at least in supplemental materials, so that the study could be replicated. Ideally, a working application and training protocol code would be available.
- No techniques against overtraining are discussed. How was the model validated?

Small issues:
* function f_theta(A) is used before it is introduced. 
* noun is missing: using our trained with deep learning.
* articles missing: that vary from amino acid to amino acid.
* KL abbreviation not explained
* MCMC abbreviation not explained
* wrong artile:  that processes the set of atom representations.
* misprint: resolution fine r than 1.8
* wrong word, has to be “lower”: sequence identity greater ˚
* everyday word: break out
* abbreviation not explained: t-SNE
* misprint case: Similarly, In contrast to our work

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper proposes an Energy-Based-Model (EBM) for scoring the possible configurations of amino acid side chain conformations in protein structures with known amino acid backbone structure. The energy of the side-chain conformation (the chi-angle) for a given amino acid in the structure is calculated as a function of a local neighbourhood of atoms (A), where each atom is embedded into a 256d vector using its cartesian coordinates, atom identity, atom side-chain position and amino acid identity. The model is trained using approximate likelihood where the model samples are generated using precalculated table (from literature) of possible Chi angles conformations conditioned on the back-bone amino acid identity and back-bone angles. The results seem comprehensive comparing the transformer based energy function parameterization with two sensible baselines as well as the Rosetta energy function  which is the de facto standard tool for these types of calculations. Using rotamer recovery accuracy as the benchmark measure the empirical results are close to performance as the Rosetta energy model however always slightly worse. Further visualizations of the energy levels for different Chi angles seems to support that the learned energy function captures well known characteristics of the rotamer configuration energy landscape.

Score:
Overall I think that the paper is solid and tackles an interesting problem of learning energy functions for physical systems from data. The experimental results are comprehensive and mostly supports the claims made in the paper. My main (slightly minor) concern about the paper is the obsession with discarding years of learned knowledge about handcrafted energy functions with fully learned functions. It seems to me that combining domain knowledge with a learned model should close the last small gap (and presumably surpass) the performance of e.g Rosetta. Combining I think the paper should be accepted at the ICLR.

Comment/Questions:

Motivation
Q1.1) Paper motivation: The paper several times seems to suggest that energy functions derived from physical knowledge are problematic (e.g. abstract). For many physical systems I don’t think this is true and would like the author's comments on why learned energy functions are preferable - arguably they can only capture properties present in the data and not any prior knowledge?  

Q1.2) Given that the paper is motivated by fully learning an energy function from data without injecting any prior knowledge i would like the authors comment on the construction of the q(x|c) distribution. This is based on essentially a contingency table between Phi/Psi/Chi angles and to me seems like injecting prior knowledge about physically possible rotamer configurations into the learned procedure? 

Method / Experimental Results:
Q2.1) With respect to the primary results in table 1) and table 2). The authors claim comparable results to the Rosetta Energy function (page 5. Sec 4.3, page 9, sec 6). However the experimental results are always slightly worse than the Rosetta energy function and I (strongly) suggest that the authors rephrase those statements to reflect that.

Q2.2 With Respect to Table 3) Firstly, Why are the only results for 16 amino acids in the table ? Secondly, just naively counting the Atom Transformer are better than Rosetta in 11 of 16 amino acids - this seems slightly at odds with the main result where the Atom Transformer is performing slightly worse than Rosetta?

Clarity
Q3): The notation in section 3 and 4 is slightly confusing. Especially I think q(x|c) is slightly misleading since it, (to my understanding) is a conditional distribution over Chi, conditioned on Psi/Phi/AminoAcid and not rotamer (x) and surrounding molecular context (c). 
review_title: Official Blind Review #2

Meta-review: The paper proposes a data-driven approach to learning atomic-resolution energy functions. Experiment results show that the proposed energy function is similar to the state-of-art method (Rosetta) based on physical principles and engineered features. 

The paper addresses an interesting and challenging problem. The results are very promising. It is a good showcase of how ML can be applied to solve an important application problem. 

For the final version, we suggest that the authors can tune down some claims in the paper to fairly reflect the contribution of the work. 
Decision: Accept (Spotlight)