Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for automated model building of protein structures in cryo-EM maps. The method utilizes amino acid sequence data and prior knowledge about protein geometries in addition to cryo-EM data. Specifically, it starts with approximate locations of individual residues using a CNN and then uses a GNN that takes in information from sequence data, cryo-EM map and initial positions of atoms to refine the full atom positions of each residue. The main innovation is utilizing several aspects of protein data in contrast to traditional sequence only or cryo-EM only approaches. The results look promising, achieving better sequence recall and backbone root mean squared deviation compared to Deeptracer, though with only 28 evaluations and a longer execution time. The main limitations discussed are sensitivity to resolution, inability to achieve full atomic modeling of nucleic acids, and inability to construct models with unknown sequences. 
strength_and_weaknesses: Overall, the paper shows promising results. I am, however, concerned by the way the method is presented for a machine learning audience. The method section is vague and does not provide sufficient detail to be able to fully understand and reproduce the method. There is also a lack of detailed descriptions about the neural network architecture and ablation studies. The paper would benefit from further discussion, detailed explanation, and clarification.
		
1. Other works combining sequence with cryo-EM: The authors mentioned several other works utilizing protein structure prediction programs to improve output predictions to fit the cryo-EM map (page 3). What are the main differences between those works and ModelAngelo? The authors claimed that “Such approaches are likely to propagate errors in the structure prediction. ”. What is the basis of this claim and what makes ModelAngelo not susceptible to the same problems?

2. Methods section 3.1 describes the coarse location estimation module, but there are many important details left to the readers imagination. How is the dataset split into train, test and validation sets? How is the neural network evaluated and validated? What is the performance of this network? What are the choices of layers for this CNN and what is the reason behind these choices? Are there any ablation studies done for the design? 

3. Methods section 3.2: Again, the description of the GNN is vague. There is no description about the training, testing and validation processes nor evaluation of the model performance. Specifically how is this module designed?

4. For eq(1),  χ is the characteristic function of whether x contains a Cα atom. How is this function defined? 

5. Time and computing resources are briefly discussed, but there does not seem to be information regarding training time for each part of the method nor the computing resources used. What kind of hardware is needed to train and generate predictions from ModelAngelo?

6. The test dataset is small and the similarity between the train and test structures is unclear. It would be helpful to see evaluation of ModelAngelo on a larger number of structures and to stratify performance based on similarities to the proteins in the training set. Is it possible that some of the test proteins are structurally similar to proteins in the training set? How well does ModelAngelo perform on structures substantially dissimilar to those seen during training? Structural splits based on structural alignment scores or SCOP/CATH classes would be illuminating.

7. How does ModelAngelo compare with more conventional approaches to model building such as Phenix or Rosetta?  we would like to see more metrics in comparison as well as comparison with more approaches such as Phenix and Rosetta. What about MAINMAST?

8. The description of each algorithm could be more detailed. For example, how do get_cryo_key_vectors and get_cryo_point_features work? What does get_query_vector_in_local_frame mean?	 

clarity,_quality,_novelty_and_reproducibility: The method is novel and the results appear to be good. The paper is clear but far too high level for an ML venue. It would not be possible to implement this model from the paper, because insufficient details are provided. The evaluation also does not sufficiently explore and justify some of the modeling decisions.
summary_of_the_review: The method is interesting and works well. However, the manuscript needs significant work to be better presented to the ML community. Model architecture and training details are vague and experiments do not demonstrate the value of individual model components. With some significant work on the text, I think this can be a strong contribution, but I think the manuscript as currently written is of limited value to the ML community.

What would raise my score: revise the manuscript to provide detailed architectural and training details and expand the evaluation to demonstrate the value of the components of the model and provide further characterization of the performance as it relates to structural similarity to proteins in the training set.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work leverages recent advances in large-scale biological machine learning to the task of atomic model fitting into cryo-EM density maps. There are three stages of this pipeline: 1) a residue segmentation model that identifies C_alpha locations in the cryo-EM density map, 2) a GNN-based model which refines the C_alpha locations by sequentially attending over the cryo-EM volume, the underlying protein sequence, and the predicted C_alpha locations, and 3) a postprocessing step where HMMER is used to infer the sequence and relax the geometry of the final atomic model. The model was applied on 28 test cryo-EM maps and compared against DeepTracer.

strength_and_weaknesses: Strengths include novelty and good empirical performance. This work is to my knowledge the first example of integrating learning over protein sequences to the task of atomic model fitting into cryo-EM maps. The algorithm is designed to iteratively incorporate information from the cryo-EM volume, geometric considerations from modeling the protein chain, and the underlying sequence.

One weakness is that the writing could be improved. The writing is clear and explains the design of the method well, however, some statements tend to be anecdotal and not substantiated. e.g. It would be nice to show ablations of the different model components. The details of the algorithms are also missing. 

Questions: 
It was not clear to me how/when side chain atoms are fit
The postprocessing step is also a bit unclear. How is the chain inferred? Why is the HMMER sequence search needed if the input sequence is known?
How does the model deal with multiple sequences in a large protein complex?
clarity,_quality,_novelty_and_reproducibility: The quality and novelty of the work is high. The clarity of the paper could be improved.
summary_of_the_review: This is fantastic work that should be accepted at ICLR. It is a novel application of machine learning to the manual, labor-intensive task of atomic model fitting in cryo-EM maps. Empirical results are strong, and the algorithms are well-designed. The writing of the paper could however be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ModelAngelo, a two-step pipeline to predict the protein structure using cryo-EM maps and protein sequences as input. To do this, ModelAngelo first employs a 3D CNN to predict positions of alpha carbons in protein sequences taking cryo-EM maps as input. Apart from this, they train a separate graph neural network to finish the task of position de-noising by using data augmentation methods, which can be used to fine-tune the protein structure after training. These two steps are separately trained. The employed models of CNN and GNN are task-driven but not new.
strength_and_weaknesses: ## Strengths
1. They propose to use protein sequences as input to better the structure prediction results, which is different from previous work named Deeptracer. 
2. They employ a GNN to integrate information from different domains including Cryo-em map, protein sequence, and alpha carbon positions.

## Weaknesses:
1. This paper is not well-written and hard to follow. The problem definitions of these two steps are unclear. (1) Clear definitions of inputs, outputs, and data structures are missing. It is hard to guess how they use these different features from different domains and what information is exactly used for the second step. (2) Some operations are hard to follow. How can they use literature bonds to update the coordinates?
2. The experimental part is not convincing. The comparison with Deeptracer is problematic because (1) Deeptracer and ModelAngelo use different training data, which means this is not a fair comparison, and (2) compared with Deeptracer, additional protein sequence information is used as input for ModelAngelo, which is a huge assumption to the problem setting.
3. Ablation studies are missing. The positions of alpha carbons can be obtained by the first stage only. Ablation studies are needed for a better understanding of the whole framework.
4. The proposed method is not novel enough. The methods for the first stage and the second stage are borrowed from previous papers with some modifications to better fit the problem settings, with limited novelty for the model part.

clarity,_quality,_novelty_and_reproducibility: **Clarity** : Not clear enough. As I mentioned in above weakness point 1.

**Quality** : Fair but not enough.

**Novelty** : Limited as I mentioned in above weakness point 4.

**Reproducibility** : Hard to reproduce, no architecture details provided. 

summary_of_the_review: Overall, because of the major concerns listed in the above weaknesses, I vote for the rejection of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for the automated building 3D protein models from cryo-EM data. The proposed model combines a CNN with a GNN component in an attention-based architectures and also requires a HMM based post-processing step.
 
As a first step, the authors try to approximate the location of residues by solving a segmentation task. The model is a 3D FPN trained on Cryo-EM maps in a supervised fashion, using a customized weighted loss to address data imbalance problems. Training starts using a small, manually curated subset of the data. Results are later used to prune larger datasets. 
Subsequently, the resulting map is processed by 3-stage GNN, which aims at correcting the CNN output. First module of the GNN is Cryo-EM Attention, which uses local neighborhood information by utilizing a novel graph-attention. Second is a Sequence Attention module, which is a BERT-like transformer taking as input the embedding computed by a pre-trained language model. The third and final module is Spatial Invariant Point Attention (IPA), inspired by similarly named module from AlphaFold2. The whole model is trained sequentially using several loss terms and produces 2 types of output. In the unpruned output, predicted chains are almost as-is from the output of the model, with minimal post-processing. This allows for or a larger percentage of the model to be built and gives higher recall. However, this may produce overhead for users in form of manually pruning longer sequences. 
Therefore, the authors provide an additional post-processing mechanism based on hidden Markov models, which produces lower recall results, but higher sequence matching percentages, Authors argue that this is better for end users.
strength_and_weaknesses: Strength:
1. The description of the problem in question is sufficiently thorough.
2. The authors argue that the proposed method allows to build an atomic model de novo in the matter of hours compared to weeks in case of doing it manually.
3. ModelAngelo outperforms current state-of-the-art DeepTracer in terms of sequence recall and RMSD, although it’s slower due to increased overall computational needs. (Personally I don’t know if there are other baselines the authors could/should compare against.)
4. The proposed system can also provide confidence of its own predictions thanks to a special loss term.
5. Contrary to DeepTracer, the proposed method outputs side-chain coordinates.
6. Authors provide extensive analysis of limitation of the proposed method and possible directions of future work. 

Weaknesses:
1. No schematic description of the model architecture and training procedure. This made it hard for me to be sure what is really proposed and to check if I understood the overall setup of the proposed method (which certainly has many moving parts).
2. Overall, the proposed algorithm is rather vaguely described. 
3. The same thing is also true for the proposed loss, which I would love to see better on more thoroughly described.
4. I do know some things about cryo-EM data and workflows, but still I would have appreciated a better description of the data that goes into the proposed model.
5. Hard to interpret and somewhat limited description of the obtained results.
6. No ablation studies are presented, leaving many canonical questions on how important different parts of the proposed method are completely unaddressed.
clarity,_quality,_novelty_and_reproducibility: In general, the authors provide a good explanation of the importance of the task and its complexity. They also report promising results. However, the paper lacks many details, crucial for readers to understand the presented method and its training procedure. Given only the information provided in this manuscript, I have serious doubt the proposed method can be fully understood, let alone be reproduced.
summary_of_the_review: The paper proposes a multi-faceted method for solving an important and challenging task. The method contains several technical novelties but fails to fully clarify their details. The problem is described in sufficient detail, but several important issues with the way material is presented make it hard to completely understand the details of the implementation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a machine learning tool ModelAngelo for automated model building in cryo-EM maps. The tool consists of a sequence of three components, including an improved Residual network for residue segmentation, a GNN-based framework for refining and an HMM for the final-mapping. These components are separately trained in a supervised fashion. ModelAngelo is evaluated in a test dataset of 28 map-model pairs, and shows better performance in sequence recall and backbone RMSD. The authors also discuss limitations and future research of the current implementation. 
strength_and_weaknesses: Strength
    1. Automated model building in cryo-EM maps is an important problem worth studying. ModelAngelo is among the initiated works that use deep learning techniques. 
    2. Deep learning approaches used in ModelAngelo are technically sound. Some improvements in model architectures and learning objectives are successfully applied.  
    3. The performance of ModelAngelo is empirically verified by a test dataset. 

Weaknesses
    1. A mathematical formulation for the entire problem is missed. Though the problem is complex and difficult to be solved with an end-to-end framework, the original formulation is still needed at the beginning of the Methods section, followed by a brief introduction of the entire framework, i.e. how to split the entire task into several components and what specific role does each component play. I also suggest authors present a figure to illustrate the overall computing procedure of ModelAngelo.
    2. Quantitive evaluation results in Figure 3 only reflect middle outputs rather than the final outputs. Figure 4 illustrates the comparison of final results with a single data sample. Thereby, current evaluations are not convincing enough to confirm ModelAngelo’s superiority to competitors. Is it possible for a quantitive comparison on the final outputs?
    3. Many losses (e.g. (RMSD) loss, backbone RMSD loss, amino-acid classification loss, local confidence score loss, torsion angles loss, and full atom loss) are involved in the learning of GNN. What are the definition of these losses? How do you get the ground truth labels required (if so) for these losses?
clarity,_quality,_novelty_and_reproducibility: The paper involves a lot of specialized knowledge and terminologies. The overall paper is difficult for me to read.  The machine learning part is mostly clear, but lacks a formal definition of the entire problem and some sub-problems (3.2 and 3.3). 
The work is novel and of good quality in terms of a deep learning based cryo-EM analysis tool. 
summary_of_the_review: The paper introduces ModelAngelo for automated model building in cryo-EM maps. Compared with previous work (deeptracer), ModelAngelo largely improved deep learning techniques. It achieves better results at the cost of affordable increased complexity. Overall, I think this work is useful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work the authors proposes a new pipeline leveraging deep learning to automate the building of protein structural models from cryo-EM density maps. Their approach integrates information from protein sequence, EM density map, and graph topology to help improve the performance of model building and the authors tested and validated their method in a decent number of targets. There are still room for improvement in terms of complete clarity of the method details, more demonstration and discussion of the benchmarks. Overall this work developed a good and useful machine learning tool for an important real-world problem in protein structure biology.

While the initial reviews had a more positive tone, most reviewers were not too enthusiastic about this work due to several technical and presentation issues. It is commendable that the authors were able to improve significantly over the initial submission thanks to the reviewers' comments.
justification_for_why_not_higher_score: While the whole process is an innovation, the method is a well-designed workflow based on existing ideas, which the authors applied to a key problem in a new way.
justification_for_why_not_lower_score: The authors have extensively revised their work during the discussion and addressed most of the major concerns that several reviewers have.
Decision: Accept: poster