Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a complete protein learning method based on hierarchical representations: amino acids, backbone, and all-atom level. 
strength_and_weaknesses: This paper showcased a hierarchical graph network, "ProtNet", which computes the protein representations at different hierarchical levels. The authors investigated and integrated the physical protein (amino acid chains) conformational properties into their methods. Their experiments indicated the current method is largely on par with other SOTA methods, with ${maybe}$ marginal gain in some metrics. 

1. The whole method is largely based on a very recent 2022 Neurips paper ComENet. Of course, ProtNet exploits unique properties of proteins and further adapted the ComENet to specialize for proteins. But the argument in 3.1 about "significantly different from ComENet" is not convincing enough and I would like to see the authors could further elaborate on the innovations and novelties in the current work.
 1.a) The first reason is ProNet is "based on the unique structural properties of protein". Thus, it is a special case of ComENet. Certainly, protein is a polymer of amino acids, thus providing more structural information for leveraging than general molecules (in which case we can also argue for greater generalization of the latter though). All properties (e.g., dihedral angles, side chain torsion) the authors discussed in the paper are long-known physical measures in protein characterization and have been exploited in various ML methods. 
 1.b) Regarding ProtNet can represent backbones and all-atoms, this again has been seen in many previous ML methods for protein representations and while torsion angles can uniquely identify the conformation, I would like to understand better how simply adding them as node attributes archives "all-atom representation" 
 1. c) The local coordinate system that is based on neighbor amino acids (they are linked by bonds) is a natural property of chains/sequences/polymers/proteins. The correct analogy should be atoms linked by chains in ComENet. 
2. Moreover, I did not find any new discoveries, insights or learning opportunities in this work. The authors might be able to strength the paper by providing deeper analyses and understandings of the methods, and fundamentals ...
3. As the experiments largely followed (Townshend et al., 2021), it would be more convincing if we do all protein tasks exhaustively (e.g., RES, MSP, PSR, LEP are missing). 
4. Additionally, in table 4, the 60% similarity cutoff is higher than the original 30% setup. (a) is there a reason we see higher RMSE for Atom3D? (b), doesn't it mean ProNet has less generalization power on more similar proteins?
clarity,_quality,_novelty_and_reproducibility: Part 4 (and the ref to Appendix C) is very brief and lacks clarity, can be expanded to be more clear. 
As I have mentioned above, the paper would benefit from more clarification of its novelty and several experiments.
summary_of_the_review: The authors provide a new protein representation learning method. But I am mainly concerned about its novelty and significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hierarchical graph neural network for protein property prediction. It models a protein at three levels -- amino acids, its backbone atoms, and its side chain atoms. Each level is modeled by a ComENet layer that learns a complete representation of the geometric structure. The method is evaluated on multiple protein classification tasks: fold classification, reaction classification, and ligand binding affinity prediction and compared with a variety of existing methods. 
strength_and_weaknesses: Strength:
1. The evaluation is fairly comprehensive. It included most of the state-of-the-art methods published recently.
2. Authors provide completeness proof for each layers, which is important for understanding the power of these geometric neural networks.

Weakness:
1. The improvement of the proposed method over previous baselines are relatively small. In particular, on the ligand binding affinity prediction task, the model underperforms the 3DCNN model (which is not equivariant) under 30% sequence identity cutoff. The 30% identity cutoff is a harder setup that requires more generalization to novel protein structures and therefore more important than 60% identity cutoff. 
clarity,_quality,_novelty_and_reproducibility: The method description is relative clear.
summary_of_the_review: Overall, I vote for a weak acceptance of the paper because it provides a completeness proof for the proposed architecture, which can be valuable to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a framework, ProNet that integrated different level of protein information for protein related tasks. It showed improved performance on available datasets. ProNet is thus consisted by multiple components. The author also noticed that different tasks may require different level of representations.
strength_and_weaknesses: Strength:
This paper is well written. It also provide complete analysis of complete graph.
They conducted lots experiment to show the performance difference compared with previous methods.
They method also focus on amino acid rather than atoms that make both the data and the model to be light.

Weakness:
The method aims to integrate all level of protein information. But however, the performance increase is rather marginal compared with existing methods. That raised some question, whether such different level of information is needed. If so, how to integrate them. However, I do not think the author address these questions. For example, how about different integration methods.
This method has many components and lots of hyperparameters. I would recommend to conduct a more through ablation study. It could also bring some insights on why integrating different level of information.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. The author did not provide code during reviewing period.
summary_of_the_review: I regard this paper is quite complete. But I find the contribution is quite marginal and lack some more in-depth analysis.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This concerns learning the hierarchical representation (e.g., a.a., backbone, all-atom) for proteins with 3D structures. A novel hierarchical graph network, termed ProNet, was developed. ProNet is flexible, efficient and effective. It is shown that ProNet representations are complete at all levels if the base 3D graph network is complete. The experimental result shows the proposed model is able to achieve state-of-the-art for multiple tasks. It also shows that different downstream tasks my require different levels of representations.
strength_and_weaknesses: Strength: The hierarchical design of the representation improves the computational cost and model performance at the same time. Representations at different level can be used to solve different tasks.

Weaknesses: The all-atom level representation share $F(G)_{bb}$ with the backbone level representation. The backbone level representation share $F(G)_{base}$ with the amino acid level representation. The importance of such redundant design is unclear.
clarity,_quality,_novelty_and_reproducibility: The model architecture in Figure 4 is confusing. All three level representations are sent into the Hier-Geom-MP block. How to evaluate the contribution of each representation? In addition, $F(G)_{all}$ contains $F(G)_{bb}$ that contains $F(G)_{base}$. Why need to feed all of them to the Hier-Geom-MP block?
summary_of_the_review: This paper provides an effective way to represent proteins. Its idea is straightforward, and the results are promising. There are a few places that need more explanations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to perform hierarchical representation learning of protein structures, where are each level a complete structure representation is learnt. Extensive experiments are performed that demonstrate the performance and efficiency of the approach and illustrates how different tasks may benefit from different levels. 

The paper makes some interesting contributions and convincingly demonstrate the value of the proposed hierarchical approach. The AC is very familiar with this area and also thoroughly checked the submission as well as the discussion. 

The author response and updates to the manuscript greatly improve the quality of the contributions and clarify novelty compared to ComENet. We also encourage the authors to incorporate their new results on four Atom3D additional protein tasks (RES, MSP, PSR, LEP) into the appendix as they suggested. 
justification_for_why_not_higher_score: While the method is novel and experiments promising, the contributions are not groundbreaking as they build on prior architectures and protein features. 
justification_for_why_not_lower_score: The approach is very interesting to the community, will inspire researchers in the growing area of protein representation learning.  The authors convincingly demonstrate great performance and efficiency compared to existing approaches. The authors have satisfactorily addressed all concerns raised in the reviews.
Decision: Accept: poster