Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors consider the problem of learning protein representations which jointly leverage features at multiple scales: sequence, structure, and surface. To accomplish this, a graph construction is proposed with interconnected structure and surface layers, and physicochemical node/edge features. A proposed model encodes the graph sublayers into a unified representation of the protein.  Computed representations are evaluated on two tasks: prediction of (1) ligand binding affinity and (2) enzyme class.  Finally, the authors propose the notion of “molecular superpixels” - adapting an image segmentation algorithm to partition molecular surfaces into compact, geometrically and chemically homogenous zones.
main_review: 
The paper presents a new approach to multi-scale protein representation learning, with a creative adaptation of superpixels for partitioning molecular surfaces into compact, homogenous regions.

Main Strengths:



* The multi-scale graph construction - uniting sequence, structure, and surface - is well motivated, conceptually simple, and a novel contribution.
* The application of superpixels to biomolecules is a novel contribution, and is compelling for: (1) its connection to a successful method in an outside field, and (2) its general applicability to any feature set.

Major Weaknesses:



* Final result impact is diluted by underperformance relative to (Hermosilla et al, 2021) on enzyme class task, despite Holoprot’s additional inclusion of surface features.
* Results in Tables 1 and 2 are hard to parse without indicators of model type and input data (sequences, structures), which are important to contextualize results.  The results are less impressive when ignoring the sequence-only rows and including other strong baselines in the Hermosilla baseline source: HHSuite (82.6%) and Rao - LSTM (79.9%).

Based on the novelty + conceptual simplicity of the proposed method, but with the weakness of experimental results limiting the impact: I recommend 6: Marginally above acceptance threshold.


### Setup, approach, originality



* Problem setup is good (forward-looking and impactful): structure-based representation + surface graph, evaluated on reasonable tasks.
    * [-]  But, rather than tasks relying on fixed-size embeddings of the full protein, predicting protein surface interaction location like the tasks in Gainza et al 2020 seems more promising and appropriate for this model class - was this considered?
* The multi-scale graph construction is conceptually simple, and well motivated.
* The application of superpixels to biomolecules is a novel contribution, and is compelling for its simplicity and generality.
* Though, as the author’s allude to in section [4 - Superpixels on Molecular Surfaces] there is a long history of methods for identifying regions on three-dimensional molecular surfaces.  Molecular superpixels are stated to be advantageous due to better geometric compactness relative to some baselines, but no quantitative analysis, nor claim is made of general superiority over alternative algorithms - fair.


### Quality

Major:



* [+] Ablations were informative.  It is nice to see that more powerful pooling on molecular superpixels was unnecessary.  And it is interesting that the marginal benefit of the structure / surface layers is so different across tasks.
* [+] [5.4 - Limitations]  Important call-out that this method requires structures is appreciated.
* [?] [Abstract]  It is claimed that Holoprot is more _consistent_ than other methods. How is this reflected in the results? The level of variation looks comparable to that of Hermosilla/Gainza/others.
* [-] The list of features in the surface and structure layers node/edge features seem extensive.
    * What is the reasoning behind the node/edge features chosen in the structure layer?
    * It would be informative to know how helpful the 7 edge features in the surface network (Hanocka et al 2019) are.  Since it seems Gainza did not use these. (e.g. Are features or the model the driver of the boost over Gainza?) 

Minor:



* [Table 1] Townshend et al. (2020) results are missing for the scaffold-level split?
* Line 540: “structure” is a typo.

### Clarity


Major:


* [+] Paper + supplement are well organized.  Explanations are clear, and the appendix concisely adds much useful detail.
* [Table 1 + 2]
    * The tables would benefit highly from organization by model / input-data type, as opposed to the current organization by chronology.
        * Partial example: Table 1 of the baseline results source (Hermosilla, et al., 2021), which specifies architecture + special notes on data sources.
    * [-] Other top baselines from that paper should not be omitted: HHSuite (82.6%) and Rao - LSTM (79.9%).
* The superpixel ERS method is hard to follow, could probably be expanded in appendix, and a high level intuitive explanation be provided in the maintext.

### Significance

* Multi-scale graph construction and representation algorithms are conceptually clean and well motivated.  The focus on explicitly incorporating protein surface features seems a useful and forward-looking approach.
* The superpixel algorithm is not quantitatively shown as superior to alternative algorithms, but is compelling for its simplicity and general applicability.  Future researchers may find this to be a useful primitive.
* The final result impact is diluted by underperformance relative to (Hermosilla et al, 2021) on the enzyme class task, despite Holoprot’s additional inclusion of surface features.
* Low data regime argument is interesting [Conclusion], but could be improved:
    * Is it possible to scale the current model, for better performance?
    * Is it possible to evaluate on a task that is more in the low data regime?
limitations_and_societal_impact: Yes.  We appreciate the acknowledgement in [section 5.4 - Limitations] and [section 2 - Related Work] that baseline models were trained with very divergent input data modalities, and that this method explicitly requires structures.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce a representation learning approach to protein structures which encompasses primary sequence, secondary structure, inter-residue distances, and surface features captured by triangulations. These different scales are encoded using two linked graph representations, and message passing networks are trained to learn geometric features from this system. Applications to protein-ligand binding and enzyme reaction classification provide evidence the model can learn effectively with few parameters. 
main_review: This paper pursues an interesting and novel direction in protein representation learning, seeking to integrate multiple existing approaches to feature extraction in a single architecture. Given the plethora of sequence and structure-based models, as well as recent work on surface representation, this is a natural line of investigation to pursue. The execution is simple and clean, making the paper easy to follow. Evaluations and empirical results are good but improvements or further analysis will strengthen the overall work.

Strengths:
1. The features chosen at each input scale are well-chosen. The graph structures representing each scale and the model choices for linking these scales are also reasonable.
2. Applying superpixels to protein surfaces is an interesting idea in its own right, and the authors systematically evaluate its impact on performance throughout the paper. The particular choice of superpixel generation algorithm is motivated well by the authors.
3. Dataset choices were good and proper attention was paid to splitting of train/eval/test sets. 
4. The ablation study in Table 3 is thorough and provides context for model performance.

Weaknesses (elaborated on in Limitations):
i. The language model baseline for protein-ligand binding affinity needs to be improved. Given that ProtBERT-BFD (Elnaggar et al 2020) is included in Table 2, this would be a natural choice to include for Table 1 as well. This model has proven substantially better than previous language models (such as Rao et al or Bepler and Berger), so it is necessary to get an informative comparison to pretrained sequence models.
ii. A number of analysis questions should be explored further, as they are central to the story. Major ones include: why is structure-only in Table 3 comparable to Holoprot across all tasks? why does Holoprot do worse on 30% sequence identity splits but better on 60% and scaffold-based splits in Table 1?
iii. The authors argue their model is more parameter efficient in settings where it loses to the Hermosilla et al baseline. Is it instead possible for them to increase the parameters of Holoprot by an order of magnitude and see if this improves performance? 

Minor comments:
- The formatting of Table 2 could be improved to highlight which methods are sequence-only, which are structure-only, etc.

Things that would raise my score:
1) Inclusion of ProtBERT-BFD for the protein-ligand binding results.
2) Expanded analyses identifying successes or challenges of the model. Some are suggested in this review, but these are just suggestions. This is particularly valuable for future work from the authors or others in the community.
limitations_and_societal_impact: 1. Choice of language model in Table 1 could be improved. Either use of ProtBERT-BFD (Elnaggar et al 2020) or ESM (Rives et al 2020) are appropriate. Given the use of Elnaggar et al in Table 2, this might be the simpler choice.
2. The change in performance from structure-only to Holoprot across Table 3 is favorable to Holoprot but not drastic. This seems to suggest that the addition of surface-level representation is improving performance in specific regimes or subproblems. Understanding this phenomenon would be very useful for future multiscale representation learning. Currently the authors claim "HOLOPROT with and without molecular superpixels improve over the performance of structure and surface representations" but this claim seems a bit strong, especially given the numbers on protein-ligand binding.
3. The authors do not provide any suggestion for why they lose to Townshend et al at 30% sequence identity splits but beat them otherwise. Is this due to over-reliance on sequence features by Holoprot? Again, some analysis would be illuminating here.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel model incorporating sequence-level, structure-level, and surface-level information from a protein. This information is combined through an MPNN with a hierarchical structure. The authors additionally propose a molecular superpixel representation of the protein surface in order to lower computational/memory costs, with no loss of performance. The model achieves generally good results while using a very small number of parameters on both binding affinity and enzyme-catalyzed reaction classification.
main_review: This is an interesting paper in an important emerging field. Given recent advances from CASP, I expect that structure-based prediction / representation learning will play an increasingly large role in protein property prediction and protein design. This paper has two major contributions - the architecture combining structure- and surface-based features, and the use of molecular superpixels. These ideas are likely to be generally useful to the structure representation community. The evaluations are reasonable and the model achieves decent performance using relatively few parameters. The ablations also suggest both structure and surface information are useful, although the effect seems minimal in the case of ligand binding affinity.

Here are some suggestions for improvements to the paper:

- The method proposed here uses a one-hot representation of amino acid identity as the sequence level feature for each residue. Why not instead use the output of a protein language model, such as ESM-1b or ProtBERT-BFD? These feature vectors are known to contain a good deal of biologically relevant information (including e.g. secondary structure) so it seems like this is an easy way to get the benefits of a large scale language model incorporated into your model.
- In the ligand binding affinity task you do not benchmark against a recent sequence-based method (e.g. ESM-1b, ProtBERT-BFD, ProTrans-T5). It would be nice to have this evaluation.
- How was the model size for HoloProt chosen? It seem as though the number of parameters is much smaller, and you make an argument for the parameter efficiency. However, could the method be scaled up to achieve SOTA performance? In other words, was the model size chosen due to computational constraints or is there a more fundamental limitation preventing scaling the model and achieving better results?
- The paper introduces molecular superpixels as a method to reduce computational / memory costs, but the improvements are not benchmarked.
- I am hesitant to suggest additional experiments since I understand they are likely infeasible in the author response period. However, if time permits or for future publications it may be worth looking into the Atom3D suite of tasks (Townshend 2020).

Minor issue:

- The citation provided for AlphaFold2 is the Senior 2020 Nature paper describing AlphaFold1. The correct citation is found at the end of [this blog post](https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology). 

Finally, I note that you describe training several of these models takes a significant amount of time (40 hours). For some of these models (e.g. the pretrained language models) a better option is to embed each sequence and save the features to disk (assuming you are not fine-tuning the entire model). Then during training, simply loading the features and training the MLP will likely allow you to train the model in minutes. 
limitations_and_societal_impact: The authors discuss limitations of the work in the paper, and some additional suggestions are provided in my review above.

This paper is a basic computational biology work and has no immediate social impact.
review_title: null

Meta-review: The paper proposes a very interesting approach for protein representation learning. The AC and reviewers greatly appreciated the author feedback and we urge the authors to incorporate their points into the manuscript.

In particular it would be important to include the comparison with ProtBERT-BFD  and the extended analysis results. 

If the authors cannot include results for larger number of parameters as promised, it is critical that they include comments on limited resources, where the bottlenecks are and path to resolve these.

Decision: Accept (Poster)