Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies unsupervised contrastive learning on protein structures, using sub-structure sampling as the data transformation strategy in contrastive learning. The protein structure representation from contrastive learning is then evaluated for three tasks: fold classification, enzyme classification, and protein similarity.

The main contributions are:
1) Demonstrating that contrastive learning with sub-structure sampling is a viable strategy for learning protein structure representations.
2) Improving empirical results for fold classification and enzyme classification when compared to existing methods.
main_review: Representation learning for 3D protein structures is a broadly relevant problem. The authors proposed using contrastive learning with sub-structure sampling as the data transformation. The empirical results on fold classification and enzyme classification show that the proposed contrastive learning method is able to meaningful representations that capture certain properties of protein structures. The paper is clear and includes detailed ablation studies.

One major concern is that the learned representation from the proposed approach might only capture "global" properties of protein structures and might not be able to capture finer-grain features of protein structures. This might be baked in from the choice of using subgraph sampling as the data transformation. For example, if two antibodies could be structurally very similar to each other but bind to completely different antigens, and it is likely that the learned representation here won't be able to distinguish two structurally similar antibodies. Extending the method to capture finer-grain features in protein 3D structures could enable more use cases for the learned representation.

While learned distance metrics on the protein structure representation space is an important research direction, likely how the distance should be defined would depend on the downstream application. It would be valuable for the authors to clarify when the proposed distance metric in this paper would or would not be a good fit.

While the evaluation tasks (enzyme classification and fold classification) are clearly defined, they are not particularly well motivated and it would strengthen the paper to further explain the biological/clinical significance of these tasks. 

The conclusions towards the end of the paper are not well-supported by the empirical results. For example, it is unclear how such a representation would help with solving protein-protein interaction related tasks. More context would be appreciated.

Question: How are variable lengths handled in contrastive learning? Could it be that the representation is implicitly using length to classify fold/EC while baselines such as hhsuite or TMalign cannot make use of the length information?

summary_of_the_review: Interesting approach. One major concern is that the learned representation from the proposed approach might only capture "global" properties of protein structures and might not be able to capture finer-grain features of protein structures. Proposed approach shows improvements on fold and enzyme classification tasks, but these tasks may have only limited practical utility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an unsupervised deep learning approach for learning representations of 3D protein structures. They use an objective function motivated by the recent contrastive learning approaches (from computer vision). They show the utility of their model in two downstream applications: protein fold classification, enzyme classification and protein similarity prediction. The main contributions are:
1. they show how the contrastive learning framework can be used in the context of protein structures
2. they analyze the learned representations
3. they show the utility of the model to downstream applications
4. they show how fine-tuning their model leads to an improved performance
main_review: Strengths:
1. The work tries to address an important proteomics problem
2. The writing is generally clear and succinct, but can be improved in some parts (see comments).
3. The approach is interesting and seems like a novel application of existing contrastive learning approaches, adapted for proteins
4. There is a reasonable analysis of the learned models

Weaknesses:
1. Their approach of using contrastive learning seems ill-fitted to learn precise representations of protein structures. They consider sub-chains of proteins as examples. But substructures of the same protein can look very different and have very different functionality? Does it make sense to minimize the cosine distance between them? for instance: an alpha helix and a beta sheet from the same protein can be quite different, so if the cosine distance between these is minimized, it is not clear what the model is learning
2. Can they restrict the similarity constraints within a single domain from a protein, maybe?
3. The evaluation is weak and is done on very coarse labels. For instance, there are currently 1549 different folds (classes) in SCOP and the paper evaluates on 7 high-level folds (classes) that are very different. Many computational papers show results on at least 27 folds (Ding and Dubchak, 2001) or a few hundred folds.
4. Some of the baselines that they compare against were developed for totally different problems. For ex: Baldassarre was for Protein Quality assessment and is specifically tuned for that problem and seems ill-suited for fold classification?
5. The paper's goal is to use unlabeled protein structures that other supervised approaches can't use, but in Section 4.1 they say "chains are removed based on annotations". So, do they only use annotated structures?
6. Table-2 shows that their approach does not work well on protein structural similarity prediction. 

Other comments:
------------------------
1. How would embeddings from other unsupervised representation learning approaches like TAPE, or maybe AlphaFold compare?
2. In Table-2, TMalign has either similar or better performance. But in Table-1, TMalign does much worse on fold classification. This seems quite odd. How is it being used for both the tasks?
3. Section 4.3: why does the method not do as well on unseen proteins subclasses for enzyme classification?
4. Section 4.3, computational time comparisons -- Can you compare the training time taken by your approach to the setup for DaliLite and TMAlign that don't need gpus and can be parallelized easily on multicore machines to make them faster.
5. Which data was used for reaching the 20-40%, 40-60% etc, in Section 4.4? Was it a validation set? If the test set is used to check the performance, then this will overfit for the particular task chosen.
6. The "network architecture" section needs better writing to convey the ideas. It is very confusing. Elaborate using a diagram or flow chart? It's not clear how many features and what all is done to the input.
7. How is representation at protein level obtained? Average over all amino acids? And for each amino-acid, is it 16 dimensional? So protein representation is also 16 dimensional?
8. Why not just sample p% starting from a given amino acid? What's the point of getting a window left and right of a specific amino acid?
9. What are the strengths of the 
10. Intro in Section 3.3, ""noise" and cropping seem irrelevant to the approach being used.:,

summary_of_the_review: I recommend 'weak accept' due to the concerns raised in the above section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the general problem of learning a compact embedding of a protein based on its 3D structure.  The key novelty here is to use a contrastive learning procedure: "minimize the cosine distance between the learned representations of the sub-structures sampled from the same protein, while maximizing the cosine distance between representations from different protein chains."
main_review: I found the abstract misleading. It talks about the wider availability of proteins sequence information compared to structural information, leading me to think that the proposed method would learn from both.  But it doesn't: it learns only from 3D sturctural information.

Furthermore, a key missing component of the story is the notion of "annotations."  This concept is mentioned in the abstract ("Moreover, this number is reduced even further, when only annotated protein structures can be considered") but never clearly defined.  A protein can be annotated in many ways, including annotations along the amino acid chains (domain boundaries, protein or DNA binding sites, active sites in enzymes, secondary structural elements, etc.) or annotations of the entire protein sequence (GO terms, KEGG pathways, SCOP annotations).  My best guess is that the authors intend to use the term "annotation" to refer to assignment of protein domains to a protein sequence or structure. ("Note that, since our framework learns from unlabeled data, we do not sample specifically protein domains from the protein chain, which would require annotations.") I do not actually know how protein domains get annotated, but I think it's mostly by sequence similarity.  If this is the case, then it's ironic that the authors are trying to avoid using these annotations during training, since they can presumably be easily generated from 3D structures.

Overall, I found this to be an overly complicated model to test what seems like a straightforward hypothesis (i.e., that this contrastive learning approach is beneficial).  Much of the model seems to derive from a previously published approach (Hermosilla 2021), but with not insignifcant modifications, particularly in the protein encoder.  This approach makes it hard to keep track of what primary claim or claims are being tested here.  In particular, I am not sure if the authors are simply trying to create a state-of-the-art fold recognition engine (in which case the results in Table 2 suggest that they have failed) or simply demonstrate that their contrastive learning approach is beneficial.

The paper needs to be more careful about defining terms before using them. For example, we are told on p. 4 what a protein domain is, but the text refers to protein domains on pp. 1 and 2.

I could not understand what the "Protein" task consists of in the SCOP superfamily setup ("... and Protein, in which proteins of the same family are present during training."): what are the test sequences?

I realize that the enzyme classification task has been presented previously, but I still think it should be described more clearly. I do not understand what a query is in this case, nor whether the train/test split is stratified by class.

I did not find the qualitative evaluations in Section 4.2 to be informative.

summary_of_the_review: This is a complicated model or set of models, coupled with results that do not clearly support the central hypothesis of the paper, namely, that using contrastive learning outperforms other methods for learning informative embeddigs of protein structures.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an unsupervised method for learning low dimensional representation of proteins borrowing ideas from contrastive learning in the computer vision literature. Experiemnts on three data sets compares the performance of such embedding with baselines algorithms. 
main_review: positives:
1) The paper addresses an important problem of inferring protein functions and structures using limited number of labeled data points.
2) Using contrastive learning as an unsupervised method for protein embedding seems to be novel.
3) Experimental evolutions are extensive and ablation studies have been conducted.
4) The paper is well-written and understandable.

negatives:
1) Perhaps the main concern is that the results do not justify that the learned embedding has superior advantage over simpler baselines. The embedding unfortunately falls short on the Protein data set in Table 2 compared to baselines. Also in Table 1, applying classifiers (SVM, MLP, and 1-NN) on the learned embedding does not result in a competing classifier suggesting than the learned low dimensional representation does not fully reflect the nonlinearities present in the target classification tasks. Not sure how to justify from these results the embedding is useful beyond an initialization for the Fine-Tuning procedure. 
2) The novelty of the method is also limited; the contrastive learning is directly applied to the protein problem and several details of the method including sub-structure sampling and protein encoder are borrowed from Hermosilla et al. and Ingraham et al.

questions:
1) How sensitive are the results of the embedding to the choice of p?
2) Is it possible to train simple classifiers (e.g., SVM, etc.) on the Protein data of Table 2 similar to Table 1?
3) Is there any benefit towards using larger number of proteins in PFAM as the main unsupervised data set? The paper motivates the presence of millions of proteins in PFAM, however, uses couple of hundred thousands as the main data set --- this could enable better results.
4) Can the authors comment on possible interpretability of their embedding regarding features for specific 3D (sub)structures that might emerge from their method? and this give additional edge over the competing methods beyond the quantitative scores?



summary_of_the_review: Overall while the problem that the paper addresses is important and the method is reasonable, unfortunately, as it stands the proposed embedding method does not yield very convincing results compared to existing baselines in downstream tasks. Added with limited novelty of the method the paper in my opinion is marginally below acceptance; although I remain open to authors discussions in the rebuttal.
review_title: null

Meta-review: In this paper they adapt unsupervised contrastive learning to the problem of representation learning for proteins from 3D structure, using sub-structure sampling for the data transformation. The reviewers have concerns that the application tasks used for evaluation are not particularly impactful tasks, and that additionally, they are likely to not require protein representations that require more nuanced information. There are also concerns about the clarity of the manuscript, and novelty of the technical approach.
Decision: Reject