Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To overcome the semantic gap between protein sequences and natural language, this paper proposes KeAP (Knowledge-exploited Auto-encoder for Protein) to perform knowledge enhanced protein representation learning. Specifically, it performs protein-centric knowledge exploitation via the attention mechanism, and uses the MLM objective to make it easier to optimize. The generalization ability of the learned protein representation are evaluated by fine-tuning the pre-trained model on a wide range of downstream applications. Ablations and failure analyses are provided to facilitate the understanding of the model.
strength_and_weaknesses: Strength:

1.This paper proposes a new knowledge-enhanced method of Knowledge-exploited Autoencoder for protein representation learning.

2.Experimental results show that the proposed approach achieves better performance than baselines.

3.This paper also provides ablations and discussions of failure cases, which help readers understand the model.

Weakness:

1.It looks like the authors have made some small improvements on the basis of the existing method OntoProtein. Specifically, both of these two methods aim to incorporate external domain knowledge (KG) into protein representation learning. It is questionable whether KeAP actually provides a great contribution or innovation to the community.

2.The three colors in the legend of the Figure1 are not very distinguishable, so it takes a lot of effort to distinguish the corresponding method of each color. In addition, I think the positions of the figures and the tables should be moved to the corresponding part of the text to make it easier for readers to read. For example, Figure1,2,3 and Table1,2,6,7,8. Figure3 is too small to be seen clearly, affecting the reader's understanding of the experiment.

3.I think there are some problems with the author's writing logic. Since the concepts of ProteinKG25 and Gene Ontology are mentioned for the first time in Section4, if readers are not familiar with the settings of OntoProtein, they will not know what relations are described in the protein-related knowledge graph. 
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and well-motivated. The code is without running scripts.
summary_of_the_review: The paper introduces how domain knowledge can be implicitly incorporated into protein representation learning. The authors use several popular ideas, such as the attention mechanism and MLM objective and show its effectiveness on multiple downstream tasks. Unfortunately, the author's writing logic is not clear enough, making the article not particularly smooth to read.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: LM pretraining on protein sequences has become popular in recent years. One difference between proteins and NLP is that for proteins there is often lots of structured/unstructured knowledge graph information for each pretraining example. This paper extends masked language modeling (BERT/MLM) by allow the model to also attend to an encoding of textual side information. This improves the performance of models on a variety of benchmark tasks.
strength_and_weaknesses: =Strengths=
Thorough evaluation on lots of downstream tasks
Lots of citations to recent work in NLP that uses similar ideas.
The general modeling idea (established in the OntoProtein paper) is good.

=Weaknesses=
Methodological contribution is very minor.
Writing is confusing. Lots of poorly-defined jargon (see below).
Potential issues regarding data leakage for test set (see below).

clarity,_quality,_novelty_and_reproducibility: The paper often uses exaggerated and poorly-defined terms that over-emphasize the novelty of the method. These include  'implicit knowledge encoding', 'protein-centric representation', 'primary structure reasoning'  'proteincentric knowledge exploitation', etc. These are not established jargon in the field. Similarly, the complexity/significance of the paper's contribution is over-stated in section 3.2, which explains a standard attention mechanism. In modern ICLR papers, only a couple of sentences should be necessary to describe this. The details could be in an appendix.

The paper considers benchmarking tasks that are publicly available, so they are reproducible in that sense. 

The paper modifies the architecture of a model that was presented previously at ICLR. This modification provides large performance improvements, but the change in architecture is standard (using a standard attention mechanism) and won't be of general interest to the ICLR community.

summary_of_the_review: Can you comment on the relationship between your work and ProGen (https://arxiv.org/abs/2004.03497)? This also uses protein KG information in a large language model. 

When you use your model to get per-residue protein embeddings for downstream tasks, I'm assuming that you pass correct / ground truth KG information. For some of the tasks, does this present a data leakage problem, where there is information provided at evaluation time that reveals information about test-set labels? For example, for the protein-protein interaction task, you're providing GO terms for proteins in the test set. With this, are you effectively learning a model about which GO terms interact with other GO terms?

I was disappointed that, besides Table 7, the experiments don't provide a direct ablation that removes the ontology information. The ProteinBert results are from a separate paper. Surely there are other differences between the systems, such as hyper-parameters.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary of the paper
OntoProtein is prior art that integrates additional information from Gene Ontology into protein presentation. OntoProtein chooses the MLM and TransE as two training objectives when learning protein representation on the Gene Ontology knowledge graphs.
The authors claim that the TransE objective has an issue because, in Gene Ontology, a triple usually concerns (Protein, Relation in natural language, and Property in natural language). While the initial embedding space of proteins and texts differs,  "OntoProtein did not take into account the domain gap between protein and natural language, and this semantic gap may make the TransE objective a suboptimal choice ".
Therefore, the authors proposed a new learning objective where only masked language model loss is considered during the training process. Specifically, random amino acids are masked as usual. Initial embeddings of masked protein sequences and texts (relations and properties) are obtained using pretrained models for proteins and texts respectively. 
These initial embeddings are used by a decoder where attention layers (QKV Vaswani et al.) are used to obtain relation/property attention, given the initial embedding of proteins as a query.  The main objective of the decoder is to learn attentive weights to knowledge graph triples associated with the protein such that it helps predict masked amino acids in the sequence.
The authors claimed that this solution learned to incorporate knowledge implicitly into the protein representation and solved the semantic gap issues. The authors demonstrated the significance of the results compared to OntoProtein in 9 different downstream tasks.


strength_and_weaknesses: Strengths
This a well-written paper.
Significant experimental results compared to OntoProtein.
A good application paper.

Weaknesses

Additional comparisons to state-of-the-art language models are needed:
Recent language models for proteins such as ESM-1B, ESM-2, and MSA-Transformer demonstrate better results on TAPE benchmark datasets. I see you compare your approach with these models on only similarity prediction tasks. Why don't you compare these methods to the other benchmark datasets like contact prediction, homology, stability, protein-protein interaction, and affinity binding? 

A potential flaw in experiments:
Potential leakage happens if proteins in the test sets of the benchmark datasets are not removed from the training, do you check the overlapping between test sets and the ProteinKG25. If that is not the case, the reported results might be over-optimistic.

Related to semantic gaps:
The authors claim that semantic gaps are the main issues of OntoProtein but it is very hard to see any evidence demonstrated in the paper to support that claim. Since this is the main claim against OntoProtein, this needs to be supported by a piece of strong evidence with careful analysis.

Reproducity:
I checked both the paper and the appendix version, this information is needed to clarify the implementation details:
+ what is the representation of proteins used in the encoder? I understand it is a bert-like encoder but whether you use pretrained models or train everything from scratch? 
+ potential leakage happens if proteins in the test sets of the benchmark datasets are not removed from the training, do you check the overlapping between test sets and the ProteinKG25 ?


clarity,_quality,_novelty_and_reproducibility: Well written paper and original work.
There is an issue with the description of implementation details.

summary_of_the_review: Summary of the comments:
Although I see this as an interesting research work I am concerned about its evaluation methods with potential leakage. I am also concerned about its baseline methods to compare. More implementation details are needed to reproduce the results and stronger evidence to support the claim about "semantic gaps"
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel protein sequence pre-training method, Knowledge-exploited Auto-encoder for Proteins (KeAP). Instead of injecting biomedical knowledge by directly applying knowledge embedding constraints as OntoProtein, KeAP employ a cross-attention mechanism to attentively inject relevant relation and attribute representations into protein representations. Upon such enhanced protein representations, masked residue type prediction loss is applied for representation learning. On contact prediction, homology detection, stability prediction and PPI prediction benchmarks, authors demonstrate the superiority of KeAP over OntoProtein and ProtBert (the initial protein sequence encoder of KeAP).  
strength_and_weaknesses: Strength:
1. It is a novel idea to enhance protein representations with associated biomedical texts (e.g., the relevant molecular functions and biological processes). The proposed cross-attention fusion scheme is technically sound to achieve this goal.
2. KeAP achieves some obvious performance improvements over the previous SOTA OntoProtein on contact prediction and PPI prediction tasks.  


Weakness:
1. KeAP does not fully utilize the biomedical texts associated with a protein. For each forward pass, KeAP adopts a single (protein, relation, attribute) triplet and enhance the protein representation with only the relation and attribute texts in this triplet. However, in the biological KG used by authors, each protein can appear in multiple triplets, where these triplets describe orthogonal properties of the protein. Therefore, it is more promising to combine different text descriptions associated with a protein and inject relevant textual information across the combined text. I am curious and willing to see the performance of KeAP under such an improved input format. 
2. Some important baselines are missed in performance comparison. Authors are suggested to include the results of the SOTA protein language models, ESM-1b and ProtT5, into Tables 1, 2 and 3.  

clarity,_quality,_novelty_and_reproducibility: Quality:

In general, this paper is well-written. The motivation and derivation of proposed techniques are clear to the audience.


Novelty:

The idea of enhancing protein language models with biomedical texts is novel and worth more explorations in the future. 


Reproducibility:

Authors submitted source codes for reproducing the results. The codes look well-organized according to my brief check. 

summary_of_the_review: In summary, I am convinced by the motivation of learning better protein representations guided by biomedical texts and the cross-attention technique to achieve this goal. However, I have concerns on the input data organization during pre-training and the completeness of performance comparison. Therefore, I think the current manuscript is on the border and expect authors' efforts during rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of learning representations for proteins for a number of biological downstream tasks while exploiting background knowledge in the form of a knowledge graph. The proposed contribution extends a previous pipeline for this problem, OntoProtein, by proposing a number of engineering modifications -- a simplified loss and the use of cross-attention --  that proves to be successful in boosting the performances of several downstream tasks.

Reviewers appreciated the results and the efforts of the authors and overall liked the contribution. Some concerns about the presentation of paper (e.g., use of jargon, contributions can be summarized more) and its incrementality w.r.t. OntoProtein were raised. Furthermore, risks of potential data leakage were highlighted. During the rebuttal authors were quick to respond and addressed all the major concerns, including the leakage for which new experiments were run. 

The only point of concern left was the incrementality of the work. I personally value the proposed engineering contributions as they, while seeming simple, require effort to be properly implemented and experiments rigorously carried out. I believe the work can be useful for people working on protein representations and it is worth presenting at the conference.

The paper is therefore accepted conditioned on the fact that the authors will include the discussion and results coming from the rebuttal phase. Furthermore they will address the jargon used in the paper as to simplify presentation.
justification_for_why_not_higher_score: We agreed with reviewers that the contributions are limited in scope and focus on a specific application. While protein successful representation learning methods can have a huge impact in real-world applications, this paper is not a breakthrough of the field.
justification_for_why_not_lower_score: I believe the paper is worth a publication, but I understand if this paper makes space for other papers once the acceptance threshold is being drawn.
Decision: Accept: poster