Prediction of Protein-protein Contacts with Structure-aware Single-sequence Protein Language Models

Authors: Derek Huang, Peicong Lin, Sheng-You Huang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: protein bioinformatics, protein language models, protein-protein contact prediction, protein representations, deep neural networks

Abstract:
Accurate prediction of the interface residue-residue contacts between interacting proteins is valuable for determining the structure and function of protein complexes. Recent deep learning methods have drastically improved the accuracy of predicting the interface contacts of protein complexes. However, existing methods rely on Multiple Sequence Alignments (MSA) features which pose limitations on prediction accuracy, speed, and computational efficiency. Here, we propose a transformer-powered deep learning method to predict the inter-protein residue-residue contacts based on both single-sequence and structure-aware protein language models (PLM), called DeepSSInter. Utilizing the intra-protein distance and graph representations and the ESM2 and SaProt protein language models, we are able to generate the structure-aware features for the protein receptor, ligand, and complex. These structure-aware features are passed into the Resnet Inception module and the Triangle-aware module to effectively produce the predicted inter-protein contact map. Extensive experiments on both homo- and hetero-dimeric complexes show that our DeepSSInter model significantly improves the performance compared to previous state-of-the-art methods.

Decision: reject
Reviewer scores: 3, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730108748263

thread_type:
official_review

<thread_object>
creator: reviewer_aiel

<content>
summary:
In this paper, the authors present a state-of-the-art model for protein-protein contact map predictions, without the use of MSA, which indeed are hard to come by in the case of heteromers and represent a big improvement in tackling protein-protein contact map problem.
To do so they essentially built a model that processed each binding  partner separately as well as the complex formed by the partners, and  they did it using different types of information:
- An EGNN is used to discover useful amino acid representations of both partners as separate entities, from structure information.
- A RBF is used to enrich the description of distances between residue at different scales, leading to the building of an intra-distance matrix for both partners as separate entities.
- ESM embeddings are used to derive useful amino acid representations of both partners as separate entities, from evolutionary information.
- ESM attention is used as a counterpart for the distance-based matrix from the RBF transformation.
- The same that was done using ESM is now done using SaProt for retrieval of amino acid information more focused on the local structural environment.
- Finally, both ESM-derived features and SaProt-derived features are also produced for the partner pair. For ESM-related features, it comes with the addition of a linker, and for SaProt, a sequence of unknown tokens as foldseek has no complex structures to work with.
All of those part are then run through a ResNet Inception module followed by a triangle aware module, to produce a protein-protein contact map.

strengths:
Apart from the fact that authors present a model that reaches state-of-the-art performance in most conditions and that it doesn't need MSA (and that's not a small achievement), the paper's strength lays in how it breaks down its conclusions: homodimers, heterodimers, real pdbs, alphafold-generated pdbs, and ablation study. The study by pdb type is super interesting, and seems to show that this model is still useful in the alphafold pdb case but probably more in the homodimeric case, and in the heterodimer case one can still work with this model MSA-based counterpart model.

weaknesses:
This paper has 3 main caveats which I believe are not deal-breakers but for which at least 2 would (and could) benefit for improvement in how it is presented in the paper. 
The first one is that it is a very massive pipeline and understanding what really matters seems hard.
The second one is the lack of a clear description of how to use the model at prediction time.
The last point is about the overall presentation of results and data, mainly incorporating more evaluation metrics and a bit deeper description of the data involved.

questions:
Indeed, it is quite hard to make sense of the ablation study (not the author's fault, the results are what they are). For example for the homodimer case: no_gt_esm as good or even better than the full model? Reverse conclusion for heterodimers? I left the reading with the feeling: it works great but is all of that really necessary? Of course, having a (performant !) model straying away from MSA is very welcomed and necessary, but what matters in the model is not obvious. A way maybe to add to this ablation study would be to have a comparison (just plotting side by side with some metrics for example: F1 score , area under the ROC, difference between contact map, where is the difference on the structure ) for a few complexes for different models.

The second one is the lack of a clear description of how to use the model at prediction time. By that I mean the following. Sequences have to be truncated when they are too big, for obvious memory and time complexity reasons, but those truncations, at least given how they are explained, rely on some ground truth of a sequence window that has maximum PPI. At prediction time if we have a pair of big proteins and no idea how to choose this window how do we do? Moreover, how can we assess the confidence of those pairings in the contact map? Could we have then a study where for long protein sliding or random windows are chosen and distribution of predicted contact residue numbers are plotted: in that case is the model most of the time predicting the maximum number of contact at the right ground truth window? How often does it not? Is this a strategy at prediction time, and can we use this distribution to have a confidence about the window chosen?

The last point is about the overall presentation of results and data:
- Results: Could we have more metrics (F1 score, area under the ROC) than the precision which would give more information about the different types of errors? Maybe a few confusion matrices for some cases where the model works fine and for when the model works worse. Also, a few of those contact maps predicted as well as how that translates on the structures. I find it really hard to draw a good picture from a paper when only one metric (a statistical one in this case) is used for the whole discussion.
- Data: Some data analysis of the training data. For example, I have no idea what is the distribution of number of contact interactions given some protein length. Mainly as the sequence has to be cropped around part of the sequence with maximum contact with the other protein partner, I would like to have an idea of what that maximum represents:  are we still in the super imbalanced class regime, how easy it is to reach 80% precision in that case by just putting every pairs of residue  as contact residue and so on. Is this cropping strategy leading to a very different distribution of contact fraction than the non-cropped one? How sensitive is it to this choice of 320 residues and so on...

Side questions: 
Would a more stringent but identical preprocessing step for both pdb types (real vs alphafold) be a possible way to reduce the drop in performance?
For the drop of performance between homodimers and heterodimers, did the author try to bump up the loss weight for heterodimers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730124532003

thread_type:
official_review

<thread_object>
creator: reviewer_xbjr

<content>
summary:
The paper presents DeepSSInter, a novel transformer-based deep learning model designed to predict protein-protein interface contacts using structure-aware single-sequence protein language models. This model's efficiency and predictive power mark a significant advancement in protein interaction prediction, particularly beneficial for cases where MSA is unavailable or inefficient.

strengths:
Originality: The model creatively combines advanced components like graph-based features, ResNet-Inception, and a triangle-aware module, innovatively moving beyond MSA. These elements effectively address challenges in both computational complexity and prediction accuracy, especially for heterodimers, which are historically difficult for MSA-based models.

Quality: The authors validate DeepSSInter rigorously through extensive experiments on homodimeric and heterodimeric datasets, consistently outperforming five state-of-the-art methods in precision metrics. Ablation studies and evaluations with AlphaFold2-predicted structures further strengthen the model’s demonstrated robustness and confirm the necessity of each architectural component.

Clarity: The paper is well-organized, with clear descriptions of model components, experimental setup, and visual aids like architecture figures. Though some technical terms could use more explanation, the methodology and results are easy to follow.

Significance: By eliminating the need for MSA, DeepSSInter is both faster and more accurate, marking a significant advancement for applications in structural biology and drug discovery. This approach paves the way for single-sequence modeling in protein interaction research, making it impactful for high-throughput and resource-efficient studies.

weaknesses:
1. Model Limitations with AlphaFold2-Predicted Structures
DeepSSInter's performance declines with AlphaFold2-predicted structures, especially for heterodimeric complexes, where it underperforms compared to DeepInter. This discrepancy suggests that single-sequence protein language models may lack the coevolutionary information MSA provides, which is crucial for some complex predictions. The paper would benefit from deeper analysis on this limitation, perhaps through: Alternative or augmented data inputs: Exploring ways to integrate low-dimensional coevolutionary features from paired MSAs, even minimally, could help bridge the gap between structure-aware single-sequence models and MSA-based predictions. Fine-tuning: Considering an additional fine-tuning phase on AlphaFold2-generated complexes could improve robustness to predicted structures, enhancing real-world applicability.
2. Limited Interpretability of Model Outputs
Although the paper shows improved prediction precision, the interpretability of the model outputs remains unaddressed. For practical applications in structural biology, understanding why certain contacts are predicted is often as important as the predictions themselves. Including feature attribution methods, like attention weight visualizations or feature importance analyses on the ESM2 and SaProt modules, could provide insights into how the model makes predictions, aiding users in the interpretation of contact predictions.
3. Broader Benchmarking and Testing
The paper focuses on precision metrics in homodimeric and heterodimeric complexes, but testing could be broadened to include more diverse protein types, such as oligomeric or multimeric complexes. These more complex structures are increasingly common in real-world biological contexts. Expanding benchmarks to these protein forms could provide a fuller view of DeepSSInter’s performance and generalizability.

questions:
1.Could you provide more interpretative insights into which features or attention patterns are most important for your model's predictions?
2.Given the reliance on specific protein language models (ESM2 and SaProt), have you evaluated the model’s adaptability to other 
3.Have you considered evaluating DeepSSInter on other types of protein complexes, such as oligomeric or multimeric structures? If so, what were the outcomes?
4.Could you elaborate on the choice of using both ResNet-Inception and triangle-aware modules? How do these components interact, and do they provide overlapping or distinct contributions to model performance?
5.Given the performance decline with AlphaFold2-predicted structures, have you considered fine-tuning the model on such predicted structures? If so, what were the results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618021461

thread_type:
official_review

<thread_object>
creator: reviewer_yodi

<content>
summary:
The paper introduces DeepSSInter, which predicts protein-protein interactions by combining existing protein language models.

strengths:
evaluation performances are better than selected baselines.

weaknesses:
The paper seems more like a wrapped-up of existing methods and provides little new insights on the problem, making the submission a workshop-level paper instead of research paper.

The paper largely ignores existing methods in multimeric protein structure predictions with MSAs ([1,2]) or PLMs ([3,4]). These methods share identical motivations to the proposed one, especially [4] which proposes to predict protein-protein interactions using protein language models. Discussions and comparisons should be made against these methods. Also comparisons with protein-protein docking methods ([5,6]) should be discussed, as these methods are already broadly used in real applications.

[1] AlphaFold Multimer https://www.biorxiv.org/content/10.1101/2021.10.04.463034v1 ;
[2] RosettaFold https://www.science.org/doi/10.1126/science.abj8754 ;
[3] ESM-Fold https://www.science.org/doi/10.1126/science.ade2574;
[4] Uni-Fold MuSSe: https://www.biorxiv.org/content/10.1101/2023.02.14.528571v1 ;
[5] HDock: https://www.nature.com/articles/s41596-020-0312-x ;
[6] Haddock: https://rascar.science.uu.nl/haddock2.4/ ;

questions:
Figure 3 and 4 should be reorganized  for readers to extract information more easily. Currently it is hard for readers to accurately compare all the colored bars.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693323378

thread_type:
official_review

<thread_object>
creator: reviewer_n9py

<content>
summary:
This work proposes DeepSSInter, a transformer-based deep learning model for predicting protein-protein interface contacts without relying on Multiple Sequence Alignments (MSA). DeepSSInter combines single-sequence and structure-aware protein language models, utilizing intra-protein distance and graph representations to capture the structural and evolutionary properties of interacting proteins. In particular, the model leverages ESM2 and SaProt representations, a ResNet-Inception module, and a geometric triangle-aware module to enhance prediction accuracy. Experiments on homo- and heterodimeric complexes demonstrate that DeepSSInter outperforms state-of-the-art methods, particularly in challenging heterodimeric cases.

strengths:
The work's primary strength lies in its approach to protein-protein contact prediction, addressing the limitation of existing methods by eliminating the need for Multiple Sequence Alignments (MSA). The method’s reliance on single-sequence data and its use of structure-aware language models (ESM2 and SaProt) offer computational efficiency and scalability, broadening its applicability to proteins with limited or no homologous sequences.

weaknesses:
The whole work is like a variant of DeepInter [1] with incremental improvements. The architecture of the proposed model only changes the way of extracting embeddings from input when compared to DeepInter. The datasets and experimental settings are all the same as those in DeepInter.

The description of the proposed method is not clear. It is unknown about the details of the "ResNet-Inception module" and "triangle-aware module". It seems that they are directly borrowed from DeepInter, but the authors didn't mention that in the manuscript. 

In Table 4, it is misleading that the results of the proposed DeepSSInter are all highlighted but they all perform worse than DeepInter.


[1] Lin, P., Tao, H., Li, H., & Huang, SY. Protein-protein contact prediction by geometric triangle-aware protein language models. Nature machine intelligence, 5, 1275-1284 (2023).

questions:
How the ligand 1D and receptor 1D features are combined with those 2D features?

How the attention representations are generated by ESM2 and SaProt? Each hidden layer of PLMs can have attention representations.

Are there multimers for the receptors in the datasets? If so, how are they processed by the PLMs?
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper proposes DeepSSInter, a transformer-based deep learning model designed to predict inter-protein residue-residue contacts by integrating single-sequence and structure-aware protein language models (PLMs). It aims to improve the prediction of interface contacts between interacting proteins without relying on Multiple Sequence Alignments (MSA), which can limit prediction accuracy, speed, and computational efficiency. 

(b) Strengths of the paper include its innovative attempt to move beyond MSA for predicting protein interactions, thereby addressing challenges in computational complexity and prediction accuracy. The model's design incorporates advanced components like graph-based features, ResNet-Inception, and a triangle-aware module, which are intended to enhance performance. The authors also provide experimental validation across different datasets, demonstrating improvements over several baselines.

(c) Weaknesses and missing elements
- Lack of novelty: The work appears to be a variant of DeepInter with incremental changes. Key modules like the "ResNet-Inception module" and "triangle-aware module" are directly borrowed, and the datasets and experimental settings are similar.
- Initial method description: The original description of the proposed method was unclear, particularly regarding the details of the borrowed modules.
- Performance with AlphaFold2-predicted structures: The model's performance declines with AlphaFold2-predicted structures, especially for heterodimers, indicating potential limitations in handling complex structures and suggesting a lack of coevolutionary information compared to MSA-based models.
- Interpretability: The interpretability of the model's outputs is not addressed, which is crucial for practical applications in understanding the basis of contact predictions.
- Testing scope: Testing is mainly focused on homo- and heterodimeric complexes, and could be broadened to include more diverse protein types like oligomeric or multimeric complexes.

(d) The most important reasons for rejecting this paper are the limited novelty in the proposed architecture compared to existing models, inconsistent performance improvements over baseline methods, and insufficient exploration of the model's limitations and generalizability. These factors raise concerns about the paper's contribution to advancing the field.