Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses contact prediction for PPI by proposing a geometric deep learning framework. They use a k nearest neighbor representation for each protein and compute geometric-based attention scores to convolute the messages over this graph. They validate this approach on the PPI task, using the DIPS-plus dataset. 
main_review: Strength : 
- The approach is original and performs better than a naive GNN implementation.
- Using a more comprehensive data set to train the methods should lead to increased results, it would be an interesting avenue in general to retrain former data-driven methods over this larger data set
- The tackled problem is interesting and the results presented here are claimed to be sota.

Weaknesses : 
- Even though the approach is novel, there is not enough evidence that it is needed. Several choices seem to be counter-intuitive and make the approach hard to follow, even with explicit equations. For instance, the attention mechanism is split into successive attention computations. The authors build several geometrical invariant descriptors, why cannot they be simply concatenated into one vector and compute a single attention over this pairwise geometric vector ? In this formulation, I think the approach would more or less reduce to graph transformers.
- The claim that this method could be used on  "other 3D graphs" is too strong because to my understanding, the invariant encoding of the geometric information heavily depends on the ordering of the points. A canonical ordering is not available in general.
- A general equivariant graph based already exists."SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks" Fuchs et al. Another more recent and simpler equivariant method can be found in "E(n) Equivariant Graph Neural Networks" (Sattoras et al.). These methods seem to be applicable in your setting here.

- The data used looks very relevant, but since the learning leverages geometric features, I would suggest conducting the splitting of your dataset based on geometric similarity as was done for instance in Maasif (you could use CATH classes or TM-score splitting).
- The choice of the methods against which the tools is benchmarked should be expanded and these methods properly introduced. I think it should include PINET ("Protein interaction interface region prediction by geometric deep learning" Dai et al.) that is to my knowledge the state of the art for this task. Despite what was stated, to my understanding PINET does target-specific prediction. The fact that the performance of the concurrent methods might be corrupted because of improper splitting should be addressed by authors, for instance by removing from the test sets interfaces too close to the training set (see structural splits above).
summary_of_the_review: This paper pursues two complementary aims. A first one is building a predictor for contact residues using the structures of both proteins interacting. To do so, the authors propose leveraging a larger data set along with a novel architecture. However the choice of the methods included in the benchmark is not motivated enough and should include more recent state of the art.

The second one introduces a novel invariant geometry-based attention for use in graph transformers. It also includes using edges as pseudo-nodes in this graph. These ideas seems very interesting and could be validated onto other classical tasks involving protein structure such as binding site prediction. However the chosen formulation sounds a bit complicated and could be motivated by comparing it to a simpler version. Such an analysis should be complemented by a more extensive comparison to existing equivariant methods. I think such a contribution could be a standalone contribution, but the current version of the paper is not motivating this approach enough.

I would advise to reject the paper in the current form, and encourage authors to pursue the validation of their paper along these two avenues.


########################

After an interesting discussion with the authors and inclusion of further results, I changed my evaluations twice and now recommend this paper for acceptance. I would advise the authors to work more on the writing of the paper using the supplementary for the equations and emphasizing more the novelties of their methods compared to previous work (backed up by ablation studies disproving the usefulness of these technical choices).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a state of the art deep learning architecture for predicting mutual contacts between proteins.
Specifically, this work presents a Geometric Transformer for rotation and translation invariant protein interface contact prediction,
given 3D protein structures.


main_review: Strengths:
1. The strengths of this work stem from the architecture used, consisting of attention mechanisms and invariant representations.
2. This work is uses a novel geometric neighborhood representation which considers edges as pseudo-nodes, with geometric feature updates.
3. The exposition and presentation are clear, and the work makes available the data, models, and code. 

Weaknesses:
1. The performance metrics and benchmarks are lacking:
- Specifically, the evaluation metric is precision; whereas recall is missing.
- The results are not compared with standard benchmarks, and require porting different results for comparison.
2. The network trained is relatively small and may not scale well.
3. The work does not predict distograms or a sequence of pose updates.

Missing is a reference to recent related work by Costa et al. from MIT:
End-to-end Euclidean equivariant Transformers for protein docking, NeurIPS Workshop on Learning Meaningful Representations of Life, 2021.

Minor typo: Section 5.2 should read "To identify..
summary_of_the_review: In summary, the proposed deep learning architecture for protein interface contact prediction is somewhat new,
though aspects of this contribution exist in related work. The performance metrics used and baselines may be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper aims to improve the prediction performance of the protein-protein interaction, which involves predicting partner-specific protein interface contacts (i.e., inter-protein residue-residue contacts) given the 3D tertiary structures of two proteins. In this paper, the authors present a graph transformer to encode the protein geometric features and perform interface contact prediction. The proposed method is validated on two benchmarking datasets, i.e., DIPS-Plus and CASP-CAPRI datasets.

main_review: 
Weakness:

1. The key contribution of this paper is not clear. The author should clarify the difference between the proposed method and the graph transformer (Dwivedi & Bresson (2021). Several related works should also be discussed, such as [1] and [2].

2. The method section as well as the figures is hard to read. The author should use equations to describe the computations of the DL model.

3. More than two important baselines (such as [2] and [3]) are not included. The author should also conduct the experiments on the DB5 dataset, which is widely used for protein-protein contact prediction.

 
[1] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8:13890, 2017.

[2] Liu, X., Luo, Y., Li, P., Song, S., & Peng, J. (2021). Deep geometric representations for modeling effects of mutations on protein-protein binding affinity. PLoS computational biology, 17(8), e1009284.

[3] Yi Liu et al. “Deep learning of high-order interactions for protein interface prediction”. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, pp. 679–687.

Overall, I am inclined to reject. The proposed architecture seems not novel, and I have some serious concerns regarding the experiments.

summary_of_the_review: Considering the noticeable improvement made by the authors, I reconsider my original view and agree with the other reviewers on the significance of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel geometry-evolving graph transformer for protein interface contact prediction, named DeepInteract. In particular, it predicts partner-specific protein interface contacts (i.e., inter-protein residue-residue contacts) given the 3D tertiary structures of two proteins as input, showcasing its effective use in learning representations of protein geometries to be exploited in downstream tasks. The experiments on challenging protein complex targets also demonstrated the proposed method achieves SOTA results for interface contact prediction.
main_review: ##########################################################################

Pros:

- The paper provided the first example of graph self-attention applied to protein interface contact prediction, which is an interesting idea for the protein bioinformatics area.
- The proposed Geometric Transformer can be used for tasks on 3D protein structures and other 3D graphs. The author trains the Geometric Transformer to evolve a geometric representation of protein structures simultaneously with protein sequence and coevolutionary features for the prediction of interchain residue-residue contacts, demonstrating the merit of the recently released Enhanced Database of Interacting Protein Structures (DIPS-Plus) for interface prediction.
- The paper is well-written and the design decisions are clearly explained. The comparison of benchmark methods is also interesting to read. 

##########################################################################

Cons:

- The core idea of this paper is Geometric Transformer, however, it seems that the difference between Geometric Transformer and Graph Transformer is not very big except for the last layer, because the essence of these two models is multi-head attention. Although the author stated in the paper the new modules of Geometric Transformer, including EDGE REPRESENTATION INITIALIZATION, CONFORMATION MODULE, NODE REPRESENTATION INITIALIZATION, and INTERACTION MODULE, it will be more convincing if the author can compare with Graph Transformer.
- In the model description, the author lacks the statement of the objective function and loss function.
- In the experiment comparison, if the author can add an ablation study experiment, it will be more convincing. In addition, from the experimental results in Table 1 and Table 2, Geometric Transformer does not seem to be significantly improved compared to Graph Transformer. Can the author provide more explanations?
- From Figure 1, the author provided the visualization of interacting protein chains, can the author also provide visualization of the predicted results? Such as qualitative analysis.
summary_of_the_review: Considering the above pros and cons, my recommendation of the paper is marginally above the acceptance threshold.
review_title: null

Meta-review: This paper presents a novel neural network architecture to predict interacting residues among two interacting proteins, and evaluates its performance on benchmarks. While the reviews were initially mixed, there has been a productive discussion and significant improvements in the paper during the discussion, including in particular much needed clarifications about the proposed methods, and more experimental results with an ablation study to better assess the benefits of various design choices. While no reviewer is willing to champion this paper as a "strong accept", due to the relatively modest novelty compared to existing methods, there is a consensus towards "weak accept" given the final quality of the work presented and potential usefulness of the method for the problem tackled.
Decision: Accept (Poster)