Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces Relative Molecule Attention Transformer, adding distance-, bond- and a neighborhood-embedding  in the attention matrix, and pretrain the model on 4M molecules via context pretraining and on 200 rdkit-descriptors.
main_review: Adding distance information directly to the learned "distance" before computing the softmax as well as additional information as in MAT is a very good idea (similar to ALiBi [1]). Results in Table 3 are supportive for adding distance-, bond- and a neighborhood-embedding in the attention matrix.
There are datasets the support a performace increase of the proposed method over baselines and established methods, however
 1. Distance Matrix in MAT vs. In R-MAT: is that it is first embedded, then a learn projection; rather than taking the distance matrix directly times an e.g. a learned scalar; is there a clear advantage of one over the other?
 2. Table 1 reports the best performance for ESOL and FreeSolv, given that R-MAT was pretrained on 200 rdkit-descriptors and in one model-variant uses those descriptors in the last layer, please make sure to also include a baseline variant of e.g. RF with the 200 rdkit-descriptors (a few of whom describe solubility), (additional to using fingerprints)
 3. Please state why between the Experiments (1-2) splits were changed from random to scaffold, as well as the normalization of the labels
 4. In Table 9: "R-MAT outperforms other variants across the three tasks" however in ESOL type 1 performs best/not significantly differently
 5. One disadvantage is the costly calculation of the distance matrix, which should be mentioned at least once in the paper
 6. Reproducibility: Code to reproduce the results is not provided

 
Minor Issues:
- "we report the mean test score" but the values in brackets for table 1, 2, .., are they std, var, or SD?
- How were the fixed-hyperparameters for Experiment 1, small-hyperparameter budget selected?
- 8 nVidia A100 GPUs --> Nvidia
- Table 2: QM7 should be changed to QM9
- Equation (1): by adding to vanilla self-attention e_{ij} , doesn't have a variance of sqrt(d_z), so the denominator in the softmax might be adjusted
- The distance matrix might change dependent on calculation, any improvements from running multiple runs?; distance is not static; rotatable bonds

[1] Press, O., Smith, N.A., & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ArXiv, abs/2108.12409.
summary_of_the_review: Several issues have been discussed: Experimental results should be compared to additional baselines, changes in the experimental setup are not justified and seem unnecessary, drawbacks are not adequately discussed.  In the current form, the paper is marginally below the acceptance threshold but the reviewer is inclined to change the rating, in light of new supportive evidence.

**update**: The authors successfully tackled the issues I raised with the manuscript, therefore I am happy to raise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new transformer network architecture to pre-train the molecule datasets. Based on the Molecule Attention Transformer, the proposed model, R-MAT, incorporates a few handcrafted features into the self-attention layer of a transformer architecture. The features can incorporate distances between atoms from multiple perspectives. The experimental results show that the pre-trained model is useful to predict various properties of molecules.
main_review: - Pros
    - The experimental results are comprehensive.
    - The paper is easy to follow. Well written.

- Cons
    - Lack of intuitive understanding of proposed features w.r.t. tasks to solve.
    - Theoretical justification on why it is impossible to learn these features with the vanilla Transformer or how difficult to learn such features.
    - Difficult to reproduce the result due to the large scale experiments

- Questions
    - How many overlaps between the task datasets and pre-trained datasets? Does this overlap influence the test performance?
    - Increasing the maximum neighborhood order does not improve the performance.
        - The results seem inconsistent. How do we interpret the result?
    - What makes EGNN perform the best with the QM9 dataset (Figure 3) and why the proposed method cannot achieve a similar result?
summary_of_the_review: Based on the pros and cons written above, I think the paper is marginally above the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a relative self-attention layer for the Transformer model. The relative self-attention of two atoms consists of their relative distance, their shortest path distance in the molecular graph, and their physiochemical. The proposed relative molecule attention Transformer can be first pretrained with a contextual property prediction task and then a graph-level prediction task. The pretrained Transformer can be finetuned on downstream molecular property prediction tasks and achieves excellent performance.
main_review: Strengths:

The proposed relative self-attention is carefully designed to consider three factors: relative distance, shortest path distance in the molecular graph, and physiochemical. Inspired by the positional bias and content bias from NLP methods, a novel relative molecule self-attention is proposed and helps the Transformer model achieve good experimental performance. 

Weaknesses:

Some concerns about the evaluation. In Table 1, why do only two datasets (BBBP and Estrogen-$\beta$) use scaffold splits, and the other four datasets use the random split? The baseline method GROVER evaluates on multiple graph classification datasets, why only BBBP is evaluated in Table 1? While another dataset BACE is only evaluated in Table 2? Could the authors include results on more datasets like GROVER? Evaluating only on selected datasets makes the readers suspicious of the generalization of the proposed method.


summary_of_the_review: The paper proposes a novel relative molecule self-attention layer for the Transformer model. The model can be first pretrained and then finetuned on downstream molecule property prediction tasks. Experimental results demonstrate the effectiveness of the proposed method. However, there are some concerns about the evaluation as described in the Weaknesses section.
review_title: null

Meta-review: This paper presents an extension to the MAT model by using relative attention that considers graph-level distance, geometric distance, and bond type between nodes during the attention computation.  This is shown to lead to improved performance on several benchmark datasets against MAT and GROVER.  The inclusion of additional information into the attention computation is a sensible and natural choice for transformer with the success of relative positional embedding in the NLP and vision domains. But it is somewhat a straightforward extension of the existing ideas from other domains to MAT, hence the novelty is somewhat limited.  It is also worth noting that the proposed method should be considered in the context of the larger body of research on 3D GNNs. The authors drew inspiration from DimeNet's design in the encoding of geometric distances but do not consider it in the empirical comparisons. Instead, it focused exclusively on transformer based models. This limits the scope of conclusions that we can draw from these experiments and makes it difficult to gauge the practical impact of RMAT in comparison to many other GNN methods that uses 3D geometries of the molecule.
Decision: Reject