Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel network architecture called Substructure-Atom Cross Attention (SACA) for molecular graphs that effectively combines Transformers and GNNs. Substructural patterns are important in molecular tasks, as we see in traditional chemoinformatics fingerprints like ECFP and fragmentation like BRICS/RECAP, but it is also widely known to be essentially hard for GNN to recognize substructure isomorphism. GNN is known to be upper bounded by 1-WL, whereas higher-order WL usually requires heavy computational cost. So explicitly incorporating substructural motifs can be a nice inductive bias to improve the prediction of GNNs in molecular tasks. The paper's simple idea is 1) prepare a motif vocabulary (e.g., MACCS keys), 2) for an input molecule, motif patterns detected from SMILES are fed as tokens to Transformers, 3) in parallel, the molecular graph is fed to GNN to get the node features representing local information around each node. 4) Then, GNN node features are used as keys & values in a Cross Attention layer with Transformer features as queries, 5) build a stack of Self Attention -> Cross Attention + a stack of Self Attention along with the CLS token for global output. The empirical studies demonstrated that this substructure-Transformer + molecule-GNN performed well in 11 downstream tasks after pretraining with a large ChEMBL + PubChem dataset.

strength_and_weaknesses: [Strength]

- The paper presents a simple, easy-to-understand, and effective design pattern to fuse Transformers and GNNs for molecular tasks with good empirical supports. It can incorporate the predefined knowledge of motif vocabulary into representation learning from molecules.

- It was very nice to see the presented method didn't try to explicitly match the predefined subgraph motifs to the input molecules, and this enables a computationally efficient and clean architecture. It would require large-scale pretraining but the relationship between motifs and input molecules is indirectly acquired in a data-driven way. In contrast, typical research considering substructure-aware methods often requires a bit costly computations such as graph matching, subgraph isomorphism, motif tree compositions, etc.

- The details are carefully followed by ablation studies (without GNN vs End-Concat vs Begin-Concat vs Random Embedding vs proposed, the results when we use different GNNs instead of GIN), experimental validations cover a wider range of competitive methods including several Graph Transformers, the pretraining is done with a large 1.8M record dataset collected from ChEMBL and PubChem, complementary analysis for practical uses such as visualization of learned features by tSNE, visualization of attention weights, and computation time.

[Weaknesses]

- One of the concerns is a potential risk of data leakage in evaluations. The paper's empirical studies are grounded on the test performance on 11 tasks of MoleculeNet. But the base model is pretrained with a large (self-collected?) dataset of 1,858,081 molecules from PubChem and ChEMBL that might also cover some/large parts of the MoleculeNet dataset. Though I understand that the pretext tasks for pretraining (to predict 200 RDKit-calculated molecular descriptors) are completely different from the MoleculeNet tasks.

- Another concern is about the use of MACCS keys. It is a set of 166 keys and usually considered to be a bit weak to represent molecules in the entire ChEMBL and PubChem database. This is basically why PubChem developed their own version of substructural keys called PubChem Fingerprint that is a 881-bit-long structural key defined at https://pubchemdocs.ncbi.nlm.nih.gov/data-specification   
   
If the input molecule doesn't have any of 166 structural keys, the input to Transformer branch (the main branch) becomes empty, and this model doesn't make any sense. This would be one of the most well-known disadvantages to use predefined motifs (as in MACCS keys) rather than motifs occurring in the dataset (as in ECFP). So what was the average number of on-bit keys out of 166 motifs? This case would not be likely to happen?  
  
Also, it seems no problem if we just use data-driven motifs, by BRICS or ECFP, if the on bits are hashed into a fixed number of bits. Have you tested any other options than MACCS keys, particularly data-driven ones such as BRICS motifs or ECFP keys?

- Related to the above point, the following three related methods [1] [2] [3], which were not included in the empirical evaluation of the paper ([1] was cited though), use data-driven motifs with GNNs or Transformers or both. Of particular interest, the "Structure-Aware Transformer (SAT)" [3] is based on Transformer + GNN like, First, extract k-hop subgraphs, process them by GNNs, and fed outputs into Transformer as Q and K with V as input molecule. So it would be highly appreciated to discuss the relationship. Is this pattern "End-Concat"..? (First, motifs occurring in the input molecule are fed into GNNs, and their outputs are fed into Transformer. See the picture at https://github.com/BorgwardtLab/SAT)

[1] Zhang et al, Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. (NeurIPS 2021)
    https://arxiv.org/abs/2110.00987

[2] Yu and Gao, Molecular Representation Learning via Heterogeneous Motif Graph Neural Networks (ICML2022)
    https://arxiv.org/abs/2202.00529

[3] Chen et. al, Structure-Aware Transformer for Graph Representation Learning (ICML2022)
    https://arxiv.org/abs/2202.03036

Disclaimer: I have no relationships with the authors of [1] [2] [3], just in case.
clarity,_quality,_novelty_and_reproducibility: This paper's writing is very good and carefully organized. The figures and examples are clean and persuasive. They promised in the Introduction that "The source code and pretrained networks will be released in the public domain upon the paper acceptance", and reproducibility would follow.

As for the novelty, although individual techniques are widely known, the paper presents an interesting, simple, and novel architecture by effectively combining relevant techniques. 

summary_of_the_review: This paper presents a novel network architecture called Substructure-Atom Cross Attention (SACA) for molecular graphs that effectively combines Transformers and GNNs. It is based on a simple idea of the use of predefined motifs (MACCS keys). The on-bit motifs are fed into Transformer as tokens (via embedding layers), while the input molecular graph is processed by a GNN and feedback to the Transformer through the Cross Attention layer. Even though I have several concerns, as described in the [Weaknesses] section, I liked the paper overall. It would always be nice to see that a simple idea worked. On the other hand, I believe that MACCS keys should be replaced by better ones to make inputs (almost) always valid.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework that incorporates Transformer and GNN for molecular representation learning. In particular, the Transformer branch encodes molecular substructures by using molecular fingerprints as input, and the GNN branch extracts local node features from molecular graphs to provide embeddings for the fusion network's cross-attention in the Transformer branch. 
strength_and_weaknesses: Pros:
1. The paper is well-organized and easy to follow. 
2. The ablation study is well-designed. 

Cons:
1. The idea of combining the information from molecular fingerprints and node embeddings learned from GNN is not novel and has been proposed in [1, 2]. 
2. The results shown in Table 2 are not convincing enough. The authors use random scaffold split and rerun the experiments 3 times following GROVER. Since the results between GROVER and the proposed model are very close (usually smaller than the standard deviation), it would be necessary to know if the differences are statistically significant. Otherwise, a fixed scaffold split may be used for fair comparisons. Besides, when comparing the proposed model to MPG, the proposed model outperforms MPG as shown in Table 2, while their performances are on par as shown in Table 2. I'm concerned about whether the other models would also have such different performances when changing the splitting way. Lastly, there is work [3] using pretraining strategy that has better results on the datasets being compared.
3. The authors claim that their model has O(n) complexity according to the self and cross attention map in the Transformer branch. However, for the cross attention map, the complexity is actually O(nm), where n is the number of atoms, and m is the number of substructures. Since the number of substructures should be linear to the number of atoms in a molecule, I think the resulting complexity should be O(n^2).
4. There is a mismatch between the shape of the rectangles and the size of the matrics as shown in Figure 3. For example, a 3*4 grid and a 2*4 grid all represent an m*d matrix, which is misleading.

[1] Fang, Y., Yang, H., Zhuang, X., Shao, X., Fan, X. and Chen, H., 2021. Knowledge-aware contrastive molecular graph learning. arXiv preprint arXiv:2103.13047.

[2] Cai, H., Zhang, H., Zhao, D., Wu, J. and Wang, L., 2022. FP-GNN: a versatile deep learning architecture for enhanced molecular property prediction. arXiv preprint arXiv:2205.03834.

[3] Xia, J., Zheng, J., Tan, C., Wang, G. and Li, S.Z., 2022. Towards effective and generalizable fine-tuning for pre-trained molecular graph models. bioRxiv.
clarity,_quality,_novelty_and_reproducibility: Due to the existing related works, the proposed model has limited novelty.
summary_of_the_review: Given the limited novelty and the unconvincing comparisons to the baselines, I would not recommend an acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: # Summary
Substructure-Atom Cross Attention for Molecular Representation Learning

## What is the problem?
Prediction of molecular properties via deep learning methods is important for AI-aided drug discovery.

## Why is it impactful?
Drug discovery is expensive, so better machine learning methods to solve this task could help accelerate drug discovery at lower costs.

## Why is it technically challenging/interesting (e.g., why do naive approaches not work)?
Processing molecular graphs is challenging, as important features exist at various scales within the input graphs, and we have a puacity of labeled data to train high-capacity methods.

## What is this paper's contribution?
This paper proposes to use extracted molecular substructures (extracted via MACCS keys) and atom-level features simultaneously via a two-branched neural network approach, where molecular substructures are processed via a transformer architecture and atoms via a GNN architecture.

In addition, the authors use a large-scale multi-task pre-training approach to initialize their architecture, thereby helping ameliorate for the small dataset scale available here.

## How do these methods compare to prior works?
While there are prior works that use both GNNs and transformers together, and approaches that leverage graph transformers directly, I am not aware of any papers that jointly leverage atomic features and MACCS substructures simultanesouly.

## How do they validate their contributions?
The authors validate their contribution against a variety of baselines from the MoleculeNet benchmark, finding that their approach achieves rank 1.8 on average and 1.6 on average in classification and regression tasks, respectively. Baselines appear to be compelling, and span both pre-trained and non-pre-trained approaches.

strength_and_weaknesses: # Strengths and Weaknesses

## Key Strengths (reasons I would advocate this paper be accepted)
  1. I think this approach is well motivated, aimed towards an impactful problem, and well presented.
  2. The results you present are compelling, and compare against meaningful baselines using appropriate evaluation metrics.

## Key Weaknesses (reasons I would advocate this paper be rejected)
  None.

## Minor Strengths (things I like, but wouldn't sway me on their own)
  1. I like that you include ablation studies motivating the design choices of your system.
  
## Minor Weaknesses (things I dislike, but wouldn't sway me on their own)
  1. There are some missing details; for example, how are positional encodings generalized for their substructure transformer in this case? In addition, while hyperparameter search is described, architecture search is not (e.g., how many layers are used).

clarity,_quality,_novelty_and_reproducibility: # Clarity, Quality, Novelty, and Reproducibility
## Clarity
The paper is clear and well written.

## Quality
The experiments used appear to be of high quality and well support the final conclusions.

## Novelty
To the best of my knowledge, this paper is meaningfully novel, and the presented final results certainly suggest this approach has significant merit.

## Reproducibility
This may be challenging to reproduce given the complexity of the problem, and it is not stated in the work whether or not code is planned to be released.

summary_of_the_review: I think this is a very strong, well-written paper proposing a meaningful solution to an important problem and demonstrating the utility of that solution via compelling results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Summary of the paper
The paper proposed a simple molecule representation learning method that combines representation from a transformer and a GNN network to leverage the strengths of each method. While transformers are good at learning frequent substructures or motifs in a molecule database, GNN provides additional information about missing structures due to the linearization of molecule graphs.

The authors also pretrained their proposed networks with 1.8 M molecules with the objective of predicting 200 real-valued descriptors of physical-chemical properties from the pretraining datasets using RDKit.

The experiments on 11 downstream tasks demonstrated significant results  of the proposed methods compared to different transformers  and GNN networks both with pretrained/non-pretrained models 






strength_and_weaknesses: Strengths
The paper is well written. The results look promising. The idea of the fusion of the representation from two different architectures to leverage the strengths of both methods is a good practice worth sharing with the application community.

The examples in Figure 1 are an interesting motivation example for leveraging motif substructure in molecules to improve the disadvantage of GAT GNN.

Weaknesses
This is rather a simple approach trivially combining existing techniques. For the ICLR conference, I would like to see more depth on technical contribution. I would suggest comparing the following simple baselines:

 + Ensemble of GNN and Transformers Models, just consider a linear model with the same weight as all the baseline models

+ Recent works have considered learning a space that combines information from 2D and 3D data GraphMVP, please look at https://arxiv.org/abs/2110.07728, this is a good baseline as GraphMVP help fix the issues that models learned from 2D data that miss information about the 3D arrangement of the atoms in the molecules.


Potential leakage during pretraining. It is important to remove test molecules from the data used for pretraining. This removal is important to make sure that there is no leakage happens.


clarity,_quality,_novelty_and_reproducibility: The paper was well written. Novelty and technical contribution are limited. The code was not included in the submission. 
summary_of_the_review: Summary of my comments
The paper was well-written and I see that its method for combining strengths from two different representations of learning architecture is an interesting practice but a combination of these two known techniques is not technically strong enough for an ICLR paper. 
I would recommend the authors validate the fusion approach against a simple linear ensemble of the baseline methods listed in your paper. Also, it is important to remove the test molecules of the benchmark datasets from the pretraining data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a transformer-based architecture for molecular property prediction. The proposed method represents a molecule at two different levels: substructure-level and atom-level. The substructures of a molecule are extracted through MACCS fingerprint. Each substructure is embedded as a token and encoded by a transformer. The atom-level embedding is learned through a standard GNN and it is fused into the substructure embedding. The model pre-trained on a large collection of unlabelled molecules and fine-tuned on MoleculeNet benchmark datasets. 
strength_and_weaknesses: Strength
* The empirical evaluation is very comprehensive and includes most of the state-of-the-art baselines. 
* The results showed that the method outperforms all the baselines (averaged over 15 datasets).

Weakness
* The novelty of the proposed architecture is a bit limited. Similar architectures have been proposed before, e.g. MolFormer [1]. It would be good to discuss the technical differences and compare with MolFormer on the same datasets.
* The proposed architecture treats each substructure as a discrete token and there is no relational attention between two substructures (e.g., the distance between two substructures, how many shared atoms, etc). In other words, the substructure transformer is a set transformer with no relational information between the substructures. It that's the case, the proposed model is essentially a transformer over MACCS keys + GNN + cross attention between the two. I believe that relational attention should be included, which could further improve the method.

[1] Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs, arxiv 2021
clarity,_quality,_novelty_and_reproducibility: The novelty of the work is detailed in the "strength and weakness" section. The paper is written fairly clearly. However, I disagree that the time complexity of the cross-attention is O(N). Based on equation 1, the cross attention does not include the substructure-atom membership information. Each substructure looks at all atoms rather than atoms that belong to its substructure. Therefore, if there are N substructures and M atoms, the cross attention should be O(NM). I would recommend the authors to clarify on this point.

summary_of_the_review: Overall, there are several concerns of the paper that needs to be addressed:
1) Novelty compared to MolFormer and other previous work
2) The cross attention is not linear time complexity based on equation 1, which contradicts with author's claim.
Therefore, I vote for weak rejection of this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: For this paper, we have four reviews with rather neutral scores. These four reviewers mentioned a couple of strong points, but also some problems, such as 
- the use of predefined substructures (the 166 substructures of MACCS keys) and situations in which an  input molecule doesn't have any of these. At the end of the discussion phase one  reviewer (who initially assigned a slightly positive score) expressed that she/he was convinced by the answers provided be the authors, and to be more inclined to the negative after the discussion phase.
-  the potential risk of data leakage in evaluations. Although the authors explained in detail their procedure of excluding any duplicated molecules in the external training set, I still think that this answer does not fully address potential leakage problems, since there might still be extremely similar (but not completely identical) molecules left.  
-  missing novelty. I also think that the mentioned conceptual similarity to MolFormer [1] can be somewhat problematic. Although the authors present some comparisons in the revision, some question marks remain (because it is hard to see truly substantial conceptual differences between the approaches).  
-  issues with the statements about computational complexity. It seems that some of these issues have be confirmed by the authors added some further details (which, however, do not look as impressive as the initial claim of linear complexity).
-  unclear  /lacking technical contributions.

I think it is important to note that none of these four initially "neutral" reviewers could be convinced to assign a clearly positive score (if any, there was even a tendency to the negative side).

Then there is another (initially) clearly positive review. After the discussion phase, however, this reviewer also saw some potential problems (such as unclear novelty) and indicated that the original score was too high (still on the "positive side, though less so than originally").

After going over all reviews and discussions again, I come to the conclusion that for this paper the negative aspects dominate the positive ones. Even after the rebuttal, there seem to be too many open questions, and some of these are truly fundamental questions regarding the conceptual basis / novelty / scientific depth of this work. Therefore, I vote for rejection.  

 


  
justification_for_why_not_higher_score: There are too many open questions, some of which concern severe conceptual problems.
Decision: Reject