Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors develop a graph neural network model that breaks a module up into fragments over non-ring single bonds, and trains a both a contrastive bag-of-fragments objective as well as prediction of the torsion angle of the respective broken bond.
strength_and_weaknesses: I appreciated the number of baselines presented, and I think the proposed model results are rather good.

I really like that the objective is straightforward and pragmatic. The approach is straightforward and more generally applicable. Since the model inputs and objectives are quite interpetable, I think Section E (Table 9) is useful as an ablation study.

Questions: 

What is the distribution of 3D bond angles used in the objective? Was only a single bond angle used, and where was this generated? Would it be possible to learn using the energetically favorable distribution of bond angles?

The loss for the torsion angle is interesting and pragmatic. However, by using a binned loss for the torsion angles, it is not radially symmetric, and may not be ideal. Can the angle loss be parameterized with a von Mises distribution?
clarity,_quality,_novelty_and_reproducibility: Figure 1 paragraph description is a bit confusing. It is unclear upon the first read that the words in parenthesis correspond to one another (incomplete / negative).

What is the ratio of hard to soft negatives in the objective? How is that determined or optimized?
summary_of_the_review: I find the approach novel, pragmatic, and of general interest. I also find the work well written and easy to understand.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper decomposes the 2D and 3D molecular graphs into fragments. The pretraining objective is composed of the fragment-based CL on 2D and 3D graphs, as well as a torsion angle reconstruction from 2D to 3D.

strength_and_weaknesses: Strength:
- The existing works on utilizing the domain knowledge to help molecule representation is either focusing on the 2D structure (like fragments) or general 3D and 2D structure. This paper introduces using the fragments from two views, which is interesting and promising.
- This paper further introduces using the torsion angle reconstruction, which can help the molecule representation intuitively.


Weakness:
- There exists information leakage between the SSL objective and backbone model.
  - The authors are using SchNet for the geometric backbone. It is an SE(3)-invariant model, using the pairwise distance information, and adding torsion angle to it has been expected to be helpful. This has been verified in GemNet[1] and SphereNet[2].
  - Meanwhile, there are more advanced SE(3)-invariant GNNs (like GemNet[1] and SphereNet[2]). They all explicitly use the torsion angles directly in the backbone models. Other SE(3)-equivariant GNNs are also preferred, like TFN [3], since the angle information is simultaneously modeled in the spherical harmonics basis.
  -  In other words, using the SchNet as backbone is not sufficient to support the effect of using torsion angle reconstruction. Now the information released in the experiments is that for geometric model with distance modeling only, adding torsion angle pretraining is beneficial. This is different from the main claim of the paper, because this may fail if the backbone models [1,2,3] explicitly/implicitly encodes the torsion angle.
  - For improvements, the authors should use more advanced backbone models with torsion angle modeling, either implicitly or explicitly. Then it can better verify the effectiveness of using torsion angle for pretraining.

- The experiments are not solid for the 3D downstream.
  - First, the 7 tasks are actually from QM9, with 12 tasks and 130K molecules.
  - So the pretraining has 50K molecules, and the downstream has 130K molecules. I don’t think this is a valid pretraining and finetuning setting.
  - The data preprocessing has been widely used on QM9 (SchNet, DimeNet, SphereNet) etc, but the authors are not applying that. So this result can be hardly compared with other existing works.
  - There are 12 tasks, I’m wondering why only 7 tasks are reported here?
  - Thus, combined all the 4 points above, I am not convinced by the validness of QM9 downstream results.

- Questions on 2D downstream. I found that authors are reporting results on 2D-FragCL, FragCL, and 3D InfoMax, and the other results are from GraphMVP paper, which is fine. But I noticed that the main benefits between the newly reported results and existing baselines are from Clintox, which is >0.1 ROC-AUC better. Other than this, the improvements are quite modes, within 0.01 ROC-AUC. Do the authors have any hypothesis for this? Could it be the hyper-parameter issue (seed, learning rate, etc)?

[1] Gasteiger, Johannes, Florian Becker, and Stephan Günnemann. "Gemnet: Universal directional graph neural networks for molecules." Advances in Neural Information Processing Systems 34 (2021): 6790-6802.

[2] Liu, Yi, et al. "Spherical message passing for 3d graph networks." arXiv preprint arXiv:2102.05013 (2021).

[3] Thomas, Nathaniel, et al. "Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds." arXiv preprint arXiv:1802.08219 (2018).

clarity,_quality,_novelty_and_reproducibility: - The method naming is confusing. FragCL only implies the contrastive learning, but actually there is an torsion angle part, which is not reflected in FragCL.
- Some references are mis-match. I would recommend authors double-check this.

summary_of_the_review: I think the main motivation of this paper is reasonable, but the experimental set-ups are not valid. The authors may double-check the SE(3)-invariant/equivariant modeling, and the QM9 dataset.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper develops a method for constructing representations of small molecules using contrastive learning. For a particular molecule, the molecule is split into two equal-sized fragments by breaking a single non-ring bond. The method attempts to maximize similarity between the original molecule’s representation and the weighted average of the representations of the two fragments. It also attempts to minimize similarity between the original molecule’s representation and the individual fragments on their own or the average of other molecules’ fragments. The authors also train two separate neural networks—a 2D GNN and a 3D GNN—and encourage representations learned between the two GNNs to be similar, and also add on a loss to predict torsional angle of the cut bond.

The authors use this method to pretrain and then attempt to predict several benchmark molecular properties, comparing to other similar methods of generating molecular representations. They demonstrate modest improvements in some of the tasks. 
strength_and_weaknesses: The paper proposes a novel scheme for fragmentation (as opposed to previous fragmentation schemes) for contrastive learning on molecules. The scheme is interesting, and worth publicizing (at least on arXiv). The improvements on benchmarks are also nice to see.

Below are some weaknesses which I believe need to be addressed before publication:
### The complexity of the objective suggests a lack of robustness
In addition to the fact that there are two separate GNNs, there are a lot of loss functions in the objective (i.e. contrastive loss for the 2D GNN, contrastive loss for the 3D GNN, contrastive losses between the two GNNs, and a complex loss function for the torsion angle which consists of both regression and multiclass classification objectives. Not only does this greatly increase training and inference time (and computational resource use), it also casts doubt on the robustness of the neural network. Especially because the improvements in downstream prediction tasks are only modest (if present at all), it suggests that all this extra complexity may be the result of overfitting to these specific benchmarks.

For example, it is difficult to justify predicting the torsion angle both as a regression problem and as a multiclass classification problem (over bins of the angle). What happens if the binned objective is removed?

The ablation analysis also suggests that each piece of this complex framework is necessary, even for the few modest improvements.
### Incorrect chemical justification for the proposed fragmentation scheme
The authors attempt to justify the fragmentation scheme using several incorrect statements. For example, the claim is made that fragmenting a molecule along a single bond is “likely to preserve chemically informative substructures”. They also claim that “single bonds are not directly involved in reaction pathways in general”, and that the "fragmentation scheme described in Section 3.1, satisfies $F(G\_{i}) = F(G'\_{i})\cup F(G''_{i})$ with a high probability” (where $F(G)$ is the set of functional groups in $G$). These statements are all dubious if not incorrect. 

Single bonds are oftentimes highly important in reaction pathways. Nucleophilic attacks such as SN1 and SN2 rely on the breaking of single bonds almost ubiquitously, not double or triple bonds as the authors claim. If anything, single bonds are perhaps even more likely to participate in reaction chemistry (compared to double or triple bonds) because the bond energy is usually lower.

Breaking single bonds also can very easily break functional groups. For example, consider dibenzyl ether. Based on the fragmentation scheme proposed by the authors, this molecule would be split into two fragments which completely destroy the ether functional group. Groups such as esters, amides, and ketones would also easily be split up. The statement that $F(G\_{i}) = F(G'\_{i})\cup F(G''_{i})$ with high probability should be quantified over the dataset if it is true (although I have very strong suspicions that it is not).

Furthermore, the distinction between single and multiple bonds in organic molecules is not as clear as the authors suggest. For example, consider a highly conjugated alkene capped with aldehydes (or another resonant functional group). Although the graph representation of this molecule will have several single bonds, the distinction between the single and double bonds in reality are actually minimal, as the entire structure is in resonance (which would be broken by fragmentation). Thus, the choice to fragment molecules by single bonds only is not particularly well justified.

It is also good to note that there are many molecules which would not satisfy the fragmentation scheme suggested by this paper, such as benzene, naphthalene, anthracene, etc. These molecules could not be fragmented at all using this scheme.
### Areas lacking in explanation, clarity, or typos
- In the penultimate sentence of Section 3.1, $G''_{2D}$ is a typo, I think
- Eq. 10 – 11 should specify what the $h$s mean. I assume they are the node representations from the last layer of the 2D GNNs
- In Table 1, the Avg column is not informative and should be removed. These are very different benchmarks with varying difficulties, and averaging over the scores doesn’t make sense. After all, a model could get a very high Avg just by optimizing very specifically for a single very easy task.
- The bolding in Table 1 and 2 is not clear. Why are there multiple numbers bolded in a single column?
- In Section 4.3, clarify what the loss value is. Is this just the NT-Xent value? If so, this metric does not make much sense because the fragmentation scheme is different between different experiments, and so the NT-Xent values will be based on different inputs and therefore are not comparable across ablation experiments
- In Appendix A, the definition of $r^{mix}_{i}$ contradicts the main text
- There are typos in the title of Appendix B, and in the Table 9 caption
- What is the form of the projectors $g$ which map from node embeddings to torsional angle? I assume they are single linear projection heads (i.e. a single linear layer) on the concatenation of node embeddings
- Once pretraining is done, how are the final tasks (e.g. the ATOM3D regression tasks) trained? Is it another linear projection head on the average node representations? This doesn’t seem to be explained anywhere
- The computation of the score of these benchmarks should be explained: how are these scores computed?
clarity,_quality,_novelty_and_reproducibility: There are some issues with the writing. In addition to some typos, the flow of the paper needs some work. For example, the torsional loss is introduced very early in the paper, but is not explained until the end of Section 3.3. Additionally, several details of the implementation are missing from the supplement. A non-comprehensive list of clarity issues are listed above.

There are also some claims and justifications for the fragmentation scheme, which are chemically dubious or incorrect (listed above).

The novelty of the work is also limited. Other works before have used many of the pieces in this paper (e.g. molecule fragmentation for contrastive learning, combining 2D and 3D GNNs, etc.). The main novelty of the paper is a different way of applying fragments to the contrastive loss.
summary_of_the_review: The work is interesting, and there are some modest improvements in some of the benchmarks shown. However, there is limited novelty in the technical contributions. Additionally, the method is very complex, combining 2 GNNs and 3 – 4 distinct types of loss functions. Even with the complex architectures and training, the improvements in the benchmarks are not particularly impressive. Combined with the numerous issues with clarity and incorrect justifications of the fragmentation scheme (one of the main focuses of the paper), it is very difficult to support its passage.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The paper proposes a fragment-based contrastive learning algorithm for learning the discriminative representations of the molecules. They propose to decompose a molecule into two fragments by breaking a single bound (none ring). The bound is chosen such that two fragments have a similar number of atoms. Then the complete bag of fragments of the molecule is considered as a positive view while the incomplete or the complete bag of fragments of the other molecule is considered as a negative view. They also  Incorporated 3D structure by including a reconstruction loss on the torsional angle. 
strength_and_weaknesses: Strength:
The paper addresses an interesting problem such as how to define a good positive and negative view when we apply contrastive learning on the molecule. They also propose an interesting idea that seems to be an easy direct way to incorporate 3D information about molecules. 

Weakness: 
Experimental results seem weak especially when trained on 2D molecule graphs. 
Compare to the existing baselines, the proposed scheme to construct a negative and positive view of molecules does not seem to have many advantages.  
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and clearly written. The proposed approach to constructing the positive and negative view of molecules and the idea of incorporating  3D structure by reconstructing torsional angle seem to be novel.
With the details in the appendix, the paper seems to be reproducible.  However, I feel like there is a bit of repetition in the main text about the proposed method around positive and negative views while many modeling details are left in the appendix. 
summary_of_the_review: Overall, the paper is clearly written and easy to follow, however, I have some detailed questions as following:

1. How often does the assumption " breaking a none-ring single bound preserve the  chemically informative substructures" 

2, Is the negative view which is the complete fragments of another molecule really useful, seems to be a very obvious negative view that is easy to discriminate.

3. It is not very clear here: "Note that the semantic of a fragment g_i is ensured to be sufficiently different from the original molecule g'_i  since ....... ".  According to the paper, F(g'_j) and F(g''_j)  are complementary, not necessarily similar. And I would think also that the negative view which is constructed by the incomplete bag of fragments is not hard to discriminate from science dropping half of the molecule could change the functional group of the original molecule drastically. 

4. When looking at the experimental results in Table1, it feels to me that the model did not have much advantage over the other models when working on 2D representation,  but it shows better performance when trained on both 2D and 3D. Could it be possible that the improvement comes from the fact that the model also tries to incorporate the reconstruction of the torsional angle but less due to introduced strategies about constructing a positive and negative view of the molecule? 

5. The ablation study section is a bit confusing to me, so the author first used the incomplete fragment as a positive view, and on top of that used the complete fragments as a positive view, and then on top of that used the incomplete fragment as a negative view? Also, it would be interesting to see what is the effect of the complete fragments of another molecule as a negative view.

6. As table 3 shows, when all settings are kept the same and only changing where we cut seems not to have such a big difference in terms of performance,e 72.1 to 72.4 (on average), instead adding the loss on torsion angle seem to improve a lot (74 to 75).

7. It would be great to see what the value of alpha is in the best model, and what happens if we set alpha to zero which means there is no reconstruction of the torsional angle
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a novel contrastive learning approach for learning unsupervised molecule representations, based off of an algorithm for splitting molecules into fragments. Reviewers praised the intuitive nature of the objectives, and the writing of the manuscript.

However reviewers were more mixed in their views when it came to evaluating the empirical performance. Some reviewers felt that the performance gains were relatively modest given the additional complexity; others had concerns regarding the experimental settings and the relation to the baselines.

The authors did a good job of responding to these points in the rebuttal and the updated paper. However, none of the three lean-reject reviews increased their score from the comments or discussion. That said I believe the updated manuscript, with the revised experimental details, is in a good position to be accepted when re-submitted at an upcoming conference.
justification_for_why_not_higher_score: Three of the reviewers had concerns regarding the strength of the experiments; one had severe concerns about the validity of the experimental setting and baselines. While the authors did a good job of responding to these concerns, they were not sufficient to cause the reviewers to raise their scores.
Decision: Reject