Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present 3D Infomax, a graph neural network (GNN) pre-training solution that leverages 3D information to generate better learned embeddings and improve performance on down-stream prediction tasks where 3D information would be useful but not easily obtainable.  The approach is useful for a range of downstream tasks involving molecules, including ones that are quantum mechnical, biological, and pharmacological in nature.  They also demonstrate that the use of multiple 3D conformations (thereby encoding the inherent flexibility of molecules) further improves performance.
main_review: The major strength of this work is the extensive empirical evidence supporting the claims that the proposed self-supervised learning scheme (mutual information between 2D GNN and 3D GNN learned embeddings) significantly improves performance on downstream tasks as compared to other pre-training strategies.  In addition, the language is simple and clear, and the figures provide easily understandable graphical depictions of the method.  

The major weakness, which is addressed by the authors, is that these 2D models, even when pre-trained with 3D information, still significantly underperform methods that can leverage 3D information in the downstream task.  This implies that in settings where accuracy is key, one should still opt for the computational expense of generating the 3D conformers directly.  While the authors argue that the gold standard methods of generating 3D conformers can be prohibitively expensive, I would like to see what the performance of 3D GNN operating on conformers such as those that might be generated by a fast method like GeoMol.  This would allow for better evaluation of the tradeoff between accuracy and speed provided by explicit 3D conformer modeling.
summary_of_the_review: Overall, I think this is a strong paper due to the compelling results and clarity of communication.  I currently recommend weak accept as I would like to see the experiment I mentioned above (using lower quality, but quick-to-generate 3D conformers) to better understand the tradeoffs involved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper uses the 3D structures of molecules to pre-train graph neural networks (GNN) representation of 2D molecules. This improves the molecular property prediction of some quantum mechanical (from 8 tested 8 were improved) and non-quantum mechanical properties (from 10 tested 4 were improved). 
main_review: I found the introduction and background good to read and easy to understand. The paper is supported by great images to explain what was done. It also admits the weakness of the method with biological systems. The non-QM property part is not as clearly written and I found it harder to understand which dataset belongs to which statement. 
-	Page 4 line 5: I believe the statement that it learns quantum mechanical (QM) interactions is not backed up. The 3D structures of molecules are partially determined by steric hindrance and not purely by QM interactions. E.g. when the model is pretrained on a smaller set of elements and used for elements it has not seen this shows it learned the structure and not QM. 
-	Page 3 line 3: What is SE(3) symmetry? 
-	Table 1: Is there a reason why two properties (cv and alpha) are better predicated with the pretraining of QMugs than with the same dataset (QM9)? 

Some very minor points:
-	For the review is would be with advantage to add a column with line numbers. Further, exchanging numbers for the citations instead of name and year, this improved the fluent reading experience. 
-	Page 1 figure 1: MI could be mentioned in the Figure comment or before on page 1 paragraph 4 (our solution) as this abbreviation was not used yet. 
-	Abstract, Page 2 line 4, page 6: Inconsistency with the number of quantum mechanical properties. 8 are given in the tables but often 10 are mentioned. 
-	Page 2 last line: Year is missing in the citation. 
-	Page 4 last line: The word hardest is confusing here. 
-	Page 7 Table 1: The targets are never mentioned or explained. Only the units are given in the SI. Table 6, here also the names of the variables would be great. 
- Page 8 after table 4, last sentence of the paragraph: Incomplete and I do not understand it.  
-	Page 10ff: The citations are not normed, e.g. PMID 

summary_of_the_review: For molecules the 3D information is essential and this paper shows an affordable method to include this for improved property prediction. There are two main advantages, firstly it can be used for a dataset where 3D information is not available and secondly it is cheaper than implementing 3D information in the representation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper starts with an interesting direction, to use 3D geometry to augment the 2D representation. From the technical point of view, it adopts SSL to maximize the representations of 2 views, so as to enable the 2D GNN to encode 3D geometry information, which can be beneficial for downstream tasks.

main_review: Strengths:
This paper starts with an interesting direction, to use 3D geometry to augment the 2D representation. This is an interesting and well-motivated problem.

-----

Weaknesses:

(1) The empirical result is the most important point, but it has the following problems.
1. Motivation contradiction. The empirical result does not match the motivation. In the abstract, it says that `fine-tuning on molecules with unknown geometry, …`; in the intro, it says `After pre-training, … fine-tuned on molecular datasets where no 3D information is available.` However, in Sec 5.1, it is tested on `50k molecules from QM9 or 140k from GEOM-Drugs`, where both include 3D information. Besides, as we can tell from the last column of Table 1, 3D geometric GNN is overwhelmingly better, so it is not quite clear why authors want to test on this dataset. Similar concerns on Table 2-3.
2. The empirical results are confusing. If we ignore the motivation contradiction issue, then according to Method section, this work pre-trains one 2D GNN and one 3D GNN, and downstream tasks include both 2D and 3D information. However, in Table1-3, the authors only check the 2D fine-tuning. If the main focus is on performance improvement, why not focus on 3D downstream tasks? If the performance is about 2D downstream tasks, then why consider these tasks with 3D geometry available? Same concerns on Table 2-3.
3. Performance not good enough and not enough SSL baselines. As pointed out by authors in Intro and Related Work, there are many SSL works along this research line, like AttrMasking, ContextPred, JOAO. GraphCL is more recent than the previous two, but JOAO is more recent. Besides, GraphCL does not show an overwhelming best performance on these tasks, and instead, it shows better performance on different types of graph data. So I’m just wondering why authors didn’t try this? And Table 4 result shows that 3D InfoMax has very limited improvement, or sometimes has even worse performance. This fails to support the usefulness of 3D Infomax.
4. Confusing results. Table 1 result is confusing. It doesn’t specify which datasets are used for the pre-training baselines.

(2) Some descriptions are not clear and terminologies should be specified more clearly.
1. GNN can be adopted on both 2D and 3D graphs, which is not clearly specified in the paper. For example: the abstract says `GNN still generates implicit 3D information …`, and the GNN here refers to the 2D GNN.
2. In Intro, `We pre-train a GNN to encode implicit 3D information …`. This expression is not clear. A better way would be: This work is training 2 GNN (one 2D GNN and one 3D GNN) at the pre-training stage; with MI (SSL), the 2D representation can encode implicit 3D information. Then only 2D GNN is adopted for fine-tuning because downstream tasks have no geometry info.
3. In Intro, `This way, the GNN learns to generate latent 3D information using only the information given by the 2D molecular graphs.` The GNN here also refers to the 2D GNN.
4. The authors mention `2D GNN generate 3D information` at multiple places, like in Intro and Method. But the main solution is to adopt InfoMax, which is a contrastive SSL method; the word `generate` typically implies generative SSL [1]. Authors might better tune these words carefully or explicitly specify them in the paper.

Without specifying these terminologies explicitly and clearly (e.g., at the beginning of the paper), it is hard for readers to follow the logic of this work.

(3) Some claims about the related work in this paper are too bold.
In Intro, `It is not clear why these learned representations should be informative and generalize well.` But all the papers, AttrMasking, ContextPred, GraphLoG, GraphCL, JOAO have claimed the intuitions of the proposed methods, some are supported with ablation studies.
I acknowledge that 3D info is helpful for 2D, but the above claims on existing works are not correct.

(4) Some text words in Fig 1, Fig 2, Fig 3 seem to be hand-written? If so, authors may as well update them in a more formal way.

(5) The 3D Net / 3D GNN representation is wrong.
In Sec 4.1, the equation on 3D Net / 3D GNN is wrong. The representation should be f^b(V, R), i.e., the input to 3D GNN includes both the atom types and atom positions. This holds for SchNet, DimeNet, EGNN, SphereNet.

(6) The objective function is wrong.
The NTXent loss doesn't seem correct to me.
The authors can verify this from the definition of Cross-Entropy, and here I put a more straightforward explanation.
In eq(1) in [2], the numerator is the positive pair; the denominator includes 1 positive pair and 2(N-1) negative paris, i.e., 2N-1 pairs in total.
In eq(1) of this paper, however, the numerator is the positive pair, and the denominator only includes N-1 negative pairs.
Thus, this is definitely not NTXent (similar for the following objective functions).

I haven’t found the codes attached in SI, so I just played around with it on my own implementation, and sometimes (on some datasets) it can lead to negative loss. I’m not sure which one is used in this work, and maybe it can still have some positive effects on downstream tasks, but definitely not NTXent loss.

-----
Minor points:

(1) Sec 4.2 can be moved to Sec 5.

-----
[1]  Liu, Xiao, et al. "Self-supervised learning: Generative or contrastive." IEEE Transactions on Knowledge and Data Engineering (2021).

[2] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

summary_of_the_review: This paper is an empirical work, so the empirical results are the most important. However, as listed above, there are some key issues with the empirical performance (point 1). In addition, there are also some other concerns: the writing issues (point 2&3), visualization issues (point 4), and equation issues (point 5-6).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a 3D pretraining method for molecular property prediction. As 3D information is infeasible to compute at the scale required by real-world applications, this paper reasons about the geometry of molecules given only their 2D molecular graphs. 

During pretraining with molecules whose 3D information is known beforehand, this paper uses a 2D GNN to encode these molecules. Then, it maximizes the mutual information between 3D summary vectors and the encoded representations for injecting 3D information into the representations. During fine-tuning, the model can take 2D molecules as input. The pretraining phase ensures that the representations during fine-tuning contain latent 3D information. 

The paper pre-trains on three datasets, QM9, GEOM-Drugs, and QMugs, and tests on both quantum mechanical properties (QM9 and GEOM-Drugs) and non-quantum properties (10 datasets, e.g. HIV and BACE). The paper claims that significant improvements are obtained for quantum mechanical properties. Also, the method does not suffer from the negative transfer. 
main_review: Strengths:

1. The method proposed in this paper is novel and sound. Incorporating 3D information for molecular property prediction should improve its performance as demonstrated by this and previous work.

2. The experiments are well designed. The authors choose to pre-train on three datasets (QM9, GEOM-Drugs, and QMugs) and evaluate for both quantum mechanical properties and non-quantum properties. Yet some setting descriptions and results are missing (see weaknesses). 

3. The paper is clearly written and easy to follow (with some exceptions shown below). 


Weaknesses:

1. Some experimental settings are unclear. For example, for QM9 and other datasets, what are the input features for your method and the baselines? There is a bunch of work using QM9 for evaluation. If they follow the same setting, the authors may consider listing their results as well for a comparison. Otherwise, it is hard to tell whether this paper has followed the standard way for evaluation on QM9 and whether the evaluation is meaningful and fair. 

2. Some evaluation results are missing. For example, in Table 2, the results for Rand Int and GraphCL are shown, yet PropPred is not shown. Is there a specific reason for this? Similarly, in Table 4, the results of Dist-pred and Conf-gen are missing. 

3. What does PropPred mean in the paper? The paper states that "we pre-train baselines by either predicting the properties of GEOM-Drugs’ pre-training subset (labeled Prop Pred)." Which baselines are referring to? What do the properties of GEOM-Drugs correspond to here?

4. Standard deviation is missing in Tables 1, 2, and 3.

5. The settings for baselines are unclear.  Which GNN is used for the baselines like GraphCL? How many layers are used? 

6. The paper claims that conformation methods are slow and shows a previous method that takes 6 hours/molecule for conformation. I am not sure whether this claim is correct. Are neural conformers (e.g. https://arxiv.org/abs/1904.00314) as slow as these methods? In my opinion, the inference of neural models shall be relatively faster.

Minor point:

I think the method has some similarities with knowledge distillation. The model takes a 3D GNN as 'teacher', while the 2D GNN learns from the 3D GNN to obtain latent 3D information. Clearly, the methods proposed in this paper have differences from knowledge-distillation-based methods. The paper might discuss these differences and compare with a knowledge distillation baseline if possible. 

Typo:

Incomplete sentence: Including 3D molecular structure as input to learned models their performance for many molecular tasks.
summary_of_the_review: The method proposed in this paper is sound. However, the evaluation has some flaws as discussed in the main review section.

I will consider raising my scores if these questions are properly solved during the discussion period.
review_title: null

Meta-review: This paper aims to use pre-training to bridge the gap in performance between 2D GNN and 3D GNN. Specifically, during pretraining, it trains both 2D GNNs and 3D GNNs on data equipped with 3D geometry to maximize the mutual information between the 2D GNN representation with the 3D GNN representation. The proposed approach is interesting and novel and the paper presents some promising results showing that the pre-training does provide some benefits for downstream tasks where 3D geometry information is not available in comparison to several other baseline pretraining methods. While the reviewers agree that property prediction without only 2D graph is a practically important setting for high throughput screening, there are concerns about whether the current set of results paint a clear picture on the benefits and superiority of the proposed methods to alternatives (e.g., vs conf-gen) even after the revision. This is not due to lacking of results, but more of a presentation issue where results are not organized and discussed clearly to provide a coherent story.  We do see clear and strong potential for this paper but it needs a careful rewrite/re-organization to tease out the key messages and how the experiments support them.
Decision: Reject