Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a simple and efficient pretraining strategy, which predicts the added noise into 3D positions. The paper further shows the connection between denoising and molecular force field learning. Experiments demonstrate the effectiveness of the proposed method. 
strength_and_weaknesses: Strengths:
- The proposed method is simple and effective
- The analysis of the connection between denoising and force field is insightful
- The experiment is comprehensive, and the results are impressive.

Weaknesses & Questions:
- Downstream applications. There are many molecules that do not have DFT 3D conformations, and thus cannot use the proposed 3D denoising task. Therefore, the proposed method seems cannot be used in many traditional molecular property prediction tasks, like MoleculeNet.  
- Some experiment settings are unclear, for example, the data split setting for QM9.
- In Sec.5.1, Only TorchMD-Net model is tested. Did you try some transformer-based architectures, like Graphormer?
- will you open-source the model/code?

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and easy to follow.

Quality: The proposed method, the experiment design, and the result analysis are all look good.

Novelty: Although the proposed method is simple, it is effective.

Reproducibility:  The paper provides details of the experiments, some are unclear (refer to "Weaknesses").
summary_of_the_review: Overall, I think this is a good paper, and recommend accepting it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a pretraining method to encode 3D small molecules into atom-level encodings which can be useful for downstream prediction tasks. The method is essentially to start with the 3D coordinates of a molecule, add random Gaussian noise to the atom coordinates, and train a 3D GNN to predict the added noise. The authors connect this method to denoising score matching, which has been proposed previously in the context of diffusion models. The authors also offer an interpretation of the method as a way to approximate the local force field of the atoms, which the GNN is attempting to predict. The paper demonstrates that pretraining using this method (along with a few other tricks like Noisy Nodes (NN) and Tailored Activation Transformation (TAT)) can be used in conjunction with pretraining to achieve better predictive performance on a variety of downstream tasks.
strength_and_weaknesses: Overall, the paper demonstrates a solid contribution. It provides several compelling analyses which show how the proposed method can generate embeddings which assist in several downstream tasks. The authors compare their method to many existing architectures, and even show that pretraining can assist another architecture (which is compatible with their proposed method). The paper also analyzes a few design choices (e.g. model size and dataset size) to show the effect of these choices on the model’s performance.

There are a few areas which I feel are weaker:
### Limited comparisons with other models from prior work
The comparisons with prior work that are shown in the paper are good to see (i.e. Table 1 and Table 8), as they directly compare GNS and/or GNS-TAT with the performance of other models. However, Table 1 only shows the comparison with other models on a single benchmark, QM9. The improvements brought on by GNS-TAT + PT + NN are certainly there, but not ubiquitous and not always huge. The comparisons in Table 8 notably do not include models such as TorchMD-NET. All other analyses in the paper are effectively only comparing GNS/GNS-TAT with itself (when PT/NN are added), or when other models like TorchMD-NET are pretrained using this method. This suggests that although the improvements in performance offered by this paper are there, they may be limited to certain datasets/benchmarks, and are not necessarily as strong. Since one of the main claims made by the paper is that PT/NN/TAT help performance, it is important to directly compare the performance of GNS/GNS-TAT (with PT/NN) with all other models (especially the strongest ones like TorchMD-NET) on more than just one benchmark. Of course, showing that TorchMD-NET performs better with pretraining (which is already done in a limited capacity) is also helpful to expand upon.
### There are several models being analyzed
Most analyses seem to use either GNS or GNS-TAT, and many of the analyses add either PT or NN (or both). Unfortunately, some analyses use GNS-TAT and others use GNS (with different flavors of adding PT or NN). This can be a little confusing for the reader, and also casts a bit of doubt on the robustness of the method. It is very possible that some benchmarks/tasks do better with TAT and others do not (or with or without NN, etc.), and if so, this should be stated explicitly as a potential caveat (i.e. TAT or NN may be better suited for some tasks and may be worse for others).
### Force-field prediction is expected to be significantly better
The connection between the denoising objective and Boltzmann-distribution force fields is a good insight offered, and it certainly suggests that the model should yield large improvements in force-field prediction. However, the main text only offers one analysis on this, which is that pretraining on TorchMD-NET seems to improve the MAE of the predicted force field for a single molecule, aspirin. Appendix G offers some more results in improving force-field predictions comparing GNS with and without pretraining. The paper would be stronger if these results were included in the main text, and it would be even better if the authors could show significant improvements in force-field prediction with GNS (or GNS-TAT) with pretraining, similar to Table 1.
clarity,_quality,_novelty_and_reproducibility: Although the paper is well-written, there are a few areas which could be improved in terms of clarity:
- Since many of the analyses focus on different models (e.g. GNS, GNS-TAT, GNS-TAT + PT, etc.), it would be helpful to clearly list the model being used in each table, figure, and caption
- In Fig. 3 (Left), the shading around the black line is green, which is confusing and I suspect should have been gray
- $R_{cut}$ is not defined anywhere in the paper
- Bolding the best numbers would be helpful in Table 8
summary_of_the_review: This paper presents a novel method of pretraining for small molecules, which has some nice theoretical connections to denoising score matching, and has a direct connection with statistical mechanics. The authors show several compelling analyses which demonstrate the effectiveness of pretraining in a variety of situations. The improvements over current SOTA are perhaps modest, but present. The authors also explore the effects of pretraining combined with other techniques such as Noisy Nodes, and an application of Tailored Activation Transformation to GNNs. The analyses are limited in that there are very few direct comparisons of their GNNs with pretraining to current SOTA models, certain analyses focus on different GNNs for unknown/unjustified reasons, and the task which pretraining should theoretically benefit the most does not seem to enjoy these benefits as significantly as one would hope. Regardless, I believe the authors have shown that their method of pretraining is functional and works well, and their numerous analyses on their method help give the reader intuition on how and why pretraining, Noisy Nodes, and Tailored Activation Transformation can be helpful for molecular prediction tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates (1) the denoising objective for pretraining and (2) tailored activation transformation (TAT) for 3D graph neural networks. The authors show how their denoising objective is equivalent to learning molecular force field based on interpretation of denoising objective as a score function. The proposed approach demonstrates a notable improvement over several tasks. 
strength_and_weaknesses: Strength 1: The problem of pretraining 3D molecular representation using a small number of labels seems very significant. 

Strength 2: The proposed denoising objective is quite simple and works well across varying datasets.

Strength 3: Tailored activation transformation is also useful trick for improving 3D GNN performance in general.

Weakness 1: The experiment does not consider applying the denoising objective for the recently proposed 3D graph neural network architectures like GemNet, SphereNet, or SEGNN. It is unclear whether if the proposed approach is useful for the recently proposed architectures too.

Weakness 2: The interpretation of denoising objective as a score matching objective is not very novel.

Weakness 3: The experiments do not compare with the recently proposed non-3D molecular representation learning frameworks. While attribute masking is considered as a baseline, there exist more recently proposed works, like GraphMVP or 3D infomax, which considered 3D information during training. 

Weakness 4: While I appreciate how the authors provide a detailed description of their hyperparameter in Appendix E and F, the description could be made more clear for the future works to build on. For example, in Appendix E, it is unclear what the authors mean by tuning on “approximately” 5 values (hyperparameters) on the QM9 dataset. What do the authors mean by approximate? Also, is the validation set used for tuning the hyperparameters? I hope the authors clearly describe this since future works will likely follow similar protocols.
 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear to read. The problem being solved, i.e., 3D molecular representation pretraining, is novel and significant. The proposed method is not very novel, but this is okay given importance of the problem. I also think the proposed method being simple is very useful for being applied to practice. 
summary_of_the_review: I recommend acceptance for this work given the following strengths:
- significance of the problem being solved 
- the proposed methodology being simple and easy to implement 
- well-executed experiments demonstrating benefit of the proposed approach over various settings
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a simple and efficient pretraining strategy, which predicts the added noise into 3D positions. The paper further shows the connection between denoising and molecular force field learning. Experiments demonstrate the effectiveness of the proposed method.

All reviewers speak highly of the paper. They like the simplicity and elegancy of the proposed method, the physical insights of denoising, and the solid experimental results.  Although they also mentioned some weaknesses, including the insufficient comparison with more baselines, and the lack of experimental details (including hyperparameters) in the main body of the paper. Overall speaking, the paper is clearly above the bar, and should be accepted.

justification_for_why_not_higher_score: The paper still has some weaknesses, including the insufficient comparison with more baselines, and the lack of experimental details (including hyperparameters) in the main body of the paper. 
justification_for_why_not_lower_score: The strength of the paper includes: simplicity and elegancy of the proposed method, the physical insights of denoising, and the solid experimental results.
Decision: Accept: notable-top-25%