Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised method for 3d molecular data. The authors started with maximizing mutual information, and obtained an objective loss (denoising score matching). The idea is supported by several 3d geometric pretraining experiments.
strength_and_weaknesses: 
Strengths
1. The derivation from mutual information maximization to denoising score matching is novel.
2. The experiments are convincing.

Weakness
1. In Sec 4.3.1 Eq.(5), the definition of $d$ and $r$ are missing
2. The calculation of $d$ is pariwise distances of all points (or atoms). If the number of points is very large, it could be very slow (quadratic complexity)?
3. It seems that the method can also be used with general point cloud data (no edges between vertices), can the authors elaborate more about this? The complexity seems to be an important problem if we want to apply the method to large point clouds.
4. The main contribution is the denoising distance matching. Is it appropriate to put "SE(3)-invariant" in the title?
5. Why is this method superior to a vanilla denoising autoencoder (the simplest method we can come up with when dealing with SSL)? More discussions are needed
6. How is the memory consumption given different sizes of input graphs?
7. If I am understanding this correctly, the row marked with a dash (-) means not using any pretraining method (Table 1-3). It is just equivalent to training from scratch.
8. In D.1, how many v100 cards do you use for a single experiment?
9. Can the authors discuss a bit more about the limitations of the proposed method?
clarity,_quality,_novelty_and_reproducibility: The description of the method is clear and easy to follow. However, the code is missing. It would be good if the authors can include the computing resources for each experiment.
summary_of_the_review: The paper proposed an interesting SSL method. But I am concerned about the computing resources needed due to the quadratic complexity.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use denoising as pretraining for SE(3)-invariant neural network models. The main idea is to add some noise to the coordinates, and use a score model to predict the added noise. The pretrained model are used in several different tasks, including molecule property prediction, force field and binding affinity prediction. The proposed pretraining method was proved better than other pretraining methods in the tasks.
strength_and_weaknesses: Strength: It is a good idea to use denoising in pretraining neural networks in molecular tasks, and the results showed that the idea is valid. The paper is written clearly.

Weakness: The advantage of GeoSSL compared to other pretraining methods seems minor, as seen in Table 1, 2 and 3. I think it is necessary to include two more experiments: (1) ablation study on hyperparameters (2) error bars over multiple random seeds. With these two experiments, it will be more convincing that the advantages are really from pretraining, not hyperparameter-tuning or randomness.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly with high quality. The idea is moderately novel. The reproducibility can be improved by providing ablation study on hyperparameters and error bars.
summary_of_the_review: This paper proposed to use denoising in training 3D roto-translation invariant neural networks, and proved its superiority in molecular property prediction, force field and binding affinity tasks. However, the experiments can be more convincing if ablation study and error bars are provided
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In the presented work, the authors present an approach to perform self supervised learning on tasks where molecular 3D geometry is relevant. They do so by taking advantage of equivariant neural networks in combination with a denoising score matching method on molecular distances. 
strength_and_weaknesses: Strengths:
* Strong empirical results
* Novel idea
* Clear methodology
* Evaluation on several relevant biophysical datasets

Weaknesses:
* Hard to read at times
* Does not reference a very related work, while claiming to be the first study to perform pretraining on 3D molecular data.
* Does not provide code to support analyses

clarity,_quality,_novelty_and_reproducibility: The work is novel, albeit it needs to be put in the context of other related studies (see summary of the review).

Additionally, the manuscript could use additional time for rewriting, as it reads very clunkily at times (see last paragraph of section 2, or the energy-based model paragraph on section 3 for examples). 

There is no code provided to support the claims or help reproduce the study, as authors claim this will be released in the future.
summary_of_the_review: Overall, the work is original and in my belief deserves publication. However, I think the authors should probably reconsider some of the claims presented in the study, specifically those regarding them being the first to use pretraining strategies on 3D molecular data. Specifically, I believe that claim should be adapted based on the following works:

https://arxiv.org/abs/2206.00133?context=q-bio

The authors should either compare to these approaches or alternatively discuss why their approach is theoretically different, or whether there are computational disadvantages in either of them.

Some other comments/questions:

•	It is not clear how the perturbed geometry g2 is obtained from g1 in Algorithm 1. 

•	Do the authors have any suggestions on how to choose the different noise levels \sigma_l in Algorithm 1?

•	It is not clear from the text whether the Molecule3D dataset is a subset of the PubChemQC database.

•	It is also not clear what the authors are referring to when they refer to the so-callsed “supervised pretraining baseline”. In particular I do not see where the pretraining is done here. From the text it appears that this is simply a supervised baseline?

•	It would be fantastic if the authors could provide standard deviations for the numbers reported in Tables 1, 2, as they are reported in 3.

•	In the downstream tasks on binding affinity prediction, is the pretraining done on both the pocket + ligand graph, or only on the ligand? Have the authors experimented with both of these approaches? In this section too, it is not clear to me what the second task (LEP) is. What is an active/inactive conformer of a protein? Are the authors referring to an active/inactive classification tasks for ligands (i.e. as in virtual screening)? Are the same molecules considered in both tasks?

•	Some references do not seem to be present in the main manuscript (e.g. 20)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a self-supervised representation learning method for small molecules based on denoising the pair-wise distances. The learned representations are shown to improve predictions on downstream tasks. The method was deemed to be novel by the reviewers, and the evaluations convincing. However this was still a borderline paper, mainly due to concerns that the technique is an obvious application of SSL techniques already applied to other problems in ML.

On the strength of the convincing performance of the method, paired with its novelty, I vote for acceptance.
justification_for_why_not_higher_score: Given the borderline reviews, I cannot advocate for a stronger acceptance (spotlight/oral).
justification_for_why_not_lower_score: On the strength of the convincing performance of the method, paired with its novelty, I vote for acceptance.
Decision: Accept: poster