Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method to incorporate 3D information into molecular representation learning, Uni-Mol. Particularly, Uni-Mol includes three parts: 

1. Two models (a molecular model and a pocket model) built upon the SE(3)-equivariant transformer architecture; 
2. Two large-scale datasets (a 209M molecular conformation dataset and a 3M candidate protein pocket dataset) and corresponding pretraining strategies; 
3. Finetuning strategies for various downstream tasks. 

In the experiments, Uni-Mol outperforms SOTA in most of the tested tasks and demonstrates its ability in few-shot learning settings. 
strengths_and_weaknesses: **Strengths**:

* (originality) The proposed method, Uni-Mol, is original and has reasonable designs: (1) For the backbone model architecture, modifications (including invariant spatial positional encoding, pair representation, and a coordinate head) have been made to the standard Transformer to adapt to 3D molecular representation; (2) Two new large-scale datasets as well as the proposed 3D position denoising task for pretraining; (3) Detailed guidances of how to fine-tune Uni-Mol on downstream tasks. 
* (quality) The experiments were conducted thoroughly and touched four aspects of molecular representation learning: molecular property prediction, molecular conformation generation, pocket property prediction, and protein-ligand binding pose prediction. For each task, the datasets and baselines are chosen carefully. 
* (significance) The proposed method Uni-Mol outperforms previous work in almost every tested task, which is impressive. In some tasks, like molecular conformation generation and pocket property prediction, the improvements are significant. 
* (clarity) The paper is presented clearly and easy to follow. Contributions are properly emphasized. 

**Weaknesses**:

* (originality) The design of Uni-Mol is largely based on previous work. For the backbone model architecture, it is not new to use Transformers in molecular representation (e.g., GROVER), and most modifications can also be found in previous work (e.g., pairwise Euclidean distance matrix representation, the SE(3)-equivariant head as in EGNN, the masked atom prediction task). This limits the novelty of the proposed method. 
* (significance) Some experiment results didn't show significant improvement over previous methods (e.g., molecular property prediction tasks and ESOL). But this is acceptable since tasks like molecular property prediction have already been studied for long, and the baseline models are very competitive. 
* The related work focuses more on molecular representation methods, while the literature on pretraining has been ignored. 
questions: * In line 114 and line 119, it is mentioned that the 3D conformations and pocket binding positions are obtained by using simulation or optimization toolkits. Will these toolkits introduce bias to the data? If yes, do you have any idea for alleviating such bias?
* In line 130, it states that for the 3D position denoising task, the noise is uniformly sampled from [-1 A, 1 A]. How is the sampling strategy determined? Have you tried other sampling strategies like using a Gaussian distribution?
* In line 204, it is concluded that 3D information helps the model to learn better representations. Are there any ablation study results that can justify this more convincingly? 
* Will the datasets be released for academic use?
limitations: The limitations are not highlighted or summarized in the paper. 

To me, my most concern is about the data used to pretrain the model. As mentioned in the Questions section, since the 3D information in the data mostly comes from computational results instead of ground-truth annotations, how do we know such bias in data won't influence the model performance? If bias is inevitable, what benefits can be guaranteed by using such data?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a pretraining / self-supervised learning method for 3D molecular representation learning. The backbone is Transformer based. The pretraining is adding noise to atom coordinates. The pretraining  Experiments on molecular property prediction, molecular conformation generation, pocket property prediction and protein-ligand binding pose prediction show the effectiveness of the proposed method. 

strengths_and_weaknesses: 1. Among the early efforts, the paper proposes a pretraining / self-supervised learning method for universal 3D molecular representation
learning framework. 

2. The proposed method achieves SOTAs on several tasks and datasets.

questions: 1. The novelty of the proposed method seems not that significant. 
- One difference or advantage that the paper claims is 3D encoding. However, the Euclidean distance map (e.g.,[1]) or Gaussian functions for distance processing [2,3]  has been widely used in protein 3D structure learning. 

[1] Highly accurate protein structure prediction with AlphaFold

[2] GraphQA: protein model quality assessment using graph convolutional networks 

[3] Learning from Protein Structure with Geometric Vector Perceptrons

- There have been a few methods for the atom-level molecule or protein 3D structure modeling (e.g., [4]). The proposed method should be compared with them via experiments.

[4] Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures

- The pretraining method seems very simple. It is more like a trick. 
- (Minor) There are a few existing pretraining / self-supervised learning methods for protein representation learning. The authors might want to add a discussion about them.

[5] Structure-aware Protein Self-supervised Learning

[6] Contrastive Representation Learning for 3D Protein Structures

[7] Protein Structure Representation Learning by Geometric Pretraining


2. The motivation to employ Transformer is not clear. The explanation that "Transformer is the default backbone in representation learning" is not convincing or even incorrect. Moreover, as a Transformer-related work, it could be better to visualize the self-attention map to verify the motivation for employing Transformers. 


3. Besides adding the pair representation, is there any other difference with the original Transformer?  Moreover, using Transformers for 3D modeling has been explored in 3D vision. Some important competitors should be compared, such as [8], to show the effectiveness of the proposed method. 

[8] Point Transformer.



4. Some important experiments or ablation studies are missing. 

a) Accuracy w/ and w/o pertaining. 

b) Effect of invariant spatial positional encoding.

c) Effect of pair representation.

d) Effect of pair representation.

e) Effect of SE(3)-equivariance coordinate head.

limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work provides representation learning for 3D molecules (like BERT in NLP). The authors provide two pretrained models using the same model architecture. The first one is trained on 209M molecular conformations, and the second one is trained on 3M protein pocket data. The model architecture is a modified Transformer, and the pretraining tasks include masked atom prediction and 3D position denoising. The pretrained models can be used for various tasks, such as molecular property prediction, molecular conformation generation, protein-ligand binding pose prediction, and pocket druggability prediction. The performance is very good.
strengths_and_weaknesses: Strengths:
1. This work is well-written and easy to follow.
2. It provides two good pretrained models for small molecules and protein pockets. The models can be finetuned for various downstream tasks.
3. The experiment results show that the provided models can outperform previous methods on various tasks.

Weakness: 
1. About the model architecture: There are many existing Transformer architectures like [1] for graph representations. The authors should discuss the differences with these methods.
[1] Ying, Chengxuan, et al. "Do transformers really perform badly for graph representation?." 
2. About baseline results: For the protein-ligand binding pose prediction task, there are several graph DL-based methods like [2], I suggest the authors include such papers.
[2] Stärk, Hannes, et al. "Equibind: Geometric deep learning for drug binding structure prediction." 

questions: 1. What is the difference between the proposed model architecture and existing Transformer architectures like Graphormer?
2. I suggest the authors compare (or discuss) the newly-proposed graph DL-based methods on the protein-ligand binding pose prediction task.
3. Can we extend protein pockets to whole proteins since in some cases, we don’t know the pocket? If not, what is the challenge?
limitations: The authors have addressed the limitations of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed the same SE(3)-equivariant transformer variant, called Uni-Mol, for both molecular pretraining and protein pocket pretraining. To make the model SE(3)-equivariant, they devise an invariant spatial positional encoding for the transformer variant.  Specifically, a affine transformation is used to fuse euclidean distance of atom pairs and types of atom pairs, then a Gaussian density function is used to get continuous representations of the atom pairs. After that, the self-attention mechanism takes the pair representation as a bias term in softmax calculation to better leverage 3D information of atoms. Following the idea of EGNN, the authors also propose a SE(3)-equivariant coordinate head to to directly predict coordinates of atoms. According to different downstream tasks, the two models are used independently or jointly.
strengths_and_weaknesses: This manuscript describes a universal pretrained model for both molecular and protein pocket representation. Specifically, it shows an interesting way of integrating attention mechanism with pair Euclidean distance matrix and atom pair type matrix, keeping the transformer variant SE(3)-equivariant. The authors conduct many downstream tasks to evaluate the performance of the pretrained model and the results are mostly promising.  However, it is a little pity that some aspects of the approach are not clearly communicated. 
questions: 1、For molecular conformation generation task, what is the input of the model for fine-tuning? That is, does the model need some kind of ‘random atomic coordinates’ as input for molecular conformation optimization? Moreover, how to randomly generate 10 conformations as the authors mention in Sec. 3.2? I guess ETKGD is utilized again to generate coordinates as inputs of Uni-Mol. If so, do other models need 3d coordinates as input, i.e., using ETKGD in a similar way? Do the authors check whether any molecules exist in both pretraining data set and test set?
2、For pocket property prediction task, the authors construct a regression dataset based on Fpocket tool. What is the difference between Fpocket Score and Druggability Score? Moreover, it seems only Fpocket Score is conducted in Table 4. Besides, how well Fpocket tool can perform in reality? Can the authors test this tool on NRDLD ( if possible )?
3、For 3D prediction tasks of protein-ligand pairs, the authors claim that they use a simple differential evolution algorithm to sample and optimize the complex conformations. More details should be revealed.
4、The authors should investigate the impact of the pair Euclidean distance matrix in ablation studies.
5、The authors should demonstrate the effectiveness of pretraining process. That is, how much improvement can pretraining bring to model performance?
6、The authors mention that 3D spatial positional encoding leaks chemicl bonds in Sec. 2.2 which is contradictory to some content in Sec. 2.1 (third to last sentence in invariant spatial positional encoding part).
7、In Table 2 in supplementary material, max number of atoms is 256 for protein pocket. Is this value big enough for a protein? Moreover, why vocabulary size is different between molecules and proteins (30 for molecule and 9 for pocket )?


limitations: 1、 The proposed SE(3)-equivariant transformer architecture has a limitation on the length of input molecules. In another word, it can not handle with molecules or protein pockets with length bigger than the max atom numbers (in this paper, the number is 256).
review_title: null

Meta-review: metareview: This paper proposes a new framework for molecular representation learning (MRL) using both 2D and 3D molecular data. This framework is general and applied to various problems (e.g., protein-ligand binding pose prediction and molecular conformation prediction). I believe this paper is potentially quite impactful and able to reshape how research is conducted for MRL research. 

However, the contribution of this paper is unclear in its current form. 
- The proposed methodology is not very novel and uses a combination of existing methods. 
- While the main contribution of this paper is to propose a new framework for MRL, the experiments focus on evaluating a single algorithm (i.e., SE(3)-equivariant model + 3 self-supervised tasks) compared to existing algorithms under different frameworks. In other words, the experiments do not deliver new information since (1) existing works demonstrated how combining 2D & 3D data improves downstream task performance and (2) pretraining is useful for the considered downstream tasks.

Overall, I recommend rejection for this paper. However, I believe this paper can be a very strong submission for the next conference if the authors clearly demonstrate their contribution. For example, I think the proposed idea would be pleasantly presented as an important "benchmark" paper, rather than a framework with superior performance.
confidence: Certain
Decision: Reject