Exploring Molecular Pretraining Model at Scale
Authors: Xiaohong Ji
Zhen Wang
Zhifeng Gao
Hang Zheng
Linfeng Zhang
Guolin Ke
Weinan E
Venue: NeurIPS 2024 poster (2024)
Keywords: Molecular Pretraining
Scaling Law
Molecular Property Prediction
Primary area: machine_learning_for_healthcare
TL;DR: we systematically investigate the scaling laws on molecular pretraining and scale the model to 1.1 billion parameters  through pretraining on 0.8 billion conformations, making it the largest molecular pretraining model to date.

Abstract: In recent years, pretraining models have made significant advancements in the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the expansion of model parameters and data size, a phenomenon now recognized as the scaling laws. However, research exploring scaling law in molecular pretraining model remains unexplored. In this work, we present an innovative molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, examining the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale the model to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show the consistent improvement on the downstream tasks as the model size grows up. The model with 1.1 billion parameters also outperform over existing methods, achieving an average 27\% improvement on the QM9 and 14\% on COMPAS-1D dataset.

Exploring Molecular Pretraining Model at Scale
Xiaohong Ji"; Zhen Wang'; Zhifeng Gao'; " Hang Zheng'
Linfeng Zhang' Guolin Ke' Weinan 2
DP Technology. Beijing. 100080, China.
2 AT for Science Institute, Beijing 100080, China.
3 School of Mathematical Sciences, Peking University, Beijing, 100871, China. *Center for Machine Learning Research, Peking University, Beijing 100084, China. [jixh, wangz, gaozf, zhengh, zhanglf, kegl) tech weinan@math. pku. edu. en
Abstract
In recent years, pretraining models have made significant advancements the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the recognized
expansion of model parameters and data size, a phenomenon the scaling laws. However, research exploring scaling law in molecular pretraining innovative
models remains unexplored. In this work, present Uni-Mol2, molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show consistent improvement the downstream tasks as the model size grows. The Uni-Mol2 with 1.1B parameters also outperforms existing methods, achieving an average 27% improvement on the QM9 and 14% COMPAS-ID dataset.
Introduction
With the exponential growth of available biological data, there arises a critical need for innovative computational methodologies to utilize this wealth of information effectively. While traditional molecular representations like fingerprint-based models [T lack the ability to capture fine-grained structural features and struggle to handle large complex molecules effectively. Molecular Repre- sentation Learning (MRL) using molecular pretraining emerges as promising approach, leveraging the power of machine learning to imbue algorithms with deep understanding of molecular structures and functions. Various modalities of molecular representation by pretraining have been extensively studied in the past. The typical approach for representing molecules involves two main strategies. One strategy is to represent molecules as one- -in-dimensional sequential strings, such as SMILES [3] 4 and InChI [5). The representative work is SMILES-BERT (3), which learns from large-scale unlabeled data through the masked SMILES recovery task. Another strategy is to represent molecules two-dimensional graphs 66 7.8 MolCLR (8) typical method, learns the representations from
unlabeled data by contrasting positive molecule graph pairs against negative ones. Additionally, a growing trend is to leverage three-dimensional information in MRL to enable tasks like 3D geometry
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

prediction or generation 9 10 III. The pursuit of molecular pretraining has sparked a wave of exploration and innovation across the field, marking a new era of discovery within the discipline.
While in the past few years, scaling up pre-trained language models [12 13 been achieved remarkable progress in natural language processing (NLP) and computer vision (CV). The exponential growth in model size and the richness of training data have significantly enhanced the capabilities and performance of LLMs across various NLP and CV tasks. Despite extensive research on molecular pretraining, the majority of prior studies have been conducted on relatively small scale, utilizing limited parameters and datasets. Learning scalable molecular representation learning is rarely explored and remains challenging problem. The recent [19]'s 's work conducts a series of data-centric experiments to demonstrate scaling behaviors in various aspects. The exploration of the molecular pretraining model is limited to the GIN (20), SchNet (2I). whose model scale and data scale are comparatively small.
15
16
17
18] has
To delve deeper into the scaling of molecular pretraining foundational models, our preliminary investigations have yielded notable insights within this domain. We summarize the contributions of this work as follows:
We have curated and organized a dataset comprising upproximately 884 million 3D conformations, which contains 73 million scaffolds for pretraining. To the best of our knowledge, this is the largest dataset of molecules with 3D conformations for molecular pretraining to date, which provides the foundation ingredient for training large-scale molecular models.
We systematically study the scalability and flexibility of Uni-Mol2 in terms of model parameters, which range from 84M 1.1B parameters, and characterize the relationship between validation loss and model size, dataset size, and computational resources. It is the first time to demonstrate the scaling law of molecular pretraining and Uni-Mol2 currently the largest billion-scale molecular pretraining model to date.
We present an in-depth analysis of scaling trends about fine-tuning on downstream tasks as the results are shown in Tablegand Uni-Mol2 demonstrates consistent improvement in downstream task performance with increasing model parameters. The 1.1 billion parameters model also achieves significant improvement over the existing method.
2
Related Work
Molecular representation learning Previous research has extensively investigated various modali- ties for molecular representation. A range of methods have been proposed based on different types of information utilized during pretraining. SMILES-BERT (3) uses the smiles sequence in pretraining to capture the representation. Due to SMILES representation lack of explicit encoding of molecular structural information. To address this limitation, GROVER integrates Message Passing Networks into a Transformer-style architecture and learns from unlabeled molecular data through carefully designed self-supervised tasks at different levels of molecular topology (node, edge, and graph). MGSSL (22) developed a motif-based graph self-supervised learning strategy that predicts both the topology and label of motifs as they are generated the motif tree process. Furthermore, GEM(6) incorporates three-dimensional (3D) spatial structure information, atoms, bonds, and bond angles simultaneously to model the molecular representation. SphereNet 23) utilizes spherical coordinates and introduces spherical message- passing approach, providing an effective framework for 3D molecular learning. The survey (24) discusses GNN shows great efficiency in processing molecular graph structures and their strong capability capture local relationships within a molecule. However, the locally connected graph fails to adequately represent long-range interactions between atoms. In contrast, transformer-baase models have shown exceptional performance in various tasks within the molecular domain, demonstrating remarkable representation capabilities.
Foundation models Recently, there has been considerable interest in developing foundational models to consolidate and expand representations. The significant advancements in scaling up pre-trained language models [2
[3 [4] have fundamentally reshaped the field of natural language 26) also prove that the foundation model demonstrates strong performance on many NI.P datasets, sometimes reaching or exceeding the human performance. Some works in
processing. [25
15
18
CV[27] 28) demonstrate the potential for "L.LM-like" scaling in vision and underscore significant
improvement via model and data scaling. And Sora 29 30 multi-modal foundation model exhibits

---

Table 1: The different scale of Uni-Mol dataset and Uni-Mol2 dataset
Datasets
SMILES
Scaffold
Data Source
Uni-Mol Dataset
19M
4,224.621
ZINCI5, ChemBL, Commercial Database
Uni-Mol2 Dataset
884M
73,725,454
Uni-Mol Dataset, ZINC 20(31)
the capacity to offer sophisticated understanding regarding the intricate interplay of physical and contextual dynamics within depicted scenes.
3 Pretraining
The pretraining stage of molecular involves learning from vast amounts of molecular data to acquire comprehensive understanding of molecular representations. By pretraining on a large and diverse unlabeled dataset, the model can develop a rich understanding of molecular structures and properties, which can subsequently be fine-tuned or applied to specific downstream tasks, such drug discovery. materials design, or chemical synthesis. The section provides details of the data curation process for pretraining, the detailed pretraining architecture, the well-designed self-supervision tasks, and the specific training procedures employed for scaling the model.
3.1
Data
To augment the richness and diversity of the dataset, we integrated the two parts we have collected. One part consists approximately 19 million molecules sourced from Uni-Mol [TT], while the other is derived from ZINC20 [31] which includes 1.4 billion compounds. We downloaded the subset with standard reactivity, which contains 884 million compounds from website Table | shows the enrichment compared with Uni-Mol dataset. The overall Uni-Mol2 dataset has increased by over 40 times compared to the Uni-Mol dataset, with the number of scaffold increasing by 17 times, greatly expanding the diversity of the data. Figure Tigure[[oop Top) shows the numeric distributions of the top 40 skeletons in Uni-Mol dataset and the number corresponding in Uni-Mol2 dataset. To prevent data leakage in evaluating pretraining performance, the Uni-Mol2 dataset as the validation set to evaluate the effectiveness and investigate the scaling relationship.
randomly sampled 520k molecules from
As illustrated in the visualization depicting the frequency distribution of the top 40 Murcko scaffolds in Uni-Mol2 dataset (refer to Figure (Bottom)), it is observed that the molecular scaffold conforms to a distribution characterized by a Tong-tail pattern. To create more balanced training dataset, we categorize the SMILES of Uni-Mol2 training set by Murcko scaffold, resulting in 73,725, ,544 scaffolds along with frequency distribution. Then, we utilize the temperature- based sampling method 32 33 as described in equation to select molecules from Uni-Mol2 training set.
N Σ
P.
(I)
Pscaf focaf fold, = softmax
Where Na; represents the number of molecules with i-th scaffold in Uni-Mol2 training set. The temperature modulates the smoothness of the molecular distribution across scaffolds. We use an empirical value r = 0.005 as the temperature to effectively balance the proportion of molecules with high-frequency and low-frequency scaffolds.
3.2 Architecture
As depicted in Figure Uni-Mol2 essentially adheres to the model design of Uni-Mol+ 34), acting as a two-track transformer that concurrently processes atom features and pair features. Consistent with
3

---

Dutasel
Un Datasol
Figure 1: Top: Comparison of scaffold frequency between Uni-Mol and Uni-Mol2 dataset. Bottom:
Scaffolds distribution on Uni-Mol2 dataset
Pretraining tasks
Coordinate Denoising
Masked Token Prediction
N
Backbone Block
Atom Type
Position- Prediction Head
FFN
Set Attention Por Hiss
Backbone
Triangular Update
Acom Representatio
Gourd Natvoe
Mask ToRce
Pai Reprosemttt
Product
Masked Atoms
Bond+ SPO
Noke Coordinates
Atam Regreserttaion
Pair Representation
molesules
Figure 2: Left: The overall pretraining architecture. Middle: Atom and Pair representation. Right: The details of backbone block
Uni-Mol2 employs two self- seff-supervised tasks: masked token prediction and molecule coordinate denoising. The detailed framework is presented as follows:
Uni-Mol 11
Feature Representation and Position Encoding Given molecular M = (xr, e, r), where r e Baxd, denotes atom features, e E Rax denotes bond features and r E R=x denotes coordinate features. Following Uni-Mol+, employ RDKit to obtain atom token x "ooen atom degree x d'degree and atomic features i Asooii for each atom. The atom embedding 4 then initialized as:
d Embedding
Embedding (# (ddegree + Embedding (2 (
For pair features, we utilize RDKit obtain bond features Bood by Embedding(#$/1))) We adopt the 1.)
to encode the shortest path distance SPDD of atom pair (i. j) in the molecular graph by Embedding 3.j Additionally, employ the Gaussian kernel approach with pair type,
method from [3
34
as described in [36 [1]. to encode the Euclidean distance of the atom pair (i. j) by 1885, The pair i.j embedding pair is then initialized as:

---

f pair
Embedding bond + Embedding(I$á)) + vtd
Two-track Transformer Layer The backbone of Uni-Mol2 has N blocks, each block handles atom representation and pair representation concurrently. Formally, for the 1-th block, Uni-Mol2 update atom representation x² by
Self SelfAAttenttomaaiiiii((NN ") x x' + FFN(LN(x1)
(4)
For the pair representation p¹,
p' p 1 OuterProduct(LN[[pp
p' = p + TraagglaaUppaatt((NN(11)11 p' = p' + FFN(LNN()))
(5)
The details of SaliArtentionPairBiis,, OuterProduct, and TriangularUpdatt are aligned with those of Uni-Mol+, Additionally, Uni-Mol2 adopts pre-norm layer normalization atom and pair rep- resentation, which differs from Uni-Mol+, to improve stability in the model's training dynamics. Specifically, we set atom embedding Zatom as atom representation z0 and pair embedding pair as pair representation pd for the first block.
Pretraining Tasks To effectively model the structure of molecular conformations, we set pretraining tasks basically following Uni-Mol. In detail, for each molecule, we randomly mask 15% of the atom tokens with the placeholder token [MASK]. We then add the atom token prediction head to optimize masked atom token loss Latom by
Laton H (Fatom [mask] Fpatom (mask])
(6)
where H denotes the cross entropy function, Tatom [mask] denotes the masked atom tokens and ppatom [mask] denotes the corresponding predicted atom tokens for the masked positions.
In the coordinate denoising task, to increase the challenge of the pertaining task, we introduce Gaussian noise with a standard deviation of 0.2 for all the atom coordinates. Additionally, to enhance broader applicability across downstream applications, we mask atomic features Tatomics bond features Thond- and shortest path distance features TSPD with probability of 50%. Furthermore, we align the conformation of the noised molecule, denoted as Fnoised_coor. with that of the raw molecule, denoted as Teoors using the Kabsch algorithm.
In contrast to Uni-Mol, Uni-Mol2 employs the position prediction head to predict the atom coordinates I "peoor of molecules.
A pas Dis(r moisedd cour)
Q pos FFN(LN( )) K pos FFN(LN((*))
V pos FFN(LN )), Bpos FFN(LN
attnpos pos = softmax Qpos KTT pos + Bpos) o Apos
A upos attripos Vposs ppos FFN A epos)
Ppcor Tnoised_coor Apps
where Dis denotes element-wise subtraction of positions between different noised atoms 7 Tnoised__cme Specifically, the difference in position between atoms i and j is given by (p. (i. I' "noised_coon cvor, Tnoised coor.j And denotes Hadamard product. LN denotes layer normalization. FFN denotes feed-forward network. In practice, we use multi-head attention; for simplicity in writing, we omitted the notation related to heads here. Once the predicted coordinates 'pcoor are obtained, pair-distance "pdistance can be derived by calculating the Euclidean distances between each pair of T Tpecor. We integrated coordinate prediction and pair-distance prediction with E loss into Uni-Mol2's optimization process for the coordinate denoising task:
predicted
5

---

Table 2: Architecture of Uni-Mol2 different scale
Embedding dim
Atlention heads
Pair embedding dim
hidden dim
FFN embedding dim
Learning
Batch
Params
Layers
rate
size
42M
768
48
512
768
lc-4
1024
84M
768
48
512
768
lc-4
1024
164M
768
48
512
64
768
le-4
1024
310M
32
1024
64
512
64
1024
le-4
1024
570M
32
1536
96
512
64
1536
1e-4
1024
1.1B
64
1536
96
512
64
1536
1e-4
1024
Looor
popor Tooor
|1
(8)
Ldistance pdlistance
'distance
We eliminated stabilizing regularization terms from the Uni-Mol model, yielding the final loss of Uni-Mol2:
Liota
Catom
+ Looor
Ldistance
3.3 Hyperparameter and Training Details
We study the scalability of Uni-Mol with the scale from 42M to 1.1B, and all the parameters for Uni-Mol2 at different scales are listed in Table And Uni- Uni-Mol2 trained with Adam optimzer 337 38). with the following syper-parameters: B1 0.9 and $2 0.99 and weight decay le 4. The gradient clip norm is set to 1.0 for training stability. The learning rate scheduler employed is polynomial decay scheduler during pretraining. Specifically, all models reach its maximum learning rate value le - 4 after 100,000 warm-up steps and decay the learning rate of each parameter group using a polynomial function with power 1.0. All the models are trained with mexxpprcciiion [39] for training efficiency.
Using the temperature- bassed sampling method outlined in Equation conformations as training samples from the dataset. All models were subsequently trained on these 838 million samples. All these conformations optimized with the Merck Molecular Force Field (MMFF) [41] in RDKit. For models containing parameters ranging from 42M to 310M, employed 32 NVIDIA A100 GPU cards, while for models with 570M and 1. parameters, we utilized 64 NVIDIA A100 GPU cards. Details on pretrain time complexity across varying sizes are provided in Appendix I2
sample 838 million
generated using the ETKGD method (40) and
Scaling Laws
Several studies 15 42, 43 on large language models (LLMs) investigate the power-law connections between model performance, commonly assessed by validation or test loss, and factors such as the number of model parameters, dataset size, and compute budget. Here, we aim to define the power-law of validation loss C during the model's convergence period. In Figure 3 we present the validation loss of Uni-Mol2 models with parameter counts varying from 42 million to 1.1 billion during the training process. We mainly examine the impact factors of three aspects: data scale N, model scale M, and compute budget scale C. Given that constant batch size B of 1024 is maintained for Uni-Mol2 across various scales, the number of training steps S is considered as a suitable proxy for D, as D can be approximated by the product BS.
We initially designed power term for M and separately. Additionally, we approximate the computed budget C' as M S. Notably, we have neglected the intricate relationship between actual computing costs C and MS, instead subsuming into the parameter estimation. Adhering to the design principles of (22), the loss function C(M, D) should exhibit scale invariance, limit consistency, and analyticity to ensure stability and consistency across varying parameters. As a result, we derived the following empirical pawer-law relationship:
C(M,S,C) S + +a. C
(10)

---

Loss Diferent Scale
Loes at Diferent Scale (Log-Log)
6.275
0.250
B0M
-1.4
80M
164M
164M
310M
Valid bee
0.200
1.18
570M
Volld loss (log)
1.8
570M
1.18
0.175
2.0
0.150
6.125
-2.-
6.100
200k
400k
600k
800k
11.0
11.5
12.0
12.5
13.0
23.5
Samples trained
Samp las trained (lo)
Figure 3: Validation loss curves. Training curves for Uni-Mol2 model from 42M to 1.1B parameters. Models are trained on 0.8B samples. At the convergence stage, the 84M parameters model has a loss of 0.105, and the I.IB parameters model reaches a loss of 0.087.
Loes curve for Un-Mlo2 570M1
Loss curve for Uni- 1100M
0.6
0.5
0.4
loldless
0.1
400k
200k
403k
500k
800c
Samples trained
Samples trainod
Figure 4: Graph of actual loss and prediction loss across different updates for the 570M (Ieft) 1.1B (right) models
and
We established the relationship based on the validation loss trajectory of Uni-Mol2 across different scales, as detailed in Table Specifically, we utilized the validation data from Uni-Mol2 42M, 84M, 164M, and 310M, recording the validation loss every 10,000 training steps. Furthermore. to prevent the performance during the transient period from affecting the parameter estimation, we excluded the loss of information from the first 200,000 training steps. Consequently, have:
C(M, S, C) = 2.660M 1.137 11137 + 1-8488 0.225
0.588C-1.477
(11)
As shown in Fig 4 equation 11 fits the actual validation loss well for Uni-Mol2 570M and Uni-Mol2 1.1B parameters model, particularly when the model's performance reaches convergence. To assess the scaling law's effectiveness, we calculated Relative Mean Absolute Error (RMAE), Mean Square Error (MSE), R-squared, and Pearson Correlation Coefficient by comparing predicted validation loss with actual validation loss over the last 100,000 steps for Uni-Mol2 570M and Uni-Mol2 1.1B on Table 3, The high Pearson Correlation Coefficient and R-squared we computed indicate strong linear relationship between our predicted values and the actual data. The RMAE values for Uni-Mol2 570M and Uni-Mol2 I.IB are 0.0169 and 0.0095, respectively, indicating that Equation IT accurately models the loss curve. Specifically, for the Uni-Mol2 570M 810,000 steps, the actual validation loss was recorded at 0.09, compared to a predicted loss of 0.088, yielding a predicted validation error of 2.22%. Meanwhile, for Uni-Mol2 1.1B the same step, the actual validation loss stood at 0.087, slightly below the forecast of 0.0871, with prediction error of 0.23%.
5 Downstream Experiment
Upon pretraining with extensive unlabeled datasets using the predefined task, one should acquire a highly accurate molecular representation for fine-tuning downstream tasks. In this section, we conduct experiments the ability of scaled models on downstream tasks.
7

---

Table 3: Metrics about Scaling Law for Uni-Mol2
Pearson Correlation Coefficient
Model
RMAE
MSE
R-Squared
Uni-Mol2 570M
0.0169 0.0095
2.450e-6 8.458e-5
0.92
0.85
Uni-Mol2 1.1B
0.87
0.75
Table 4: Mean absolute error(MAE, 4) results QM9 Dataset
Model
HOMO LUMO GAP
alpha
C,
mu
ZPVE
GROV E Rause
0.0079 (3e-04)
2.365 (0.302)
1.103 (0.339)
0.618 (0.002)
113.01 (4.206)
0.0035(3e-04
GROV Raugge
0.0083 (6c 04)
2.240 (0.385)
0,853 (0.186)
0.623 (0.006)
85.85 (6.816)
0.00381 04)
GEM
0.00674ee-05)
0.589(0.0042)
0.444(0.0015) 0.237(0.0137)
25.67(0.743)
0.0011(2e-05)
Uni- Mol
0.0043(2e-05)
0.363(0.009)
0.155(0.0015) 0.183)0.002)
4.805(0.055)
0.0011 05)
Uni-Mol2 84M
0.0038(5e-05)
0.376(0.027)
0.178(0.012)
(.0050000009)
4.968(0.235)
0.0010(1c-04)
Uni-Mol2 164M
2.036((e-05)
0.325(0.004)
0.093(0.0006) 0.157(0.017)
4.935(0.189)
0.0005(1e-05)
Uni Uni-Mol2 310M
0.0036(ee-05)
0.315(0.003)
0.092(0.0013) (.043
4.672(0.245)
0.0005(1e-05)
Uni-Mol2 570M
0.0036(2e-05)
0.315(0.004)
0.147(0.0007)
0.089(0.0015)
4.523000800
0.0005(e-05)
Moi- 1.IB
0.0035(1e-05)
0.305(0.0033)
0.144(0.002)
0.089(0.0044)
4.265(0.067)
0.8M05(8e-05)
5.1 QM9 Dataset
datasets evaluate the performance of the molecular pretraining model different scales and compare Uni-Mol2 with representative existing methods. QM9 dataset provides the geometric, energetic, electronic, and thermodynamic properties of the molecule, comprising 134 thousand stable organic molecules with up to nine heavy atoms. Due to QM9 containing several quantum mechanical properties with different quantitative ranges, each property is treated separate task. However, the HOMO, LUMO, and HOMO-LUMO GAP, which share similar ranges, trained together single task for simplicity
We employ QM9 44 45
Baselines We evaluate Uni-Mol2 against several baseline models, with a primary emphasis on pretraining baselines. Given that Uni-Mol demonstrates superior performance compared to these baselines in previous work (IT), our analysis concentrates on the comparison between Uni-Mol and Uni-Mol2, specifically examining the scalability of Uni-Mol2 at various scales. It is noted that we have shifted the dataset partitioning method from scaffold-base based partitioning, thereby increasing the task difficulty to evaluate the model's performance more comprehensively. The dataset is then divided into training. validation, and test sets in proportions of 80%, 10%, and 10%, respectively. Following previous work 66 we report the mean and standard deviation by the results of random seeds.
partitioning to scaffold similarity-
The results are presented comprehensively in Table where the best results are marked in bold. Uni-Mol still outperforms baselines on almost all downstream datasets. Uni-Mol2 outperforms Uni-Mol in four out of the six tasks examined. But as the model parameters increase, Uni-Mol2 demonstrates significantly improved performance, surpassing Uni-Mol across all tasks at the 1.1 billion parameter level, achieving average 27% improvement the QM9 task for all properties. We systematically investigate the scaling of Uni-Mol2 across parameter sizes ranging from 84 million to 1.1 billion. Except for the C, property prediction task, the results for other properties progressively improve as the model size increases, consistent with the patterns observed in the model's 's validation performance. This indicates that enlarging the model consistently enhances downstream performance. However, for properties such as HOMO, LUMO, HOMO-LUMO GAP, and ZPVE, the results converge the model size increases. This convergence suggests that further increases longer influence the performance ceiling for these tasks in model size.
Results
5.2 COMPAS-ID Dataset
Due to the QM9 dataset only providing the conformation, some molecules failed to generate the atom and bond feature correctly. Therefore, fine-tuning Uni-Mol2 on the existing QM9 dataset to evaluate its effectiveness with bond and edge features presents non-trivial challenge. To further validate the performance and generalization capabilities of the Uni-Mol2 pretraining model,
8

---

Table 5: Mean absolute error(MAE, 4) results COMPAS-ID Dataset.
Model
aEA
alP
dispersion
Dipmom Debye
Uni-Mol
0.0099(20-05) 0.0104(2e-05)
0.0083(9c-05)
0.0092(6e-04)
0.0198(2e-04)
Uni-Mol2 84M
0.0081(3e-05
0.0092(5e-04)
0.0196(1e-04)
Uni-Mol2 1.1B
0.0103(4c-04)
0.008(1e-05)
0.0081(1e-04)
0.0186(3e-44)
Uni-Mol2 84M *
0.0104(4c-04)
0.0077(5e-05)
0.0085(1c-04)
0.0173(6c-04) 0.0170(2-04)
Uni-Mol2 I.IB
0.0093(4c-05)
0.0074(9c-05)
0.0067(2e-04)
Table 6: Mean absolute error(MAA, 4) about HOMO-LUMO GAP on QM9 Dataset
Model
train50
train
train200
Uni-Mol2 84M
0.0062(8.1e-05)
0.0053(1.0e-06)
0.0046(1 0.0046(1.0e-06) 06--06)
Uni-Mol2 164M
0.0058(3.7c-05)
0.0050(1.4e-05
0.0044(6.9c-05)
Uni-Mol2 310M
0.0056(4.7c-05) 0.0049(0.4c-06)
0.0044(4.0c-05)
Uni-Mol2 570M
0.0057(4.2e-05)
0.0048(1.8e-00))
0.0044(8.1e-06
Uni-Mol2 1.1B
0.0056(1.ee-05)
0.0048(3.ee-05)
0.0043(4.7e-05)
utilized COMPAS-ID ID from COMPAS project [46]. COMPAS-ID offers essential computational properties crucial for comprehending the behaviour of polycyclic aromatic hydrocarbons and other organic molecules across various chemical and physical processes. Modeling the relationships of these properties has significant implications for the field of organic photoelectric materials.
We still follow the QM9 scaffold similarity-based partition and split it by ratio of 8:1:1 into the train, validation, and test sets. Table presents the predictive capabilities of Uni-Mol2 regarding photoelectric quantum properties. The model with * suffix indicates that they incorporate atom and bond features. The results indicate that Uni-Mol2 excels in all tasks except for aEA property prediction task. Additionally, consistent with findings from the QM9 dataset, Uni-Mol2 demonstrates superior performance across all tasks as the model scales up. The results also show that under the same parameter scale, models incorporating atom and bond features outperform those without these features. Uni-Mol2 1B achieves 4% improvement over Uni-Mol, while Uni-Mol2 1B with atom and bond feature achieves 14% improvement over Uni-Mol. This suggests that, in certain scenarios, these features consistently provide a significant advantage.
5.3 The Performance on Limited QM9 Dataset
In numerous fields like bio-medieine, acquiring extensive well-annotated molecular data often expensive and time-consuming, Typically, these datasets include only a limited quantity of data [47 48). To evaluate the performance of Uni-Mol2 with restricted data availability, we conducted sampling on the QM9 dataset. We sampled the training set by stratifying it according to the quantile binning of the HOMO-LUMO GAP label from the QM9 test set and then created subsets named train50, train 100, and train200 by sampling at 50%, 100%, and 200% of the test set size, respectively.
We enhanced Uni-Mol2 from 84M to 1.IB parameters using train50, train 100, and train200 datasets to predict HOMO, LUMO, and GAP properties on the QM9 test dataset. As illustrated in Table 6 two conclusions emerge from the MAE for predicting HOMO, LUMO, and HOMO-LUMO GAP on the QM9 test set. First, the model's 's performance, indicated by a decreasing MAF, progressively improves as the training dataset expands. This is evident from comparing the MAE values between the train50 and train200 rows across different scales of the Uni-Mol2 models. For example, the Uni-Mol2 84M model shows a reduction MAE from 0.0062 0.0046, marking a 25.8% decrease as the dataset grows from 50 to 200 instances. Secondly, in situations where training data is scarce, the larger Uni-Mol2 models demonstrate enhanced predictive capabilities. This is evidenced by the fact that the Uni-Mol2 I.IB parameters model, which has the largest parameters, consistently records the lowest MAE scores for all sizes of training sets. This especially apparent in the train50 scenario, where it achieves an MAE of 0.0056, marking the best performance among the models discussed. These results highlight the advantages of enlarging both the training dataset and the model scale to improve predictive accuracy in downstream finetuning tasks with Uni-Mol2.

---

6 Conclusion
In this paper, to fully investigate the scaling law in the molecular pretraining field, we construct diverse dataset of molecular structures spanning 884 million instances and present novel molecular pretraining model Uni-Mol2. We successfully scale the model size 1.1 billion parameters from 84 million parameters and characterize the power-law relationship between validation loss and model size, dataset size, and computational resources. By empowering the power-law relationship of Uni- Mol2, it can shed light on the performance of the larger model. Our largest 1.1B parameters model also outperforms the existing methods.
The scaling law paves the way for exploring larger models to achieve higher performance. We hope that our work can open avenues for further exploration of the foundational molecular pretraining model. While larger models yield substantial benefits, there are still several potential future directions. Firstly, beyond property prediction tasks, it also worthwhile to explore whether the representation can be effectively utilized to enhance generative tasks. Secondly, even though the Uni-Mol2 has shown excellent results in several domains by increasing model capacity, it remains to be explored whether the advantages of scaling are beneficial for a broader range of tasks. Thirdly, the current mainstream large language models (LLMs) are predominantly based on decode-only architecture. It is worth investigating whether there are more elegant decode-only architectures for molecular pre-training models.
References
[1] Minjian Yang et al. "Machine learning models based on molecular fingerprints and an extreme gradient boosting method lead to the discovery of JAK2 inhibitors". In: Journal of Chemical Information and Modeling 59.12 (2019), pp.
[2] David Rogers and Mathew Hahn. "Exierieedcomnectiviyy fingerprints". In: Journal of chemical informa- tion and modeling 50.5 (2010), pp. 742-754.
[3] Sheng Wang al. "Smiles-bert: large scale unsupervised pre-training for molecular property prediction". In: Proceedingy of the 10th ACM international conference bioinformatics. computational biology and health informatics. 2019. pp. 429-436.
[4] Zheng Xu al. "Seq2seq fingerprint: An unsupervised deep molecular emhedding for drug discovery". In: Proceedings of the 8th ACM international conference on bioinformatics, computational biology, and health informatics. 2017, pp. 285-294.
[5] Robin Winter et al. "Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations.. In: Chemical science 10.6 (2019). pp. (692-1701.
[6] Xiaomin Fang et al. "Geonetry-eehannee molecular representation learning for property prediction". In: Nature Machine Intelligence 4.2 (2022). pp. 127-134.
[7] Yu Rong et al. "Self-supervised graph transformer on large-scale molccular data". In: Advances in neural information processing systems 33 (2020), pp. 12559-12571.
[8] Yuyang Wang et al. "Molecular contrastive learning of representations via graph neural networks". In: Nature Machine Intelligence 4.3 (2022), pp. 279-287.
[9] Hannes Stärk et al. "3d infomax improves gnns for molecular property prediction". In: International Conference Machine Learning. PMLR. 2022, pp. 80479-20502
[10] Shengehao Liu et al. "Pre-training molecular graph representation with 3d geometry". In: arXiv preprint 1rXiv:221000771 (2021).
[11] Gengmo Zhou al. "Uni-Mol: A Universal 3D Molecular Representation Learning Framework". In: The Eleventh International Conference on Learning Representations. 2023. URL: //ope//rrr net/forum? Ptt-pddd-----22 6K2RM6wVqKu
[12] Alec Radford al. "Language models are unsupervised multitask leamerr".. In: OpenAl blog 1.8 (2019).
p. 9.
[13] Tom Brown "Languagee models are few-shot learners". In: Advances in neural information process- ing systems 33 (2020), pp. 1877-1901.
[14] Josh Achiam et al. "Gpt-4 technical report". In: arXiv preprint arXiv:2303.0872 (2023).
[15] Dee "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism", arXiv preprint arXiv:2401.0254 (2024), URL: hhtp::htp///// github. wtt. deepseek- ai ai/DeepSeck- LLM
[16] Hugo Touvron et al. "Llama: Open and efficient foundation language models". In: arXiv preprint
vxXiv2232213397 (2023).
[17] Haotian Liu al. "Visual instruction tuning". In: Advances in neural information processing systems 36
(2024).
10

---

[18] Zhe Chen et al. "Internvl: Scaling up vision foundation models and aligning for generic visual- -angul-inguiitic tasks". In: arXiv preprint arXiv:2312.14238 (2023).
[19]
Dingshuo Chen et al. "Uncovering neural scaling laws in molecular representation learning". In: Advances in Neural Information Processing Systems 36 (2024).
[20]
Keyulu Xu et al. "How powerful graph neural networks?" In: arXiv preprint arXiv:1810.00826 (2018).
[21] Kristof Schütt al. "Schnet: A continuous-filter convolutional neural network for modeling quantum interactions". In: Advances in neural information processing systems 30 (2017).
[22] Zaixi Zhang al. "Motif-based graph self- sufesuuperieed learning for molecular property prediction". In: Advances in Neural Information Processing Systems 34 (2021), pp. 15870-15882.
[23] Liu et al. "Spherical message passing for molecular graphs". In: International Conference on Learning Representations (ICLR). 2022.
[24]
Zhichun Guo al. "Graph-base based molecular representation leaming". In: arXiv preprint arXiv:200700886
(2022).
[25]
Jinze Bai al. "Qwen technical report". In: arXiv preprint arXiv:2309.16// (2023).
[26]
Albert Q Jiang et al. "Mistral 7B". In: arXiv preprint urXiv:3310.06825 (2023).
[27]
Xiaohua Zhai al. "Scaling vision transformers". In: Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition. 2022. pp. 12104-12113.
[28]
Mostafa Dehghani et al. "Scaling vision transformers 22 billion parameters". In: International Confer- ence on Machine Learning. PMLR. 2023, pp. 7480-7512.
[29]
OpenAL. Sora: Creating video from text. 2024. URL: htt.tttt/s//ppaaaiii.... com/sora
[30] Yixin Liu et al. "Sora: Review on Background, Technology, Limitations, and Opportunities of Large Vision Models". In: arXiv preprint arXiv:2402.11111 (2024).
[31] John Irwin et al. "ZINC20-a a free ultralarge-scaa chemical database for ligand discovery". In: Journal of chemical information and modeling 60. 12 (2020), pp. 6065-6073.
[32] Alexis Conneau and Guillaume Lample. "Cross-lingual language model pretraining". In: Advances in neural information processing systems 32 (2019).
[33] Hyung Won Chung et al. "UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining" " In: The Eleventh International Conference on Learning Representations. 2023. nwt.
URL: https://openrenee
[34] Shuqi Lu ct al. Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+. 2023. arXiv:
2303. 16982 [physics. [physicsschem-phh)]
[35]
Chengxuan Ying et al. "Do transformers really perform badly for graph representation?" In: Advances in neural information processing systems 34 (2021), pp. 28877-28888.
[36] Yu Shi et al. "Benchmarking graphormer on large-scale molecular modeling datasets". In: arXiv preprint arXiv:2203.0 04810 (2022).
[37] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: arXiv preprint arXiv::412.om 2.6609000000000 (2014).
[38] Ilya Loshchilov and Frank Hutter. "Decoupled weight decay regularization". In: arXiv preprint
arXiv:1711.0500 (2017).
[39]
Paulius Micikevicius et al. "Mixed precision training". In: arXiv preprint 17X1v:1710.03740 (2017).
[40] Sereina Riniker and Gregory A Landrum. "Better informed distance geometry: using what know to improve conformation generation". In: Journal of chemical information and modeling 55.12 (2015),
pp. 2562-2574.
[41] Thomas Halgren. "Merek molecular force field. Basis, form, scope, parameterization, and perfor- mance of MMFF94". In: Journal of computational chemistry 17.5-6 (1996), pp. 490-519, [42] Jared Kaplan al. "Scaling laws for neural language models". In: arXiv preprint arXiv:2001.0836) (2020).
[43]
Hui Su et al. Unraveling the Mystery of Scaling Laws: Part I. 2024, arXiv: 2403. 06563 [cs LC]]
[44]
Raghunathan Ramakrishnan et al. "Quantum chemistry structures and properties of 134 kilo molecules".
In: Scientific data 1.1 (2014), pp. 1-7.
[45]
Zhenqin Wu "MoleculeNet: benchmark for molecular machine learning". Chemical science
9.2 (2018), pp. 513-530.
[46]
Alexandra Wahab et al. "The compas project: A computational database of polycyclic aromatic sys- tems. phase I: cata-condensed polybenzenoid hydrocarbons". In: Journal of Chemical Information and Modeling 62.16 (2022), pp. 3704-3713.
[47] Keith T Butler et al. "Machine lcarning for molecular and materials science". In: Nature 559.7715 (2018),
pp. 547-555.
11

---

[48]
Han Li et al. "Improving molecular property prediction through task similarity enhanced transfer learning strategy". In: Iscience 25.10 (2022).
[49] Stefan Hougardy. "The Floyd-Warse Soadd-lrshall algorithm graphs with negative cycles". In: Information Processing Letters 110.8-9 (2010), pp. 279-281.
[50] Uni-Core Uni Core, efficient distributed PyTorch framework. 2024. URL: attps:///githin com/ dptech-corp/uste
[51] Jason Ansel al. "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transforma-
tion and Graph Compilation" ". In: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2024, pp. 929 947.
[52] Ashish Vaswani et al. "Atlention all you need", Advances in neural information processing systems
30 (2017).
12

---

A Implementation Details
A.1 Dataset Description
QM9 Dataset The QM9 dataset 44 is significant resource in the field of quantum chemistry, offering a single equilibrium conformation and 12 labels that include geometric, energetic, electronic, and thermodynamic properties. For the purpose of performance evaluation, we select the following properties: HOMO, LUMO, gap, alpha, C., mu, R², and ZPVE. The details the properties are as follows:
HOMO The HOMO (Highest Occupied Molecular Orbital) is the highest energy molecular orbital that is occupied by electrons in molecule.
LUMO The LUMO (Lowest Unoccupied Molecular Orbital) is the lowest energy molecular orbital
that is not occupied by electrons.
gap The gap, often referred to as the HOMO-LUMO gap, is the energy difference between the HOMO and LUMO. It is measure of the energy required to excite electron from the HOMO to the LUMO.
ZPVE ZPVE (Zero-Point Vibrational Energy) is the energy associated with the vibrational motion of atoms in molecule absolute zero temperature.
The a value represents the static polarizability of molecule.
Cv The C. (Heat Capacity at Constant Volume) is the amount of heat needed to raise the temperature of given amount of substance by one degree Celsius constant volume.
f The y (Dipole Moment) is the measure of the molecule's permanent electric dipole moment.
R² The R²2 (Electronic Spatial Extent) is defined the expectation value of the square of the electronic distance from the nucleus.
COMPAS-ID Dataset The COMPAS-ID dataset is part of the COMPAS Project, which is acronym for the computational Database of Polycyclic Aromatic Systems. The dataset is specifically focused on data-condensed poly-benzenoid hydrocarbons, which are type of polycyclic aromatic hydrocarbons (PAHs) with unique structure where the benzene rings are connected edge-to-edge. The COMPAS-ID [46] contains 8,678 molecules and offers essential computational properties crucial for comprehending the behavior of polycyclic aromatic hydrocarbons and other organic molecules across various chemical and physical processes. The details of the properties used in the downstream tasks are as follows:
aEA aEA (Adiabatic Electron Affinity) measures the tendency of a molecule to gain an electron.
alp aIP (Adiabatic lonization Potential) measures the energy required for molecule to lose an electron.
Dispersion Dispersion describes weak inter-molecular forces important for understanding molecu- lar interactions.
Dipmom Debye Dipmom in Debye indicates the polarity molecule, affecting its interactions and solubility.
A.2 Atom and Bond Feature for Molecules
The molecular feature used in Uni-Mol2 contains two parts: 1) Atom and bond features, we use RDkit to generate these atom and bond features input of Uni-Mol2. The detailed features are listed in Table 7 and Table R 2) Shortest path SPD,,j We employ the Floyd-Waa Warshall algorithm 49 to calculate the shortest distances between each pair of connected atoms.
A.3 Hyperparameter Settings
In line with previous methods, we employ grid search to find the optimal hyper-parameters for tasks within the QM9 and COMPAS-ID datasets. The specific hyper-parameters are detailed in Table 9 In all experiments, we select the checkpoint with the lowest validation loss and report the corresponding test set results based on that checkpoint. For the COMPAS-ID ID dataset, experiments were conducted
13

---

Table 7: Atom features
features
size
description
atom type chirality degree formal charge
119
type of atoms including C, N. O, etc, by atomic number type of chirality like Tetrahedral chirality the degree of integer electronic charge assigned to atom number of bonded hydrogen atoms number of radical electrons SP, SP2, SP3, SP3D, SP3D2
in molecule
number of H number of radical electrons
S
hybridization aromaticity in ring
whether an atom part an aromatic system whether an atom is within ring structure
Table 8: Bond features
features
size
description
bond type stereo
SINGLE, DOUBLE, TRIPLE, AROMATIC NONE, Z, E, CIS, TRANS, ANY,
6
conjugated 1
whether the bond is conjugated
using a single A100 GPU, whereas for the QM9 dataset, the experiments were run on eight A100 GPUs.
Table 9: Syyper-paramtes' for fine-tuning on QM9 and COMPAS-ID Dataset
Hyperparameter
Value or description
Leaminng rate
[4e-5, 6c-5, lc-4, 2c-4, 3c-4, 4c-4]
Batch size
[32, 64, 128]
Epochs
[40, 60, 80, 100, 200, 300]
Pooler dropout
[0.0, 0.1]
Warmup ratio
[0.0, 0.06, 0.1]
A.4 Evaluation Metrics
Diverse evaluation metrics can better help us understand and evaluate the effectiveness of our model. In this section, we introduce the evaluation metries used in this study. Given samples, where yi is the actual value and is is the predicted value.
Mean Absolute Error (MAE) calculates the average of the absolute differences between predicted and actual values regression tasks, treating errors of different scales equally.
MAE
9
36l
(12)
Relative Mean Absolute Error (RMAE) measures the average absolute prediction error relative to the actual values, providing dimensionless indication of model accuracy. By normalizing with the actual values, it removes the effect of the data scale, making it possible to compare data with different scales.
11
3k
RMAE
(13)
71
Mean Square Error (MSE) calculates the average of the squared differences between predicted and actual values, heavily penalizing larger errors.
(Ú: y:)
MSE
(14)
71
14

---

R- R-squared measures the proportion of variance in the dependent variable that can be predicted by the independent variables, highlighting the goodness of fit for a regression model. A higher R-squared indicates that the independent variables explain a significant portion of the variance in the dependent variable, while a lower R-squared indicates that the model explains less.
R-squared = E (v: 0:)²
(15)
The Pearson Correlation Coefficient (r) measures the linear correlation between two variables, ranging from -1 to 1. Larger absolute values signify a stronger linear relationship between the two variables, while values near 0 indicate a weak or non-existent linear relationship.
E (c:
(1: 9)
(16)
2)² Ei=ı (3 y)²
B Extra QM9 Property and ADME Experiment
B.1 Additional QM9 Property Experiment Results
We have undertaken additional experiments utilizing the QM9 dataset to assess molecular properties 10 Our findings indicate that the model's performance consistently improves with an increase in model size. This scalability suggests that employing larger models will further enhance the accuracy and reliability of molecular property predictions, thereby increasing the utility our approach for practical applications in the field.
Table 10: Mean absolute error(MAE, 4) results on QM9 Dataset
Madel Uni-Me22 1.00
/OOMD/LUMD/GAM GAP
appla
12
ZPVE
U
/
ILIIS (e-1X)
0.54MILEN))
6,440 1633
50000
5.5600 1241
(O4K)
4.3300000
(LICES) (6-15)
0. 6.M058.J88)
0.1460192)
(11996,8644)
4.265(L067)
0.99550-455)
4.5120.188)
5,71408.155
3.92800876)
5.88911.23)
2 1.0%
5.17%
-5.6%
5.11%
0%
26 mra
3.9%
20.5%
7.5%
B.2 Biogen ADME Dataset
The Biogen ADME dataset focuses the evaluation of drug metabolism and pharmacokinet- ies (DMPK) properties, specifically assessing absorption, distribution, metabolism, and excretion (ADME) characteristics of potential drug candidates. Table TieeTTllllaan illustrates the predictive performance of the Uni-Mol2 model regarding these ADME properties.
Table 11: Mean absolute error(MAE, 4) results on Biogen ADME Dataset
Model
1
PERM-1
SOLU-1
Uni-Mol
0.3085(0.003) 0.3017(0.007) 0.3045(0.003)
0.2886(0.008)
0.3167(0.010)
Uni-Mol2 84M
0.2853(0.006)
0.325(0.006)
Uni-Mol2 1.1B
0.2795(0.00291)
0.3062(0.005)
C Infrastructures
We utilize an efficient distributed PyTorch framework called Uni-Core (50), specifically designed for swiftly developing big--performance Py PyTorch models (5T), particularly those based on Transformer architectures Given the variability in molecule lengths. padding inputs to match the maximum molecular length is necessary during training. Consequently, the batch size for model training is influenced by the longest molecule in each batch. However, since molecule lengths follow long- tail distribution (with the majority falling within specific range), we employ dynamic batching techniques to enhance GPU utilization. By adjusting batch sizes according the maximum lengths of different batches, we can significantly boost GPU utilization with minimal effort.
15

---

The time consumption of reading data from distributed storage is often overlooked. We employ singular, dedicated process on each computational node to asynchronously replicate the training dataset of each epoch onto the host machine. This strategy effectively mitigates time overheads, thereby obscuring the duration spent on data reading from distributed storage. To resume the coruptiinn due to the infra and other factors effectively, we save model weight and optimizer state for every Ik step asynchronously. This means we will lose step training resources in the worst case of hardware instability loss spike during training. Meanwhile, any checkpoints exceeding the most recent ten files will be deleted to avoid consuming too much storage space.
D Pretrain Time Complexity and GPU Resources
We utilized computational cluster comprising 64 NVIDIA A100 GPUs, each equipped with 80GB of HBM2 memory. The GPUs were interconnected via high-speed Nvidia Infiniband fabric, offering 400 Gbps bandwidth for inter-GPU communication. The details of each model size listed in table
Table 12: Training Time of Uni-Mol2 different scale
Compute Resouce
Training Time (GPU hours)
Params
(GPUs)
84M
32
2585.6
164M
32
5120
310M
32
7680
570M
64
13824
1.IB
64
30720
E
Limitations
The major limitation of our study pertains to the absence of an exploration of the optimal batch size and learning rate. Our investigation primarily focuses on analyzing and delineating the power-law relationships among validation loss, model size, dataset size, and computational resources. The predictive accuracy performance aligns well with the scaling curve, indicating that the current optimal learning rate and batch size approximate the near-optimal values. However, existing research suggests progressive increase in the optimal batch size with augmented computing resources, while the optimal learning rate tends to decrease gradually. It is necessary to note that as we further increase the model's parameters, the final optimal values for learning rate and batch size may fall outside the currently identified range. Consequently, investigating the scaling law for optimal batch size and learning rate also paramount.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
* Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims articulated in the abstract and introduction accurately mirror the paper's s contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper outlines the limitations of the research and discusses the future direction to explore the limitations.
17

---

Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-Ee speecc-to-ttet system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper provides comprehensive details necessary for reproducing the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided not.
18

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code, model, and data are made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp) hhttps://nips.cc/publlll cctps/iccc/p/cccccccccccccccc
for more details.
While we encourage the release code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code. unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: h/nipss:////ss.../ cttpps.cc/pubbcc.. Siaddes/CooeSuubmsseee issi ssissnoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
19

---

The paper meticulously details all aspects of the training and testing process. It specifies the data splits used for training, validation, and testing, along with thorough description of the hyperparameters and their selection criteria.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: he paper uppropriately reports error bars. All the results reported with the mean and standard deviation by the results of 3 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: The paper provided the detailed information of compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, every respect, with the NeurIPS Code of Ethics hticcshtpps//eeeriiip httppp///aaaipp..cc//aa////ccttt
20

---

Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics,
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed in this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
21

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
22

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
23


Reviews:
[{"review_id": "vqhZpsNIRN", "reviewer": "Reviewer_V7dw", "summary": "This paper studies the pretraining task on molecular domain. The main contribution includes extending the size of pretraining dataset, scaling the model to 1.1B size, investigating pretraining scaling law behavior. The evaluation of the pretrained model is conducted for some molecular property prediction task.", "questions": "1. For these downstreaming evaluation tasks, the improvement of performance from small model to large model is not that high, (for example the number in Table 5 and Table 6) can the author provide some justification about these numbers? 1.1B model is significantly costly than 84M, so we would expect a larger improvement.\n2. Can the author include more downstreaming tasks?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extend the pretraining dataset size, including both strings, graphs, and 3D conformation. \n2. While the model architecture is not new and is based on existing work, the effort to scale it up to 1B model is great for the current literature. \n3. The scaling law behavior is interesting, while it is different from language model. \n4. While the downstreaming tasks are limited, consistent performance improvement is shown with increasing model size.", "weaknesses": "1. The evaluation of the pretrained model is limited, it would be great if the author can show more diverse and complicated chemistry tasks. Better include some domain-intense tasks. \n2. The model training time is not reported, the author should include enough detail for reproducing the experiment. \n3.  Pretraining tasks/losses have some hyperparameter, the author should include ablation study result for these choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer V7dw for the thorough review and insightful comments. Below, we have carefully considered your feedback and provided detailed responses to each point.\n\n**Response to Weaknesses:**\n\n1. **Limited Evaluation of the Pretrained Model**  \n   We begin by highlighting the promising results achieved in the fields of small molecules and photoelectrics, while also acknowledging the limitations of our evaluation tasks. Due to constraints on computational resources, we focused on a specific set of molecular property prediction tasks. We agree that including more diverse and complex chemistry tasks could provide a broader evaluation of the model's capabilities. To address this, we have added new Biogen ADME benchmark results to demonstrate the model's capabilities. In future work, we plan to incorporate additional domain-intensive tasks to better showcase the utility of our pre-trained model.\n\n2. **Model Training Time and Reproducibility**  \n   Thank you for pointing out the omission of detailed information regarding the model training time. We have outlined the details of the pre-training hyperparameters in Section 3.3 of the paper. Additionally, we provide comprehensive details about the training process, including hardware specifications, training duration, and computational resources used in the general rebuttal. We will subsequently integrate this information into the manuscript. If there are specific details or additional aspects you would like us to clarify, please let us know, and we would be happy to discuss them further.\n\n3. **Ablation Study on Pretraining Tasks/Losses Hyperparameters**  \n   Thank you for your valuable suggestion. We have indeed adjusted the hyperparameters of the loss function during model training to ensure the stability of the training process. In the end, we set the weight for each loss component to 1 across all model sizes. However, systematically conducting ablation studies on the loss function at expanded model and dataset scales would require substantial computational resources, which are currently beyond our capacity for comprehensive exploration in the short term. Nevertheless, we recognize the importance of this research direction and will consider it a significant area for future investigation.\n\n**Response to Additional Concerns:**\n\n1. **Justification for Performance Improvement**  \n   We fully understand your concern regarding the relatively modest performance improvement observed when transitioning from the small model (310M) to the large model (1.1B) in the downstream evaluation tasks, as illustrated in Table 5 and Table 6. To address this, we have provided further clarification of the results and supplemented our study with additional experiments. The detailed findings and explanations have been included in the general rebuttal section.\n\n2. **Inclusion of More Downstream Tasks**  \n   Yes, expanding the range of tasks will allow for a more comprehensive evaluation of the model's performance across different aspects. We will explore more applications in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnXe8JumJn", "reviewer": "Reviewer_buoE", "summary": "This paper studies large scale pretraining for molecule. The authors compose the largest molecular pretraining dataset, Uni-Mol2, and train the largest molecular pretraining model with 1.1B parameters. This paper also fits a scaling law in this domain that can accurately predict losses on a validation set. Downstream performance on QM9 and COMPAS-1D datasets demonstrate the proposed model can outperform previous approaches.", "questions": "Related questions have been asked above.", "limitations": "The authors have included a limitation section in Appendix C, on the prediction accuracy of the scaling law. However, I would like to see a discussion on scaling up models with the proposed approach when the experimental results have shown saturation with only 1.1B parameters – which may indicate that the proposed training losses cannot be used to train powerful larger and larger models.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper creates the largest molecular pretraining dataset and the authors indicate the intent to open-source the dataset in the checklist.  \n2. The authors pretrain various scales of molecular models up to 1.1B parameters on the new dataset, a scale that has not been studied previously.  \n3. The results on downstream tasks surpass previous works.", "weaknesses": "1. On the downstream numbers reported in the paper, the scores do not scale well with the model size for most of them. For example, on most of the properties Uni-Mol2 1.1B is not apparently better than much smaller models like Uni-Mol2 310M, and on COMPAS-1D the Uni-Mol2 1.1B is even comparable to Uni-Mol2 84M in some cases – it is likely that I am not familiar with these datasets and do not perceive the score difference well, yet I am wondering whether a gain like 0.0001 is really meaningful and how can we know it is statistically significant? Is this because the tasks are too simple or the large 1.1B model is not trained properly? If the reason is the former, I think more difficult tasks should be included to demonstrate the benefit of larger models – otherwise the impact of this paper on scaling up model sizes is limited.  \n2. The paper’s presentation should be improved. For example, in the results tables it is better to indicate the parameter size of the baselines (like Uni-Mol) to understand the effect of model sizes on the results. There are also  some grammar errors such as Line 39 “Most” -> “most”. Line 65 “To” -> “to”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate Reviewer buoE's careful review and thoughtful feedback. Your suggestions have greatly contributed to improving our manuscript. We respond to your comments and questions in detail below.\n\n\n**Response to Weaknesses**\n\n**1. Performance Improvements**\n\nWe acknowledge your concern regarding the insufficient improvement of Uni-Mol2 in certain downstream tasks. We have provided additional clarification for our downstream experimental results and added some new downstream experiments. Overall, Uni-Mol2 1.1B demonstrates a considerably greater performance improvement compared to Uni-Mol2 84M and Uni-Mol2 310M in downstream tasks. \n\nFor the aEA and aIP tasks in the COMPAS-1D dataset, Uni-Mol2 1.1B with graph features demonstrates a more noticeable MAE reduction compared to Uni-Mol2 84M (0.96% -> 10.58% for aEA and 1.23% -> 3.90% for aIP). On the 10 downstream tasks of the QM9 dataset, Uni-Mol2 1.1B achieved an average improvement of 8.472% compared to Uni-Mol2 310M. Detailed explanations and results supporting these clarifications can be found in the general rebuttal section.\n\n\n**2. Presentation Improvements**\n\nWe appreciate the suggestions for improving the presentation of our manuscripts. We will revise the results tables to include the parameter sizes of baseline models, such as Uni-Mol, to better illustrate the impact of model size on performance. Additionally, we will correct the identified grammar errors and thoroughly review the manuscript to ensure clarity and accuracy.\n\n\n**Response to Limitations**\n\n\nWe appreciate the reviewer's observation regarding the limitations of our proposed approach, particularly in relation to the observed saturation of experimental results with models containing 1.1 billion parameters. We agree that this finding suggests a potential limitation in the scalability of the proposed training losses.\n\nIn response, we would like to clarify that the saturation observed may be due to the specific experimental setup and dataset limitations. While some experiments of the current results show a plateau at 1.1B parameters, it does not necessarily imply an inherent ceiling of the proposed method. We added some new extra experiments in the supplementary appendix; the increase in Uni-Mol2 1B compared with 310M is still significant and does not indicate that the results have reached saturation.\n\nReturning to the starting point of this paper, we primarily investigate the relation of validation loss as the model, data, and computation size increase, which is consistent with scaling laws. In the field of molecular pretraining, establishing appropriate pretraining loss to ensure the scaling effects of the model is indeed a compelling direction[1,2,3]. Given our current result, we have ensured stable training, and this has led to consistently superior results in downstream tasks. We will discuss these considerations in the manuscript to provide a more comprehensive understanding of the limitations and to further explore this direction in future work.\n\n**References**\n\n[1] Yang J, Zheng K, Long S, et al. MOL-AE: Auto-Encoder Based Molecular Representation Learning With 3D Cloze Test Objective[J]. bioRxiv, 2024: 2024.04. 13.589331.  \n[2] Ni Y, Feng S, Ma W Y, et al. Sliced Denoising: A Physics-Informed Molecular Pre-Training Method[J]. arXiv preprint arXiv:2311.02124, 2023.  \n[3] Feng, Shikun, et al. \"UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning.\" arXiv preprint arXiv:2405.10343 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqFwHKKLvC", "reviewer": "Reviewer_XhCf", "summary": "In this work, the authors propose Uni-Mol2 , a molecular pretraining model that leverages a two-track transformer to integrate features at the atomic level, graph level, and geometry structure level. The authors also investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources.", "questions": "Can the authors more clearly describe (1) the contributions of their work, and (2) their design choice behind considering an LLM model as opposed to GNN model. Why not also consider a combination of both? \n\nWhat is the time complexity of training time of the LLM model. Also, how many GPU resources were used?", "limitations": "The work sufficiently describes the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The work investigates an important problem in chemistry and machine learning, and proposes a useful LLM model. The figures are very informative, especially Fig. 2 on the architectural pipeline.", "weaknesses": "In the Related Work, it would also be relevant if the authors can discuss graph neural network (GNN) models that have recently been used to aid in improved representation learning for molecular graphs e.g., YieldGNN. Furthermore, GNNs are significantly more computationally efficient as compared to LLM-based models. \n\nThis paper provides a survey on molecular representation learning, and seems to be a relevant recent reference: \n\nZhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, Yihong Ma, Olaf Wiest, Xiangliang Zhang, Wei Wang, Chuxu Zhang, and Nitesh V. Chawla. 2023. Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 744, 6638–6646. https://doi.org/10.24963/ijcai.2023/744", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer XhCf for the detailed and constructive review. Below, we respond to all your comments.\n\nWe appreciate the opportunity to clarify a key point at the outset: typically, when referring to LLM models, the term denotes large language models that primarily aim at next-token prediction within the natural language processing domain [1][2]. However, our manuscript builds upon a pre-trained model specifically tailored for the small molecule domain. Fundamentally, our model is a two-track transformer aimed at masked token prediction and coordinate denoising. Our research focuses on examining the scaling laws relevant to this category of molecular pre-training models. Hence, we believe that a more appropriate comparison would be between transformer-based models and GNN-based models within the small molecule domain.\n\n**Response to Questions**\n\n1. **Contributions, Design Choice of Uni-Mol2 and Comparision with GNN**  \n   Our research's main contribution is to investigate the scaling laws relevant to this category of molecular pre-training models. Our results reveal power-law relationships between validation loss and model size, dataset size, and computational resources. Additionally, we observe consistent improvements in downstream tasks as the model size increases.\n\n   We recognize the advantages of GNNs, particularly their efficiency in processing molecular graph structures and their strong capability to capture local relationships within a molecule. However, the locally connected graph fails to adequately represent long-range interactions between atoms. These long-range interactions are crucial in molecular representation learning (MRL). In contrast, transformer-based models have shown exceptional performance in various tasks within the molecular domain, demonstrating remarkable representation capabilities. Some researchers even view transformers as a form of fully-connected GNN [3]. Additionally, recent advancements in transformer engineering optimization have further enhanced their effectiveness [7].\n\n2. **Time Complexity and GPU Resources**  \n   The time complexity of training the Uni-Mol2 model primarily depends on the number of parameters, dataset size, and computational resources. We provide the details of the training process, including hardware specifications, training duration, and computational resources used in the general rebuttal. We will subsequently integrate this information into the manuscript.\n\n**Response to Weaknesses**\n\n1. **Discussion of Graph Neural Network (GNN) Models**  \n   We appreciate the suggestion to include a discussion on Graph Neural Network (GNN) models, such as YieldGNN [9], in the Related Work section. Specifically, we will reference the survey on graph-based molecular representation learning by Guo et al. (2023) [8] and highlight the advantages of GNNs, particularly in terms of representation power and computational efficiency. We believe that including this related work will significantly enhance the completeness and context of our manuscript.\n\n\n**References** \n\n   [1] Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.  \n   [2] Reid M, Savinov N, Teplyashin D, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context[J]. arXiv preprint arXiv:2403.05530, 2024.  \n   [3] Min E, Chen R, Bian Y, et al. Transformer for graphs: An overview from architecture perspective[J]. arXiv preprint arXiv:2202.08455, 2022.  \n   [4] Zhou, Gengmo, et al. \"Uni-mol: A universal 3d molecular representation learning framework.\" (2023).  \n   [5] Yu, Q., Zhang, Y., Ni, Y., Feng, S., Lan, Y., Zhou, H., and Liu, J. Unified molecular modeling via modality blending. arXiv preprint arXiv:2307.06235, 2023.  \n   [6] Luo S, Chen T, Xu Y, et al. One transformer can understand both 2d & 3d molecular data[C]//The Eleventh International Conference on Learning Representations. 2022.  \n   [7] Dao T, Fu D, Ermon S, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness[J]. Advances in Neural Information Processing Systems, 2022, 35: 16344-16359.  \n  [8] Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, Yihong Ma, Olaf Wiest, Xiangliang Zhang, Wei Wang, Chuxu Zhang, and Nitesh V. Chawla. 2023. Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 744, 6638–6646. https://doi.org/10.24963/ijcai.2023/744  \n[9] Shi R, Yu G, Huo X, et al. Prediction of chemical reaction yields with large-scale multi-view pre-training[J]. Journal of Cheminformatics, 2024, 16(1): 22.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJ7xIBCTl1", "reviewer": "Reviewer_ER2p", "summary": "# Summary\n\nThis paper presents Uni - Mol2, a molecular pretraining model, and systematically investigates the scaling law within molecular pretraining models.\n\n# Contributions\n\n1. The largest dataset for molecular pretraining: Curated a dataset of approximately 884 million 3D conformations for pretraining.\n2. Scaling law: It is the first time to demonstrate the scaling law of molecular pretraining and its impact on downstream task performance.\n3.  Significant improvement: Uni - Mol2 is the SOTA model and demonstrates consistent improvement in downstream task performance with increasing model parameters.", "questions": "1. What is the computational cost and time complexity of training the Uni - Mol2 model with different parameters on the large dataset?\n2. Could you please provide more details on how the temperature - based sampling method affects the balance of the training dataset and the performance of the model?", "limitations": "See the weakness and questions plz.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Presents Uni - Mol2, a novel molecular pretraining model that leverages a two - track transformer to integrate features at multiple levels.\n2. Conducts the first exploration of the scaling law in molecular pretraining models.\n3. Curates a large dataset of approximately 884 million 3D conformations, providing a solid foundation for training large - scale molecular models.", "weaknesses": "Limited details of pretraining hyper-parameters: The paper primarily focuses on analyzing the power-law relationships, but does not give details of the pretraining hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere thanks to Reviewer ER2p for the positive evaluation and the thoughtful time invested in reviewing our manuscript. Your encouraging feedback is greatly appreciated, and we have carefully addressed each of your comments in the detailed responses provided below.\n\n**Response to Weaknesses**\n\n1. **Limited Details of Pretraining Hyper-Parameters**  \n   Thank you for pointing out the lack of detailed information regarding the pretraining hyper-parameters in the current manuscript. We have outlined the details of the pre-training hyperparameters in Section 3.3 of the paper, which includes comprehensive details on the hyper-parameters used, including learning rates, batch sizes, number of epochs, optimizer settings, and regularization techniques. If there are specific details or additional aspects you would like us to clarify, please let us know, and we would be happy to discuss them further.\n\n**Response to Questions**\n\n1. **Computational Cost and Time Complexity**  \n   The time complexity of training the Uni-Mol2 model primarily depends on the number of parameters, dataset size, and computational resources. We provide the details of the training process, including hardware specifications, training duration, and computational resources used in the general rebuttal. We will subsequently integrate this information into the manuscript.\n\n2. **Impact of Temperature-Based Sampling**  \n   In our current study, we faced a significant imbalance in the skeletal clustering results of the data. Specifically, the first two categories had a disproportionately high proportion of molecules, while the subsequent categories had much lower proportions. Our dataset contains 73 million scaffolds, with the top two scaffolds alone accounting for 2.78% of the total number of molecules. In contrast, the last 25 million scaffolds together represent only 2.8% of the molecules. To address this issue and ensure that rare scaffolds can be sampled, we referred to some sampling techniques [1][2] used in language models. We adopted a temperature-based sampling approach that reduces the sampling probability of selecting the top two scaffolds by a factor of 19,000, which in turn increases the chances of sampling rarer scaffolds.\n\n   However, due to current computational resource constraints, we have not been able to conduct extensive ablation studies on different sampling strategies. Nevertheless, based on recent advancements in Large Language Models (LLMs) [3][4][5], it is evident that various data sampling methods and the mixture of data types present a promising direction for future research. We appreciate your suggestion and consider it a valuable area for further investigation.\n\n**References** \n\n\n   [1] Wang X, Tsvetkov Y, Neubig G. Balancing training for multilingual neural machine translation[J]. arXiv preprint arXiv:2004.06748, 2020.  \n   [2] Fan A, Bhosale S, Schwenk H, et al. Beyond english-centric multilingual machine translation[J]. Journal of Machine Learning Research, 2021, 22(107): 1-48.  \n   [3] Shao Y, Li L, Fei Z, et al. Balanced Data Sampling for Language Model Training with Clustering[J]. arXiv preprint arXiv:2402.14526, 2024.  \n   [4] Ye J, Liu P, Sun T, et al. Data mixing laws: Optimizing data mixtures by predicting language modeling performance[J]. arXiv preprint arXiv:2403.16952, 2024.  \n   [5] Gu J, Yang Z, Ding C, et al. CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models[J]. arXiv preprint arXiv:2407.17467, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Rebuttal
(R IDs: R1=V7dw, R2=buoE, R3=XhCf, R4=ER2p)

We thank the reviewers for the detailed and helpful reviews. Next, we address the main concerns from reviewers.

1. **Time Complexity and GPU Resources**  
  
We utilized a computational cluster comprising 64 NVIDIA A100 GPUs, each equipped with 80GB of HBM2 memory. The GPUs were interconnected via a high-speed Nvidia Infiniband fabric, offering 400 Gbps bandwidth for inter-GPU communication. The details of each model size are listed as follows.

| Params | Compute Resouce(GPUs) | Training Time(GPU hours) |
|-----------------|---------------|-----------|
| 84M           | 32        |  2585.6 |
| 164M         | 32        | 5120 |
| 310M         | 32        | 7680      |
| 510M         | 64        | 13824  |
| 1.1 B          | 64        | 30720     |

2. **Performance Improvement Concern**  
   
Similar to other works on scaling laws [1][2], our paper explores the power-law relationship between the validation loss of pre-trained models in the molecular domain and the scales of data, model, and computation. This represents the first validation at the scale of billions of data points and pre-trained model parameters. 


   On the other hand, while our proposed Uni-Mol2 consistently outperforms other baseline models across various downstream tasks, the performance on diversified downstream tasks is influenced by several factors such as data partitioning, data quality, and label noise levels [3][4][5]. Consequently, in some downstream tasks, there isn't always a substantial performance improvement as the model scale increases because the data quality and quantity may exert a more significant impact on the final results compared to the model scale. A typical example is the aEA and aIP tasks in the COMPAS-1D dataset, where adding features to Uni-Mol2 1.1B leads to significant performance improvements, achieving an average 14% improvement compared with Uni-Mol.


Regarding the differences in model performance on the QM9 dataset that you mentioned, we have supplemented four additional property prediction tasks (U0, U, G, and H) in the supplementary material. Overall, when comparing Uni-Mol2 310M with Uni-Mol2 1.1B on the QM9 dataset, eight of ten tasks showed improvements exceeding 2.7% with Uni-Mol2 1.1B, while two tasks achieved improvements above 26%. It is worth noting that in the fine-tuning process for QM9, we only used atomic features and conformational features. We believe this is one of the potential reasons contributing to the observed convergence in model performance during scaling up Uni-Mol2.


In summary, our experiments affirm that scaling models is effective and leads to noticeable improvements. The results clearly demonstrate that increasing model size yields significant performance gains in downstream tasks.

**References**

[1] Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint arXiv:2001.08361, 2020.  
[2] Hoffmann J, Borgeaud S, Mensch A, et al. Training compute-optimal large language models[J]. arXiv preprint arXiv:2203.15556, 2022.  
[3] Sultan A, Sieg J, Mathea M, et al. Transformers for molecular property prediction: Lessons learned from the past five years[J]. arXiv preprint arXiv:2404.03969, 2024.  
[4] Deng J, Yang Z, Wang H, et al. Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study[J]. arXiv preprint arXiv:2209.13492, 2022.  
[5] Martinez-Mayorga K, Rosas-Jiménez J G, Gonzalez-Ponce K, et al. The pursuit of accurate predictive models of the bioactivity of small molecules[J]. Chemical Science, 2024, 15(6): 1938-1952.

Meta-review: This paper seeks to evaluate the effectiveness of scaling up pretraining data and parameter size in generative models of molecular structure and to establish initial scaling laws analogous to those already established for generative models of natural language. The paper trains the largest generative molecular model to date and, in experiments, demonstrates improved performance on downstream tasks as pretraining data and parameter scale are increased. 

The majority of reviewers found the contributions of this paper to be valuable, specifically: (1) the sheer scale of these experiments, including the additional pretraining data collected, (2) the establishing of new scaling laws that are interesting in how they differ from natural language scaling laws, and (3) the consistent performance improvements on downstream tasks.

Concerns were raised about (1) the breadth and depth of downstream tasks for evaluation, (2) missing training details like training time and effect of hyperparameters, and (3) missing comparisons with GNN-based modeling approaches. These concerns were all addressed adequately in rebuttal.
Decision: Accept (poster)