NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation
Authors: Zhiyuan Liu
Yanchen Luo
Han Huang
Enzhi Zhang
Sihang Li
Junfeng Fang
Yaorui Shi
Xiang Wang
Kenji Kawaguchi
Tat-Seng Chua
Venue: ICLR 2025 Poster (2025)
Keywords: 3D molecule generation
molecular conformer generation
large language models
diffusion models
geometric deep learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We develop a foundation model that combines the strengths of 1D language modeling and 3D diffusion for 3D molecule generation.

Abstract: 3D molecule generation is crucial for drug discovery and material design. While prior efforts focus on 3D diffusion models for their benefits in modeling continuous 3D conformers, they overlook the advantages of 1D SELFIES-based Language Models (LMs), which can generate 100\% valid molecules and leverage the billion-scale 1D molecule datasets. To combine these advantages for 3D molecule generation, we propose a foundation model -- NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation. NExT-Mol uses an extensively pretrained molecule LM for 1D molecule generation, and subsequently predicts the generated molecule's 3D conformers with a 3D diffusion model. We enhance NExT-Mol's performance by scaling up the LM's model size, refining the diffusion neural architecture, and applying 1D to 3D transfer learning. Notably, our 1D molecule LM significantly outperforms baselines in distributional similarity while ensuring validity, and our 3D diffusion model achieves leading performances in conformer prediction. Given these improvements in 1D and 3D modeling, NExT-Mol achieves a 26\% relative improvement in 3D FCD for de novo 3D generation on GEOM-DRUGS, and a 13\% average relative gain for conditional 3D generation on QM9-2014. Our codes and pretrained checkpoints are available at https://github.com/acharkq/NExT-Mol.

Published as a conference paper at ICLR 2025
NEXT-MOL: 3D DIFFUSION MEETS ID LANGUAGE MODELING FOR 3D MOLECULE GENERATION
Zhiyuan Liu', Yanchen Luo Han Huang², Enzhi Zhang", Sihang Li², Junfeng Fang", Yaorui Shi?, Xiang Wang Kenji Kawaguchi', Tat-Seng Chua! National University of Singapore,
University of Science and Technology of China, Chinese University of Hong Kong. *Hokkaido University zhi ahiyuan@nus .eu.euuug....... sg, Luoyanchenêmail. www.ll..ttcee....... edu.
ABSTRACT
3D molecule generation is crucial for drug discovery and material design. While prior efforts focus on 3D diffusion models for their benefits modeling contin- uous 3D conformers, they overlook the advantages of ID SELFIES- -ased Lan- guage Models (LMs), which can generate 100% valid molecules and leverage the billion-scale 1D molecule datasets. To combine these advantages for 3D molecule generation, we propose a foundation model - NEXT-Mol: 3D Diffusion Meets ID Language Modeling for 3D Molecule Generation. NExT-Mol uses an tensively pretrained molccule LM for ID molccule generation, and subsequently predicts the generated molecule's 3D conformers with 3D diffusion model. We enhance NEXT-Mol's performance by scaling up the LM's model size, refining the diffusion neural architecture, and applying ID to 3D transfer learning. No- tably, our ID molecule LM significantly outperforms baselines in distributional similarity while ensuring validity, and our 3D diffusion model achieves leading performances in conformer prediction. Given these improvements in ID and 3D modeling. NExT-Mol achieves 26% relative improvement in 3D FCD for de novo 3D generation on GEOM-DRUGS, and a 13% average relative gain for con- ditional 3D generation on QM9-2014. Our codes and pretrained checkpoints are available https: com/ awwaakchaaaagaaaa Baeaakkg//Ee--taa /////NEEXXM.../ -Mol.
INTRODUCTION
Molecule discovery is crucial for designing new drugs and materials. To efficiently navigate the astronomical chemical space of molecules, generative deep learning methods have been extensively explored. While promising progress has been made in generating 2D molecular graphs (Jin al., 2018; Vignac et al., 2023a), recent research has shifted toward 3D molecule generation due to its broader application scope. For example, understanding the 3D molecular geometry crucial for structure-based drug design (Zhang et al., 2023), prediction of molecular quantum chemical proper- ties (Zhou et al., 2023), and molecular dynamic simulation (Hansson et al., 2002).
3D molecule generation aims to predict 3D molecular conformers along with their 2D graphs (Hoogeboom et al., 2022). These generated 3D molecules are typically evaluated based on their molecular validity and stability, ensuring adherence the chemical valency rules. Recent advancements in 3D diffusion models (Vignac et al., 2023b; Hua et al., 2023; Huang et al., 2024) have improved these metrics by better modeling continuous 3D conformers, yet they still occasion- ally generate invalid molecules. This validity issue hinders distribution learning of valid molecular structures, like pharmacophoric functional groups. For improvement, we draw inspiration from ID molecule generation (Fang et al., 2024b; Polykovskiy et al., 2020) studies, which reliably ensure 100% validity. By representing 2D molecular graphs as linear strings of SELFIES (Krenn et al., 2020), these approaches typically leverage 1D language models (LMs) for 2D molecule generation. Due to SELFIES' inherent robustness, the generated molecules are guaranteed to be 100% valid. Inspired by these studies, natural solution for improving 3D molecule generation is to incorporate a ID SELFIES-based LM into a 3D diffusion model (Jing et al., 2022), thus leveraging the chemical
"qqual contribution.
Correspondence Xiang Wang. xiangwang| /223mnggg22@@...m @2.3@t 223:22mai..coom gmail.com.

---

Published as a conference paper at ICLR 2025
Billin-sccle Sequence Dataset
NExT-Mol
Task: 1D Molocule Generation
S USinS) an di Popc C3R [0
MoLLama
100% Volidity FILFIES:
Fline-tune
[G]C]]-Bessh ([G]--100)
Pretrain
Pretrained Weight
Generate
LoRA Weight
ZINC15, ZINC22
Task: 3D Conformer Prediction
Physics-computed 3D Conformer Dataset
Regresenttion
Cross-modal
Projector
Generate
Train
Diffuslon
Molecule
Transformer
Task: Molecule Generation
GEOM-DRUGS, QM9
Figure 1:
Overview of our NExT-Mol foundation model for 3D molecule generation. NExT-Mol consists of three key components: (1) MoLlama, large LM for generating ID molecule sequences; (2) DMT, a diffusion model to predict 3D conformers from the ID sequences; and (3) NExT-Mol leverages transfer learning enhance DMT's 3D prediction with MoLlama's 1D representations.
validity of 1D representations while improving 3D conformer prediction. To our best knowledge, few prior research has thoroughly explored this incorporation for 3D molecule generation.
To bridge the research gap above, we explore a two-step solution for 3D molecule generation: ini- tially generating a ID molecule (a subset a 3D molecule) using an LM and subsequently predict- ing its 3D conformer with diffusion model. Here we focus on three key strategies scaling up ID molecular LMs, refining the architecture of 3D diffusion models, and utilizing transfer learning between ID and 3D modeling to resolve the following three challenges faced by prior studies:
The Development of An Effective 1D Molecular LM. This can be done by training an autore- gressive transformer LM (Vaswani et al., 2017) on large SELFIES corpus. However, existing studies have the following limitations: some use noneauvoreeressivee pretraining, rendering them unsuitable for de novo generation (Fang et al., 2024b; Irwin et al., 2022: Born & Manica, 2023; Yüksel al., 2023); some do not have 100% validity (Bagal et al., 2021); and others are con- strained by small model sizes and employ non-transformer architectures, limiting their scalabil- ity (Polykovskiy et al., 2020; Eckmann al., 2022; Arús-Pous et al., 2019; Jin et al., 2018).
The Design of A Powerful 3D Diffusion Model. This is to accurately generate the 3D conformers for the ID molecules generated by the ID molecular LM in the earlier step. Existing works can be improved by adopting scalable architectures (Jing et al., 2022: Corso et al., 2024; Xu et al., 2022; Ganea et al., 2021) or leveraging the full information of 2D molecular graphs (Wang et al., 2024).
Transfer Learning between 1D Molecule Sequences and 3D Conformers. It has the potential to offer a significant improvement to 3D conformer prediction, given the greater availability of ID sequences compared to high-accuracy 3D conformers, which are typically derived by expensive physics-based computations. For example, 7INC22 (Tingle et al., 2023) now includes over 54.9 billion 1D sequences and GEOM (Axelrod & G.omez-Bombared
2022) holds only 37 million 3D conformers. Although this 1D to 3D transfer learning was successfully applied to 3D) protein structure prediction (Lin et al., 2023; Wu et al., 2022), similar methods remain mostly unexplored for small molecules, indicating significant research opportunity.
To address the challenges above, we propose a foundation model - NExT-Mol: 3D Diffusion Meets ID Language Modeling for 3D Molecule Generation, as illustrated in Figure NExT-Mol con- sists of three key components: (1) To achieve effective autoregressive ID molecule generation, we pretrain a Molecular Llama LM (MoLlama) (Touvron et al., 2023; Zhang et al., 2024) on large collection of 1.8B SELFIES sequences. This extensive pretraining empowers MooLaam to effec- tively capture the desired ID/2D molecular patterns (e.g., scaffolds and fragments) downstream datasets, laying strong foundation for the subsequent 3D conformer prediction. (2) To achieve high-accuracy 3D conformer prediction, we introduce a novel diffusion model Diffusion Molecular Transformer (DMT). DMT combines the power of scalable neural architecture (Wang et al., 2024) and retains the full information of 2D molecular graphs by incorporating the Relational Multi-Head Self- Self-Attention (Huang al., 2024) that extends the standard self-attention by incorporating pair in-
2

---

Published as a conference paper at ICLR 2025
formation describing atomic interactions. We show that DMT achieves leading performance for 3D conformer prediction, surpassing prior works by 1.1% COV-R on GEOM- -UDGS Further, it accu- rately reveals the 3D structures of MoLlama-geneeeee in 3D FCD and significant improvements in geometric similarity and stability on GEOM-DRUGS. (3) We show that transfer learning between 1D molecular sequences and 3D conformers improves conformer prediction by 1.3% COV-R on GEOM-DRUGH This improvement is driven by transfer- ing MoLlama's is pretrained ID representations, which encode rich molecular knowledge, to DMT for better molecular representation. The ID-to-3D modality gap in transfer learning is bridged by our proposed cross-modal projector and the corresponding training strategy (Liu et al., 2024a).
ID molecules, providing 26% relative gain
Collectively, our NExT-Mol foundation model is a versatile multi-task learner, and demonstrates leading performances for de novo 3D molecule generation, conditional 3D molecule generation, and 3D conformer prediction on the GEOM-DRUGS, GEOM-QM9 (Axelrod & li 2022) and QM9-2014 (Ramakrishnan et al., 2014) datasets. The strong performance highlights NEXT-Mol's 's effectiveness and its potential impact foundation model in the field. We further present extensive ablation studies to demonstrate the significance of each component NEXT-Mol.
RELATED WORKS
A complete molecule includes atoms, bonds, and the 3D coordinates of atoms (i.e,, 3D conformer). 3D conformers (Axelrod & Honez-Bombbarlli, 2022), many studies focus on generating atoms and bonds without 3D conform- ers, representing molecules as ID sequences or 2D graphs. Here we begin by reviewing 1D and 2D) molecule generation, then discuss 3D molecule generation and 3D conformer prediction.
However, due to the expensive computation for obtaining
ID and 2D Molecule Generation aims to generate the atoms and bonds of a molecule. ID gener- ation works are mostly based on LMs. However, they usually apply -on-autoregressvve pretraining such as span-prediction (Irwin et al., 2022; Fang et al., 2024b; Born & Manica, 2023), making them unsuitable for de novo generation. Other works use non-transformer architecture (Arás-Pous et al., 2019; Polykovskiy et al., 2020; Flam- Shen-hh et al., 2022: Bombr-Bomaarrlli et al., 2018; Eck- mann et al., 2022; Popova et al., 2018), which are unsuitable for scale-up (Vaswani et al., 2017). 2D molecule generation works typically decompose molecular graphs as functional fragments (or atoms), and train models recurrently generate or edit these fragments (Jin et al., 2018; Xie et al., 2021; Luo et al., 2021; Shi et al., 2020; Sun et al., 2022: Liu et al., 2018; You et al., 2018; Popova et al., 2019; Jin et al., 2019). However, due their non-transformer architectures and domain- specialized training methods, these 2D generation models also face challenges with scalability and transfer learning. We refer readers to (Du et al., 2022) for a comprehensive survey in this area.
3D Molecule Generation is dominated by diffusion models (Hoogeboom et al., 2022; Bao et al., 2023; Huang et al., 2023a; 2024; 2023b; Vignac et al., 2023b; Hua et al., 2023). While autoregres- sive methods have been explored (Gebauer et al., 2019; 2022; Luo & Ji, 2022; Simm et al., 2020), they underperform diffusion models, potentially due their inability to model bonds and the error accumulation when autoregressively generating 3D coordinates. Diffusion models typically employ 3D equivariant neural networks (Satorras al., 2021) to denoise the variables of atoms, bonds, and 3D coordinates within single diffusion process. However, they predict molecules without validity constraints and are limited by insufficient 3D data. To address these issues, we aim to integrate the two advantages of ID SELFIES sequences- 100% validity and the more abundant dataset (Sterling & Irwin, 2015; Tingle et al., 2023) into 3D molecule generation for improvement.
3D Conformer Prediction is to predict the 3D conformer given the atoms and bonds of molecule (Xu et al., 2022; Ganea et al., 2021; Zhou et al., 2023; Jing et al., 2022; Corso et al., 2024). The current state-of-the-art approach scales up diffusion model using a general-purpose transformer architecture (Wang et al., 2024), but overlooks the chemical bond information and uses lossy representation of molccular structures. We address these issues by introducing the DMT architecture that maintains scalability and retains the full information of 2D molecular graphs.
3D DIFFUSION MEETS ID LM FOR 3D MOLECULE GENERATION
NEXT-Mol for 3D Molecule Generation. NExT-Mol is a foundation model that generates 3D molecules with two-step method: initially generating the 1D molecule sequence (a subset of 3DD
3

---

Published as a conference paper at ICLR 2025
Original 3D Conformation
Jowe Pore
Alve
Alom Representationa
Norm
Glil
Scale
MiD
Scare
Condition
Teaushop Champng phopp
Motscule Transformer (DMT)
Home-iite Curput
Clausaian rolse
Gaussian
Far 4iee
MPP
Shail
Conformation
Representations
N lyyers
(e) DMT's diflusion process.
(b) Delailed view of each DMT layer.
Figure 2: Overview of DMT's neural architecture. (a) DMT is diffusion model learning denoise random Gaussian perturbations e applied on the 3D coordinates of atoms. (b) DMT relies on the RMHA module to iteratively update atom representations H and pair representations E.
molecule) using the MoLlama LM and subsequently predicting its 3D conformer using the DMT diffusion model. Here we begin by introducing the MoLlama for 1D molecule generation and then proceed to DMT. Finally, we detail the transfer learning method to incorporate MoLlama's ID repre- sentation to enhance DMT's 3D conformer prediction. Appendix C includes implementation details.
3.1 1D MOLECULE GENERATION WITH MOLECULAR LLAMA LM
Data Preparation. Following (Irwin al., 2022), we collect 1.8 billion molecules from the ZINC- 15 database (Sterling & Irwin, 2015), significantly more than the 100 million molecules used in previous studies (Irwin al., 2022; Fang et al., 2024b). We preprocess the molecules to transform them into SELFIES and perform data filtering to avoid overlap with the downstream datasets. The resulting dataset contains billion SELFIES tokens.
Pretraining MoLlama. Our MoLlama is a 960M parameter LM with the popular decoder-oni Llama-2 (Touvron et al., 2023) architecture. We pretrain from scratch for ID molecule generation with the next-token prediction objective. 'The pretraining takes 555K global steps, processing 145 billion tokens, which amounts to approximately 1.6 passes through the pretraining dataset.
Randomized SELFIES Augmentation. We use randomized SELFIES as data augmentations dur- ing fine-tuning MoLlama for ID molecule generation. A molecule can have multiple valid SELFIES, because they are generated by traversing the 2D molecular graph in different orders. Randomized SELFIES are generated by traversing in random orders. This approach improves sample diversity and mitigates overfitting compared to using the canonical traversal order (Arús-Pous et al., 2019). The intuition is that the atoms in a molecule are inherently unordered, therefore an ideal LM should generate different orderings of the molecule with equal likelihood.
3.2 3D CONFORMER PREDICTION WITH DIFFUSION MOLECULAR TRANSFORMER
Here we elaborate on the three key components of our proposed DMT: (1) the diffusion process governing the training and inference; (2) the neural architecture; and (3) the rotation augmentation.
Diffusion Process. A molecule G = (x, h, e) is represented by its 3D coordinates E RR Nxd (e.g., atom types), and pair features e Nx x (e.g., chemical bonds), where N is the number of atoms and d1 and d2 are the feature dimensions. For 3D con- diffusion model (Kingma et al., 2021) that denoises a molecule's 's 3D coordinates x based on its atom and pair features. As Figure 2a shows, in the for- ward diffusion process, noises are gradually applied to the original 3D coordinates x()) = such that ((x(1) (x) = (x) (0) (1 (1-a(")I), where / E (0, is the diffusion's 's time-step, and (1) is hyperparameter controlling the noise scale at the step. Based on the reparameterization trick (Ho al., 2020), we can sample (1) (w) + v1 (1)) where (!!) NN I). Given the perturbed coordinates x(!), DMT is trained to predict the noise e(!) by minimizing the MSE loss L = (t) (MT(G()) .t)ll, where (t)) (x(() h, c). After training. DMT can be employed for 3D conformer prediction through ancestral sampling (Ho et al., 2020).
atom features
former prediction, we use continuous
Neural Architecture. As Figure 2b illustrates, DMT adopts Relational Multi-Head Self-Attention (RMHA) (Huang et al., 2024) and adaptive layernorm (adaLN) (Perez et al., 2018; Peebles & Xie, 2023). adaL.N replaces the learnable scale and shift parameters in standard layernorm (Ba, 2016)
4

---

Published as a conference paper at ICLR 2025
Belemerr
MoLiama
Projector
DNT
Moliona
Cross Medal Projector
Difuutoon
Stage 2: Projector Varmup
Traneformar
Malecula
MoLiaama
Crase-Mood Moddll Projector
DNTT
3: Transfor
1mg
Mol lama
Projector
DMT
(4] Crose-modal projector bridgas leme and DMT
(b) Three training stages for 1D to tranafer karming
Figure 3: Transfer learning between MoLlama's 's ID representations and DMT's 3D prediction. (a) cross-modal projector bridges the gap between MoLlama and DMT. Grey atoms have no cor- responding SELFIES tokens, and are replaced by learnable token. (b) Transfer learning's 's three training stages. Snowflake denotes frozen parameters while flame denotes trainable ones.
with adaptive ones that are generated from the condition embedding C, which combines the time- step and optionally a desired chemical property. For simplicity, we omit adaLNs in discussion below.
The philosophy behind DMT's neural architecture generally follows the "bitter lesson" recently re- vealed by MCF (Wang al., 2024) that large scalable models outperform domain-specifer inductive biases. Notably, MCF shows that it is unnecessary to have an architecture built-in 3D equivari- ance for conformer prediction. However, MCF is limited employing lossy representation of 2D molecular structures and overlooks bond information, by relying on the top-k eigenvectors of the graph Laplacian (Maskey et al., 2022) to represent 2D molecular graphs. To address this issue, DMT retains the full information of 2D molecular graphs in its atom representation H E IR xd and pair representation E E T xd and then applies RMHA learn and distinguish the 2D graph struc- tures. Specifically, the atom representations H are initialized by concatenating the atom features h and the perturbed 3D coordinates x("), the pair representations E are initialized by concatenating the pair features and the distances between each atom pair. H and E are then iteratively refined by RMHA. The single-head RMHA is defined below with the multi-head version in Appendix C.2:
QQ: W W W. W.
(1)
V tanh( W
W E.))
(2)
O O K.
N
0i. = softmax,(
(3)
a. O V2), O, = Σ
(4)
Vd
where @ denotes element-wise product; softmax, denotes softmax along the j dimension: linear projectors Wg, WR. and W, generate queries, keys, and values for atom representations, W og and W ev generate queries and values for pair representations; O is RMHA's output for the i-th atom; and Q V E Fd are the query and value for the atom pair representation (i.j).
RMHA uses the lair- query O and key V of to modify the atom-level query Q and value V, through element- siiment-wise multiplication (O), enabling RMHA fully incorporate pair representa- tions. Specifically, the pair E affects attention scores via (Q Q K, and affects the aggregated attention values via V F @ V,. this way, the output O is adaptively informed by the structural and interaction information in E. After RMHA, O, is passed to MIP to update the atom representa- tion H, and the linear combination of O, and O, used update the pair representation E,,- As Figure 2b illustrates, residual connections and adal.Ns are included for improved performance.
Random Rotation Augmentation. Following AlphaFold3 (Abramson et al., 2024), we apply the same random rotation augmentation on both the input 3D coordinates (x(*) and the target 3D co- ordinates (e()) to help DMT obtain equivariance to rotated inputs by learning. While (Wang al., 2024) report decreased performance given random rotations, DMT benefits from it, potentially due the improved neural architecture.
3.3 MOLLAMA REPRESENTATION IMPROVE DMT's 3D CONFORMER PREDICTION
We explore the transfer learning between molecular ID sequences and 3D conformers. As Figure illustrates, we leverage MoLlama's 's pretrained representation to improve DMT's 3D conformer pre- diction. This is achieved by our cross-modal projector and the corresponding training strategy.
5

---

Published as a conference paper at ICLR 2025
Cross-Modal Projector. Following Liu et al. (2023a). DMT uses a projector to leverage MoLlama for atom representation, addressing two challenges: (1) MoLlama uses causal self--ttention, where each token only perceives preceding tokens, limiting the representation quality; and (2) SELFIES tokens do not map directly to individual atoms. Mitigating the first issue, we feed Mal.aam''s SELF- IES representations into single-layer bi-directional self-attention (Vaswani al., 2017), expanding the receptive field of SELFIES tokens. Further, we program the SELFIES-to-atom mapping using the SELFIES and RDKit software. For atoms corresponding multiple SELFIES tokens, we obtain its representation by mean pooling; for hydrogen atoms without corresponding SELFIES tokens, we use a learnable token as a replacement. The output of the SELFIES ----E--tt--ttom mapping then fed into an MLP and concatenated with DMT's original atom representations for 3D conformer prediction.
Training Strategy. As Figure 3b illustrates, to save computation, we fine-tune a pretrained DMT to incorporate Mol.lama representations, instead of training new DMT from scratch using MoL.ama representations. Throughout the process, Maml uses LoRA tuning (Hu et al., 2021) to save memory. The training strategy consists of three stages. In the first stage, we train standalone DMT without MoLlama until convergence. In the second stage, we attach MoLlama and the cross-modal projector to the pretrained DMT, keeping the DMT parameters frozen, and train for 10 epochs to warmup the random parameters in the projector and LoRA. This step prevents the gradients from the random parameters from distorting the pretrained DMT parameters (Kumar al., 2022). In the final stage, we fine-tune the entire integrated model until convergence.
When incorporating MoLlama representations into DMT, we find that canonical SELFIES performs better than randomized SELFIES. This may be because bridging the gap between ID MoLlama and 3D DMT is challenging, and using the fixed canonical representations leads to faster convergence.
EXPERIMENT
In this section, we evaluate NExT-Mol's performance on de novo 3D molecule generation and con- ditional 3D molecule generation. Further, we report results of 3D conformer prediction, the critical second step our two-step generation process. Finally, we present ablation studies to demonstrate the effectiveness of each component of NExT-Mol.
4.1 EXPERIMENTAL SETTINGS
Datasets. As Table shows, we evaluate on the popular GEOM-DRUGS (Axelrod & Gomez-Bonbarelli, 2022), GEOM-QM9 (Ax-
Table Datasets for each task.
Task
Dataset
elrod & Gomez- Bombarllo 2022), and QM9- 2014 (Ramakrishnan et al., 2014) datasets.
novo 3D Mol Gen
GEOM-DRUGS, QM9-2014 QM9-2014
Conditional 3D Mol Gen
3D Conformer Pred
GEOM-DRUGS, GEOM-QM9
Among them, we focus on GEOM-DRUGS, which is the most pharmaceutically relevant and largest one. Due to different tasks incorporating different dataset splits, we separately fine-tune NExT-Mol for each task without sharing weights.
Baselines. For de novo and conditional 3D molecule genration, we use baselines of CDGS (Huang et al., 2023a), JODO (Huang et al., 2024), MiDi (Vignac al., 2023b), G-SchNet (Gebauer et al., 2019), G-SphereNet (Iuo & Ji, 2022), EDM (Hoogeboom et al., 2022), MDM (Huang et al., 2023b), GeoLDM (Xu et al., 2023), EEGSDE (Bao et al., 2023), EQGAT-diff (Le et al., 2024), MolGPT (Ba- gal et al., 2021), and MolGen (Fang et al., 2024b). For 3D conformer prediction, of OMEGA (Hawkins, 2017), GeoMol (Ganea et al., 2021), GeoDiff (Xu et al., 2022), Torsional Diffusion (Jing et al., 2022), Particle Guidance (Corso et al., 2024), and MCF (Wang et al., 2024). More details on experimental settings are in Appendix D.
use baselines
NEXT-Mol. Throughout the section, NExT-Mol fine-tunes the pretrained 960M MoLlama for 1D molecule generation. We have trained two versions of DMT: DMT-B of 55 million parameters and DMT-L of 150 million. For the de novo and conditional 3D generation molecule tasks (cf. Section 4.2 and Section 4.3), NExT-Mol uses DMT-B. DMT uses 100 sampling steps by default.
4.2 De Novo 3D MOLECULE GENERATION
Experimental Setting, Generating a complete 3D molecule involves generating the 2D molecular graph and the corresponding 3D conformer. Therefore, evaluate both the predicted 2D molecu-

---

Published as a conference paper at ICLR 2025
Table Performances for de novo 3D molecule generation. denotes our reproduced results using their source codes. Other baseline results are borrowed from (Huang et al., 2024). 2D-Metric evaluates the directly predicted 2D molecular graphs, whereas the 3D-Metric evaluates the predicted 3D coordinates or the 2D molecular graphs reconstructed from the 3D coordinates.
(a) Performances on the GEOM-DRUGS dataset.
2D-Metric
FCD4
AtomStable
MolStable
V&C
V&U
V&U&N
SNN
Frag
Scaf
Train
0,251
1.000
1.000
1.000
1.000
0.000
0.585
0.999
0.584
MolGPT
0,888
0.979
0.977
0.957
0.955
0.918
0.520
0.991
0.539
MolGen
0.655
1.000
0.995
1.000
0.993
0.759
0.513
0.993
0.549
CDGS
22.051
0.991
0.706
0.285
0.285
0.285
0.262
0.789
0.022
JODO
2.523
1.000
0.981
0.874
0.905
0,902
0.417
0.993
0.483
MiDi
7.054
0.968
0.818
0.633
0.654
0.652
0.392
0.951
0.196
EQGAT-diff*
6.310
0.999
0.998
0.959
0.993
0.702
0.368
0.986
0.147
NExT-Mol, ours
0.334
1.000
0.999
1.000
0.999
0.945
0.529
0.999
0.552
3D-Metric
FCD4
AtomStable
Bond length
Bond angle]
Dihedral angle.
Train
13.73
0.861
1.56E-04
1.81E-04
1.56E-04
EDM
31.29
0.831
4.29E-01
4,96E-01
1.46E-02
JODO
19.99
0.845
8.49E-02
1.15E-02
6.68E-04
MiDi
23.14
0,750
1.17E-01
9,57E-02
4,46E-03
EQGAT-diff>
25.89
0.846
1.23E-01
5,29E-02
2.17E-03
NExT-Mol,
14.69
0.848
2.05E-02
8.18E-03
2.31E-04
(b) Performances on the QM9-2014 dataset.
2D-Metric
FCD4
AtomStable
MolStable
V&C
V&U
V&U&N
SNN
Frag
Scaf
Train
0.063
0.999
0.988
0,989
0.989
0.000
0.490
0.992
0.946
MolGPT*
0.461
0.982
0.976
0.977
0.937
0.763
0.523
0.958
0.923
MolGen
0.085
1.000
0.988
1.000
0.955
0.479
0.500
0.988
0.934
CDGS
0.798
0.997
0,951
0.951
0.936
0.860*
0.493
0.973
0.784
JODO
0.138
0.999
0.988
0,990
0.960
0.780*
0.522
0.986
0.934
MiDi
0.187
0.998
0.976
0.980
0.954
0.769
0.501
0.979
0.882
EQGAT-diff*
2.157
1.000
0.972
1.000
0.996
0.695
0.479
0.949
0.707
NExT-Mol, ours
0.070
1.000
0.989
1.000
0.967
0.802
0.530
0.992
0.945
3D-Metric
FCD4
AtomStable
Bond length.)
Bond angle.
Dihedral angle..
Train
0.877
0.994
5.44E-04
4.65E-04
1.78E-04
G-SchNet
2.386
0.957
3.62E-01
7.27E-02
4.20E-03
G-SphereNet
6.659
0.672
1.51E-01
3.54E-01
1.29E-02
EDM
1.285
0,986
1.30E-01
1.82E-02
6.64E-04
MDM
4.861
0.992
2.74E-01
6.60E-02
2.39E-02
JODO
0.885
0.992
1.48E-01
1.21E-02
6.29E-04
MiDi*
1.100
0.983
8.96F-01
2,08F-022
8.14F-04
EQGAT-diff*
1.519
0.988
4.09E-01
1.91E-02
1.14E-03
NExT-Mol, ours 0.879
0.993
1.15E-01
7.32F-03
1.95E-04
Table 3: Performance of conditional 3D molecule generation on the QM9-2014 dataset. We report MAE L between the desired properties and the predicted properties of the generated samples. Base- line results are from (Huang et al., 2024). We bold the best performance.
Method
(D)
(Bohr")
C,
K)
EHоMO (meV)
ETAMM (meV)
As (me
L-Bound 0.043
0.09
0.040
39
36
65
EDM
1.123
2.78
1.065
371
601
671
EEGSDE
0.777
2.50
0,941
302
447
487
GeoLDM
1.108
2.37
1.025
340
$22
587
JODO
0.628
1.42
0.581
226
256
335
NExT-Mol, ours
0.507
1.16
0,512
205
235
297
relative improv.
19.3%
18.3%
11.9%
9.3%
8.2%
11.3%
lar graphs (i.e. 2D-Metric), and the predicted 3D coordinates (i.e., 3D-Metric), following (Hooge- boom et al., 2022: Huang et al., 2024). 2D-Metrics can be roughly grouped into three types: (1) stability and validity: atom stability, molecule stability, and validity & completeness (V&C); (2)

---

Published as a conference paper at ICLR 2025
Table 4: 3D conformer prediction results. Baseline results are from (Jing al., 2022; Corso al., 2024; Wang et al., 2024). denotes reproduction using their codes. EecRl and PP-Prrecisin.
(a) Performances on the GEOM- dataset. TD w/ PG denotes torsional diffusion with particle guidance.
COV-R (%)t
AMR-R +
COV-P (%)f
AMR-P
Method
Model Size
Mean
Median
Mean
Median
Mean
Median
Mean
Median
Model size - 100M
OMEGA
53,4
54.6
0.841
0.762
40.5
33.3
0.946
0.854
GeoMol
0.3M
44.6
41.4
0.875
0.834
43.0
36.4
0.928
0.841
GcoDiff
1.6M
42.1
37.8
0.835
0.809
24.9
14.5
1.136
1.090
Torsional Diffusion
1.6M
72.7
80.0
0,582
0.565
55.2
56.9
0.778
0.729
TD w/ PG
16M
77.0
82.6
0.543
0.520
68.9
78.1
0.656
0.594
TD w/ P
16M
73.8
79.3
0.566
0.539
65.2
70.8
0.680
0.615
MCF-S
13M
79.4
87.5
0.512
0.492
57.4
57.6
0.761
0.715
MCF-B
64M
84.0
91.5
0.427
0.402
64.0
66.2
0.667
0.605
DMT-B, ours
55M
85.4
92.2
0.401
0.375
65.2
67.8
0.642
0.577
DMT-B, PC samp.
55M
85,5
91.2
0.396
0.370
67.6
71.5
0.623
0.546
Model size > 100M
MCF-L
242M
84.7
92.2
0.390
0.247
66.8
71.3
0.618
0.530
DMT-L, ours
150M
85.8
92.3
0.375
0.346
67.9
72.5
0.598
0.527
(b) Performances on GEOM QM9 dataset.
COV-R (%)f
AMR-R +
COV-P (%)f
AMR-P +
Method
Model size
Mean
Median
Mean
Median
Mean
Median
Mean
Median
OMEGA
85.5
100.0
0.177
0.126
82.9
100.0
0.224
0.186
GeoMol
0.3M
91.5
100.0
0.225
0.193
86.7
100.0
0.270
0.241
GeoDiff
1.6M
76.5
100.0
0.297
0.229
50.0
33.5
0.524
0.510
Torsoinal Diffusion
1.6M
92.8
100.0
0.178
0.147
92.7
100.0
0.221
0.195
MCF-B
95.0
100.0
0.103
0.044
93.7
100.0
0.119
0.055
DMT-B, ours
55M
95.2
100.0
0.090
0.036
93.8
100.0
0.108
0.049
diversity: validity & uniqueness (V&U), and validity & uniqueness & novelty (V&U&N); and (3) distribution similarity between the generated molecules and the test set: similarity to nearest neigh- bor (SNN), fragment similarity (Frag), scaffold similarity (Scaf), and Fréchet ChemNet Distance (FCD) (Polykovskiy et al., 2020). For 3D-Metrics, we follow (Hoogeboom al., 2022) evaluate the predicted 3D molecules by assessing atom stability, and FCD of the 2D molecular graphs re- constructed from predicted 3D coordinates. Additionally, 3D-Metrics includes the maximum mean discrepancy (MMD) (Gretton et al., 2012) for bond lengths, bond angles, and dihedral angles to evaluate geometric similarity to the test set. Training set performance also reported for reference. The experimental results are presented in Table 2. We can observe that:
Obs. 1: NExT-Mol Demonstrates Leading Performances for 3D Molecule Generation.
achieves the best performance across all metrics GEOM-DRUGS, and achieves the best per- formance out of 14 metrics on QM9-2014, Although CDGS shows a higher novelty score on QM9-2014, it significantly underperforms NExT-Mol for other metrics. This observation shows that NExT-Mol highly effective at generating chemically valid and diverse 3D molecular structures. Its strong performance both large (i.e., GEOM-DRUGS) and small (i.e., QM9-2014 molecules highlights its robustness and potential as a foundation model for various tasks.
Obs. 2: NExT-Mol is Powerful in Capturing 1D/2D Molecular Characteristies, including SNN, Frag, Scaf, and FCD. Notably, it improves the FCD from 0.655 to 0.334 on GEOM-DRUGS. achev- ing a 49% relative improvement. This good performance is attributed MoLlama's extensive pre- training, which lays strong foundation for the subsequent 3D conformer prediction.
4.3 CONDITIONAL 3D MOLECULE GENERATION WITH QUANTUM CHEMICAL PROPERTIES
Adatping NEXT-Mol for Conditional Generation. We employ NExT-Mol for conditional 3D molecule generation targeting quantum chemistry properties. To adapt NExT-Mol incorporate numerical conditions, the desired property values are encoded into vector embeddings using MLPs. These embeddings prepended to the SELFIES sequences during MoL. fine-tuning, serving
8

---

Published as a conference paper at ICLR 2025
Table Incorporating MoLlama's ID representations to improve DMT's 3D conformer prediction.
COV-R (%)f
AMR-R R 4
COV-P (%)f
AMR-P
Dataset
Method
Mean
Median
Mean
Median
Mean
Median
Mean
Median
DMT-B
85.4
92.2
0,401
0.375
65.2
67.8
0,642
0,577
GEOM-
+MoLlama
86.1
92.1
0.383
0.367
66.2
68.6
0.626
0.566
DRUGS
DMT-L
85.8
92.3
0.375
0.346
67.9
72.5
0.598
0,527
+MoLLama
87.1
93.0
0.360
0.334
68.1
71.8
0.595
0.525
Table 6: 3D conformer prediction performance on GEOM-DRUGS's test subsets, split by scaffold frequency in the training set. 68 low-quality sam- plcs are filtered following (Jing ct al., 2022).
(a) Case 1. to R: GT, DMT, Lama
Test subset
#Mol
Method
AMR-R
AMR-P
DMT-B
0,450
0.785
unseen scaffold
348
+MoL.lam
0.422
0.755
(b) Case 2. to R: GT, DMT, DMT+MoLlama.
DMT-B
0.364
0.549
scaf. freq. 21
584
+MoLlama
0.359
0.548
Figure 4: Visualization of 3D conformers.
DMT-B
0.348
0.515
We select the predicted conformers with the least RMSD to the ground truth (GT).
scaf. freq. 2 10
285
+MoLlama
0.347
0.513
as a soft-prompt to condition its output (Li & Liang, 2021), and are also fed into the DMT through the condition MLP module (cf. Figure 2). See Appendix D.4 for details of this methodology.
Remark. Quantum chemical properties (e.g.. HOMO-LUMO gap) often vary across a molecule's different 3D conformers. As result, the ID molecules generated by MoLlama alone cannot achieve errors lower than the average across molecule's 's different conformers. To address this, we condition DMT on the desired property value when predicting the 3D conformer, enabling DMT to find the conformer that best matches the target property.
Experimental Settings. Following (Hoogeboom et al., 2022; Huang et al., 2024), we focus on six properties of heat capacity C.,, dipole moment μt, polarizability a, highest occupied molecular orbital energy <HoMO, lowest unoccupied molecular orbital energy fLUMO, and HOMÓ-LUMO gap Ac. For evaluation, report the mean absolute error (MAE) between the desired property values and the predicted values of the generated molecules, using a property classifier network &e (Hoogeboom et al., 2022). QM9-2014's training set is split into two halves: Da and Db. each containing 50k molecules. de is trained on D, and NExT-Mol is trained on D3- We report Φ performance on D; as the performance's lower-bound (L-Bound). Table 3 shows the results.
Obs. 3: NExT-Mol Outperforms Baselines for Conditional 3D Molecule Generation. NExT- Mol's improvements are consistent and significant, with average relative gain of 13% MAE, demonstrating its ability to effectively capture quantum chemical properties. This good performance partially attributed DMT, which finds the 3D conformer that best aligns the desired property.
4.4 3D MOLECULAR CONFORMER PREDICTION
Experimental Setting. Our setting follows (Jing et al., 2022). Evaluation metrics include Aver- age Minimum RMSD (AMR), which measures the distance between a predicted conformer and ground truth, and Coverage (COV), which measures the proportion of predicted conformers that are sufficiently close ground truth. Due to 2D molecule can have multiple ground truth and predicted conformers, we report both precision (comparing a prediction to its most similar ground truth) and recall (comparing a ground truth to its most similar prediction) for AMR and Coverage. For DMT-B, we report performance with the predictor-correcte sampler (PC samp.; Song et al. (2021)), comparing with the particle guidance sampler (TD w/ PG).
Obs. 4: DMT Demonstrates Leading Performance for 3D Conformer Prediction. Table 4 com- pares DMT and baselines for 3D conformer prediction. We can observe that DMT-B outperforms MCF-B, and DMT-L surpasses MCF-L, even though DMT-L is only 60% of the size of MCF-L. This improvement demonstrates that DMT can better utilize 2D molecular graph structures than MCF. Further, DMT-L. improves upon DMT-B, demonstrating DMT's scalability. Both the improvements

---

Published as a conference paper at ICLR 2025
Table Enhancing 3D molecule generation with MoLlama representations on GEOM-DRUGS,
Method
3D Pred.
FCD + AtomStable Bond length
Bond anglel
Dihedral angle.
DMT-B
14.69
0.848 0.852
2.05B-02 1.48E-02
8.18 8.18E-03 8.08E-03
E.31E-04 1.81E-04
NExT-Mol
+MoLLama
14.32
Table 8: Ablating randomized SELFIES augmentations for 1D molecule generation on QM9-2014.
2D metrics
FCD4
AtomStable MolStable
V&C
V&U V&U&N
SNN
Frag
Scaf
Mom.aama
0.070
1.000
0.989
1.000
0.967
0.802
0.530
0.992 0.945
w/o randomized aug.
0.074
1.000
0.988
1.000
0.948
0.395
0.491
0.989 0.939
above are attributed to DMT's meticulously designed architecture, combining the power of scalabil- ity while effectively leveraging the full information of 2D molecular graphs.
Obs. 5: MoLlama's 1D Representation Improves DMT's 3D Conformer Prediction. As Ta- ble 5 shows, MoLlama enhances DMT on GEOM-DRUGS. Table 12 shows integrating MoLlama into DMT also improves performance on GEOM-QM9. This observation demonstrates the potential to leverage the abundant ID) molecule sequences to improve 3D generation and design tasks, mit- igating their data scarcity issue. Further, this observation highlights MoLlama's value to generate expressive molecule representations for 3D tasks, beyond its ID molecule generation ability. Al- though MoL.aam is pretrained only on ID molecules, we hypothesize that large-scale pretraining helps it develop chemical heuristics useful for 3D prediction.
4.5 ANALYSIS AND ABLATION STUDIES
MoLlama's ID Representation Improves 3D Prediction for Unseen Scaffolds. Scaffold split is widely used to evaluate a molecular model's generalization ability to unseen structures (Hu et al., 2020). We divide GEOM-DRUGS's test set into subsets based on the test molecule's scaffold frequency in the training set. As Table 6 shows, DMT--'ss performance drops significantly for molecules with unseen scaffolds: AMR-R and AMR-P increase by 0.086 and 0.236, respectively, compared to molecules with scaffold frequency 2 1. However, incorporating MoI.aam mitigates this issue, reducing AMR-R and AMR-P by 0.028 and 0.030, respectively. This improvement stems from MoLlama's 's exposure to diverse scaffolds during pretraining on a large molecular dataset, en- abling better generalization for transfer learning. Figure 4 highlights cases where MoLlama signifi- cantly enhances conformer prediction, particularly by improving torsion angle predictions.
Enhancing 3D Molecule Generation with MoLlama Representations. NExT-Mol uses DMT-B for conformer prediction by default for de nove 3D molecule generation. Here we show that enhancing DMT-B with Mol lama further improves its performance on 3D-metrics. Ta- ble 7 shows significant gains in geometric measures (i.e., bond lengths, angles, and dihedral angles), highlighting MoLlama's s ability to enhance DMT's 3D geometry prediction.
without
Random SELFIES Augmentation. As Table 8 shows, using randomized SELFIES augmentation significantly improves the novelty (i.e., V&U&N) of the generated samples. It also improves other metrics, like SNN and FCD, highlighting its importance for ID molecule generation.
CONCLUSION AND FUTURE WORKS
In this work, we presented NEXT-Mol, a foundation model for 3D molecule generation that inte- grated the strengths of ID SELFIES basedd LMs and 3D diffusion models. NExT-Mol demonstrated leading performances in de novo 3D molecule generation, 3D conformer prediction, and conditional 3D molecule generation. These good performances are attributed our focus on incorporating chemical inductive biases without compromising model scalability, and they highlight NEXT-Mol's promising potential as a foundation model in the field. Additionally, NExT-Mol showed that trans- fer learning between ID molecule sequences and 3D conformers can significantly improve 3D con- former prediction performance, underscoring the value of leveraging the abundant 1D molecular data to enhance 3D prediction tasks. Looking ahead, plan to extend NExT-Mol to process mul- tiple molecular inputs, aiming to tackle structure-basse molecule design and modeling interactions between small molecules and proteins RNAs, with real-world applications in drug discovery.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our research advances 3D molecule generation with the NExT-Mol model, aiming to enhance gener- ative deep learning methods for molecular design. This work primarily technical and foundational, with applications in drug discovery and materials science. We have carefully considered potential societal impacts and do not foresee any direct, immediate, or negative consequences. We are com- mitted to the ethical dissemination of our findings and encourage their responsible use.
REPRODUCIBILITY STATEMENT
All the results this work are reproducible. We provide the necessary code to replicate our results in anonymous GitHub repository https://githbb.em httggggiw..o..aaaagg//////////// includes environment configurations, run scripts, and other relevant materials.
The repository
We discuss the experimental settings for various tasks in Section 4, including details parameters such as sampling steps. Additionally, detailed experimental settings are provided Appendix D.
ACKNOWLEDGEMENT
This research is supported by the National Natural Science Foundation of China (92270114), This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-24-1-400 I, and this research is partially supported by the Singapore Ministry of Ed- ucation Academic Research Fund Tier 1 (Award No: Tl 251RES2207). This research is supported by NExT Research Center.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models. CoRR, abs/2209.01712. 2022. doi: 10. www.iv..209.000 48550/ARXIV.2200001722 URL https:://ot / wtt.//....5000....0
Josep Arús-Pous, Simon Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized SMILES strings improve the quality molecular generative models. Cheminformatics, 1(())71:11-1:13,, 2019.
Simon Axelrod and Rafael Bombarll Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: molecular generation using a transformer-ecooder model. Journal of Chemical Information and Modeling, 62(9):2064- 2076, 2021.
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided SDE for inverse molecular design. In ICLR. OpenReview.net, 2023.
Jannis Born and Matteo Manica. Regression transformer enables concurrent sequence regression and generation for molecular language modelling, Nat. Mac. Intell., 5(4):432 444, 2023.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In The Tenth International Conference Learning Representattonss, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https:/ http///pppneeeeccce/eeee net net/foruutttteeeeeeeeee forum?ide F
11

---

Published as a conference paper at ICLR 2025
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Hongshuai Wang, Yongge Li, Mujie Lin, et al. Sciassess: Benchmarking Ilm proficiency in scientific literature analysis. arXiv preprint arXiv:2403.01976, 2024.
Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi S. Jaakkola. Particle guidance: non-ii.d. diverse sampling with diffusion models. In ICLR. OpenReview.net, 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference Learning Representations, 2024. URL https:// openreview net/ forum? fotett/oormmmmmm///22/2 2amm7n22Xyyh...
Jacob Devlin, Ming-Way Heig-Weiilo Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACI-HIT, pp. 4171-4186. Associ- ation for Computational Linguistics, 2019.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, pp. 8780-8794, 2021.
Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 2022.
Peter Eckmann, Kunyang Sun, Bo Zhao, Mudong Feng, Michael K. Gilson, and Rose Yu. LIMO: latent inceptionism for targeted molecule generation. ICML, volume 162 of Proceedings of Machine Learning Research, pp. 5777-5792. PMLR, 2022.
Junfeng Fang, Shuai Zhang, Chang Wu, Zhengyi Yang, Zhiyuan Liu, Sihang Li. Kun Wang, Wenjie Du, and Xiang Wang. Molte: Towards molecular relational modeling in language models. In ACL (Findings), pp. 1943-1958. Association for Computational Linguistics, 2024a.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan Hc, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
/popreevieww net wesss//formmmdddde (tte://orrrmm...tSvsvvvv33
Yin Fang, Ningyu Zhang, Zhuo Chen, Lingbing Guo, Xiaohui Fan, and Huajun Chen. Domain- agnostic molecular generation with self--eedbbak.. In ICLR. OpenReview.net, 2024b.
Daniel Flam-Shepherd, Kevin Zhu, and Alán Aspuru-Guzik. Language models can learn complex molecular distributions. Nature Communications, 13(1):3293, 2022.
Gordon L Flynn. Substituent constants for correlation analysis in chemistry and biology. by corwin hansch and albert lco., 1980.
Octavian Ganea, Lagnajit Pattanaik, Connor W. Coley, Regina Barzilay, Klavs F. Jensen, William H. Green Jr., and Tommi S. Jaakkola. Geomol: Torsional geometric generation of molecular 3d conformer ensembles. In NeurIPS, pp. 13757-13769, 2021.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Sympetry-adapted generation 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1):973, 2022.
Rafael Gómez Bombarell,,,,, Jennifer Wei, David Duvenaud, José Miguel Hernández-Lobato Timothy D Hirzel,
Benjamín Sänchez-Lengeling Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Gude Guzik. Automatic chemical design using a data-driven contin- representation of molecules. ACS central science, 4(2):268-276, 2018.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
12

---

Published as a conference paper at ICLR 2025
Tomas Hansson, Chris Oostenbrink, and WilfredF van Gunsteren. Molecular dynamics simulations. Current opinion in structural biology, 12(2):190-196, 2002.
Paul C. D. Hawkins. Conformation generation: The state the art. J. Chem. Inf. Model., 57(8):
1747-1756, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation in 3d. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 8867-8887, PMLR, 2022.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD, pp-
ACM, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. and Weizhu Chen. Lora: Low-rank adaptation large language models. CoRR, abs/2106.09685, 2021.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020.
Chenqing Hua, Sitao Luan, Minkai Xu, Zhitao Ying, Jie Fu, Stefano Ermon, and Doina Precup. Mudiff: Unified diffusion for complete molecule generation. In LoG, volume 231 of Proceedings of Machine Learning Research, pp. 33. PMLR, 2023.
Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Conditional diffuuson based on discrete graph structures for molecular graph generation. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 1302243111 2023a.
Han Huang. Leilei Sun, Bowen Du, and Weifeng Lv. Learning joint 2-d and 3-d graph diffusion models for complete molecule generation. IEEE Transactions Neural Networks and Learning Systems, 2024.
Lei Huang, Hengtong Zhang, Tingyang Xu, and Ka-Chun Wong. MDM: molecular diffusion model for 3d molecule generation. In AAAI, pp. 5105-511111 5112. AAAI Press, 2023b.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: pre- trained transformer for computational chemistry. Mach. Learn. Sei. Technol., 3(1: 2022.
Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 21-22322222222 2237.. PMLR, 2018.
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi S. Jaakkola. Learning multimodal graph- to-graph translation for molecule optimization. In ICLR (Poster). OpenReview.net, 2019.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi S. Jaakkola. Torsional diffusion for molecular conformer generation. In NeurlPS, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 14-21696-21707, 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru-Guzia Self- referencing embedded strings (SELFIES): A 100% robust molecular string representation. Mach. Learn. Sci. Technol, 144):45024, 2020.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-off-iistribttin.. In International Con- ference on Learning Representations, 2022. URL https: wwwp//viepeeeeeeee/ee/eeee.eeeee net/ forum?
13

---

Published as a conference paper at ICLR 2025
Tuan Le, Frank Noé, and Djork-Amé Clevert. Representation learning biomolecular structures using equivariant graph attention. In LoG, volume 198 of Proceedings of Machine Learning Research, pp. 30. PMLR, 2022.
Tuan Le, Julian Cremer, Frank Noé, Djork-Amé Clevert, and Kristof 'T. Schütt. Navigating the de- sign space of equívariant diffusion-based generative models for de novo 3d molecule generation. In ICLR. OpenReview.net, 2024.
Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, pp. 13052-13065, 2022a.
Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang. Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. 3d-molm: Towards 3d molecule-text interpretation in language models. In ICLR, 2024a. URL https: //wp/ppeeeeieeeeeeeeeeee wwwtteouu//niieeeee http://oorroariaaaAaaaaalaaaaaaa
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/IJCNLP (1). pp. 4582-4597. Association for Computational Linguistics, 2021.
Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua. Interventional video relation detection. In Proceedings of the 29th ACM International Conference Multimedia, pp. 4091-4099, 2021.
Yicong Li, Xiang Wang. Junbin Xiao, and Tat-Seng Chua. Equivariant and invariant grounding for video question answering. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4714-4722, 2022b.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2928-2937, 2022c.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Transformer-empowe invariant grounding for video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Yicong Li, Junbin Xiao, Chun Feng. Xiang Wang, and Tat-Seng Chua. Discovering rationales for video question answering. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 13869-13878. 2023b.
Yicong Li, Xun Yang, An Zhang, Chun Feng, Xiang Wang, and Tat-Seng Chua. Redundancy- aware transformer for video question answering. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 3172-3180, 2023c.
Yicong Li. Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language- guided affordance segmentation on 3d object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14251-14260, 2024b.
Zeming Lin, Halil Akin, Roshan Rao. Brian Hic, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Fxolutionary-scale prediction of atomic-level 2023.
protein structure with a language model. Science, 379(6637):
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph varia- tional autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In ICLR. OpenReview.net, 2022.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang. and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-mode uni-modal adapter. In EMNLP, 2023a. URL https: /ttp///eeeeee/eeeee/e/ee net forum? dd=
4WRhMNG
14

---

Published as a conference paper at ICLR 2025
Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder in masked graph modeling for molecules. In NeurIPS, 2023b. URL https:tt/////. http:///ppppeeeeeeeeeees net/ forumoormmm//mm idd WWW.
Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang. Kenji Kawaguchi, and Tat-Seng Chua Reactxt: Understanding molecular "reaction-ship" via reaction-come contextualized molecule-text pretraining. In Findings of the Association for Computational Linguisties: ACL 2024. Association for Computational Linguistics, 2024b. Forum?id=v-en jD fi. we.
Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. Prott3: Protein-to-text generation for text-based protein understanding. In ACL. Associa- tion for Computational Linguistics, 2024c. URL hwtph:ttttpspreeeeeeeeee ZmII jOPIjOPiZA 11.b.
net formm?id=
Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular prop- erty prediction: A multilevel quantum interactions modeling perspective. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firsi Innovative Applications of Artificial Intelligence Conference, IAAI 2019. The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 Febru- ary 1, 2019, pp. 1052-1060. AAAI Press, 2019. doi: 10.1609/AAAL 133101 13000 1052. URL. https: /to.//iiio///// org/ 10. 1099 aaai.aia......... v33101. 33011052.
Yanchen Luo, Sihang Li, Zhiyuan Liu, Jiancan Wu, Zhengyi Yang, Xiangnan He, Xiang Wang, and Qi Tian. Text-guided diffusion model for 3d molecule generation. 2023.
Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3d molecular geometry generation from scratch. In International conference learning representations (ICLR), 2022.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 7192-7203.
PMLR, 2021.
Sohir Maskey, Ali Parviz, Maximilian Thiessen, Hannes Stärk, Ylli Sadikaj. and Haggai Maron. Generalized laplacian positional encoding for graph representation learning. arXiv preprint vrXiv:2210.15966, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-88 8171. PMLR, 2021.
OpenEye, Cadence Molecular Sciences. OMEGA 5.0.1.3. OpenEye, Cadence Molecular Sciences, Santa Fe, NM. http://www.eyessopen..com
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, pp. 19992-995111 AAAI Press, 2018.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru- Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molec- ular Generation Models. Frontiers in Pharmacology, 2020.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeap7785, 2018
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularmn: Generating realistic molecular graphs with optimized properties. arXiv preprint arXiv: /305. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:1 10:10 2020.
15

---

Published as a conference paper at ICLR 2025
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Zachary A Rollins, Alan Cheng, and Essam Metwally. Molprop: Molecular property prediction with multimodal language and graph fusion. Journal of Cheminformatics, 16(1):56, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323 93322. PMLR, 2021.
Kristof T Schütt, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 2018.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In ICLR. OpenReview.net, 2020.
Yaorui Shi, An Zhang, Enzhi Zhang, Zhiyuan Liu, and Xiang Wang. ReLM: Leveraging language models for enhanced chemical reaction prediction. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: openreview. net forum?id- gJ2cSREV21.
Yaorui Shi, Sihang Li, Taiyan Zhang, Xi Fang, Jiankun Wang, Zhiyuan Liu, Guojiang Zhao, Zheng- dan Zhu, Zhifeng Gao, Renxin Zhong, al. Intelligent system for automated molecular patent infringement assessment. arXiv preprint arXiv:2412.07819, 2024.
Gregor Simm, Robert Pinsler, and José Miguel Hernández-Lobare Reinforcement learning for molecular design guided by quantum mechanics. International Conference on Machine Learn- ing. pp. 8959-8969, PMLR, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR. OpenReview.net, 2021.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Li6. 3d infomax improves gnns for molecular property prediction. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 20479-20502. PMLR, 2022. URL https: http:///oocceeeieeeeeeoooo
mlr ppe.s/ppee////ccee.eoe.ee vt/p/v12//////// htt/s//ar22222 html.
Teague Sterling and John J. Irwin. ZINC 15 ligand discovery for everyone. J. Chem. Inf. Model., 55(11):2324-2337, 2015.
Mengying Sun, Jing Xing, Han Meng, Huijun Wang, Bin Chen, and Jiayu Zhou. Molsearch: Search- based multi-objective molecular generation and property optimization. In KDD, pp. 4724 ACM, 2022.
Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and cranslaton-equiiarian neural networks for 3d point clouds. CoRR,
2018.
Benjamin Tingle, Khanh G Tang, Mar Castanon, John Gutierrez, Munkhzul Khurelbaatar, Chin- zorig Dandarchuluun, Yurii S Moroz, and John Irwin. Zine 22 free multi-billion- billion-scale database of tangible compounds for ligand discovery. Journal of chemical information and modeling, 63 (4): 166-1 176, 2023.
16

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, hts/2307.092883 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp.
2017.
Clément Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In ICLR. OpenReview.net, 2023a.
Clément Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. Midi: Mixed graph and 3d denoising diffusion for molecule generation. In ECML/PKDD (2), volume 14170 of Lecture Notes in Computer Science, pp. 560-576. Springer, 2023b.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular con- trastive learning of representations via graph neural networks. Nat. Mach. Intel., 4(3):279- 287, 2022. doi: 10.10085442236002-0477...
URL https:/ / org/10.1038/ /038/
a42256-022-00447-2
Yuyang Wang, Ahmed A. A. Elhag, Navdeep Jaitly, Joshua M. Susskind, and Miguel Ángel Bautista. Swallowing the bitter pill: Simplified scalable conformer generation. In ICML. OpenReview.net, 2024.
David Weininger. Smiles, chemical language and information system. 1. introduction method- ology and encoding rules. J. Chem. Inf. Comput. Sci., 28(1):31-36, 1988.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2): 2018.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: markov molecular sampling for multi-objective drug discovery. In ICLR. OpenReview.net, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geomet- ric diffusion model for molecular conformation generation. In ICLR. OpenReview.net, 2022.
Minkai Xu, Alexander S. Powers, Ron O. Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 38592-38610. PMLR, 2023.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor W. Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi S. Jaakkola. Klavs F. Jensen, and Regina Barzilay. Analyzing learned molecular repre- sentations for property prediction. J. Chem. Inf. Model., (188)3370-3388 2019. doi: 10.1021/ ACS.JCIM.9900037. URL https:tt////occ...t. ststtps::/doi. .org/10 021 /acs jaim. 9b00237.
17

---

Published as a conference paper at ICLR 2025
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
Atakan Yüksel, Erva Ulusoy, Atabey Ünlü, Gamze Deniz, and Tunca Dogan. Selformer: Molecular representation learning via SELFIES language models. CoRR, abs/2304.04660 2023.
Peiyuan Zhang, Guangtao Zeng. Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. CoRR, 002335,, 2024.
Zaixi Zhang, Yaosen Min, Shuxin Zheng, and Qi Liu. Molecule generation for target protein binding with structural motifs. In ICLR. OpenReview.net, 2023.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shlcifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. 11207..117.... 2023.
Xiaofan Zheng and Yoichi Tomiura. bert-based pretraining model for extracting molecular struc- tural information from a smiles sequence. Journal of Cheminformutics, 16(1):71, 2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In ICLR. OpenReview.net, 2023.
18

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND MORE FUTURE WORKS
NExT-Mol has several limitations that have not been addressed due to our limited computational resources and other technical challenges. We outline these limitations below:
Explore Generalization of 3D Conformer Prediction to Unseen Scaffolds. Table 6 shows that DMT-B's performance drop significantly on test molecules with unseen scaffolds in the training set. While our proposed transfer learning using MoLlama's pretrained ID representations can mitigate this issue, there is still room for improvement. Future work could explore advanced generalization techniques and the integration of chemical inductive biases to enhance performance on unseen scaf- folds. Additionally, developing a more comprehensive evaluation benchmark with a stricter scaffold split would provide deeper insights into model generalization. We leave these for future research.
Explore Randomized SELFIES Data Augmentation in Pretraining. Although randomized SELFIES augmentation shows promising results when fine-tuning Mol.ama for ID molecule gener- ation, we do not use this augmentation technique during pretraining due to our limited computational resources. We believe applying this technique in pretraining could lead to different outcomes. We leave this exploration for future work.
Explore Pretrained Molecular Large LM with Bi-directional Self-Attention. MoLlama uses causal self-attention, where each token only attend to previous tokens. While this approach is good fit for ID molecule generation, constrains MoLlama's 's potential for molecule representation learning. To mitigate this issue, we have attached bi-directional self eeff-attention layer after MoLlaaa (cf. Figure 3). However, a more natural solution would be to use a molecular LM with built-in bi- directional self-attention. Due to resource constraints, we do not pursue this, and existing works are often limited in scale (Irwin et al., 2022; Zheng & Tomiura, 2024). We hope this work draws more attention to this area and encourages the development of more foundation models for biochemistry.
Explore MExT-Mol for Struture-based Molecule Generation. We do not explore NExT-Mol for based molecule generation (Zhang et al., 2023) due the limited scope of this work. However, NExT-Mol could be extended for this task by conditioning the generation process on the structural embeddings of target pockets, potentially using techniques like cross- attenntntn,,on, adaptive layer normalization, or soft-prompting (Li & Liang, 2021). We leave this for future works.
Limited Exploration on Diffusion Guidance. Our DMT model utilizes i.i.d. sampling, without exploring advanced sampling method like classifier guidance (Dhariwal & Nichol, 2021) and particle guidance (Corso et al., 2024). However, particle guidance demonstrates that well-tuned guidance method can improve the conformer prediction by 10% precision. This is because the 3D molecular conformational space is large, and a guidance method with appropriate chemical inductive bias can improve the sampling efficiency. We leave this exploration as a future work.
Computational Cost when Incorporating MoLlama for 3D Conformer Prediction. Incorporat- ing MoLlama, a large LM with 960M parameters, increases training time. For example, training DMT-B alone (55M parameters) takes 52 seconds per epoch on an A100 GPU, while DMT-B with MoL.ama takes 210 seconds. We mitigated this problem by using a pretrained DMT-B, instead of training it from scratch, to reduce the training epochs when incorporating MoLlama. Yet, we will need improvement when transferring ID representations from large LM.
Quadratic Memory Complexity of DMT's Pair Representation. This pair representation incurs an additional ( GPU memory cost than the standard transformer, compared to the standard transformer's ' O(N) memory complexity when using FlashAttention, where N is the node number of molecular graphs. While we encountered no memory issues on the GEOM-DRUGS dataset (molecules with hundreds of nodes), this could be a bottleneck for molecules with thousands of nodes. Potential solutions include smaller batch sizes and model parallelism.
More Future Works In future, we plan to build multi-mode foundation models (Liu et al., 2024c; Li et al., 2024a) with NExT-Mol as the essential backbone for 3D molecule generation (Luo et al., 2023), in order to support tasks like chemical reaction prediction (Liu et al., 2024b; Shi et al., 2023) and drug-drug interaction prediction (Fang et al., 2024a). This is to support the broad application of LLMs for scientific discovery (Cai et al., 2024; Shi et al., 2024) and multi-modal LLMs (Li et al., 2024b). We will also explore model editing techniques (Fang et al., 2025) to locate the influential parameters and update its knowledge according the application requirement, and explore causal
19

---

Published as a conference paper at ICLR 2025
interventional methods (Li al., 2021) to find the rationale part (Li et al., 2023a;b;;; 2022b;e) of 2D molecules that influence 3D conformer prediction.
B
MORE EXPERIMENTAL RESULTS
B.1 ABLATION STUDY
Ablating MoLlama Pretraining. As Table 9 shows, pretraining significantly improves MoLlama's performances the ID distribution similarity metrics of SNN, Scaf and FCD, but slightly decreases novelty score (V&U&N). This may be because the model without pretraining prefers a more random sampling, increasing the novelty but reducing the similarity to the desired molecule distribution. Pretraining does not significantly influence stability and validity measures, because they are mostly guaranteed by the SELFIES representations.
Table 9: Ablation study for the MoLlama pretraining for ID molecule generation on the GEOM- DRUGS dataset.
Method FCD4
AtomStable
MolStable
V&C
V&U
V&U&N
SNN
Frag
Scaf
MoLlama
0.334
1.000
0.999
1.000
0.999
0,945
0,529
0.999
0.552
w/o pretraining
0.586
1.000
0.995
1.000
0.999
0.974
0.495
0.999
0,534
Table 10: Ablating random rotation augmentation for 3D conformer prediction on GEOM-QM9.
COV-R (%)f
AMR-R +
COV-P (%)1
AMR-P +
Method
Mean
Median
Mean
Median
Mean
Median
Mean
Median
DMT-B
95.2
100.0
0.090
0.036
93.8
100.0
0.108
0.049
w/o rand rot aug.
95.2
100.0
0.095
0.040
93.3
100.0
0.113
0.053
Random Rotation Augmentation. Table 10 shows that DMT benefits from random rotation aug- mentations. Unlike MCF (Wang et al., 2024), which relies on fixed canonical rotations, this is a key improvement because real data may be out-of-distribution and do not follow canonical rotations.
B.2 MOLECULE PROPERTY PREDICTION RESULTS FOR MOLLAMA
Experimental Settings. To evaluate MoLlama's capabilities beyond molecule generation, we apply it to molecular property prediction tasks (Liu et al., 2023b; Li et al., 2022a), highlighting the quality of its molecular representations. Following the setup in (Rollins et al., 2024). we fine-tune MaLlaaa four MoleculeNet (Wu et al., 2018) datasets: FreeSolv, ESOL, Lipo, and QM7. We adopt the same experimental settings and dataset splits as (Rollins et al., 2024), reporting mean performance and standard deviation over 10 random seeds. Following Section 3.3, we attach single-layer bi-directional self-attention layer after MoLlama to improve its encoding ability. After that, we apply a linear layer the mean embedding of all molecule tokens for property prediction. For each run, MoLlama is trained for 100 epochs, with test performance selected based on the validation dataset. We use a fixed learning rate of le-4 with the AdamW optimizer, and fine-tune MaLa using LoRA (Hu et al., 2021) (LoRA 8 and = 32) applied to all linear layers of the model. Following Section 3.1, we apply the random SELFIES augmentation during training- During inference, final prediction.
the average prediction values of 20 differently augmented SELFIES the
Observation. As shown in Table 11, MoLlama significantly outperforms baseline methods, achiev- ing relative improvements of 6.5%, 4.7%, 3.5%, and 16.9% on the FreeSoly, ESOL, Lipo, and QM7 datasets, respectively. Notably, our baselines include LM-based, based, and pretrained GNN- based methods, and MoLlama's 's better performance demonstrates its advantages derived from the extensive pretraining.
20

---

Published as a conference paper at ICLR 2025
Table 11:
Molecule property regression results on four MoleculeNet datasets (Wu al., 2018).
Baseline results are from (Rollins et al., 2024). Lower, is better.
Method
FreeSolv (RMSE) ESOL (RMSE) Lipo (RMSE) QM7 (MAE)
Supervised Learning Methods
RF (Wang et al., 2022)
2.03 3.1420.00
1.07 1.5020.00
0.8820.04
122.7a4. 156.940.0
SVM (Wang et al., 2022)
0.82=0.00
Supervised GNN-based Methods
GCN (Kipf & Welling, 2017)
2.87+0.14
1,4320.05
0,85
122.9
GATv2 (Brody et al., 2022)
3.14x0.00
1.00
0.89=0.00
113.3+0.0
GIN (Xu et al., 2019)
2.76#0.18
1,45#0.02
0.85=0.07
124.8+0.7
SchNet (Schlitt et al., 2018)
3.22+0.76
1.05#0.06
0,91=0.10
74.2=6.0
3D Infomax (Stärk et al., 2022)
2.23+0.26
0.95+0.04
0.74=0.01
MGCN (Lu et al., 2019) 9)
3.35+0.01
1.27
1.11
77.6s4.7
D-MPNN (Yang al., 2019)
2. 2.181
0.98+0.26
0.65=0.05
105.8+13.2
Pretrained GNN-based Methods Pretrain-GNN (Hu et 2020)
2.83+0.12 12
1,22+0.02
0.74=0.00
110.2+6.4
MolCLR (Wang et al., 2022)
2.20+0.20
1.11
0.65=0.08
87.2+2.
LM-based Methods
ChemBERTa-2 (Ahmad et al., 2022)
2.047#0.00
0.889#0.00
0.798=0.00
172. 840.00
MolPROP (Rollins et al., 2024)
1. 1.70s0.09
0.777=0.02
0.733=0.02
151,8+100
MoLlama, ours
1.59
0.740=0.01
0.627s0.01
63.5=1.6
Table 12: Incorporating MoLlama's ID representations to improve DMT's 3D conformer prediction.
COV-R (%)1
AMR-R |
COV-P (%)1
AMR-P L
Dataset
Method
Mean
Median
Mean
Median
Mean
Median
Mean
Median
GEOM-
DMT-B
95.2
100.0
0.090
0.036
93.8
100.0
0.108
0.049
QM9
+MoLlama
95.6
100.0
0.083
0.036
94.2
100.0
0.097
0.044
B.3 3D MOLECULAR CONFORMER PREDICTION
Table 12 presents the results of integrating MoLlama's pretrained 1D representations into DMT-B for 3D conformer prediction, using the same experimental setup as Table 5. The results demonstrate that MoLlama's pretrained representations can enhance DMT-B's performance.
B.4 INFLUENCE OF HYPERPARAMETERS
More Results on the Crerictor-Corrector Sampler. Table 4 reports DMT-B's performance with the predictor-corrector sampler with the hyperparameter of snr=0.3. To provide a more comprehen- sive analysis, Table 13 presents results for additional hyperparameter settings.
Different Noise Schedules at Inference Time. We test DMT-B's robustness to different noise schedulers at inference, using two representative options: the linear (Ho et al., 2020) and poly- nomial (Hoogeboom et al., 2022) schedulers. The original noise scheduler, based on the cosine function, follows (Nichol & Dhariwal, 2021). In this study, without retraining the model with these new schedulers, so the results are suboptimal.
the existing DMT-B checkpoint
GEOM-QM9
GEOM-Drugs
AMR-Recall AMR-Precision
AMR
0.12
ADMR
0.6
AMR-Recal
0.10
AMR-Precisian
0.4
10
20
50
100
'10
20
50
100
Sampling Slep (log scale)
Sampling Step scale)
Figure 5: Effect of sampling steps on AMR for 3D conformer prediction using DMT-B. Observation. As shown in Table 14, the polynomial scheduler achieves performance close to the cosine scheduler, likely because their curve shapes are similar. However, the linear scheduler results
21

---

Published as a conference paper at ICLR 2025
Table 13: Performances of 3D conformer prediction on the GEOM-DRUGS dataset.
COV-R (%)f
AMR-R +
COV-P (%)
AMR-P +
Model
Mcan
Median
Mcan
Median
Mcan
Median
Mcan
Median
DMT-B, PC samp., sur=0.2
85.3
91.5
0.398
0.372
66.5
69.2
0.633
0.560
DMT-B, PC samp., snr=0.3
85.5
91.2
0.396
0.370
67.6
71.5
0,623
0.546
DMT-B, PC samp., snr=0.4
73.8
79.9
0.535
0.501
68.0
72.1
0.621
0,548
Table 14:
DMT-B's 3D conformer prediction performances on the GEOM-DRUGS dataset when using different noise schedulers at inference time.
COV-R (%) f
AMR-E R J
COV-P (%) 1
AMR-P P J
Noise schedule
Mean
Median
Mean
Median
Mean
Median
Mean
Median
linear
62.7
62.7
0.648
0.634
60.3
60.6
0.726
0.624
cosine, original
85.4
92.2
0,401
0.375
65.2
67.8
0.642
0.577
polynomial
84.9
91.7
0.454
0.421
64.5
66.2
0.685
0.619
Time Comparison
DATL
Omega (CFU)
DMT
DNT-B
DMT-L
GFN2-xTB
TD w/ PG
Mollama
aga (CPU)
800
1000
Time (minutes)
Figure 6: Comparison conformer generation time on the test set of the GEOM-Drugs dataset using various methods.
a significant performance drop, suggesting that retraining DMT-B with the linear scheduler is necessary to achieve better results.
Sampling Steps. We evaluate 3D conformer prediction performance given different sampling steps.
Observation. As shown in Figure 5, we observe an improving trend in AMR for both recall and precision the sampling steps increase from to 100. The most significant improvements oceur between 5 and 20 steps, with diminishing returns beyond 50 steps. This indicates that our model can half the inference cost by trading off a small amount of performance.
The Influence of Batch Size to 3D Conformer Prediction. We evaluate the performance of DMT- with different batch sizes. The original batch size of 256 was chosen to maximize GPU utilization. To assess the impact of batch size, tested two variations: (1) reducing the batch size to 128, and (2) increasing it to 512 using gradient accumulation.
Observation. As shown in Table 15, the performance with 512 batch size is slightly worse than the original model. This likely due to underfitting caused by fewer training steps. We keep the number of training epochs the same the original experiment (256 batch size), therefore the larger batch size results in fewer gradient updates, leading to reduced model performance. Other than this observation, using the 128 batch size does not lead to significant difference than the original model.
22

---

Published as a conference paper at ICLR 2025
Table 15:
DMT-B's 3D conformer prediction performances on the GEOM-DRUGS dataset when using different batch sizes.
COV-R (%) f
AMR-R L
COV-P (%) f
AMR-P
Batch size
Mean
Median
Mean
Median
Mean
Median
Mean
Median
128
85.5
92.4
0.395
0.366
65.1
68.0
0.644
0.575
256. original
85.4
92.2
0.401
0.375
65.2
0.642
0.577
512
85.1
92.0
0.410
0.377
64.9
67.7
0.645
0.582
Table 16: 3D Molecule stability performances.
denotes our reproduced results.
GEOM-DRGUS dataset.
(b) QM9-2014 dataset.
3D-Metric
MolStable
3D-Metric
MolStable
Train
0.028
Train
0.953
EDM
0.002
G-SchNet
0.681
JODO
0.010
G-SphereNet
0.134
MiDi
0.003
EDM
0.817
EQGAT
0.025
MDM
0.896
NEXT-Mol, ours
0.027
JODO
0.934
MiD
0.842
EQGAT
0.889
NExT-Mol, ours
0.946
B.5 COMPUTATIONAL TIME COMPARISON
We conducted a time comparison between model and representative baselines for conformer generation on the test set of the GEOM-Dran Drugs dataset, which includes 1000 molecules. The base- lines include the OpenEye Omega (OpenEye, Cadence Molecular Sciences), TD w/ PG (Corso et al., 2024), and xTB The results are shown in Figure 6. platform with 8-core Intel Xeon Processor@* @2 2.90GHz
These experiments were performed CPU and an NVIDIA A100 GPU and the time is measured in minutes and seconds. Please note that the Omega and xTB are run the CPU only, while DMT and Mollama on the GPU. So the results may vary depending on the hardware.
B.6 3D MOLECULAR STABILITY PERFORMANCE
We do not report the 3D molecule stability metric (Hoogeboom et al., 2022) the main part of this work, because this metric presents significant limitation on the GHÖM-DRUGS dataset, showing only 2.8% for the ground truth training set. We present the results here for backup purposes.
B.7 MORE VISUALIZATIONS
Visualization of Random Samples. Visualizations of complete molecules sampled from NExT-Mol on GEOM-Drugs and QM9 are shown in Figure 8 and Figure 9, respectively. These samples are ran- domly selected to illustrate the diversity and effectiveness of our model. The visualization includes ID SELFIES sequences, 2D molecular graphs, and 3D conformers highlighting the spatial arrange- ment of atoms within the molecules. Notably, in the complex GEOM-Drugs dataset, NExT-Mol demonstrates its robustness by consistently generating molecules without disconnected components and effectively preserving the stable geometric planes aromatic ring structures. These visualiza- tions not only demonstrate the fidelity of the molecules generated by NExT-Mol with ID SELFIES sequences along with 3D spatial coordinates, but also emphasize the ability of our model to produce stable and chemically valid conformers accommodating wide range of molecular weights.
Visualization of 3D Conformer Prediction. To gain more insights on how transfer learning us- ing MoLlama's 1D representations improve 3D conformer prediction, we present more visu-
23

---

Published as a conference paper at ICLR 2025
(a) Ground truth.
(b) DMT-B's prediction (RMSD (c) DMT-B MoLlama 's predic- = 0.90).
tion (RMSD = 0.05).
(d) Ground truth.
(c) DMT-B's prediction (RMSD (f) DMT-B MoLlama's predic- 0.87).
tion (RMSD 0.06).
(g) Ground truth.
(h) DMT-B's prediction (RMSD (i) DMT-B MoLlama's 's predic- 0.84).
tion (RMSD 0.07).
(j) Ground truth.
(k) DMT-B's prediction (RMSD (l) DMT-B MoLlama's predic- 0.86).
(RMSD 0.07).
Figure 7: Visualization of 3D conformers. From feft-to-right, have the ground truth conformer, the conformer predicted by DMT-B, and the conformer predicted by DMT-B-MoLlama. For each model, we select the predicted conformer with the least RMSD the ground truth.
alizations of 3D conformer prediction Figure The samples are selected from the test set of GEOM-DRUGS with unseen scaffolds in the training set.
C FURTHER DETAILS ON METHODOLOGY
C.I ID MOLECULE GENERATION WITH MOLECULAR LLAMA LM
Data Preparation. Following (Irwin et al., 2022), collect 1.8 billion molecules from the ZINC- 15 database (Sterling & Irwin, 2015), significantly more than the 100 million molecules used in previous studies (Irwin al., 2022; Fang et al., 2024b). We keep only molecules with molecular
24

---

Published as a conference paper at ICLR 2025
lowir is vicecedccc Weltl B0UCR
CeoRtroa is Beleee medieas Brant
cleng
Eiotowe rntercaa HEG as Elccii ta 1i
ERGER EREERE Eatoed CONIRICHE TENC
ERERE sud Dwed BE and
ERTERROL Bhe BRago sea Soorr 2a00 Ir
AEMAOCCE
Mi
Kaokoon COOO Revest CETTEREEE PEBEE---OO Rigll
C BUROOO Bescell oo baa 18lb
Figure 8: Visualization of random samples generated by NExT-Mol trained on GEOM-DRUGS.
Table 17: Hyperparameter for pretraining MoLlama.
hidden
2048
hidden
silu
intermediate size
5632
batch size
512
max position emheddings
512
warmup steps
2000
num attention heads
32
min Ir
4.00E-05
hidden layers
22
init Ir
4.00E-04
num key value heads
weight decay
1.00E-01
n query groups
4
grad clip
1.0
550ght=500 Daltons and LogP<5 (Flynn, 1980), and transform them into SELFIES (Krenn al., 2020) sequences. After canonicalizing the SELFIES and removing hydrogen atoms, the dataset contains 90 billion tokens. We further filter the molecules in the valid and test sets of the GEOM- QM9 and GEOM-DRUGS datasets (Axelrod & GomezzBombarelli, 2022) and randomly sampled 1% of the remaining data as the validation set.
Randomized SELFIES Augmentation Details. In order to generate randomized SELFIES. we first generate the randomized SMILES (Weininger, 1988), and transform the SMILES into SELF- IES. We follow (Arás-Pous et al., 2019) for the implementation details of random SMILES, and use restricted random sampling of SMILES. Similarly, we also generate canonical SELFIES by transforming canonical SMILES.
Pretraining Details. We train MoLlama from scratch for ID molecule generation using a next-token prediction objective. The code and hyperparameters are based (Zhang et al., 2024), utilizing (Dao, 2024) and FSDP (Zhao et al., 2023) for faster training. We use a max context length of 512, concatenating multiple SELFIES sequences into the same context, with any overflow
25

---

Published as a conference paper at ICLR 2025
Iireri Brotkderdd
Edtdi i ffratisnnn totaet iffral Borevil
Borent tikeel el
Milwer Io The Reeeiti Th Eikenn
KeHrOOse: Biral CEDERCCEENN
Eair vevi HeB.Ria poure g
SEseir Barer ECEDer Sentii Ehardi ng'
kee rUD wii TE 2eree2eee/// II
Edel Toltor EE Enwsd TRO
Eerre du di IERIGOORIEE IE'S Teatocent TE COITEIN TCCEEBI OF ELRORONIIT
H
Peg' Pelliiieeett EdS: liteli barch Wi3 Phrahd
Eliiilee Brrekd
Rig BiBeeviiii
Ze coca- Ric Drandic Iried
Wibaren' 1-2 Dup (ill HineeciiiBBsccc
Eetrr ze Klare HiBreed CII thete
Elirura 1-C cari DIEAS Daano Thanet Prlaattte
2E-CRanee SICICR 21-6arch
Povr
Figure 9: Visualization of random samples generated by NExT-Mol trained on QM9-2014.
trimmed and used in the next context. We use the AdamW optimizer and a scheduler with linear warmup and cosine decay. The key parameters are included in Table 17, We train the model for 555k global steps. The training was done on 4 NVIDIA A100-40G GPUs and took approximately two weeks. The training log shown Figure 10.
On the Advantages of Acheving 100% Validity beyond Validity Itself. We employ the ID SELF- IES representation for LM training. Here we elaborate on the other advantages beyond 100% valid- ity, which are also crucial for real-world applications:
Improving validity could improve other 2D metrics, like SNN, Frag, and Scaf. These metrics measure the distributional similarity of 2D molecular structures of valid molecules. If a model still generate invalid molecules, it likely the model does capture the true target distribution, which contain only valid molecules. 100% validity helps the model learn from and sample from the valid molecular structures, which essential for molecule generation tasks. This is demonstrated by our improved FCD, SNN, Frag, and Scaf metrics in Table 2.
Improving validity could improve 3D geometry learning. The improved validity also leads to better learning of 3D molecular geometry, because grounds 3D structure prediction on valid 2D structures. Other joint 2D and 3D prediction methods (Huang et al., 2024; Vignac et al., 2023b) can easily encounter invalid 2D structures when sampling 3D structures, therefore leads to worse 3D structure prediction. This is demonstrated by NExT-Mol's significant improvements in geometry similarity metrics (e.g., bond angle and bond length) Table 2.
26

---

Published as a conference paper at ICLR 2025
Train and Validation Perplexity vs Step
Train PPL
4.5
Perplexity
5.
20
10
10'
10'
Step (log scale)
Figure 10: Visualization of MoLlama's training and validation PPL. log during pretraining.
C.2 3D CONFORMER PREDICTION WITH DIFFUSION MOLECULAR TRANSFORMER
Diffusion Process. Here we elaborate on the details of our diffusion process. Following (Nichol & Dhariwal, 2021; Huang et al., 2024), we use the cosine scheduler controlling the noise scale for the diffusion process:
f(t) f(0)
t +
[(I)
COs
(5)
1
where t (0, 1] is the time step, and is hyperparameter empirically to 0.008, follow- ing (Nichol & Dhariwal, 2021).
Our pseudo codes for training and sampling shown in Algorithm and Algorithm 2 below. Following (Ho et al., 2020), we have the following hyperparameters used in the pseudo-codes for training and sampling:
( = -((()))/ (((1)11 σ(0) V1- a!f).
(6)
Algorithm Training
1: t ~ 1()
(Sample a time step)
2: G(0) (x("), h,e) Training Set
Sample 3DD molecule
3: x()) + (()) (U)
(Centering molecule coordinates)
4: x()) + xx(0) where R E SO(3) randomly sampled
Random rotation augmentation!
5: ( ~N(O
6: (() ()) +
Forward diffusion)
7: G + (x((), e)
8: Minimize loss L = -DM0) 1
RMHA. Here we define the multi-head version of RMHA. Similar to the single-head version, we first generate the queries, keys, and values for atom representation H, and generate the queries and values for pair representation E:
QQK;
W gi W. (7)
[Q v tanh( W.r (8)
27

---

Published as a conference paper at ICLR 2025
Algorithm 2 Sampling 3D Conformers
Require: time steps (44)== a molecular graph GTD <- (h, e)
x(11) IO)
(Set the initial noise conformer
2: for to M do
3: 1 8
(Set time step)
4:
G + (x((), h,
5:
z~ NO, 1) i < M else z =0
6:
x(1)
1-a0
DMT(G(O, + tol0z
(Update conformer
voi
Vi-a00
7: end for
8: return (M)
Subsequently, we define the Relational-Attention (R-Attention) module, which is the combination of Equation 3 and Equation 4:
R-Artention ( Q,K, vR),
(9)
where O2
Σ ==1
ai 3 Vj),
(10)
05 K
C.jj = softmax,
(11)
vd
After this, the muli-head version of RMHA can be written as: RMHA(Q, V, V²) = Concat(O' O where Of ff = R-AAttetion, WarQ. wEyK, W., W Wer Q", W V
(12)
(13)
where h is the number of head; f E [1, hl; W, is the linear projector combining outputs of different heads; and W af, WaF, and W.f are linear projectors for the f-th head of atom representations; and oaf and Wegf are linear projectors for the f-th head of the pair representation.
C.3 MOLLAMA REPRESENTATION IMPROVE DMT'S S 3D CONFORMER PREDICTION
Details of SELFIES-to-Aoon Mapping. The mapping process is not straightforward with existing software, so we have to manually code a significant portion. For details on the full implementation, please refer to our code. brief, the SELFIES software provides mapping between SELFIES and SMILES tokens, and RDKit gives the atom order when generating SMILES. We manually convert this atom order into a mapping between SMILES and atom indices, then combine the SELFIES- to-SMILES and SMILES-to-att handle missing hydrogen atoms in both SMILES and SELFIES during the mapping process.
mappings into the DELFIES-tO-ate mapping. Additionally,
Rationale behind Transfer Learning between ID Molecule Sequences and 3D Conformers. The final goal this transfer learning is to leverage the billion-scale ID/2D molecule dataset to improve the 3D conformer prediction performance, which is constrained by limited 3D data. For clarity, decompose the rationale into the following chain arguments:
3D conformers are theoretically governed by 2D molecular graphs under quantum mechan- ics (QM). 3D molecular properties and structures are fundamentally rooted in QM. Using (ap- proximated) QM-based methods, like DFT, we can accurately predict 3D conformers from 2D molecular graphs, though high computational cost. This establishes the critical role of 2D representations in determining 3D structures.
3D conformer prediction relies on high quality 2D molecule representations. Deep learning models predict 3D conformers from 2D graphs, and their performance is heavily influenced by the quality of 2D molecular representations. Transfer learning can enhance 2D molecular representa- tions, demonstrated by prior works (Hu al., 2020; Liu al., 2022; Hou et al., 2022).
28

---

Published as a conference paper at ICLR 2025
ID molecular representations be converted to 2D molecular representations, and tribute to 3D prediction. 1D molecule sequences encode the same information as 2D molecular graphs, and the ID to 2D transformation can be achieved by deterministic toolkit, like RDkit. Leveraging RDkit and our proposed cross-modal projector (ef. Section 3.3), we can transform ID molecular representations to 2D molecular representations, and therefore contribute the 31) prediction. We have demonstrated this improvement in Table 5, where using the pretrained ID representations improve 3D conformer prediction.
ID pretraining scales more effectively than 2D. Given the billion-scale ID/2D molecule dataset, we mostly prioritize the scalability when selecting the pretraining method. After literature review, we find that ID LM-based pretraining methods, like Llama (Touvron al., 2023) and BERT (De- vlin et al., 2019), are extensively demonstrated for scalability and effectiveness. Therefore, we opt pretraining instead pretraining.
D EXPERIMENTAL DETAILS
D.1 BASELINES
Here present a brief introduction for the baselines used in our experiments, We categorize base- lines by their benchmarks.
De Novo and Conditional 3D Molecule Generation.
G-SchNet (Gebauer et al., 2019): G-SchNet autoregressively generates 3D molecules by consid- ering molecular symmetries through the SchNet (Schütt et al., 2018)
G-SphereNet (Luo Ji, 2022): G-SphereNet autoregressively generates 3D molecules, in which each step determines the atom type, bond length, angle, and torsion angles.
EDM (Hoogeboom et al., 2022): EDM pioneers the diffusion methods for 3D molecue generation. constructs a diffusion model with the EGNN (Satorras et al., 2021) architecture and the VDM diffusion process (Kingma al., 2021).
MDM (Huang et al., 2023b); MDM is diffusion model for 3D molecule generation. Through specialized edge construction module, leverages both global interatomic interactions and local interatomic interactions for 3D modeling.
CDGS (Huang al., 2023a) and JODO (Huang al., 2024): CDGS is diffusion model for 2D molecular graph generation. It models discrete vairables (e.g. atom types and bond types) using one-hot encoding and applies continous diffusion for generative modeling. JODO extends CDGS by studying joint 2D and 3D molecule generation. It features a RMHA module for enhanced relational molecular graph modeling.
MiDi (Vignac et al., 2023b): MiDi is joint 2D and 3D diffusion model for 3D molecule genera- tion. It leverages two diffusion processes of discrete diffusion (Vignac et al., 2023a) and continous diffusion (Hoogeboom al., 2022) for the corresponding data types in a molecule.
EQGAT-diff (I.e al., 2024): EQGAT-diff modified the EQGAT (Le et al., 2022) architecture for joint 2D and 3D molecular generation. EQGAT is based on the Tensor Field Networks (Thomas et al., 2018) to achieve 3D rotational and translational equivariance.
GeoLDM (Xu et al., 2023): GeoLDM explores the idea of latent diffusion model (Rombach et al., 2022) for 3D molecule generation.
EEGSDE (Bao et al., 2023): EEGSDE explores conditional 3D molecule generation with diffusion guidance by an energy function.
MolGPT (Bagal et al., 2021): MolGPT is decoder-only molecule LM pretrained on ID SMILES sequences.
MolGen (Fang et al., 2024b): MolGen is encoder-decoder molecular LM pretrained ID SELFIES sequences. Following (Raffel et al., 2020), it is pretrained and evaluated using a span- corruption objective.
3D Conformer Prediction.
29

---

Published as a conference paper at ICLR 2025
Table 18: Hyperparameters of the DMT-B and DMT-L models.
DMT-B
DMT-L
layers
10
12
atom hidden size
512
768
atom intermediate size
2048
3072
pair hidden size
128
192
pair intermediate size
512
768
heads total params optimizer init Ir min Ir
8
8
55M
150M
Adam
1.00E-44
1.00E-05
warmup warmup steps weight decay
1.00E-06
1000
0.05
OMEGA (Hawkins, 2017): OpenEye OMEGA is a commercial software that employs a com- bination of fragment-based methods and torsional sampling, guided by empirical force fields or customized energy functions, to predict 3D conformers.
GeoMol (Ganea et al., 2021): GeoMol is an SE(3)-invariant model for 3D conformer prediction. In the first step, it predicts the bond angles and bond lengths for all the neighbors of each non- terminal atom. Next, it assembles the local structures together by predicting their torsion angles.
GeoDiff (Xu et al., 2022): GeoDiff is diffusion model that leverages roto-translational equiv- ariant GNN for 3D conformer prediction.
Torsional Diffusion (Jing et al., 2022): Torsional diffusion is diffusion model defined on the dihedral angles of 3D molecules. samples seed conformers using RDkit, and applies diffusion only on the dihedral angles of molecular bonds, while fixing the bond lengths and bond angles.
Particle Guidance (Corso al., 2024): Particle guidance is diffusion guidance method designed to improve the sampling diversity compared to the vanilla i.i.d. sampling. It modifies torsional diffusion'' 's sampling process for 3D conformer prediction, without changing its training process.
MCF (Wang al., 2024): MCF explores the power of scaling law 3D conformer prediction. Instead of following prior works and leveraging neural architecture with built-in 3D equivari- ance, scales up general-purpose purpose transformer, and demonstrates strong performances.
D.2 DMT CONFIGURATIONS
Hyperparameter: Table 18 shows the key hyperparameters used for training the DMT-B and DMT- L models. Other hyperparameters, like batch size and training epochs, separately listed for each task in the following sections.
Features. We use the same atom features and pair features as (Jing et al., 2022). For the GEOM- DRUGS dataset, the atom feature has 74 dimensions; for the QM9-2014 and GEOM-QM9 datasets, the atom feature has 44 dimensions. The bond feature has 4 dimensions.
D.3 TASK: De Novo MOLECULE GENERATION
For De Novo molecule generation, we separately train NExT-Mol for the GEOM-DRUGS and the QM9-2014 datasets. This process involve training both the MoLlama and DMT of NExT-Mol.
MoLIama Settings. For QM9-2014, we use a batch size of 512 and train for 100 epochs, while for GEOM-DRUGS, we use batch size of 256 and train for 20 epochs. For sampling, we employ a sampling temperature of 1.0 and, beam size of I, and we sample 10,000 molecules for evalua- tion. We use the AdamW optimizer and learning rate scheduler with linear warmup and cosine decay. The optimizer hyperparameters are as follows: init Ir=le-4, min Ir= le-5, warmup Ir=le-6, warmup steps=1000, and weight decay=0.05.
30

---

Published as a conference paper at ICLR 2025
DMT Settings. We use dropout rate of 0.1 for QM9-2014 and 0.05 for GEOM-DRUGS, Follow- ing (Huang et al., 2024), we select only the conformer with the lowest energy for training on the GEOM-DRUGS dataset. For both datasets, train DMT-B for 1000 epochs. The batch size for QM9-2014 is 2048 and the batch size for GEOM-DRUGS is 256.
Details the Evaluation Metrics. We the MMD distance when computing the distributional similarity of bond lengths, bond angles, and dihedral angles. Note that, we do not perform Kekuliza- tion and Sanitization when computing molecule and atom stability for 2D and 3D molecules. We use canonicalized SMILES for both the generated molecules and the training dataset when computing novelty and uniqueness of molecules. All the baselines are consistently evaluated under the same setting above.
D.4 TASK: CONDITIONAL MOLECULE GENERATION
Details for Adapting NExT-Mol for Conditional Generation. For conditional molecule gener- ation on the QM9-2014 dataset, we modify the NExT-Mol architecture to incorporate property- specific information into both the Mo language model and the DMT conformer prediction model. This approach allows us to generate molecules with desired properties in both ID sequence and 3D structure spaces.
Condioning MoLlama. We implement a condition MLP to encode property information into soft prompt. This MLP consists of two linear layers with a GELU activation function in between. It transforms single property value into 4-token sequence embedding, each token having the same dimensionality as the model's hidden size. The resulting soft prompt is prepended to the input sequence embeddings of SELFIES before being fed into the language model. We adjust the attention mask accordingly to ensure the model attends to these conditional tokens.
Condioning DMT. We use an MI.P to process the property value, followed by a linear projection match the time embedding dimension. This processed condition is then added to the time embedding, allowing the diffusion process to be guided by the desired property throughout the denoising steps.
MoLlama Setting, For conditional molecule generation, train MoLlama with batch size of 256 for 100 epochs on the QM9-2014 dataset. We use a sampling temperature of 1.0, beam size of and we sample 10,000 molecules for evaluation of each desired property.
DMT Setting. For the DMT-B model, we train with a batch size of 512 for 1000 epochs on the QM9-2014 dataset. We employ dropout rate of 0 with 100 sampling steps for evaluation.
The optimizer and learning rate schedule are consistent with the de novo generation task, using AdamW with linear warmup followed by cosine decay. We train the conditional generation model for six different quantum properties using the same optimization strategy as in the de novo generation task. Each model trained on 4 NVIDIA A100-80GB GPUs.
D.5 TASK: 3D CONFORMER PREDICTION
Training Details. We elaborate the training details for each of the three training stages in Sec- tion 3.3.
Stage 1: DMT Training, For GEOM-QM9 we train the DMT-B model for 2000 epochs with batch size of 2048. For GEOM-DRUGS. we train both the DMT-B and DMT-L models for 3000 epochs with batch size 256. Note that, for each epoch, we randomly sample 3D conformer for each molecule, but not enumerate all the 3D conformers of that molecule. The resulting models (i.e., DMT-B and DMT-L) are used directly for evaluation in Table 4.
Stage 2: Projector Warmup. For both datasets, we train only the LoRA weights of MoLlama, and the cross-modal projector for 10 epochs. The pretrained weights of DMT and MoLlama are frozen throughout the process.
Stage 3: Integrated Fine-tuning. For both datasets, we train the integrated model for 500 epochs. We train the LoRA weight of MoLlama, the mrosl pojector, and the DMT model. The pretrained weights of Mol are frozen throughout the process.
31

---

Published as a conference paper at ICLR 2025
Evaluation. Following (Wang et al., 2024; Jing et al., 2022), the dataset split of 243473/30433/1000 for GEOM-DRUGS and 106386/13323/000 for GEOM-QM9, provided by (Ganea al., 2021). For molecule with K ground truth conformers, we generate 2K con- formers as predictions.
Evaluation Metries. Let (CT hEl,L) be the L predicted conformers and let {C*]kelL,K) be the K ground truth conformers. The evaluation metrics AMR-R (AMR-Recall) and COV-R (COV-Recall) be formally defined as follows:
COV-R = +
1
1l E [1..]] 36 [1..K], RMSD(O, Cf) < 6H},
(14)
AMR-R = T
1
Σ IE[1..]]
min RMSD(C&, C7), kE[I..K]
(15)
where 5 is a threshold that is set to 0.75A for GEOM-DRUGS and set to 0.5 for GEOM-QM9, fol- and COV-P COV-Precision)
lowing (Wang al., 2024; Jing et al., 2022). AMR-P can be similarly defined by swapping the ground truth conformers and predicted conformers.
32


Reviews:
[{"review_id": "cnfLtCmFjs", "reviewer": "Reviewer_boXt", "summary": "The paper introduces the NEXT-Mol method, which first generates 1D molecular character representations using the Mol-LLAMA model and then generates the 3D structures of the molecules. The pre-trained molecular generation model’s atomic representations improve the performance of 3D structure generation.", "questions": "I think the authors should provide new evidence of the advantages of this method as a 3D model for molecular generation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This strategy has advantages in terms of the stability and effectiveness of molecule generation because it completely ignores the influence of 3D structures during the generation process.\n2. The pre-trained molecular representations and the improved structural diffusion model achieve a new state-of-the-art (SOTA) in small molecule conformation generation, although the improvement is very slight.", "weaknesses": "1. The comparison results of 3D molecular generation in the paper are unfair because the authors completely ignore the influence of 3D structures on molecular representation during the generation process. The results of molecular generation should be compared with generation models based on 1D molecular representations.\n2. The improvement in molecular conformation generation results mentioned in the paper is actually very small. In Table 3(A), DMT-B on GEOM-Drugs only improves by 1.4% compared to MCF-B, COV-R. Compared to Par. Guid, COV-P decreases by 3.7%, but the model parameters increase by 30 times. The significance of this method is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nipa7OloyM", "reviewer": "Reviewer_ZnjA", "summary": "The paper \"NEXT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation\" introduces a foundational model for 3D molecule generation that combines the 3D diffusion model with a 1D language model trained on SELFIES representations. By integrating the advantages of both approaches, NEXT-Mol aims to address challenges in chemical validity, scalability, and data scarcity. The model comprises three components: (1) MoLlama, a large language model for generating 1D molecules, (2) DMT, a diffusion model for 3D conformer prediction, and (3) a transfer learning technique that utilizes MoLlama’s 1D representations to improve DMT’s 3D predictions. The experiments show that NEXT-Mol performs well on several datasets and tasks, including 3D molecule generation and conditional molecule generation with specific quantum chemical properties.", "questions": "Could you provide a theoretical justification for transfer learning between 1D molecular sequences and 3D conformers? An in-depth explanation could clarify why this cross-modal transfer is effective.\nWhat are the potential computational trade-offs for using larger models (DMT-L) in terms of scalability and inference speed? Including a computational analysis of DMT-B versus DMT-L could highlight the scalability limits.\nAre there specific molecular properties or types of molecules where NEXT-Mol struggles to perform as well? Identifying any limitations or edge cases where the model’s performance drops would clarify its practical scope.\nCould you elaborate on why RMHA was chosen over other potential attention mechanisms? A comparison or justification of this design choice could strengthen the architectural motivation.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Innovative Approach: The combination of a 1D language model with a 3D diffusion model is an original solution to ensuring chemical validity while efficiently generating 3D conformers. This cross-modal learning technique enhances the model’s adaptability.\nComprehensive Experiments: The authors provide extensive experimental results across different tasks and datasets, demonstrating NEXT-Mol’s versatility and effectiveness in molecular generation and conformer prediction.\nPractical Application Potential: This approach is particularly relevant for pharmaceutical applications where 3D molecular structures are critical for drug discovery and chemical analysis. The model’s strong performance on chemical validity and stability metrics suggests its practicality.\nScalability and Adaptability: The design of NEXT-Mol allows for transfer learning, making it more resource-efficient and adaptable to different datasets or molecule sizes, which is useful in a field with diverse requirements.", "weaknesses": "Limited Theoretical Insight: The paper lacks a theoretical explanation of why combining 1D and 3D modeling via transfer learning improves performance. Further theoretical analysis could provide deeper insights into the effectiveness of this architecture and potential limitations.\nAbsence of Ablation Studies on Model Size and Hyperparameters: While the paper shows promising results with two model sizes, a more detailed examination of how model size or key hyperparameters (e.g., noise schedule, batch size) impact performance would provide more guidance on model tuning.\nLimited Exploration of Alternative Architectures: The use of RMHA and the specific structure of DMT are well-motivated but not directly compared to alternative architectures. A comparative study could clarify if these design choices are optimal for all molecular generation tasks.\nLack of Discussion on Model Limitations and Future Extensions: Although the model shows improvements, potential challenges such as memory overhead in larger molecules or limitations in certain chemical property predictions are not thoroughly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s83dPSAkuE", "reviewer": "Reviewer_pyyj", "summary": "The paper introduces NEXT-Mol, a model for 3D molecule generation that combines 1D Language Models with 3D diffusion models. NEXT-Mol first uses a pre-trained LM to generate 1D molecular sequences, ensuring chemical validity. It then predicts these molecules' 3D shapes with a refined diffusion model. Enhancements in model scaling, architecture, and transfer learning between 1D and 3D representations improve 3D predictions. NEXT-Mol is claimed to generate stable, accurate 3D conformers.", "questions": "●  \tWhat is the rationale for combining 1D and 3D generation sequentially, and what benefits does this approach offer?\n●  \tCould you clarify how transferring a 1D representation contributes to the overall model performance?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "100% Validity: Ensures generated molecules are chemically valid by using a 1D SELFIES-based LM.\n●  \tImproved 3D Accuracy: Enhanced 3D conformer prediction through a refined diffusion model.\n●  \tTransfer Learning: Leverages 1D representations to boost 3D conformer prediction accuracy.\n●  \tScalability: Scales well with large molecular datasets for robust molecule generation.\n●  \tVersatility: Performs well across tasks like de novo 3D generation, conformer prediction, and conditional molecule generation.", "weaknesses": "●  \tFocus on Core Objective: In the title the paper is presented as a generative model for molecules. The model is benchmarked as a conformer generating model. Currently, benchmarks appear misaligned from the title; consider established benchmarks like CheckPose or DrugPose for 3D generation.\n●  \t1D-to-3D Transformation Claim: The claim that converting a 1D sequence to 3D adds value is questionable, as the graph already provides all necessary information.\n●  \tNot accurate statement on Rotation Augmentation: The statement, “Following AlphaFold3 (Abramson et al., 2024), we apply random rotation augmentation on 3D conformers to help DMT obtain equivariance to rotated inputs by learning. While (Wang et al., 2024) report decreased performance given random rotations, DMT benefits from it, potentially due to the improved neural architecture,” is unclear. The authors imply that rotation can be achieved without using equivariant networks (those maintaining symmetry under rotation). However, AlphaFold claimed that it is not necessary to have equivariant networks, so it is essential to clarify how DMT benefits from rotation augmentation and to distinguish it from learned equivariance.\n●  \tLimited ML Novelty: The model presents minimal innovation from an ML perspective, as it mainly combines existing components—LLaMA and diffusion models. This combination, particularly in transferring 1D information to 3D, offers limited novelty and benefit for the conformer generation part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mPdvXzWKI", "reviewer": "Reviewer_iSmq", "summary": "The paper introduces Next Mol, an innovative model for 3D molecule generation that combines the strengths of 1D molecule generation using SELFIES representations with subsequent conformer prediction. This approach is particularly timely given the scarcity of 3D annotated molecules in existing databases. While billions of molecules are cataloged in databases like ZINC or Enamine, researchers often rely on datasets such as GEOM DRUGS, which contain only about 400,000 unique molecules with approximately 40 million 3D structures.\n\nKey Contributions:\n- Novel Methodology: The integration of 1D molecule generation through SELFIES with conformer prediction addresses the limitations of current datasets and methods in 3D molecule generation.\n- Performance Improvements: Next Mol demonstrates significant enhancements on benchmarks for both conformer generation and unconditional 3D molecule generation, outperforming existing models.\n- Transfer Learning: The study shows that transfer learning between the stages of 1D molecule generation and conformer prediction positively impacts the results, suggesting a valuable strategy for future research.\n- Advancing Beyond Equivariance Restrictions: The proposed DMT (Diffusion Molecular Transformer) model pushes the boundaries of 3D molecule generation by moving beyond the equivariant restrictions that have been prevalent in recent years, potentially opening new avenues in molecular modeling.", "questions": "Questions to the Authors:\n\n**Comparison of MoLama with Other Models:** Could you provide a comparison between your 1D molecule generation model (MoLama) and other models like Equiformer on tasks beyond 3D molecule generation, such as molecule classification (or any suitable task where you can compare MoLama and let's say Equiformer)? Including such a comparison would highlight the standalone value of MoLama.\n\n**Practical Advantages of 100% Validity:** Given that validity filtering is straightforward in practice, what are the practical advantages of achieving 100% validity over, for example, 90% validity in real-world applications?\n\n**Inclusion of Performance Metrics:** Considering that computational performance is crucial in conformer generation tasks, could you include speed and efficiency metrics for your model, particularly in comparison with tools like OpenEye Omega? I would also add OpenEye Omega conformer generation + consequent xTB geometry optimization time. \n\n**Analysis of Metrics Used:** Could you analyze the metrics used in your study, especially focusing on 3D molecular stability and the FCP metric? Since the FCP metric relies on a neural network, is it capable of handling your data distribution, and is it up-to-date with recent developments?\n\n**Consistency and Clarity of Metrics:** To ensure consistency, could you provide detailed descriptions of the metrics used in your study and confirm that the same versions are applied across all comparisons? Additionally, please clarify any preprocessing steps that might affect the results.\n\n**Inclusion of EQGAT-Diffusion in Comparisons:** EQGAT-Diffusion is a recent model relevant to 3D unconditional molecule generation. Could you include it in your comparisons to provide a more comprehensive evaluation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- State-of-the-Art Conformer Generation: The model achieves state-of-the-art performance in conformer generation, demonstrated through extensive comparisons with popular models like GeoDiff, Torsional Diffusion, and MCF, as well as widely used tools such as RDKit and OpenEye-Omega.\n- High Topological Metrics: Utilizing 1D generative models significantly improves topological metrics—such as molecular stability, validity, and uniqueness—elevating them to nearly 100%.\n- Scalable Transformer Architecture: The Diffusion Molecular Transformer is a scalable model that employs a simple transformer architecture with proven efficiency, making it an excellent base model for numerous other related tasks.", "weaknesses": "Weaknesses:\n\n**Lack of Comparison with Other Molecular Language Models:** Although the paper introduces a 1D molecule generation component (MoLama), a 3D conformer generation model (DMT), and a transfer learning technique, it primarily showcases the performance of the conformer generation part and the advantages of transfer learning. However, it lacks a comparison with other molecular language models concerning the quality of the generated SELFIES representations.\n\n**Overemphasis on 100% Validity:** The paper focuses on achieving 100% validity, but in real-world applications, validity filtering is an extremely simple process due to how validity is defined. Consequently, there is no significant practical difference between achieving 90% validity and 100% validity.\n\n**Mischaracterization of Computational Complexity:** The paper states that structures were obtained using computationally intensive geometry optimization with DFT. However, the GEOM dataset was designed using the CREST software for conformer sampling, followed by geometry optimization with GFN2-xTB—a semi-empirical tight-binding method, not DFT. Moreover, compared to some deep learning models, this approach is not computationally intensive; on a reasonable workstation, geometry optimization takes about 0.5 seconds per average GEOM-Drugs structure.\n\n**Missing Performance Metrics for Conformer Generation:** In conformer generation, computational performance is extremely important. OpenEye Omega remains one of the most popular software tools for this task because of its speed. The paper lacks performance metrics related to speed and efficiency, which are necessary for a fair comparison with existing tools.\n\n**Use of Questionable Metrics:** The metric reported by JODO shows a 2.8% 3D molecule stability for the GEOM-Drugs dataset, rendering it practically meaningless. I strongly encourage avoiding the propagation of this metric in new papers. According to the code, the metric is based on a predefined bond length table, and a bond length is considered \"good\" if it is within 0.05 Å of the table value. However, the optimal distances between atoms are primarily defined by the energy landscape underlying the data—for GEOM-Drugs, it's GFN2-xTB—and depending on atom configurations, deviations in bond lengths can exceed 10%. While I'm uncertain about the validity of the 3D FCP metric, it's not entirely clear that it's completely off when compared with 3D molecule stability. Instead of attempting to report every possible metric to compare with other methods, focus on identifying the most important ones and emphasize the significance of the margins your model is achieving.\n\n**Inconsistencies in Metric Definitions and Reporting:** Providing detailed descriptions of the metrics used in the supplementary material is crucial due to inconsistencies across different papers. For example, MiDi reported the Wasserstein distance for bond angles and bond length distributions for all bonds and angles, whereas this paper (and at least JODO) uses MMD for the most frequent bonds, angles, and torsions. Additionally, the way MiDi computes atom stability and molecule stability differs from the JODO code. Performing kekulization at the beginning can alter the valencies of atoms, leading to different results (e.g., if you manually define H:O:H, where \":\" is an aromatic bond, kekulization converts it to the valid water molecule H-O-H). Some models measure atom and molecule stability for raw data, while others use RDKit preprocessing before measuring stability, which can inflate results. It is essential to be consistent in comparisons. While this may not significantly change your results, it could artificially boost JODO's stability results, skewing the comparison. Overall, ensure that the same versions of the metrics are used across all comparisons and that they are well-defined in the supplementary material.\n\n**Omission of EQGAT-Diffusion in Comparisons:** EQGAT-Diffusion is a recent model—published well before the ICLR deadline—that should be included in the comparisons for 3D unconditional molecule generation to provide a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)