3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling
Authors: Qizhi Pei
Rui Yan
Kaiyuan Gao
Jinhua Zhu
Lijun Wu
Venue: ICLR 2025 Poster (2025)
Keywords: molecule-text modeling
3D molecular tokenization
language model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The integration of molecular and natural language representations has emerged as a focal point in molecular science, with recent advancements in Language Models (LMs) demonstrating significant potential for comprehensive modeling of both domains. However, existing approaches face notable limitations, particularly in their neglect of three-dimensional (3D) information, which is crucial for understanding molecular structures and functions. While some efforts have been made to incorporate 3D molecular information into LMs using external structure encoding modules, significant difficulties remain, such as insufficient interaction across modalities in pre-training and challenges in modality alignment. To address the limitations, we propose \textbf{3D-MolT5}, a unified framework designed to model molecule in both sequence and 3D structure spaces. The key innovation of our approach lies in mapping fine-grained 3D substructure representations into a specialized 3D token vocabulary. This methodology facilitates the seamless integration of sequence and structure representations in a tokenized format, enabling 3D-MolT5 to encode molecular sequences, molecular structures, and text sequences within a unified architecture. Leveraging this tokenized input strategy, we build a foundation model that unifies the sequence and structure data formats. We then conduct joint pre-training with multi-task objectives to enhance the model's comprehension of these diverse modalities within a shared representation space. Thus, our approach significantly improves cross-modal interaction and alignment, addressing key challenges in previous work. Further instruction tuning demonstrated that our 3D-MolT5 has strong generalization ability and surpasses existing methods with superior performance in multiple downstream tasks, such as nearly 70\% improvement on the molecular property prediction task compared to state-of-the-art methods. Our code is available at \url{https://github.com/QizhiPei/3D-MolT5}.

Published as a conference paper at ICLR 2025
3D-MOLT5: LEVERAGING DISCRETE STRUCTURAL IN- FORMATION FOR -ELECULE-TEXT MODELING
Qizhi Peit.2 Rui Yan Kaiyuan Gao
Jinhua Zhu
Lijun Wu
Gaoling School of Artificial Intelligence, Renmin University of China
2 "Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education SScoool of Computer Science, Wuhan University UUniversity of Science and Technology of China Shanghai AI Laboratory (qi zhi heip ruiyas rai ruc. edu. on im ka @hust. edu. cn tes la zhu@ma i ustc. edu. en apeterswup gma com
"Huazhong University of Science and Technology
ABSTRACT
The integration of molecular and natural language representations has emerged a focal point in molecular science, with recent advancements in Language Models (LMs) demonstrating significant potential for comprehensive modeling of both do- mains. However, existing approaches face notable limitations, particularly in their neglect of three-dimensional (3D) information, which is crucial for understanding molecular structures and functions. While some efforts have been made to incorpo- rate 3D molecular information into LMs using external structure encoding modules, significant difficulties remain, such as insufficient interaction across modalities in pre-trainn tre-training and challenges in modality alignment. To address the limitations, we propose 3D-MolT5, unified framework designed to model molecule in both sequence and 3D structure spaces. The key innovation of our approach lies mapping fine-grained 3D) substructure representations into specialized 3D token vocabulary. This methodology facilitates the seamless integration of sequence and structure representations molecular sequences, molecular structures, and text sequences within a unified architecture. Leveraging this tokenized input strategy, we build a foundation model that unifies the sequence and structure data formats. We then conduct joint pre- training with multi-task objectives to enhance the model's comprehension of these diverse modalities within shared representation space. Thus, our approach signifi- cantly improves cross-modas interaction and alignment, addressing key challenges in previous work. Further instruction tuning demonstrated that our 3D-MolT5 has strong generalization ability and surpasses existing methods with superior performance in multiple downstream tasks, such nearly 70% improvement the molecular property prediction task compared to state-of of thef-th-ann art methods. Our code is available at hntps:: /tte:://gittuu com Qi www.iiiieee// 3a-ii//30MMM1100 Mo1T5
tokenized format, enabling 3D-MolTS encode
INTRODUCTION
Molecule plays pivotal role in various scientific and industrial applications, spanning from pharma-
ceuticals to materials science (Drews) 2000; Dara et al.) 2022] Al4Science & Quantum (023). In
recent years, the development of Language Models (LMs) (Achiam al. [023] Touvron et al.] 2023]
Dubey et al. (2024) has garnered significant attention towards the joint modeling of molecule and lan- guage (Edwards et al. 2022) Zeng et al., 2022 Li et al., (202c). LMs trained on textual descriptions
of molecules can acquire comprehensive know ledgeg that enhances molecular understanding. thereby improving generalization to various molecule-related tasks, such as molecule-text retrieval (Zeng
et al. 2022: Su et al.) 2022) and molecule captioning (Edwards al. 2022 Liu et al., [023b Pei ct al. 2023). Language, inherently sequential, has inspired researchers to explore autoregressive
pre- pre-training of LMs for jointly modeling molecular sequences (c.g.. SMILES (Weininger 1988).
SELFIES (Krenn et al. 2020 2022h) and text sequences (Edwards et al
2022 /Zng et al.
2022
1 CCrrrsponndig authors: Rui Yan (ww.yan@ruc....... edu. en and Lijun Wu (apeterswu@@rra

---

Published as a conference paper at ICLR 2025
Pei et al.
2023). To incorporate 2D graph information, two primary approaches have emerged:
contrastive pre-training between 2D molecular graphs and text (Edwards et al. 2021 Su et al. 2022
Scidl et al. 2023; Luo et al.
2023 Liu et al. 2023a), and alignment of 2D molecular graph encoders
with LMs (Liu et al. 2023c
Cao et al. 2023 through multi-stage pre-training inspired by BIIIP2 (Li
al.
2023b).
However, most existing works have overlooked the molecular 3D structure, which contains crucial
stereochemical information for function-related tasks (Ruddigkeit et al. 2012
Hu et al. 2021
I|
Li
et al. 2023c; Wang et al. 2005). Few attempts Tang et al., 2023 Li et al. 2023c Xiao et al.
2024 Zhao al. (024) try to eliminate this limitation by integrating external molecular structure
encoders to incorporate the 3D molecular inputs with language. Through alignment training between
the external molecular structure encoders and LMs, these approaches achieve preliminary success, but notable shortcomings exist in these methods: (1) Insufficient interaction across modalities in pre-training: The molecular structure encoder is pre-trained separately from the text, resulting in inadequate interaction between different modalities during pre-training, (2) Challenges in modality aligument: The different representation spaces of pre- traitra molecular encoders and LMs require alignment training. However, modality alignment is always challenging (Baltrušaitis et al. 2018) for different reasons (e.g.. limited molecule-tex paired data, discrete text tokens, and continuous 3D structure representation). Besides, how assess the quality of alignment well remains unclear. (3) Dependency on external encoder: Though is efficient to incorporate the pre-trained external structure encoder, the performance of the encoder can not be directly controlled within the framework. which also raises difficulty in performance alignment.
To address these limitations, inspired by the joint multi-modal modeling in Vision-Language (Team) 2024; Xie et al.) [2024] (Zhou et al. 2024), we propose 3D-MolTS, a versatile T5 framework capable of understanding 3D Molecular structure to handle various 3D-dependent tasks simultaneously with text instructions. To enable LMs to comprehend 3D molecular structures, the crucial innovation is that we introduce a 3D molecular tokenization method based on the Extended 3D Fingerprint (E3FP) algorithm (Axen et al. 2017]. Specifically, E3FP tokenizes the 3D molecular structure into discrete 3D tokens, with each token encapsulating the 3D information of a substructure centered around a specific atom. Since most ID SELFIES tokens (e.g., [C] and [O]) represent specific atoms, the tokens from both ID and 3D modalities can be directly aligned at the atomic level. The embeddings of the same atom in both 1D and 3D tokens are then summed to form the final joint representation. In this way, it enables effective learning of molecular information by leveraging both sequence and structure tokens for molecules, allowing the representation of ID molecule, 3D molecule, and ID text modalities all using discrete tokens, such that all modalities can be easily trained in LMs. Therefore, we not only remove the dependency and requirement on external molecular structure encoders but also climinate the necessity of challenging modality alignment training.
With tokenized ID and 3D molecules, we conduct comprehensive molecule-text pre-training on our 3D-MolT5 framework. The pre-training tasks are inspired by the "TT objective" (Raffel et al. 2020), which employs a "recover masked spans" objective. In 3D-MolT5, we design five types of pre-training tasks: (1) ID denoising: Apply T5 objective to SELFIES, text, and wrapped text, where molecules mentioned the text are replaced with SELFIES. (2) ID + 3D joint denoising: Apply T5 objective to the summed 1D and 3D tokens, with the target being to recover the masked 1D SELFIES tokens. (3) 3D to ID translation: Given the 3D molecular tokens, generate the corresponding ID SELFIES. (4) 3D molecule to text translation: Given the summed ID and 3D tokens, generate its textual description. (5) Text to ID molecule translation: Given the textual description, generate the corresponding ID SELFIES. Consequently, our 3D-MolT5 pre-training allows for early and extensive interaction between different modalities in the pre-training stage, enhances representation, and better integrates information across modalities.
To verify our 3D-MolT5 framework, we conduct instruction tuning after pre-training on vari- ous molccule-text tasks, including molecular property prediction (both 3D-dependent and 3D- independent), molecule captioning (3D-dependent), and text-based molecule generation (3D- independent). The results show that both the Specialist (single-task tuned) and Generalist (multi-task tuned) versions of 3D-MolT5 achieve superior performance across these tasks. For example, on the 3D-ddependett molecular property prediction task with the PubChemQC (Maho) 2015) dataset, 3D-MoITS achieves an improvement of nearly 70% compared to 3D-MoLM (Li et al. 2023c). These results underscore the versatility and efficacy of our 3D-MolT5 in both 3D-dependent and 3D DD-independent molecule-text tasks.
2

---

Published as a conference paper at ICLR 2025
aDOOo
<x> |Branch [[C][o)
SELFIES Denaising
The proportion patients aged over
75 years <X> and a positive
<X> had negative relationship with statins and ACE inhibitors <Y>
Text
association with aspirin.
Recover Masked Spans
Denoising
Interactions S (-2-haloethh))- <X> +
Wrapped
||Branch SJ+Y= [O] analogs with plasmid DNA.
<X> mercapturic acid <Y> SSICICCIOICI//saacch]]C1-0000
Text
Denolsing
cooOa
[O] [C [Branch1] |=O] <y-Ring Br=aanchhh)
3D- MolT5
[- <x>
<Y>
SELFIES
%
|=C]C[|=C]
3D Structure Denoising..
CISICICICC@ [C]N]]]][ ([NNIC[--Bancch]jj]e Branch1]jQ[[-jj0
3D to 1D
[3D]3D0]3D)..13D
Translation
00000
- C[C][-C
Translation
Phenylethanolamine also known 2-amino-1-phh 1-peeyllethhnoo is chemical compound with the molecular formula CBH1 1NO. It belongs to the class phenylethanolamins and the simplest member of this group...
[Ring I|[= Branch1]
Molecule to Text Translation
Carbonyl sulfide (COS) is colorless and poisonous ges characteristic sulfide odor. It flammable presents a moderate fire hazard due to its ability rupture violently under prolonged exposure intense heat...
[O][-C[[-S]
Molecule Transtotion
Overview of the 3D-MolT5 multi-task pre-training. The upper 4 tasks involve the "recover masked spans" task, where consecutive spans of the input are replaced with sentinel tokens such as <x>, <Y>, <Z>, The bottom 3 tasks are translation tasks. The input modalities are annotated with small icons. Tokens with 3D structure information are colored in blue, and [3D] refers 3D tokens.
Figure I:
RELATED WORK
Molecular Encoding. The 1D sequence is the most widely used form of molecular encoding, typically obtained by traversing the atoms in molecular graph in a specified order. The simplified molecular-input line-entry system (SMILES) (Weininger] 1988 Weininger et al. 1989) is the most
common, while Self-Referencing Embedded Strings (SHLFIES) (Krenn et al. 2020 2022) has recently gained popularity due to its robust nature. 2D graph representations align naturally with molecular topological structures, as molecules inherently form 2D graphs with atoms serving as nodes and chemical bonds as edges (Guo al. (023). In contrast, 3D structures provide information about the spatial arrangement of atoms, offering valuable insights into molecular geometry and interactions. Molecular fingerprints (FPs) are also widely used, especially in molecular similarity searches and virtual screening (Cereto-Massagué et al. 2015). FPs encode critical information about molecular structure as sequence of binary bits, which are useful for property predictions (Jeon & Kim 2019 Wen et al., 2022). Common examples include Morgan FPs, such as detended-connectte fingerprints (ECFPs) and functional class fingerprints (FCFPs) (Rogers & Hahn 2010a). as well as RDKit (topological) fingerprints (Landrum et al. (023). However, these fingerprints primarily capture 2D topological features and do not account for 3D structural patterns. Spherical extended 31) fingerprints (E3FPs) (Axen et al.] 2017) effectively incorporate neighboring atoms in 3D space to encode 3D information. Our structure-aware 3D molecular vocabulary built on E3FP (Axen et al.. 2017). which is then used for our atom-centric joint representation.
Molecule-Text Cross Modeling. Recent advancements have integrated LMs with molecules to
enhance the understanding of molecular structures and properties (Zhang et al. 2024b Pei et al.
2024b Taylor al.) 2022). MolT5 (Edwards et al. 2022). BioT5 (Pei al. 2023) and Bio (Pei et al. 2024a) are basba (Raffel et al. 2020) models that jointly trained on ID molecular sequences and text sequences, followed by tne-ttnnng for tasks related to molecules. Mol-Innstuuttios (Fang et al. (023) and LlaSMol (Yu et al., (024) offer instruction datasets where molecules are represented as SMILES or SELFIES for instruction tuning. Additionally, 2D molecular graphs have been utilized to infuse topological knowledge into LMs via external graph encoding modules. For instance, MoMu (Su et al. 2022). MoleculeSTM (Liu et al. 2023a),, and MolFM (Luo et al., 2023) employ cross-modal contrastive learning on 2D molecular graphs and corresponding text. MolCA (Liu et al.) 2023cb and MolX (Le et al. 2024) align 2D molecular space with text space through cross-modie tre-training. UniMoT (Zhang et al. 2024a) proposes Vector Quantization-drinn convert the 2D molecular graphs into molecule tokens, aiming at unified modeling of molecule and text. Recent endeavors have also incorporated 3D molecular information. MolBind (Xiao et al. 2024
tokenizer to
3

---

Published as a conference paper at ICLR 2025
Iteration
Iteration
Iteration 2 [converged]
SELFES
Token
[0] [+[]
[4]
[+4]
[Bracch]] (=Branch1)
K
[-Branch]]
[3
14
[+[]
(irati (-8runch2)
COOO
2729 1927
1755
1530
1344
501
Level
1310
1310
1492
2756
1409
670
3452
670
1051
4046
4066
1254
Loodl
-1
-1
Figure 2: The process of 3D molecular tokenization and alignment between ID SELFIES tokens and 3D tokens. We choose one conformer of the example. At each iteration of E3FP, each atom and its neighborhood substructure represented by 3D token. The alignment between ID SELFIES tokens and 3D tokens is shown at the bottom table.
acid (CID: 101399) as the
for example, employs contrastive learning to align the 2D graph encoder, 3D structure encoder, and language encoder, demonstrating strong performance in cross-modal retrieval tasks. In line with the BLIP2 (Li al., (203b) paradigm, 3D-MoLM (Li al. 2023c) equips the LM with external Uni- Mol encoder (Zhou et al. 2023) and curates the 3D-MolT dataset for 3D molecule-text instruction tuning. 3D-MoLM combines ID SMILES and 3D molecular representations for 3D molecule-to- text interpretation. Nonetheless, these methods do not unify the modeling of molecular sequences, molecular structures, and text sequences, as the 3D molecules are encoded by an external module, posing challenges in attaining comprehensive integration of multimodal molecular information.
METHODS
The overview of 3D-MolT5 is shown in Figure We first introduce the sequence representation of molecules in Section 3.1 as is the preliminary knowledge for our molecular tokenization. In Section 3.2. we present the 3D structure-aware E3FP fingerprint and how we adopt it in 3D- MolT5. We then introduce our 3D molecular tokenization and the integration with ID tokenization in Section 3.3 Lastly, we present our multi-task pre-training framework in Section 3.4
3.1 ID SEQUENCE REPRESENTATION
The ID sequence representation for molecules lays the groundwork for our molecular tokenization (Section 3.3) hence we give the necessary descriptions here. In this work, for given molecule M, we use SELFIES (Krenn al., 2020) as its sequence representation, offering enhanced robustness and validity compared to SMILES (Weininger) 1988 Weininger al., 1989). In SELFIES, each token, generally denoting an atom group (like [C] and [=N] or structure directive (like [Ringl] and [-Branch1] is enclosed within brackets, facilitating straightforward tokenization based these demarcations. Therefore, the ID SELFIES sequence can be represented as { 1 where 8; a SELFIES token and m denotes sequence length[] To ensure that each molecule has unique SELFIES, and thus unique atom flatten order, we employ the canonical form of SELFIES.
3.2 3D STRUCTURE-AWAR FINGERPRINT
In our 3D tokenization, the focus is transforming the continuous 3D molecular structure into discrete tokens. To achieve this, we leverage the 3D molecular fingerprint E3FP (Axen et al. 2017).
which efficiently converts 3D structures into discrete identifiers, enabling tokenized representation of spatial molecular information. For molecule conformers, the E3FP algorithm generates 3D fingerprint F, represented a bit vector, with F indicating its length. We transform the atoms of M into canonical atom sequence A {a, f==0 where a; represents one of the heavy atoms. For each atom a; with & representing the number of iterations for the E3FP algorithm, identifiers. This process, illustrated in Figure 2 and Algorithm (some details and special cases
composed n atoms and one its 3D
derive its 3D token d, composed of k + 1 non-negative
Typically, m is large than the number of atoms n, in SELFIES there some structure directive tokens.
4

---

Published as a conference paper at ICLR 2025
omitted and provided in the Appendix A encompasses several steps as follows. (1) Structure representation initialization. At iteration O, we establish a set of atomic invariants A for each atom a, within M, including attributes such atomic number, the number of immediate neighbors, and whether the atom is part of a ring, among others. These atomic invariants are hashed into identifier ds, by MurmurHash3 (Apleey 2016) algorithm to create unique identifiers for each atom a, (2) Iterative spherical shell expansion. In each iteration j. we expand the radius of the spherical shells centered on each atom a; by radius multiplier r, including all atoms within the expanded radius R j. The connectivity identifier, of, derived from Connectivity. and the stereochemical identifier, si. obtained from Stereochemistry, of these neighboring atoms are encoded relative the central atom, incorporating both bonded and non-bonded interactions. These identifiers allow E3FP to capture the 3D molecular structure, including relative atomic orientations and distances, which are absent in 2D representations. A detailed explanation of Connectivity and Stereochemistry is provided in Appendix A.I The current iteration number j. the identifier of a; from the previous iteration, and the neighbors' information (ck, d s are combined and hashed into the identifier dr,j- Each iteration produces a new layer of structural information, continuing until either a predefined maximum number of iterations k is reached or all atoms in M are included in the current shell. (3) Folding. For each atom a;, we aggregate its substructure information from each layer by concatenating the hashed identifiers, resulting in the vector d; [dr.o. d,......, d.,.). Each 32-
Algorithm E3FP Algorithm. U represents the concatenation operation.
1: Input: Molecule M, maximum iteration num- ber k, shell radius multiplier
bit element of d, is then "folded" down to [F] bit by applying the modulo operation, mathemat- ically represented as d, d, mod (F.. In 3D- MolT5, we do not use the E3FP F directly but in- stead employ the 3D tokens d, for each atom a; which are then integrated with the ID sequence. More details and analysis about the E3FP algo- rithm, including connectivity and stereochem- istry encoding, 3E33)-nnvariance, time and space complexity, the hyperparameter settings, special cases, a specific example for better illustration, information loss brought by discrete representa- tion, are introduced in Appendix A
2: Initialize D 5- O
3: // Step 1: Structure Representat ion Initialization 4: for each atom a, in M do
5: Initialize atomic invariants A,
6: di,0 MummurHash3(A)
7: d, f -d.ol
8: end for
9: // Step 2: Iterative
Spherica Shell Expansion
10: for iteration j 1 to do
11:
R+r rj
12:
for each atom a; in M do
3.3 MOLECULAR TOKENIZATION
13:
L + d. [ل,d
14:
for each neighbor og within radius R do
After obtaining the 1D SELFIES sequence S =
15:
c + Connectivity(a;; a:)
{s; {s and the sequence of 3D tokens D
16:
S + Stereochemissryy(as, a:)
[d. [=p) we combine them to form the final
17:
L + LU dx.
ID + 3D joint representation. As depicted in
18:
end for
Figure for each molecule, most ID SELFIES
19:
dij <- MurmurHash33 T.)
tokens 8; uniquely represent an atom o, Sim-
d, d dg u [dras]
ilarly, each 3D token d, a vector of non-negative identifiers, also uniquely
dimensional
20:
21: end for
corresponds to an atom a, Thus, tokens from
22: end for
both ID and 3D modalities can be aligned at the atomic level. Based on this alignment, we con-
23: // Step 3: Folding
24: for each atom a, M do
25: d; - d4 mod [F]
struct the ID + 3D joint representation, captur-
ing both the chemical sequence and the spatial configuration of the molecule.
26: D DUDUd;
27:
end for
28: Output: 3D structure identifier matrix D
We define the ID embedding EID for the 1D SELFIES tokens and the 3D embedding EaD for the 3D tokens. The 3D embedding EaD is directly indexed by the 3D tokens, as each component of d, is a non-negative integer. For each molecule, each SELFIES token 8; is mapped to its corresponding embedding vector (sp IR where H denotes the hidden dimension. For the 3D token embeddings, we map each component d in d, to the vector EaD (d.)) E H These k + 1 vectors EsD (di;;))) )) are then averaged to compute the 3D embedding for token d,, given by (d)( = (1/k + 1) Σ EaD Eap(di.) (di.)) The final joint representation E for each
5

---

Published as a conference paper at ICLR 2025
token is determined based on the available information: if only ID information is present, E EID; only 3D information is available, E = EsD: and if both 1D and 3D information are present, then E EiD + & EnD This combination captures both the sequential and spatial information of the molecule, producing comprehensive representation suitable for various downstream tasks.
3.4 PRE-TRAINING
Using the molecular tokenization approach described above, now we can train LMs on text sequences. molecular sequences, and also molecular structures all in tokens. Our LM backbone is T5 (Raffel (et al. 2020), a transformer-baset encoder-decoder architecture, which serves as the foundation for design two categories of tasks within a multi-task framework: (1) self-supervised denoising tasks aimed at recovering masked spans, and (2) translation tasks between different modalities to further enhance the model's 's capability (see ablation study in Section 5 Additional details regarding pre-training, including loss functions, configurations, and datasets, are provided in Appendix [
3D-MolT5. Detailed model configurations are provided in Appendix D For pre-training,
Denoising Tasks. The denoising pre-training tasks are divided into two categories based on input modalities: (1) ID denoising, which includes denoising on SELFIES, text, and "wrapped" text. For SELFIES, we random sample 50M molecules from the PubChem (Kim et al. 2019) database and
represent them as canonical SELFIES. For text, we use both the C4 (Raffel et al. [2020) dataset from the general domain and full articles from PubMed Central (Canese & Weis [013] White 2020) in
the biomedical domain. The concept of "wrapped" text is adapted from MolXPT (Liu et al., 2023b where molecules mentioned in the text are replaced with their SELFIES. As demonstrated in Liu al. (2023b); Pei et al. (2023) 2024a). training on such "wrapped" text beneficial, as the context rich with molecular descriptions. We follow the same pipeline as MolXPT to detect molecules mentioned in the text using BERN2 (Sung et al. 2022) on the PubMed abstracts, appending them with their corresponding SELFIES. (D) ID + 3D joint denoising, which involves denoising the combined ID SELFIES tokens and 3D tokens, aiming at recovering the ID SELFIES token We use the PCQM4Mv2 dataset from the OGB Large Scale Challenge (Hu et al. 2021) for this task, which includes 3.37M DFT-calculated (Geerlings et al.) 2003) 3D molecular structures.
Translation Tasks. We incorporate three translation tasks simultaneously to further bridge different modalities: (1) 3D to ID translation. We use the same PCQM4Mv2 dataset as in the ID + 3D denoising. The input the sequence of 3D token representations EsD of the molecule, and the output is the corresponding ID SELFIES. (2) 3D molecule to text translation. We use the pre-training split of the PubChem dataset collected by 3D-MoLM (Li al. 2023c). which contains 298K 3D molecule-texi pairs from the PubChem (Kim et al.) 2019) database. The input is the combined ID and 3D tokens of the molecule, and the output is the corresponding textual description. (3) Text to ID molecule translation. We use the same data as (2), but the input is the textual description of the molecule, and the output is the corresponding ID SELFIES.
EXPERIMENTS
We evaluate 3D-MolT5 on three types of text-based molecule-related downstream tasks: (1) molecular property prediction, including both 3D-independent properties (e.g., molecular weight, LogP) and 3D-dependent properties (e.g., HOMO-LUMO gap); (2) 3D molecule captioning; (3) text-based molecule generation. In Appendix B we further validate the effectiveness of 3D-MoIT5 across additional tasks and benchmarks. All downstream data is formatted as instructions, with tasks framed as conditional text or molecule generation based on the input instructions. The training objective remains the standard cross-entropy loss, consistent with pre-training.
Following 3D-MoLM (Li al. 2023ch and Mol-Inntructions (Fang al. 2023), we present results for two variants of 3D-MolT3: Specialist, fine-tuned for a specific task; and Generalist, fine-tuned in multi-task setup. To ensure fair comparisons, we use the same multi-task setup as the baseline models. More details the downstream datasets, Generalist settings, and baseline methods are provided in Appendix
In our work, since each 3D molecular token for atom contains & + 1 different components, we do not attempt to predict the 3D molccular tokens for denoising tasks, is also the case for the translation tasks.

---

Published as a conference paper at ICLR 2025
Table 1: MAE results of computed property prediction tasks on PubChem (Kim et al. 2019) and PubChemQC (Maho 2013) datasets. The valid answer rate is also reported as LMs may fail to generate valid numerical responses. 3D-dependent properties are colored in blue. i refers to a variant of 3D-MoLM (Li et al. 2023c) that is initially pre-trained on the original PubChem text without GPT-3.5 enrichment. represents no fine-tuning.
DATASET
PUBCHEM
PUBCHEMQC
MODEL
WEIGHT (G/MOL)
LoGP
TPSA (A9)
COMPLEXITY
HOMO (EV)
LUMO (EV)
H.L G.P (AV)
Non-LM
Uni-Mol
20.35
0.59
13:48
57.24
0.32
0.35
0.21
Spectallst
Llama2-7B
(96%)
(95%)
15.87 (92%)
(93%)
1.24 (96%)
1.4 (95%)
0.88
2D-MoLM
21.48 (94%)
0.88 (96%)
13.52 (92%)
55.74 (94%)
0.92 (98%)
0.80 (96%)
0.67 (93%)
3
(96%)
0.95 (964))
(94%)
(95%)
(981))
0.36 (96%))
0.41 (94%)
3D MoL.
14.79 (95%)
0.66 (975))
9.71 (93%)
44.85 (94%)
0.26 (97%)
0.25 (94%)
0.28 (94%)
3D-MolTS
12.30 (1001)
0.44 (100%))
3,93 (100%)
29.51 (100%)
0.08 (100%)
(10018)
0.08 (1001))
Generalist
Alama2-7B+
42.18 (82%)
2.10 (85%)
27.11 (84%)
121.87 (76%)
2.87 (70%))
1.89 (71%)
1.86 (70%)
Llama2-7B
27.42 (92%)
1.78 (93%)
17:07 (90%)
78.16 (92%%
1.89 (90%)
1.26 (90%)
1.25 (91%)
2D M
20.80 (92%)
1.36 (94%)
12.47 (89%)
52.70 (915)
1.52 (9312)
1.13 (92%)
1.09 (88%)
3f
19.54 (9.9))
0.92 (925)
11.14 (92%)
54.68 (90%)
0.65 (94%)
0.41 (92%)
0.55 (89%)
3D-MoLM
16.58 (92%)
0.78 (95%)
10.90 (90%)
45.49 (891))
0.35 (9512)
0.36 (93%)
0.32 (90%)
3D-MolTS
(1009(1008))
0.61 (100%)
6.37 (100%)
37.59 (100%)
0.11 (100%)
0.11 (100%)
0.11 (100%)
4.1
MOLECULAR PROPERTY PREDICTION
Following 3D-MoLM eLi al. 2023c), we assess 3D-MolT5 on two types of molecular property
prediction tasks: (1) Computed property prediction: We focus the MAE performance by extracting the predicted numerical value of the property from the generated text. (2) Descriptive property prediction: We evaluate text similarity metrics between the predicted text and the ground truth.
4.1.1 COMPUTED PROPERTY PREDICTION
Setup. We use three datasets evaluate the performance of 3D-MolT5 on computed property prediction task: QM9 (Ruddigkeit et al. 2012 Fang et al. 2023). PubChemQC (Maho 2015
Xu
al. 202T). ) and PubChem (Kim et al. 2019). The QM9 (Ruddigkeit al. 2012 Fang al. 2023)
dataset contains over 130,000 molecules with ground-star state 3D structures obtained through DFT computations (Geerlings et al. 2003), each molecule having fewer than nine heavy atoms. This dataset is widely used for quantum property prediction, including HOMO, LUMO, and HOMO-LUMO gap (H-L gap). The PubChemQC (Maho 2015] Xu et al. (02T) dataset is larger in scale, containing 3.37M molecules with more heavy atoms, along with their DFT-calculated (Geerlings et al. 2003) 3D structures. We use the same quantum properties as QM9 for PubChemQC. The PubChem (Kim et al. 2019) database also provides various molecular properties. Following 3D-MoLMM (Li et al. 2023c), we select four properties that can be inferred from 1D or 2D molecular information.
For the QM9 dataset, ChemQC and PubChem datasets, we use instruction data constructed by 3D-MoLM
instruction data from Mol- Mol-Innttustions (Fang et al.
2023). For Pub- (Li et al. 2023c)
Baselines & Evaluation. We compare 3D-MolTSS against three types of baseline models, categorized by input modalities. For 1D sequence-based models, we include Llama2-78 (Touvron et al. 2023).
Vicuna (Chiang al., 22023). Mol-Instructions (Chiang et al. (023), and BioT5+ (Pei al. 2024a)
For 2D graph-based models, we incorporate 2D-MoLM (Li et al. (023c) and InstructMol (Cao et al.
2023). For the 3D structure-based model, we compare against Uni-Mol (Zhou et al. 2023 and
3D-MoLM (Li et al.
2023c). Note that models based on 2D graphs or 3D structures can also accept
1D sequences as input (Li al.
2023c Cao et al. 2023).
Results. The computed property results for the PubChem and PubChemQC datasets are presented in T and for QM9 in Table Key findings from the results include: (1) 3D-MolTS outperforms all baseline methods on 3D-independent properties in the PubChem dataset (Kim et al. 2019). For molecular hydrophobicity (LogP), which depends on ID and 2D features such as functional groups, molecular connectivity, and topology, 3D-MolT5 consistently surpasses the LMs trained on ID, 2D, and 3D molecular information. (2) 3D-MolT5 exhibits substantial improvements in predicting 3D-dependent properties. For energy properties including HOMO, LUMO. and HOMO-LUMO gap. which are primarily determined by 3D molecular structures, 3D-MolT5 shows significant performance enhancements. For the Specialist version the PubChemQC (Maho 2015) dataset,
7

---

Published as a conference paper at ICLR 2025
Table 2: MAE results on computed property prediction tasks on QM9 * means direct inference without further fine-tuning.
(Ruddigkeit et al.
2012 dataset.
MODEL
HOMO (HA)
LUMO (HA)
H-L. GAP (HA)
(HA)
Generulist
Llama2-788 78 (5-shot)
0,7367
0.8641
0.5152
0.7510
Vicuna-13B (5-shot)
0.7135
3.6807
1.5407
1.9783
Mol-Instructions
0,0210
0.0210
0,0203
0.0210
BinTS+
0.0012
0.00224
0.0028
0.00125
InstructMol-G
0,0060
0.0070
0,0082
0.0070
InstructMol-GG
0.0048
0.0050
0.0061
0.0050
3D-MolT5
0.0017
0.0016
0.0016
0.0017
Table 3: Results of the descriptive property prediction task on PubChem (Kim et al., 2019 dataset. refers variant of 3D-MoLM eLi 2023c) that is initially pre-trained the original PubChem text without GPT-3.5 enrichment. means direct inference without further fine-tuning.
MODEL
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Specialist
Llama2-71 7BB
28.15
23.24
35.14 14
22.08
30.41
46.87
2D-MoLM
30.84
25.09
38.46
24.22
33.04
50.92
3D-MoLM1
30.33
24.47
23.93
32.98
51.33
3D M
32.00
26.13
40.13
25.55
34.64
52.15
3D-MolT5
51.24
43.06
52.48
56.97
Gewernlist
Llama2-78*
25.22
21.16
19.21
25.22
43.17
Llama2-7B
27.68
22. 22.81
34.73
21.55
29.91
46.39 39
2D-MoLM
30.23
24.57
37.85 85
22.95
32.28
50.08
3D-
3D 3D-MoLM
31.81
26.08
40.13
25.87
34.99
51.93
30-MoTTS
.9.84
41.42
54.23
38.45
50.39
34.76
the improvements over the previous SOTA methods (Zhou et al., 2023 Li al. 2023c) are 0.18 eV, 0.17 eV, and 0.13 eV, respectively. For the Generalist version the QM9 (Ruddigkeit al.
2012) dataset, the average improvement is 0.0008 Ha. (3) Compared to Uni-Mol (Zhou et al. 2023) 3D-MolT5 exhibits consistent improvements. Uni-Mol (Zhou et al., 2023) is specially designed for 3D molecular representation learning and pre-trained on large-scale (209M) 3D molecular data. The superiority of 3D-MolT5 demonstrates the benefit of unified 3D molecule-text modeling. By integrating structural knowledge from molecules with contextual knowledge from biological literature through comprehensive pre-training. 3D-MolT5 enhances its generalization molecular property prediction tasks. (4) 3D-MolT5 also continuously improves upon 3D-MoLM eLi al. (002c)), which integrates the Uni-Mol (Zhou et al. 2023) molecular encoder with Llama2-7B (Touvron al. (023) as the language decoder through a projector for 3D molecule-text interpretation. The superiority of 3D-MolT5 can be attributed to the extensive interaction between 3D molecular structure and text during multi-task pre-training and our joint tokenization, which significantly improves 3D- MolT5's s ability to handle complex fine-grained 3D molecular structures and enhance cross-modal understanding. (5) On PubChemQC and PubChem datasets, the Generalist version of 3D-MoITS also outperforms all baselines, though it slightly underperforms compared to the Specialist, likely due to task conflicts in multi-task training. These results demonstrate 3D-MolT5's ability to handle multiple tasks concurrently as Generalist.
4.1.2 DESCRIPTIVE PROPERTY PREDICTION
Setup. For the descriptive property prediction task, we evaluate the performance of 3D-MolT5 on PubChem dataset, as constructed by 3D-MoLM (Li al. (2023c). Unlike computed property prediction, this task involves generating natural language descriptions of molecular ID, 2D, and 3D properties, necessitating accurate and contextually relevant textual output.
Baselines & Evaluation. We compare 3D-MolT5 against the same baselines used for the Pub- ChemQC (Maho) 2015) and PubChem (Kim et al.] (2019) datasets as described in Scction 1.1. Following MolT5 (Edwards al. 2022), we employ widely used text generation metries. including BLEU (Papineni et al.) 2002), ROUGE (Lin) 2004), and METHOR (Banerjee & Lavie 2005), to assess the similarity between the generated property descriptions and the ground truth.
Results. The results for the PubChem dataset are shown in Table From the table, we have several observations: (1) 3D-MolT5 outperforms all baseline methods. The Specialist version
8

---

Published as a conference paper at ICLR 2025
Table 4: Results for the 3D molecule captioning task on PubChem (Kim et al. 2019) dataset. f refers variant of 3D-MoLM (Li et al. 2023c) that is initially pre-trained on the original PubChem text without GPT-3.5 enrichment.
MODEL
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Specialist
MorrS-Large
25.87
17.28
34,07
16.42
23.41
28.04
MoMu- Large
26.34
18.01
34.75
16.86
24.76
28.73
3D-MoLMF
29.82
22.39
37.23
22.49
31.07
32.69
3D-MoLM
30.32
22.52
36.84
22.32
31.23
33.06
MolX
31.40
24.25
44.20
28.96
38.76
39.55
3D-MofT5
42.05
34.16
48.13
33.20
42.33
44.69
Generolist
Llama2-7B
20.94
35.76
20.68
2D-MoLM
27.15
21.19
36.02
20.76
29.12
32.28
M
3D-MoLM
28.95
21.63
36.51
21.26
30.02
33.55
3D-MofTS
43.11
34.98
48.69
33.54
42.72
44.82
shows significant improvements, with BLEU-2 and ROUGE-L scores increasing by 19.24 and 17.84, respectively. The Generalist version also shows substantial gains, with BLEU-2 and ROUGE-L improvements of 18.03 and 15.4 points, respectively. (2) The Generalist version of 3D-MolTS surpasses all baselines but slightly underperforms compared to the Specialist version.
4.2 3D MOLECULE CAPTIONING AND TEXT-BASED MOLECULE GENERATION
Despite the molecular property prediction tasks, also evaluate our 3D-MolT5 on 3D molecule captioning and text-based molecule generation tasks.
4.2.1 3D MOLECULE CAPTIONING
Setup. We use PubChem eL et al. 2023c) dataset for 3D molecule captioning to evaluate 3DD-MolT5's ability to understand the 3D molecular structure. This dataset contains approximately 15,000 3D molecular structure-text pairs sourced from the PubChem database (Kim et al. 2201).. Specifically, the molecular captions include both molecular names and 3D-related descriptions assess the model's 's capability in name prediction (Favre & Powell 2013) and description prediction (Edwards)
al. 2022), offering more comprehensive evaluation compared to the descriptive properties.
Baselines & Evaluation. The compared baselines are classified into three categories based on input modalities. For 1D sequence- based models, we include MolT5 (Edwards et al. 2022) and Llama2-7B (Touvron et al. 2023). For 2D graph-based models, we compare against MoMu (Su et al.) 2022), 2D-MoLM (Li et al. 2023c), UniMoT (Zhang et al. 2024a), and MolX (Le et al. 2024). For incorporate 3D-MoLM (Li et al. 2023c). Note that the 2D graph or 3D structure- based models may also take ID sequence as input simultaneously. The evaluation metrics remain consistent with those in Section
3D structure-based models,
Results. Table shows the results for the 3D molecule captioning task. 3D-MolT5 demonstrates superior results, surpassing all baseline methods. Compared to 3D-MoLM (Li et al. 2023c), 3D- MolTS achieves an improvement of approximately 11 points in ROUGE-L and METEOL EET scores for both the Specialist and Generalist versions. Furthermore, 3D-MolT5 also exceeds baselines with ID and 2D information, highlighting the importance of 3D structure information for molecule understanding and captioning. This further validates the efficacy of our unified pre-training on ID SELFIES, 3D structure, and text with 3D molecular tokenization.
4.2.2 TEXT-BASED MOLECULE GENERATION
Setup. To further demonstrate the capability of 3D-MolT5, also evaluate its performance on the text-based molecule generation task. We use the CheBI-20 (Edwards et al. 2022 dataset, which is
widely used for this task (Edwards et al. 2022 Luo et al. 2023 Liu al. 2024 2023b Pei et al.)
2023). The imput for this task is the textual description of the molecule, and the target is to generate a ID molecular sequence that fits the description.
Baselines Evaluation. The compared baselines include Llama2-7B (Tovrro et al. 2023), GPT- 3.5 OOpeAA 2023), GPT-4 (Achiam al. 2023). T5 (Raffel al. 2020). Mol75 (Edwards et al.

---

Published as a conference paper at ICLR 2025
Table 5: Results on text-guided molecule generation task on ChEBI-20
[Ewwards el al. 2022) dataset.
MODE.
BLEUT EXACT?
LEVENSSITIIN
MACCS FTS"
RDK FTS?
MORGAN FTST
PCD]
Texe2Mout
VALIIIT
Geseralivt
Llam2277 76 (0-shot)
0.104
0.000
B4.18
0.243
0.119
0,089
42.01
0.148
0.631
Uama2-78 (2-shot)
0.693
0.022
36.77
0,808
0.717
0,609
4,90
0.149
0.761
3,5 surbo (0 (0-sbot)
0.489
0.019
52,13
0.705
0.462
0.367
2,05
0.499
0,802
Ga---3-5-arh (10-4hot)
24.910
0.847
0.887
GPT-4-0314 (10-shot)
0,903
0.805
0.593
0.899
Specsillist
T5-buse
0.762
0.069
24,950
0.73)
0.605
0.545
2.48
0.499
0.660
115-lagee
0.854
0.279
16.721
0.823
0.731
0.670
1.22
0.552
0,902
MoaTT--huse
0.769
0.081
24.458
0.721
0.588
0.529
2.18
0.496
0.772
MoalTS-larg
0,854
0.311
16.071
0,834
0.746
0.684
1,20
0.554
0.905
MoMu base
0.815
0.183
20.520
0.847
0.737
0,678
0.580
0.863
MollMd
0.210
19.445
0.854
0.758
0.697
0.583
0,892
GiT-Mel
0.756
0.051
26.315
0.758
0.582
0.928
MoXPP
0.215
0.859
0.757
0.667
0.578
0.983
BiaTS
0.867
0.413
15,097
0.886
0.801
0.734
0.43
0.576
1.000
3D-MolTS
0.549
0.487
10.527
0.884
0.506
0.744
0.41
0.574
1.000
2022), MoMu (Su et al.
2022), MolFM
(Luo et al. 2023), GIT-Mol (Liu et al. (024). MolXPT (Luu
Following (Edwards et al. 2022), the evaluation metrics include BLEU (Papineni et al. 2002). exact match score, levenshtein distance, fingerprint
2023b), and BioT5 (Pei et al.) 2023).
similarity (Durant et al. [0022] Landrum et al. 2023 Rogers & Hahn 2010b), FCD score (Preuer
et al. 2018 text2mol (Edwards et al. 2021) score, and validity.
Results. The results are presented in Table 5 Our 3D-MolT5 outperforms all compared baselines across most metries. Notably, 3D-MolT5 achieves an exact match score of 0.487, indicating that nearly 50% of the generated molecules exactly match the ground truth molecules. These results underscore that 3D-MMolT5 has acquired comprehensive molecular knowledge during tre-training, enabling it to effectively generate accurate molecular sequences based on textual descriptions.
ABLATION STUDY
3D MoTS
30 Irpa:
-10 dencsing
-Th pren
To validate the efficacy of 3D molecular to- kenization and multi-task pre-training, we conduct ablation studies focused on property prediction using the PubChemQC (Maho, 2015) dataset, a task heavily reliant on 3D structural information. The ablation results are shown in Figure
0.100
0.3463
2.0000
0.095
3.5923
5,0906
0.090
5.0855
(ev)
0.085
3] More case studies are
5,0028
shown in Appendix
0.080
5,0750
07079
Whether 3D input truly help? To assess the impact of 3D input, we pre-train and fine- tune a variant of 3D-MolT5 excluding all the 3D structure information. As illustrated in Figure 3 removing 3D information leads to performance drop on the 3D-dependent property prediction task. For example, the MAE for the HOMO-LUMO gap increases from 0.0791 to 0.0968. This indicates that integrating 3D structure information into LMs can enhance their understanding of the molecule.
0.075
0.070
HOMO (eV)
LUKO (eV)
H-l Gap (eV)
Figure 3: Ablation studies on PubChemQC Maho
2015) dataset. The evaluation metric is MAE.
Whether 3D-related pre-training help? To demonstrate the efficacy our 3D-related pre-training, we remove the 1D 3D joint denoising task and translation tasks separately. The results in Figure indicate that both of them contribute to improvements in 3D-related downstream tasks, underscoring the importance of incorporating 3D information into the pre-training process.
6 CONCLUSION
In this paper, we introduce 3D-MolT5, a unified framework that integrates molecular sequences, molecular structures, and text sequences, to enhance the capabilities language models in handling various molecular tasks. By proposing a 3D molecular tokenization method, we can effectively map 3D structures to 3D tokens. The combination of 1D SELFIES tokens and 3D tokens enables a comprehensive representation of the molecule. Through extensive pre pre-training on ID and 3D data and subsequent instruction tuning. 3D-MolT5 demonstrates superior performance in molecular property prediction, molecule captioning, and text-based molecule generation tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank the reviewers for their insightful comments. This work was supported by Beijing Outstanding Young Scientist Program NO. SUIW2YJH012011100200000 and the Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double- First Class" Initiative, Renmin University of China. Qizhi Pei supported by the Outstanding Innovative Talents Cultivation Funded Programs 2023 of Renmin University of China. Qizhi Pei is an intern at Shanghai AI Laboratory.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altan, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774 2023.
Microsoft Research A14Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4, arXiv preprint arXiv:2311.0736/ 2023.
Austin Appleby. Murmurhash3,
2016.
Seth D Axen, Xi-Ping Huang, Elena L Cáceres, Leo Gendelev, Bryan L Roth, and Michael Keiser. simple representation of three- diceeniinnii molecular structure. Journal of medicinal chemistry, 60(17):7393-7409 2017.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philipee Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41(2): 423 423-443, 2018.
Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin- Yew Lin, and Clare R. Voss (eds.), Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization @ ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pp. 65-72. Association for Computational Linguistics, 2005. URL htttts://aclaatthlllloo
org/w05 /05 09097
Kathi Canese and Sarah Weis. Pubmed: the bibliographic database. The NCBI handbook, 2(1), 2013.
He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. Instructmol: Multi-modal integration for build- ing versatile and reliable molecular assistant in drug discovery. arXiv preprint arXiv:2311.16208. 2023.
Adrià Cereto-Massagull María José Ojeda, Cristina Valls, Miquel Mulero, Santiago Garcia- Vallvé, and Gerard Pujadas. Molecular fingerprint similarity search in virtual screening. Methods, 71: 58 58-63, 2015.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL |RLhttps
/
03 30 vicuna/
Suresh Dara, Swetha Dhamercherla, Surender Singh Jaday, Ch Madhu Babu, and Mohamed Jawed Ahsan. Machine learning in drug discovery: A review. Artif. Intell. Rev. 55(3):1947 1999, 2022. doi:
URL https:://oi..rgg/
s104
Jurgen Drews. Drug discovery: historical perspective. science, 28765460) 2000.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The laama herd of models. arXiv preprint arXiv:2407.21788 2024.
11

---

Published as a conference paper at ICLR 2025
Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42(6): 1273-1280, 2002.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 1, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 595- 607. Association for Computational Linguistics, 2021. doi: 12216665//112222mmmmmip
URL https: / //to: org 1ttp://1.11655//1////0
Carl Edwards, Tuan Manh Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Translation between molecules and natural language. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022, pp. 375-413. Asso- ciation for Computational Linguistics, 2022. URL
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions-a large-scale biomolecular instruction dataset for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Henri A Favre and Warren H Powell. Nomenclature of organic chemistry: IUPAC recommendations and preferred names 2013. Royal Society of Chemistry, 2013.
Daniel Flam- Shepherd and Alán Aspuru- Guzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv:2300.05770 2023.
Paul Geerlings, Frank De Proft, and Wilfried Langenaeker. Conceptual density functional theory. Chemical reviews, 103(5): 1774-1874, 2003.
Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G Tyer, Yihong Ma, Olaf Wiest, Xian- gliang Zhang, Wei Wang, Chuxu Zhang, et al. Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 6638-6646, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representationss ICLR 2022, Virtual Event, April 25-29, 2022.
OpenReview.net, 2022. URL BRLhhtpss: prtetss://opnnaee wwwp//ppeeeeiieeeeeeeeeeeeeeeeee net hwwp:::wwwww..tt.eeee.ee.eeeeee.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGB-LSC: A large-scale challenge for machine learning on graphs. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
http:s/s//tttttttseeeecmaaaaaass ptsereeeeccccmt---oeeereeeeeeeee ngs. neur ps cc/ ptcc/paper/22211 htwp://2222222220255--eeee-- absttract wowndo. html
Woosung Jeon and Dongsup Kim. FP2VEC: a new molecular featurizer for learning molecular
properties. Bioinform., 35(23):4979 4985, 2019. doi: 10. 10..0037BBONNFORWAATIISTTTTTTTT
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2019 update: improved access to chemical data. Nucleic acids research, (7(DI))D1102-D110. 2019.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik, Self- referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.
Mario Krenn, Qianxiang Ai, Senja Barthel, Nessa Carson, Angelo Frei, Nathan C Frey, Pascal Friederich, Théophile Gaudin, Alberto Alexander Gayle, Kevin Maik Jablonka, et al. Selfies and the future of molecular string representations. Patterns, 3(10): 10588, 2022.
12

---

Published as a conference paper at ICLR 2025
Greg Landrum et al. Rdkit: Opunesource cheminformatics, 2023. URL hwwph:httssttt..t. WtLLtttps///ithub.com
Releaaeeeeeeeeeeeeee 2023 09_5 GitHub release.
Khiem Le. Zhichun Guo, Kaiwen Dong, Xiaobao Huang, Bozhao Nan, Roshni Iyer, Xiangliang Zhang, Olaf Wiest, Wei Wang, and Nitesh V Chawla. Molx: Enhancing large language models for molecular learning with a multi-modal extension. arXiv preprint arXiv:2406.06777 2024.
Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. arXiv preprint arXiv:2306.06615 2023a.
Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. Empowering
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023b. URL
mlr. wrw.srrsssss...... wttps///222211133/// html
Sihang Li, Zhiyuan Liu. Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, and Qi Tian. Towards 3d molecule-text interpretation in language models. In The Twelfth International Conference on Learning Representations, 2023c.
Shin-Yew, Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Computers biology and medicine, 171: 108073, 2024.
Shengehao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. Multi-modaa molecule structure text model for text-based retrieval and editing. Nature Machine Intelligence, 5(12): 2023a.
Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie- Yan Liu. Molxpt: Wrapping molecules with text for generative pre-training. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023b.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 15623-15638, Association for Computational Linguistics, 2023c. doi: 10.18653/V1 11116553/11/2223EMMLPPP WWW.E-MMNN....... 966. URL httttttt//oo...... 1653
emm em:1p-main..9 966
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 20/9, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Yizhen Luo, Kai Yang, Massimo Hong, Xing Yi Liu, and Zaiqing Nie. Molfm: A multimodal molecular foundation model. arXiv preprint arXiv:2307.09484 2023.
Nakata Maho. The pubchemqe project: A large chemical database from the first principle calculations. In AIP conference proceedings, volume 1702. AIP Publishing, 2015.
Piotr Nawrot. nanot5: A pytorch framework for pre-train and fine-tuning t5-style models with limited resources. arXiv preprint arXiv:2309.002333, 2023.
OpenAl. Chatgpt, 2023. URL httph:tt///// oww.ai..ccccmmmmm wtt. http:p::///peeaaacccma/a//aa/aaa
chat
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. July 6-12, 2002, Philadelphia, PA, USA, pp. 311 1-118, ACL, 2002. doi: 10.3115510733033117333 URL http:///s////aaaaaaaooooooo///g/
13

---

Published as a conference paper at ICLR 2025
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingee Xia, and Rui Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1102-1123, Singapore, December 2023. Association for Computational Linguisties. URL bettss:/aclanthoooory.oo
Yan. BioT5:
emnlp main.
Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. Biot5+: Towards generalized biological understanding with IUPAC integration and multi-task tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 1216-1240. Association for Computational Linguisties, 2024a. URL llltss://cllanhhlooyy..oo wtw.:y...rgg/ooo attpg////........innna.......... mnn.ings-aa..... acl.
Qizhi Pei, Lijun Wu, Kaiyuan Gao, Jinhua Zhu, Yue Wang, Zun Wang, Tao Qin, and Rui Yan. Leveraging biomolecule and natural language through multi-mode learning: A survey. arXiv preprint urXiv:2403.01522 2024b.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Günter Klambauer. Fréchet chemnet distance: A metric for generative models for molecules in drug discovery. J. Chem. Inf. Model., 58(9) 17111177111111, 2018. doi: 10.1021/acs.jcim.80022.. URL https:
/rg/
10.1021/acs: jeim. 8b00234
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1):5485-5555 -5515,55511100 2020.
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media. 2019, /ttetss://wwtt /ww. amazon. com/ Deep Lea reiignnng-- Li fe Bicceennees..cccrccccoor///////p
David Rogers and Mathew Hahn. Extended-cooneetivvty fingerprints. Journal of chemical information and modeling, 50(5):742-755 2010a.
David Rogers and Mathew Hahn. Extended--oomnecivity fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010b.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 1):2864 2875, 2012.
Nadine Schneider, Nikolaus Stiefl, and Gregory A Landrum. What's what: The (nearly) definitive guide to reaction role assignment. Journal of chemical information and modeling. 56(12):2336 2346, 2016
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer, Enhancing activity prediction models in drug discovery with the ability to understand human language. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. URL httph::////// htttps::/ppoceecinnge mlr wppess//2000 httpe//eei123... /ttpa///s......mppeeeeee/e/e/da. html
PMLR, 2023.
Bing Su, Dazhao Du, Zhao Yang. Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun. Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural (tt::2229000999 2022.
language. arXiv preprint
Mujeen Sung, Minbyul Jeong, Yonghwa Choi, Donghyeon Kim, Jinhyuk Lee, and Jaewoo Kang. Bern: an advanced neural biomedical named entity recognition and normalization tool. Bioinfor- matics, 38(20):4837-488
2022.
Xiangru Tang, Andrew Tran, Jeffrey Tan, and Mark Gerstein. Mollm: unified language model to integrate biomedical text with 2d and 3d molecular representations. bioRxiv, pp. 2023-11, 2023.
14

---

Published as a conference paper at ICLR 2025
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. CoRR, abx/2211.09085, 2022. doi: 448550/ARX...221000000 URL https:
Aidoi.orge 10. w0w.54/555xiii. / 2211 09085
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09286 2023.
Renxiao Wang. Xueliang Fang, Yipin Lu, Chao-Yin Yie Yang, and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, 48(12):41114113 2005.
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36. 1988.
David Weininger, Arthur Weininger, and Joseph L Weininger. Smiles. 2. algorithm for generation of unique smiles notation. Journal of chemical information and computer sciences, 29(2):97-101, 1989.
Naifeng Wen, Guanqun Liu, Jie Zhang, Rubo Zhang, Yating Fu, and Xu Han. A fingerprints based molecular property prediction method using the BERT model. J. Cheminformatics, 14 (1):71, 2022. doi: 00.1186661332110220006666 URL httpp:://6o..rrg/10..1 s 613321-022-00662
Jacob White. Pubmed 2.0. Medical reference services quarterly, 39(4):382-387, 2020.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()):513-330, 2018.
Teng Xiao, Chao Cui, Huaisheng Zhu, and Vasant G Honavar. Molbind: Multimodal alignment of language, molecules, and proteins. arXiv preprint arXiv:2403.08167, 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong I Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. arXiv preprint arXiv:21
2021.
Botao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint wxXi::442.00939/ 2024.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862, 2022.
Juzheng Zhang, Yatao Bian, Yonggiang Chen, and Quanming Yao. Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.00880 2024a.
Qiang Zhang, Keyang Ding, Tianwen Lyv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, et al. Scientific large language models: survey on biological & chemical domains. arXiv preprint arXiv:2401.144 14656, 2024b.
Zihan Zhao, Bo Chen, Jingpiao Li, Lu Chen, Liyang Wen. Pengyu Wang, Zichen Zhu, Danyang Zhang, Ziping Wan, Yansi Li, Zhongyang Dai, Xin Chen, and Kai Yu. Chemdfm-x: Towards large multimodal model for chemistry. arXiv preprint 2r0i:: 13194, 2024.
15

---

Published as a conference paper at ICLR 2025
Artem Zholus, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Daniil Polykovskiy, Sarath Chandar, and Alex Zhavoronkov. Bindgpt: A scalable framework for 3d molecular design via language modeling and reinforcement learning. arXiv preprint
2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11033 2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng. Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning frame- work. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htt.t: http:///ppeeeeeeeeeeeeeeeeeeeeee net/pdf? id= 6K2RM6WVQVV
16

---

Published as a conference paper at ICLR 2025
A MORE DETAILS ABOUT E3FP
A.1 CONNECTIVITY AND STEREOCHEMISTRY ENCODING
In Algorithm| [ there are two functions, Connectivity and Stereochemistry, which encode the con- nectivity and stereochemical information between ag and a; respectively, resulting in the oj and s The connectivity identifier of ranges from 5, which represents relative atomic distance: 1 for single bond, 2 for double bond, 3 for triple bond, 4 for aromatic bonds, and 5 for no bonds. The stereochemical identifier sj encodes relative atomic orientation. ranges from -5 to 5 based on their regions in divided unit sphere, where the division is based on the x/y-axis defined by direction vectors from the center atom to its neighbors. More details can be found in the original paper (Axen et al. 2017).
A.2 SE(3)-INARRIACE ANALYSIS
The process of mapping 3D molecular structure to hash values in the E3FP algorithm is SE(3)- invariant. The reason is: (1) The initial identifier iteration 0 for each atom is defined by atomic invariant features. (2) As shown in step of Algorithm| | during the iterative process of E3FP, the identifier for the current shell defined by E3FP is determined by iteration number j. the identifier for the same atom from the previous iteration d I. the neighbors' connectivity of and relative orientation si with respect to the center atom of the current shell, and the neighbors' identifiers dx.j-1 1 from the previous iteration. Both the initialization and iterative process are not affected by the molecule's 's rotation, translation, and reflection, thus preserving the SE())-invariance.
A.3 COMPLEXITY ANALYSIS
In this section, we briefly analyze the complexity of E3FP algorithm.
Time Complexity. The core of the E3FP algorithm involves iterating over atoms and their neigh- borhoods up fixed number of iterations k. At each iteration, the algorithm: (1) Draws shell of increasing radius around each atom; (2) Identifies neighbors within the shell; (3) Generates unique identifiers for the substructures formed. For a molecule with n heavy atoms, the time complexity for each iteration involves examining the neighbors within the shell. Typically, the number of neighbors bounded by constant b, due to the constraints of chemical valence. Thus, the time complexity of the fingerprinting process for each atom is Obb)), and for a molecule with n atoms, it becomes Onbb)). Typically, n <= 100,
4, and k = our setting.
Space Complexity. As shown in Figure 2 in our paper, without considering the storage of inter- mediate variables, we only need store the final 3D token indices for each atom each iteration. Thus, for a molecule with N atoms, the overall space complexity for a single molecule is O(Nk), Notably, in practice, the generation of E3FP fingerprints is performed offline and can be executed rapidly through multi multi-processing, achieving throughput of approximately 300 samples per second using 24 parallel processes.
A.4 HYPERPARAMETER SETTINGS AND SPECIAL CASES
In E3FP (Axen et al. 2017). ), there are three key hyperparameters, the iteration number k the shell radius multipler r, and the length of E3FP fingerprint FF). The l is set to 3, as our preliminary experiments indicate that three iterations are sufficient for the E3FP algorithm to converge, capturing all potentially occurring substructures for the vast majority of molecules. The set to 1. 1.718À fol- lowing the default setting in E3FP (Axen et al., 2017). The FI is set to 4096 rather than the default 1024 to further decrease the probability of collisions of folding.
There are two special cases where the 3D substructure identifier à will be -1, indexing a zero embedding of dimension H: (1) The corresponding SELFIES tokens do not correspond to atoms, like structure directive tokens: [Ringl] and [=Branchi]. (2) The E3FP algorithm is converged before the k iterations.
17

---

Published as a conference paper at ICLR 2025
"1 SELFIES token: [+C]
Feature Array
32-bit Identifier
Folded Identifier
Iteration Number
[6. R
1763934239
31
[ 1. 1763984239
1. 6156634635
1,
9119668869
1827
2. 410092236
.22
915857869
1990676907
015643315
-1910577370
1310
(conveggd
2-1113137791 2.
5185493
23
Figure 4: Visualization of the E3FP process the second atom d1 (Carbon) of the molecule with CID 101399 (same the case in Figure
The final number of embeddings for SELFIES tokens is 2944, and for 3D tokens 4097 as there is a special zero embedding representing no 3D information.
A.5 EXAMPLE FOR ENCODING PROCESS
To provide a clearer illustration of the E3FP process for better understanding, present a specific case here. For the case in Figure 2 let's consider the second atom (Carbon) with SELFIES token [=C] (denoted as 01 for simplicity) and its shells at each iteration. The detailed visualization of the E3FP process is shown in Figure
At iteration 0, a set of atomic invariants for each atom a; is used to initialize the structure represen- tation. For d1, these invariants include the atomic number (6), the number of immediate neighbors (3), the number of bound hydrogens (1), the difference between the atomic mass and the standard atomic weight of the corresponding element (0), the atomic formal charge (0), and whether the atom part of ring (0). These atomic invariants concatenated (16, 3, 1, O, OJ) and then hashed into identifier d 1763934239 by MurmurHash3 (Appleby 2016) algorithm.
At iteration 1, spherical shell centered on 01 with radius r is defined. The connectivity and spatial arrangement of neighboring atoms within the shell are encoded relative their central atom by 2-element header list and several 3-element lists. For the header list is [1, 1763934239), where 1 is the iteration number and 1 1769934339 is the identifier of the shell from the previous iteration, i.e., di,0- Since there are two neighboring atoms centered on a₁ in this spherical shell, two 3-element lists are defined: [1, 6615636665, for Nitrogen and [2, 410692236, 2] for Oxygen. The first position of the 3-element list (e.g.. for Nitrogen, 2 for Oxygen) is the int connectivity identifier introduced in Ap- pendix[A.1] The second position (e.g.. 61 56344635 for Nitrogen, 410692236 for Oxygen) is the iden- tifier of the shell from the previous iteration. The third position (c.g.. 1 for Nitrogen, -2 for Oxygen) the stereochemical identifier encoding relative atomic orientation as illustrated in Appendix A. Then these lists are concatenated (11, 1739334339, 1, -0-661 -615634663 1, 410692236, -2)), and then hashed to identifier d11 -915867869 by MurmurHash3 (Appleby 2016). 2016))
At iteration 2, the radius of the shell is increased to 2r, and the process similar to iteration performs again. For 0j, the resulting identifier at this iteration d = 1918577378.
In setting, the maximum iteration number is 3. However, for this molecule example, the E3FP converges at iteration 2, as the shell at iteration 2 centered on atom as (Carbon) has al- ready included all atoms. The combined hash identifier for is dj = [dr d1.1, d1,2] = [17639342399, -915867869, - 19118577378, which is then converted d1 d, mod [F] [31, 1827, 1310). The final d, = [31, 1827, 1310, -1], where -1 indicates no 3D information for iteration 3, as shown in the bottom table of Figure 2
18

---

Published as a conference paper at ICLR 2025
SELFIES Toean
[0]
[+[]
16
[+C]
(Branch1]
[+Branch1)
[C]
[+Bancch))
[C]
[+0]
131
14
[+C]
[4
[+]
[knot)
Breancha
Lecel 0
2700
11
2157
2734
2714
-1
-3
2571
-1
-1
2700
2815
1738
1738
1718
173A
Leced
2730
1827
427
1765
1520
-1
1344
1882
2616
561
1527
1527
Tecel
SELFIES Tokan
[0]
[-[]
16
[-q]
(Brunch1) [+Branch1]
[+Branch)]
[C]
[-0]
101
[4]
[+C]
[4
[+C]
[8nol]]
[-Bracch2]
Locol
2730
1827
427
1929
1802
2616
561
1527
1327
Leval 2
2646
1680
1880
1255
1255
-1
2406
2277
546
1243
4066
4046
2060
Lecel
SELFIES
[0]
PM
[:Branch]]
[=Franchl]
1:00
101
[:C]
IReg11
IeBranchZ)
level
2700
2952
2212
2571
2700
2805
1738
1759
1718
Level
2239
1827
427
1965
1520
-1
1944
1802
2616
561
1527
1527
551
Level 2
450
3101
2083
2050
5.6
3509
3909
2015
SELFIES Tocen
[0)
[+C]
1N0
K3
[+C]
[Brnchh) [-Branch]]
[S]
[+Brancht)
[]]
[+0]
101
[4]
[+C]
[4]
[+C]
[Knol] [-Brarchc]
Level 0
2700
31
2357
2714
2714
-1
2521
2700
2865
1738
1738
1738
1735
1802
2616
1527
1810
1310
1492
3452
570
4046
4046
Lovel 3
1
-1
SELFIS
1N
[+Branch1)
[+Srncch))
[-0]
[-Brarch]]
2000
2357
2714
2521
2700
2855
1738
1738
1738
Level 1
2230
1827
427
1765
1520
-1
-1
1344
-1
-1
1502
2616
561
1527
1327
551
-1
-1
Levd 2 480
3780
2142
1735
1010
-1
2406
1943
546
1243
3509
3909
754
Lovi 3
Figure 5: Visualization of conformers and their corresponding 3D tokens for molecule with CID) 101399 (same as Figure The difference among their 3D tokens are colored in red.
A.6 INFORMATION Loss OF E3FP DISCRETE ENCODING
In our 3D tokenization, we use discrete tokens to represent 3D molecular substructure, which is sparser than continuous representation and may introduce information loss. We empirically demonstrate that this sparsity does not hinder the extraction of critical information for 3D structures from the following two perspectives.
(1) We show that E3FP can effectively capture subtle variations between different conformers. We choose a molecule from PubChem and visualize its 5 conformers as shown in F Notably, these conformers exhibit slight variations in two substituent groups on the benzene ring. We can see that different conformers possess distinct 3D tokens extracted by E3FP. Thanks to the hierarchical nature of our 3D tokenization: iteration 0 primarily captures atomic invariant features, and iteration I accounts for neighbors within radius 1.718.. Since iterations 0 and I are quite local, these features are identical for atoms across different conformers. However, iteration 2 considers the neighbors within a larger radius 2r, it reveals subtle differences between conformers. This proves our 3D tokens can effectively capture subtle variations between different conformers. (2) We do empirical verifications on 3D molecular understanding tasks, which is the focus of our work, to demonstrate whether the full continuous information is necessary or the discrete token enough for the performance effect. Specifically. we compare variant of our model that does not incorporate ID SELFIES information and relies solely on 3D tokens, with Uni-Mol (Zhou et al. 2023), which employs 3D continuous information, on the H-L. gap prediction task with the QM9 dataset. Our model achieves an MAE of 0.15, outperforming the Uni-Mol's 's 0.21. This suggests that discrete 3D tokens are sufficient for 3D understanding tasks and the information loss is not heavy. (3) An empirical observation from recent work. A recent study, UniMoT (Zhang) tokenizer to convert 2D molecular graphs
ct al. 2024a) introduced a Vector Quantization-drive into sequences of molecular tokens, followed by multi-stage training, enabling joint molecule-tent modeling. Their experiments on the molecule captioning task revealed that, while quantized discrete tokens exhibit slightly inferior performance compared to continuous embeddings. the performance degradation is marginal. This demonstrates that discrete representation may indeed lead to some degree of information loss, but remains within acceptable limits.
B ADDITIONAL. DOWNSTREAM RESULTS
primarily focus 3D-related molecular tasks. However, since 3D-MolT5 can naturally adapt to tasks involving ID and 2D molecular representations, we further evaluate its versatility on broader range of benchmark datasets in this section. These benchmarks include
In Section 4
19

---

Published as a conference paper at ICLR 2025
Table 6: Results for retrosynthesis task on USPTO-50K (Schneider et al. 2016 dataset (Best,
Second Best).
SETTING
METHOD
BLEU-21
LEVENSHTEING
RDK FTS*
VALIDITY
Llama-2-7B
10.10
468.74
00.00
Infer-only
Llama-2-7B + MolX
36.73
62.33
0.4041
13.71
-lama-2-7B
80.37
16.22
0.6981
89.27
LlasMol-7B
50.09
31.28
0.7351
99.65
ChemDFM-13B
39,93
57.48
0.5380
14.04
LoRA FT
Llama-2-7B + MoMu
70.88
20.77
0.5691
90.53
Llama-2-77 1-na-22-B--a + +D-MMoLM
82.05
15,90
0.7126
91.13
Llama-2-72 + 33D-MoLM
81.31
16.21
0.7341
90.31
Llama-2-7B + MoX
82,59
15.74
0.7466
92.19
Llama-2-7B + MolX
36.73
62.33
0.4041
13.71
Chemformer SeactionTS-Largee
74.01
19.51
0.6951
97.14
Full FT
81.63
17.69
0.7400
97.58
3D-MolT5
86.23
14.08
0.7538
100.00
Table Results (AUROC) for molecule property prediction tasks on MoleculeNet (Wu al. 2018
benchmark (Best, Second Best). represents LoRA (Hu et al. (0022) tuning.
METHOD
BACE 1513
BAAP 2039
HIV 41127
Clintox 1478
MOLECULES
Models
GraphCL
75.4
69.7
78.5
76.0
74.9
GvphMVP-.C
81.2
72.4
77.0
77.5
77.0
MGSSL
MalCLR
89.0
73.8
80.6
93.2
84.2
72.4
90.1
Uni-Mol
85.7
72.9
80.8
91.9
82.8
LM bused Models
KV.PLM
84.3
MoMu
76.7
70.5
75.9
79.9
75.8
MalPM
83.9
72.9
78.8
79.7
78.8
Galactica-6.78 7B
58.4
53.5
72.2
78.4
65.6
Galactica-30B
72.7
39.6
75.9
82.2
72.6
Calactica- 120B
82.6
UniMaT
83.7
71.4
78.5
92.9
81.6
3D-MolTS
88.1
76.5
80.8
95.4
85.2
retrosynthesis on the USPTO-50k dataset (Schneider et al. 201ble 6 molecular property prediction the MoleculeNet benchmark (Wu et al. 2018) ( 7).le and tasks including forward reaction prediction, reagent prediction, and retrosynthesis on the Mossrunttts datasets (Fang et al.) 20b3) 8). The superior performance of 3D-MolTS across these diverse molecular modeling Tasks highlights its robustness and adaptability to various molecular benchmarks.
C ADDITIONAL DISCUSSIONS AND FUTURE DIRECTIONS
C.I COMPARISON WITH DIRECT COORDINATE REPRESENTATION
Directly encoding spatial molecular data as text containing atom coordinates, as demonstrated in recent works (Zholus et al., 2024 Flam-Shephert & Aspuru- Guzik 2023). is indeed simpler and more transparent approach than E3FP (Axen et al. 2017) encoding in 3D-MolT5. However, for 3D molecular understanding tasks, such as property prediction and captioning, the E3FP-based discrete token scheme offers significant advantages, which we summarize below:
(1) Input Length and Computational Efficiency. Representing spatial coordinates directly text substantially increases input sequence length, especially when dealing with large molecules. This not only introduces additional computational overhead considering the quadratic complexity of the attention mechanism, but also complicates the model's 's learning process, as longer sequences can dilute meaningful patterns within the data. In contrast, by encoding 3DD structure into 3D tokens and
20

---

Published as a conference paper at ICLR 2025
Table Results for chemical reaction-related tasks on Mol-Instructions (Best, Second Best). represents LoRA (Hu et al. 2022) tuning.
Fang et al. 2023 datasets
MODEE.
EXACTT
BLEUT
LIVINSITTINN
RDK FTST
MACCS FTST
MORGAN FTS"
VALIDITYT
Reagent Prediction
Liama70
0,000
0.001
28,040
0.037
0.001
0.001
0:001
6alactica-6-888
01000
30,750
0.136
01051
0.995
Test TS-233M
0.822
0.313
Instractions
0.237
0213
Laama788 (LoRA)
11000
53.510
0.136
0.204
0.106
1,000
InstrccMatt-C
0.070
0.890
24.733
0.469
0.691
0.226
1,000
InstractMol GS 6.98
0.129
0.610
19,664
0.464
0.539
0.400
1,000
UniMaT
0.162
0.728
14.558
0.542
0.621
0.507
1,000
3D-Mor15
0,377
0.756
13.173
0,571
0,546
0.540
1,000
Fonvard Reuction Prediction
0.000
0.001
0.001
Gallatica--677 7B
0.097
Cheae 15-223
20,413
0,700
0.052
0.762
Mot-Instractions-1
0.045
27,262
0.313
0.262
1,000
Liama (LoRA)
0,012
0.804
29 29,947
0.699
0.519
0.407
1,000
IostracMol-4G-4.98
0.153
0.906
20.155
0.519
0.717
0.457
1,000
IniraacMel--S-6..00
0.536
0.967
10.851
0,776
0.878
0.741
1,000
UniMoT
0.611
0,980
8,297
0,826
0.911
0.807
1,000
3D-MarTS
0,579
0.994
2,842
0,955
0.978
0.942
1,000
Reimeeymhheziz
0.018
0.010
Golaccics-6678
13000
34,940
0.167
0.134
0:986
Teem+Chem T5-223M
0.141
0.765
24.043
0.683
0.065
0.585
0.898
Mol Instractions 7B
0,009
0.705
31.227
0,283
0.457
0.230
1,000
LIama-78 (I oRR)
0.000
0.283
$3.510
0.136
0.204
0.106
1,000
InstractMol- G 5.50
0.114
0.586
21.271
0.422
0.523
0.285
1,000
InstractMol MarracMMo------.35 658
12407
0.841
13.967
0,753
0.852
0.714
1,000
UniMaT
0.478
0.974
11.63
0.800
0,002
0.771
1,000
30-Mof15
0,679
0,974
6.201
0,907
0,940
0,878
aligning ID and 3D embeddings at the atomic level, 3D-MolT5 maintains balanced and scalable representation while avoiding unnecessary computational complexity.
(2) Semantic Representation of Numerical Data. Tokenizing coordinates as text often results in loss of numerical semantic relationships. For instance, the tokens for the numbers "123" and "1 "124" treated entirely distinct, despite their numerical proximity. This lack of semantic similarity makes it challenging for the model to capture meaningful numerical relationships such as proximity continuity. In contrast, the E3FP algorithm encodes 3D molecular structures discrete tokens based on hierarchical and spatial substructures, preserving critical spatial relationships in a form that more interpretable and useful for the model.
(3) Preservation of Invrinnn, Representing spatial data directly as coordinates can struggle with preserving SEv3)nnnvariance, i.e., invariance to molecular rotations, translations, and reflections. Without explicit adjustments, such representations may lead to inconsistencies in encoding the same molecule under different orientations. But E3FP is inherently SE(3)-invariann (discussed in Appendix A.2). ensuring that the discrete tokens remain consistent regardless of molecular orientation, which is crucial for tasks like 3D molecular understanding.
While 3D-MolT5 currently focuses on 3D molecular understanding tasks, we recognize the potential of extending the model to support 3D structure generation. Future work could integrate both ID and E3FP-based 3D tokens into unified sequence and incorporate an external decoder to reconstruct 3D structures from generated tokens, addressing both understanding and generation tasks in molccular modeling.
C.2 SEQUENTIAL INTEGRATION OF E3FP TOKENS AND ID SELFIES TOKENS
In this section, we explore whether the E3FP tokens could be directly concatenated with 1D SELFIES tokens. We conduct preliminary experiments on the 3D molecule to text translation task using the PubChem dataset (Kim et al., 2019). Specifically, we compare our original 3D-MolTS, which sums ID and 3D embeddings, with the sequential approach, where E3FP tokens are directly concatenated with ID molecular tokens in a unified sequence. The results, presented in Table 9 show that the two approaches achieve comparable performance. However, the sequential approach significantly
https:///ccmmt openai. www.t//.c///////oo t http://wwwyyyhu 11- -i- larger than 9- 9-1 inced: b' e 869824/5
21

---

Published as a conference paper at ICLR 2025
Table 9: Results comparison between sum of ID and 3D embedding versus sequential concatenation of ID and 3D tokens.
SETTING
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L.
METEOR
Embedding Summation
37.78
29.61
42.94
27.51
37.40
38.86
Sequential Concatenation
37.65
29.34
43.25
27.64
37.45
38.98
increases the imput sequence length, particularly for larger molecules, leading to higher computational costs and longer training times. In experiments, the sequential concatenation requires more than 1.5 times the training time to converge compared to the embedding summation. Despite these drawbacks, the sequential approach offers practical advantages as it does require modifications the model's source code and aligns more naturally with tasks involving 3D molecular generation.
This finding highlights the flexibility of the E3FP-based framework and suggests potential extensions for tasks that benefit from sequential integration, such as 3D structure generation. Future work will explore the incorporation of these methods further expand the capabilities of 3D-MolT5.
D MODEL CONFIGURATION
3D-MolTS adopts the same architecture T5 model (Raffel al. 2020) with T5-1. configu--
ration. The encoder and decoder have 12 layers. The dimensions of attention and feed-forward layers
768 and 2048, respectively. The number of attention heads is 12. The size of the ID vocabulary is 35,045, including original text tokens of T5 and additional SELFIES tokens, and the size of the 3D vocabulary is 4096. The total number of parameters of 3D-MolT5 is 255M. We use nanoT Nawrot 2023) as our codebase.
E PRE-TRAINING
E. TRAINING TASK
As introduced Section 3.4, the pre-training includes the denoising and translation tasks. We give the corresponding loss functions as follows.
Denoising Tasks. Given sequence X 1- some consecutive spans of X are randomly masked by sentinel tokens, and the model learns to reconstruct these spans:
CD = =
Σ t=
log P(XM ((as)
where XM are the tokens that need to be recovered/generatee, |M is the number of masked tokens, and X the input X with the masked spans replaced by sentinel tokens.
Translation Tasks. In addition to denoising tasks, we also add translation tasks between modalities enhance the representation learning,
1 Σ
CT =
log P(Y XX)
where X is the input sequence, such as 3D molecule tokens, and Y is the target output sequence, such as the ID text sequence.
E.2 DATA AND CONFIGURATION
The pre-training is done eight NVIDIA 80GB A100 GPUs. The total number of steps for pre- training 400,000, with warm-up steps set to 10,000. AdamW (Loshchilov & Hutter 2019) with
httto::/haegiiofae...oooooggoooo httpss//githubbcco///oorro
bass
22

---

Published as a conference paper at ICLR 2025
Root Mean Square (RMC) scaling optimizer is used. The peak learning rate is 2e-3 with cosine decay, and the minimum learning rate is le-3. The maximum length for imput and output is 512. The batch size is set to 768. As shown in Table I0 the sizes of pre-traine pre-training datasets vary significantly. To balance the data from different tasks during pre-training, we implement a batch-level balancing strategy. Each batch evenly includes data from all tasks, ensuring more balanced and comprehensive pre-training process. For smaller datasets, such as molecule-text pairs from PubChem, we employ round-robin strategy repeat their usage multiple times, compensating for their limited size. For all molecular data, we first get its canonical SMILES from the provided SMILES or 3D structure using RDKit (Landrum al. 22033, and then convert to SELFIES using selfies toolkit (Krenn al., 2020 6
The resulting SELFIES are also wrapped by special tokens (bom) and (eom) to differentiate from text.
Table 10: Statistics of pre-training datasets. Deno. refers to T5 (Raffel et al. 2020) denoising task; Tran. refers to translation task. For the PubChem dataset, the 3D structure is obtained using the MMFF algorithm in RDKit (Landrum et al. 2023) and the text enriched by GPT-3.5 (OpenAl 2023
TEXT MOLECULE
DATA
TASK
S1ZE
ID
3D
PubChem SELFIES
Deno.
38,400,000
C4-English
Deno.
2,210,000
PubMed Central full articles
Deno.
38,400,000
PubMed abstracts
V
Deno.
33,404,528
PCQM4Mv2
Deno, & Tran.
3,377,055
PubChem molecule-text pairs
Tran.
298,861
F FINE-TUNING
Here introduce more details about fine-tuning, including details about datasets and baselines. The fine-tuning is done on a single NVIDIA 80GB A100 GPU.
Details for datasets for fine-tuning shown in Table TT For all downstream datasets, we follow the same pipeline as described in Appendix E to first get the canonical SMILES from the provided SMILES or 3D structure using RDKit (Landrum al.) (023),, and then convert it to SELFIES wrapped by (bom) and (eom). All reported results for 3D-MolTS are the mean value obtained from three independent random runs.
Table Dataset statistics for donwstream fine-tuning. All the datasets are in instruction format. Small differences exist between our processed datasets and the original version, as we discard the data that can not be processed by E3FP (Axen et al. 2017).
DATASET
MOLECULE
TASK
SIZE (TRAIN/VALIDATTOVVIEX)
PubChemQC
3D
Computed Property Prediction
2:463,404/300,02444388.4444
QM9
3D
Computed Property Prediction
947,774/11928/1.998
3D
Computed Property Prediction Descriptive Property Prediction
46,532/3 16.532/3,885/77,46
PubChem
3D
99,775/4,980/9.004 980/9,9409990
3D
3D Molecule Captioning
11.955/996/1988
CheB1-20
ID
Text- based Molecule Generation
26.407/3.301/3.300
For PubChemQC (Maho 2015) and PubChem (Kim et al. 2019) datasets, we use the instruction versions built by 3D-MoLM (Li et al. 2023c). The Generulist version of 3D-MolTS here is trained simultancously on these two datasets with three types of tasks: computed property prediction, description property prediction, and 3D molecule captioning. We follow the same sampling algorithm as 3D-MoLM (Li al. 2023c). where the sampling probabilities for each task are proportional to the
23

---

Published as a conference paper at ICLR 2025
fourth root of the size of its data. For descriptive property prediction, the descriptive text is generated by employing GPT-3.5 (OpenAl 2023) to read molecular captions and create five QA pairs for each molecule. The reported baseline results are derived from 3D-MoLM (Li et al. (023c). Specifically, the baseline method MD-MoLLM is a variant of 3D-MoL.M (Li et al. 2023c), where the 3D molecular encoder is replaced with 2D molecular encoder. The baseline I. Lamm2-7B (Touvron et al. 2023)
directly removes the 3D molecular encoder of 3D-MoLM (Li et al. 2023c) and uses ID SMILES as the molecular representation.
For QM9 (Ruddigkeit et al., 2012) dataset, we use its instruction version built by Mol- Instructions (Fang al., 2003). The 3D structures are downloaded from DeepChem (Ramsundar et al. 2019). The Generalist version for QM9 is trained on the direct combination of its three subsets: HOMO, LUMO, and HOMO-LUMO gap. The reported baseline results are derived from BioT5+ PPi 2024a).
et al.
For CheBI-20 (Edwards al., 2022) dataset, we manually convert it to instruction version. To avoid data leakage, exclude the molecules of the CheBI-20 test set that are also present in the PubChem 3D molecule-text pairs pre-training. this task, molecular names are removed from text to prevent the model from learning simple mapping from molecular names to ID sequences.
The reported baseline results are mainly sourced from MolT5 (Edwards et al. 2022), MolReGPT (Li
al. 2023a), , MolFM (Luo et al. 2023). GIT-Mol (Liu et al. 2024). MolXPT Liu ct al. 2023b). and
BioT5 (Pei et al. 2023
G CASE STUDY
The cases for computed molecular property prediction are shown in Table 12
We can find that 3D- MolT5 can give accurate numerical predictions about the computed properties of the input molecule. For descriptive property prediction, results in Table 13 show that 3D-MolTS successfully answers the question about the composition of the input molecule, including the attached hexacosanoyl group and sphinganine backbone.
The case for 3D molecule captioning is shown in Table 14. 3D-MolT5 successfully predicts the molecular names, composition, pH, and functional relationship. The case for text-based moleculc generation is shown in Table 15 where 3D-MolT5 generates the molecule that exactly matches the ground truth molecule.
H LIMITATIONS
In 3D- D.-M, the 3D structure information is only incorporated in the input, and 3D-MolTS can not generate 3D molecular structure directly, which is mainly caused by two factors. (1) The hash algorithm and "folding" process are irreversible and may introduce value collisions, though the probability small. (2) The pooling of embeddings at each iteration into the 3D embedding. and the pooling of ID and 3D embeddings. The unified modeling of both the understanding and generation of 3D molccular structures remains an area for future exploration.
24

---

Published as a conference paper at ICLR 2025
Table 12:
Case studies for computed molecular property prediction task.
MOLECULE
INSTRUCTION
MOLTS
GROUND TRUTH
would know the Molccular Weight of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with numerical value only.
The Molccular Weight for the input molecule 273.40 g/mol.
The Molecular Weight for the input molecule is 263.37 g/mol.
would like know the LogP this molecule, could you please provide it? If uncertain, provide estimate. Respond with numerical value only.
LagP input molecule 3.10.
LogP for input molocule 2.80.
know Topological
The Topological Polar Sur- face Area for the input molccule 49.30 A-.
The Topological Polar Surface Area for input molcculc 49,30 A
of this malecule, could you please provide it? If uncertain, provide an estimate. Respond numerical value only.
need know the Complexity this molecule, could you please provide it? If uncertain, provide numerical value only.
Complexity molecule is 384,00.
Complexity for in- molocule is 363.00.
estimate. Respond
HOMO molecule, you pleuse provide it? uncertain, provide esti- mate. Respond with the numerical value only.
The HOMO the input
The HOMO for the input
molecule -.793
molecule -5.769 eVV
Pleuse provide LUMO value for this molecule. uncertain, provide an numerical vallue only.
The LUMO for the input
The LUMO for the input
Respond
molecule 0.011 eV.
molecule 0.054 eVV.
am interested the HOMO-LUMO Gop of this molecule, could you tell me what it uncertain. provide estimatie. Respond with the numerical value only.
The HOMO-LUMO
The HOMO-LUMO Gap for the input molecule is
for the input molecule is 5.810
Please provide the highest occupied molecular or- bital (HOMO) energy of molecule.
-0.2131.
-0.2132.
What is the lowest unoccupied molecular orbital (LUMO) energy this molecule?
-0.0064.
Can you give the energy difference between the HOMO and LUMO orbitals of this malecule?
0.2744.
0.2756.
Table 13: Case studies for descriptive molecular property prediction task.
MOLECULE
INSTRUCTION
3D MOLTS
GROUND TRUTH
I-O-(alpha- Saaha--Dgaaaaa D-gallacopyranoeyl--.. What the composition
I-O-<alpha-D --ggaactopyroossy---.
hexacosanay Isphinganine hasbosaannyypphngannneee consists hexacosanoyl group attached the uitrogen atom, with a galactosyl residue connected oxygen atom at position of the sphinganine hac shone
hexacosanoyksph hohinngannnnnn compused of sphinganine backhone, hexacosanoy group attached to the nitrogen atom, and alpha D galactepyranosy residue the 0-1 position.
of I-O-talpha-D-
galactopyranosy)--M hexacosamay hww.ouannnyyyllinnnnnnneeeee
25

---

Published as a conference paper at ICLR 2025
Table 14: Case studies for 3D molecular captioning task.
MOLECULE
INSTRUCTION
3D-MOLTS
GROUND TRUTH
4-methy [henzeyl-AMPLL. is an organophosphate oxoanion obtained by deprotonation of the phosphate OH group Describe of 4-methylbenzoyll-e Ibenzoyl-AMP: major species 7.3. It is functionally adenosine 5 (mmmpphosppaat((()) and
is organophosphate oxcanion obtained by deprotonation of the phosphate OH group of 4-methylbenzool-AAM major microspecics at 7,3. functionally related to an adenosine monophosphatef? a toluate.
4-merhylhenzoyl-AMPP
molecule.
methylben/oote.
Table 15: Case studies for text-based molecular generation task.
DESCRIPTION
INSTRUCTION
3D-MorT5
GROUND TRUTH
The molecule is member the class of napitioutes that n-pphhhrate substinted at positions and 5 by hy- droxy and methyl groups spectively; major species at pH 7.3. a role as bacterial metabolite. base of a hydyyxyy-SS methyl- Inaphthoic acid.
Generate molecule that the input de- scription.
[[[C][CC][Banchi] BerrBaanncch]]BBBBani]]
JJCII-C[[BBaccch/BBBmmelll [[C]
J[
[
#BBaacb)]-=CCCC0le[[]Rkgglgll [#Bcm
conjugate
26


Reviews:
[{"review_id": "ApGSBjIO9G", "reviewer": "Reviewer_GGJY", "summary": "This paper introduces 3D-MolT5, a novel framework for integrating molecular sequence and 3D structural information within a unified language model. While recent language models have advanced molecular science applications, most neglect critical 3D structural data essential for accurately understanding molecular properties. To address this, 3D-MolT5 uses a unique 3D token vocabulary that captures detailed 3D substructure information, enabling it to encode molecular sequences, spatial structures, and natural language together in a cohesive architecture. Through multi-task pre-training, 3D-MolT5 develops robust cross-modal comprehension. Experimental results show strong generalization, with the model achieving nearly a 70% improvement in molecular property prediction compared to current methods.", "questions": "Given that multimodal approaches are increasingly common, how does 3D-MolT5 specifically leverage 3D structural data differently or more effectively than existing fine-tuned LLMs that incorporate multimodal information?\nIn the model’s tokenized format for 3D structures, what considerations were made for maintaining structural fidelity across different molecule sizes and complexities? How does the approach handle conformational flexibility within molecular ensembles?\nCould the model’s performance be further validated on additional downstream tasks, such as reaction prediction or drug-target interaction, to assess its versatility across a broader range of applications in molecular science?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work shows strength in its comprehensive comparison with various existing models, showcasing 3D-MolT5’s effectiveness across different approaches and highlighting its strong performance in molecular structure understanding. By integrating both molecular sequence and 3D structural information, the study addresses a crucial aspect of molecular modeling that is often underexplored, underscoring its relevance in capturing complex molecular properties.", "weaknesses": "While the work presents a solid approach, the concept of multimodal integration itself is not entirely new. Including a comparison with fine-tuned large language models (LLMs) that incorporate multimodal data[1] would further strengthen the evaluation, providing a clearer benchmark comparing to a boarder types of multimodal molecular representation models and other LLM as well e.g. Galactica[2].\n\n[1] Le, Khiem, et al. \"MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension.\" arXiv preprint arXiv:2406.06777 (2024).\n[2] Taylor, Ross, et al. \"Galactica: A large language model for science.\" arXiv preprint arXiv:2211.09085 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bcwfTSTGod", "reviewer": "Reviewer_7qiv", "summary": "The authors introduce a novel approach that integrates spatial molecular data by calculating a set of E3FP-based discrete tokens for each atom, then combining these tokens with corresponding SELFIES embeddings. Additionally, they propose a new pretraining scheme designed to improve the alignment and interaction between structural (1D) and spatial (3D) tokens. Finally, the authors present a thorough evaluation of the 3DMol-T5 model, which incorporates these novel methods into an encoder-decoder architecture, across various molecular tasks.", "questions": "As noted in the Weaknesses section, although the proposed method achieves significant performance improvements over baseline models, it appears overly complex and includes numerous subcomponents. The necessity of each part is not sufficiently justified.\n* The authors introduce a novel pretraining scheme comprising five subtasks. Are all these subtasks equally essential for optimal model performance, or could the model show similar performance using only one or two? Intuitively, the “3D molecule to text translation” task might be sufficient, as it encompasses all three modalities—text, 1D molecular structure, and 3D molecular structure. It would be valuable to explore this question further by showing the performance of 3DMol-T5 after pretraining on each individual task while excluding the remaining four.\n* Several recent works [a, b] have demonstrated that language models can process spatial molecular data directly as text containing atom coordinates. Compared to the proposed E3FP-based discrete tokens, this direct coordinate representation is considerably simpler, more transparent, and requires no model modifications. Furthermore, it can serve as both input and output (for example, in spatial molecular generation tasks). Providing a comparison with this approach is important to justify the use of the E3FP-based discrete token scheme.\n* The proposed alignment scheme for combining 1D and 3D molecular information involves summing their embeddings as $E=\\frac{1}{2}E_{1D} + \\frac{1}{2}E_{3D}$ which necessitates modifications to the language model's source code—requiring time and substantial programming knowledge to adapt it for other models. However, is this modification truly necessary? Could the E3FP tokens be interpreted as textual tokens and used directly within the text without altering the model? Intuitively, a sequence such as “$s_0, d_{01}, d_{02}, d_{03}, s_1, d_{11}, d_{12}, d_{13} \\cdots$” might accomplish this. It would be useful to elaborate on this question.\n\nMinor suggestions:\n* While the QM9 dataset is suitable for demonstrating the model's capability in using 3D molecular information to predict quantum properties, I recommend that the authors consider the QMUGS dataset [c] due to the greater diversity in molecular structures and quantum properties. Notably, QMUGS includes atom-wise properties, which could provide a more comprehensive evaluation. Testing model performance on this dataset could enhance the significance of the experimental section.\n\na. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files, 2023\n\nb. BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning, 2024\n\nc. QMugs, quantum mechanical properties of drug-like molecules, 2022", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow, with supplementary materials providing all necessary details to reproduce the method. The concept of integrating spatial molecular fingerprints alongside molecular sequence representations is a novel contribution in the field of language models. Experimental results demonstrate a notable improvement in model performance compared to baseline models.", "weaknesses": "Despite the comprehensive benchmarking of the model across various molecular tasks, the limited ablation studies make it difficult to assess the individual contributions of each component of the proposed approach to overall model performance. Additionally, a minor drawback is the absence of non-language model baselines in the Molecular Property Prediction tasks section. Further details on these points are provided in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZGhSvlCsb", "reviewer": "Reviewer_NdAP", "summary": "The paper introduces a unique framework, 3D-MolT5, that integrates 3D structural data and molecular sequences into a unified language model. 3D-MolT5 embeds molecular geometry directly into the language model using discrete 3D tokens generated from the E3FP algorithm. The model is pre-trained with multi-task objectives, including both denoising and translation tasks. The model achieves state-of-the-art results in tasks like molecular property prediction, molecule captioning, and text-based molecule generation.", "questions": "- Why do the authors focus solely on T5 instead of exploring other LLM architectures? Could the 3D \"tokenization\" approach work effectively with models like Llama or Mistral?\n\n- What advantages does the E3FP algorithm have over a 3D molecular encoder?\n\n- How many embeddings are used for encoding SELFIES tokens and 3D tokens?\n\n- Since the SELFIES sequence length differs from the number of atoms, how do you manage the additional SELFIES tokens?\n\n- What are the time and memory costs for running the E3FP algorithm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The introduction of 3D tokens for representing molecular substructures at atomic level removes the dependency on structure encoders.\n- The combination of 1D and 3D embeddings enables alignment of molecular sequence and structural data at atomic level.\n- The novel pre-training denoising and translation tasks enhance cross-modal training and alignment.\n- The model demonstrates performance gains over existing baselines across different benchmarks, including 3D-dependent tasks.", "weaknesses": "- The term “tokenization” may not be accurate here, as 3D-MolT5 does not achieve unified molecule-text modeling in token space as UniMoT [1] does. UniMoT can generate molecule tokens as text, but 3D-MolT5 lacks the ability to generate 3D tokens. The token generation ability is the major difference between discrete and continuous structures, with 3D-MolT5 limited to encoding rather than generating 3D data.\n\n  - The 3D “tokenization” in this paper functions more like a 3D molecular encoder, encoding each atom into a 3D embedding rather than generating 3D structures.\n  - 3D-MolT5 lacks 3D generation abilities because it cannot map 3D embeddings back to specific atoms or structures.\n  - The motivation section resembles that of UniMoT. A more detailed comparison between the two would be helpful.\n\n- The E3FP algorithm introduces additional space and time complexity than using a well-pretrained encoder.\n\n- The reasoning for further encoding $d_i$ with $E_{3D}(d_i)$ instead of using $d_i$ directly is unclear.\n\n- It’s not explained why the authors chose the E3FP algorithm over a 3D molecular encoder like Uni-Mol. And the paper does not provide an ablation comparison to illustrate any advantages of E3FP.\n\n- In Algorithm 1, it appears that $\\cup$ is used to indicate concatenation; this should be stated for clarity.\n\n[1] Juzheng Zhang, Yatao Bian, Yongqiang Chen, and Quanming Yao. Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.0086.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hYOjZCx2xr", "reviewer": "Reviewer_mQNb", "summary": "The authors propose 3D-MolT5, a unified framework designed to model molecule in both sequence and 3D structure spaces. The key innovation of our approach lies in mapping fine-grained 3D substructure representations into a specialized 3D token vocabulary. This methodology facilitates the integration of sequence and structure representations in a tokenized format, enabling 3D-MolT5 to encode molecular sequences, molecular structures, and text sequences within a unified architecture. Leveraging this tokenized input strategy, we build a foundation model that unifies the sequence and structure data formats. We then conduct joint pre-training with multi-task objectives to enhance the model’s comprehension of these diverse modalities within a shared representation space. Authors also provide source code for reproducibility.", "questions": "See above section for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors describe developments for LLMs to capture textual information in molecules, as well as graph representation models to capture 3D structure topology of molecules. Limitations of the existing work is also well described. \n\nThere are a variety of tasks evaluated both on text generation as well as structure learning, which is informative and comprehensive.", "weaknesses": "The paper seems to be missing the important citation from the IJCAI survey paper on Graph-based molecular representation learning: \n\n[IJCAI 2023] Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, Yihong Ma, Olaf Wiest, Xiangliang Zhang, Wei Wang, Chuxu Zhang, and Nitesh V. Chawla. 2023. Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 744, 6638–6646. https://doi.org/10.24963/ijcai.2023/744\n\nWhile experiment results achieve competitive results, there seem to be only 2 benchmark datasets being considered. There are more diverse and difficult datasets such as USPTO. Can the authors evaluate on more datasets to make the results more conclusive?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)