Leveraging Discrete Structural Information for Molecule-Text Modeling

## Abstract

Abstract The integration of molecule and language has garnered increasing attention in molecular science. Recent advancements in Language Models (LMs) have demonstrated potential for the comprehensive modeling of molecule and language. However, existing works exhibit notable limitations. Most existing works overlook the modeling of 3D information, which is crucial for understanding molecular structures and also functions. While some attempts have been made to leverage external structure encoding modules to inject the 3D molecular information into LMs, there exist obvious difficulties that hinder the integration of molecular structure and language text, such as modality alignment and separate tuning. To bridge this gap, we propose 3D-MolT5 , a unified framework designed to model both 1D molecular sequence and 3D molecular structure. The key innovation lies in our methodology for mapping fine-grained 3D substructure representations (based on 3D molecular fingerprints) to a specialized 3D token vocabulary for 3D-MolT5. This 3D structure token vocabulary enables the seamless combination of 1D sequence and 3D structure representations in a tokenized format, allowing 3D-MolT5 to encode molecular sequence (SELFIES), molecular structure, and text sequences within a unified architecture.
Alongside, we further introduce 1D and 3D joint pre-training to enhance the model’s comprehension of these diverse modalities in a joint representation space and better generalize to various tasks for our foundation model. Through instruction tuning on multiple downstream datasets, our proposed 3D-MolT5 shows superior performance than existing methods in molecular property prediction, molecule captioning, and text-based molecule generation tasks.
Our code will be available on GitHub soon.

## 1 Introduction

Molecule plays a pivotal role in various scientific and industrial applications, ranging from pharmaceuticals to materials science [, , ].
In recent years, the development of Language Models (LMs) [, , ] has led to a growing focus on joint modeling of molecule and language [, , , ]. This is primarily driven by the extensive molecular-related textual descriptions available in biological literature [, ] and databases []. By training on these textual descriptions, LMs can capture comprehensive knowledge that aids in understanding molecules, thus better generalizing to molecule-related tasks, such as molecule-text retrieval [, ], molecule captioning, and description-guided molecule design [, , ].
Several studies have investigated molecule-text LMs along the way.
Given that language itself represents a form of sequence, some attempts [, , ] employ autoregressive pre-training of LMs to simultaneously model 1D molecular sequences (e.g., SMILES [], SELFIES [, ]) and text sequences.
To further integrate 2D graph knowledge of molecule with text, some works leverage either contrastive pre-training between 2D molecular graphs and text [, , , , ], or align the 2D molecular graph encoders with LMs [, ] through multi-stage pre-training inspired by BLIP2 [].

However, most existing works overlooked the molecular 3D structure, which contains critical stereochemical information, such as the spatial arrangement of atoms and functional groups. This information is crucial for tasks such as molecular energy prediction [, ], 3D function prediction [], protein-ligand interaction [], and so on.
Recognizing this, several efforts have been made to integrate the 3D molecular information into LMs. The most representative ones, MolLM [], 3D-MoLM [], and MolBind [], incorporate 3D molecular inputs with language by integrating external molecular structure encoders.
They use external molecular encoders to first get the molecular representations, which are then aligned with LMs with continued training/tuning. Despite achieving preliminary success, these methods exhibit several shortcomings: (a) The molecular structure encoder is pre-trained separately, where the representation is not in the same space with LMs. (b) Though modality alignment can help mitigate the representation gap, it is challenging due to differences in model architectures, and accessing alignment quality is also difficult. (c) The final performance of the integrated model heavily relies on the performance of the external structure encoder, which can not be directly controlled.

To address these limitations, we propose 3D-MolT5, a versatile T5 framework that can understand 3D Molecular structure and handle various 3D-dependent tasks with text instructions.
To enable LMs to understand 3D molecular structure, we introduce a 3D molecular tokenization method based on the molecular E3FP (Extended 3D Fingerprint) [] algorithm. E3FP can simply tokenize the 3D molecular structure into 3D tokens,
with each token encapsulating the 3D information of a substructure centered around a specific atom.
Additionally, since most 1D SELFIES tokens (e.g., [C] and [O]) represents specific atoms, the tokens from both 1D and 3D modalities can be directly aligned at the atomic level.
Hence, the embeddings of the same atom in both 1D and 3D tokens are then summed to form the final joint representation, which can then enable the T5 model to effectively learn and comprehend molecular information by leveraging both the sequence and structure tokens, leading to improved performance across various molecular tasks.

With tokenized 1D and 3D molecules, we conduct comprehensive molecule-text pre-training for our 3D-MolT5 framework.
The pre-training tasks are inspired by the “T5 objective" from T5 [] pre-training, which uses a “recover masked spans” objective that aims at recovering each masked span.
In our 3D-MolT5, there are five types of pre-training tasks:
1) 1D denoising: apply T5 objective to SELFIES, text, and wrapped text, where molecules mentioned in the text are replaced with SELFIES.
2) 1D + 3D joint denoising: apply T5 objective to the summed 1D and 3D tokens, and the target is to recover the masked 1D SELFIES tokens.
3) 3D to 1D translation: given the 3D molecular tokens, the target is to generate the corresponding 1D SELFIES.
4) 3D molecule to text translation: given the summed 1D and 3D tokens, the target is to generate its textual description.
5) Text to 1D molecule translation: given the textual description, the target is to generate the corresponding 1D SELFIES.

After pre-training, we conduct instruction tuning on various molecule-text tasks, including molecular property prediction (both 3D-dependent and 3D-independent) on PubChemQC [, ], QM9 [, ] and PubChem [] datasets, molecule captioning (3D-dependent) on PubChem [] dataset, and text-based molecule generation (3D-independent) on CheBI-20 dataset [].
The results show that 3D-MolT5 achieves superior performance across these tasks, underscoring its versatility and efficacy in both 3D-dependent and 3D-independent molecule-text tasks.

Figure: Figure 1: Overview of the 3D-MolT5 multi-task pre-training. The upper 4 tasks involve the “recover masked spans” task, where consecutive spans of the input are replaced with sentinel tokens such as <X>, <Y>, <Z>. The bottom 3 tasks are translation tasks. The input modalities are annotated with small icons. Tokens with 3D structure information are colored in blue, and [3D] refers to 3D tokens.
Refer to caption: x1.png

## 2 Related Work

##### Molecular Encoding.

The 1D sequence is the most widely used form of molecular encoding, typically obtained by traversing the atoms in a molecular graph in a specified order. The simplified molecular-input line-entry system (SMILES) [, ] is the most used one, while Self-Referencing Embedded Strings (SELFIES) [, ] has recently gained popularity due to its strong robustness.
2D graph representations align naturally with molecular topological structures, as molecules inherently form 2D graphs with atoms serving as nodes and chemical bonds as edges.
In contrast, 3D structures provide information about the spatial arrangement of atoms, offering valuable insights into molecular geometry and interactions.
In addition to these representations, molecular fingerprints (FPs) are widely used, especially in molecular similarity searches and virtual screening []. FPs encode critical information about molecular structure as a sequence of binary bits, which are useful for property predictions [, ]. Common examples include Morgan FPs, such as extended-connectivity fingerprints (ECFPs) and functional class fingerprints (FCFPs) [], as well as RDKit (topological) fingerprints [].
However, these fingerprints primarily capture 2D topological features and do not account for 3D structural patterns.
Spherical extended 3D fingerprints (E3FPs) [] effectively incorporate neighboring atoms in 3D space to encode 3D information.
Our structure-aware 3D molecular vocabulary is built on E3FP [], which is then used for our atom-centric joint representation.

##### Molecule-Text Cross Modeling.

Recent advances have integrated LMs with molecules to enhance the understanding of molecular structures and properties [, ].
MolT5 [], BioT5 [], and BioT5+ [] are T5-based [] models that jointly trained on 1D molecular sequences and text, then fine-tuned for molecule-related tasks.
Mol-Instructions [] and LlaSMol [] provide instruction datasets with molecules represented as SMILES or SELFIES for instruction tuning.
2D molecular graphs are also used to inject topological knowledge to LMs through external graph encoding modules.
For example, MoMu [], MoleculeSTM [], and MolFM [] employ cross-modal contrastive learning on 2D molecular graphs and corresponding text.
MolCA [] aligns 2D molecular space with text space using cross-modal pre-training.
Some recent works also consider 3D molecular information.
For instance, MolBind [] uses contrastive learning to align the 2D graph encoder, 3D structure encoder, and language encoder, showing strong performance in cross-modal retrieval tasks.
MolLM [] adopt a similar learning objective but use Transform-M-style [] encoder to integrate both 2D and 3D molecular information.
Following a similar paradigm to BLIP2 [], 3D-MoLM [] equips the LM with an external 3D molecular encoder and curates the 3D-MoIT dataset for 3D molecule-text instruction tuning.
3D-MoLM combines 1D SMILES and 3D molecular representation for 3D molecule-to-text interpretation.
However, these methods do not model 1D molecular sequences, 3D molecular structures, and text in a unified manner, as the 3D molecule is encoded by an external module.
This separation presents challenges in achieving a holistic integration of multimodal molecular information.

## 3 Methods

Figure: Figure 2: The process of 3D molecular tokenization and alignment between 1D SELFIES tokens and 3D tokens. We choose one conformer of the 2-(Formylamino)benzoic acid (CID: 101399) as the example. At each iteration of E3FP, each atom and its neighborhood substructure is represented by a 3D token. The alignment between 1D SELFIES tokens and 3D tokens is shown at the bottom table.
Refer to caption: x2.png

The overview of 3D-MolT5 is shown in Figure 1.
We first give some preliminaries in Section 3.1, focusing on the 1D sequence representation of molecule.
In Section 3.2, we present the 3D structure-aware E3FP fingerprint and how we adopt it in 3D-MolT5.
We then introduce our 3D molecular tokenization and the integration with 1D tokenization in Section 3.3.
Lastly, we present our multi-task pre-training framework in Section 3.4.

### 3.1 Preliminary

For a given molecule $M$, we use SELFIES [] as its sequence representation, which is more robust than SMILES [, ].
In SELFIES, each token, typically representing an atom group (like [C] and [=N]) or structure directive (like [Ring1] and [=Branch1]), is enclosed within brackets and can be naturally tokenized based on these delimiters.
Therefore, the 1D SELFIES sequence can be represented as $S=\{s_{i}\}_{i=1}^{m}$, where $s_{i}$ is the SELFIES token and $m$ is the length of sequence(^1^11Typically, $m$ is large than the number of atoms $n$, as in SELFIES there are some structure directive tokens and other uninformative tokens.).
To ensure that each molecule has a unique SELFIES, and consequently a unique atom flatten order, we employ the canonical form of SELFIES. We realize this by firstly getting the canonical SMILES using RDKit [], then converting it to SELFIES.

### 3.2 3D Structure-aware Fingerprint

Given a molecule $M$ with $n$ atoms, and one of its 3D structure $C$, we use the E3FP [] algorithm to generate its 3D fingerprint $F$.
$F$ represents a bit vector, with $|F|$ denoting its length.
We flatten the atoms of $M$ into a canonical atom sequence $A=\{a_{i}\}_{i=1}^{n}$, where $a_{i}$ represents one of the heavy atoms.
For each atom $a_{i}$, with $k$ as the number of iterations for the E3FP algorithm, we can derive its 3D token ${\bm{d}_{i}}$, which consists of $k+1$ nonnegative integers.
This process, illustrated in Figure 2, includes several steps as follows.
(1) Structure representation initialization. At iteration 0, we define a set of atomic invariants for each atom $a_{i}$ within $M$, including properties such as the number of heavy atom immediate neighbors, atomic number, mass, and charge. These atomic invariants are hashed into integers $\hat{d}_{i0}$ to establish unique identifiers for each atom $a_{i}$.
(2) Iterative spherical shell expansion.
In each iteration, we extend the radius of the spherical shells centered on each atom $a_{i}$ by a radius multiplier $r$, capturing all atoms within this expanded radius. The connectivity and spatial arrangement of these atoms are encoded relative to their central atom, incorporating both bonded and non-bonded interactions. This allows E3FP to capture the 3D structure of the molecule, including relative atomic orientations and distances that are not evident in simpler 2D representations.
(3) Termination and outputs. Each iteration results in a new layer of structural information, extending until either all atoms in $M$ are included or a predefined maximum number of iterations $k$ is reached.
For each atom $a_{i}$, we combine its substructure information in each layer by concatenating the hashed integers, resulting in the vector $\hat{\bm{d}_{i}}=[\hat{d}_{i0},\hat{d}_{i1},\dots,\hat{d}_{ik}]$.
Each item of $\hat{\bm{d}_{i}}$ is then converted to $\bm{d}_{i}$ by taking its value modulo $|F|$, expressed mathematically as $\bm{d}_{i}=\hat{\bm{d}_{i}}\mod|F|$.
In 3D-MolT5, we do not use the E3FP $F$ but instead use the 3D tokens $\bm{d}_{i}$ for each atom $a_{i}$, which are then integrated with 1D sequence.
Specifically, the $k$ is set to 3, as our preliminary experiments indicate that three iterations are sufficient for the E3FP algorithm to converge, capturing all potentially occurring substructures for the vast majority of molecules.
The $|F|$ is set to 4096 and $r$ is set to 1.718 Å.
For more details of the E3FP algorithm, we suggest to refer the original paper [].

### 3.3 Molecular Tokenization

After obtaining the 1D SELFIES sequence $S=\{s_{i}\}_{i=1}^{m}$ and 3D sequence (i.e., the sequence of 3D tokens) $D=\{\bm{d}_{i}\}_{i=1}^{n}$, we further combine them to create the final 1D + 3D joint representation.
As shown in Figure 2, for each molecule, most 1D SELFIES tokens $s_{i}$ uniquely represent an atom $a_{i}$.
Each 3D token $\bm{d}_{i}$, a $k+1$ dimensional vector of nonnegative identifiers, also uniquely corresponds to an atom $a_{i}$.
Thus, tokens from both 1D and 3D modalities can be aligned at the atomic level.
Based on this alignment, we construct our 1D + 3D joint representation, capturing both the chemical sequence and spatial configuration of the molecule.

We define the 1D embedding $\mathbf{E}_{\text{1D}}$ for the 1D SELFIES tokens and the 3D embedding $\mathbf{E}_{\text{3D}}$ for the 3D tokens. Note that the 3D embedding $\mathbf{E}_{\text{3D}}$ is directly indexed by the 3D tokens, as each item of $\bm{d}_{i}$ is a nonnegative integer.
For each molecule, each SELFIES token $s_{i}$ is mapped to its corresponding embedding vector $\mathbf{E}_{\text{1D}}(s_{i})\in\mathbb{R}^{H}$, where $H$ is the hidden dimension.
For the 3D token embeddings, we first map each item $\bm{d}_{ij}$ in $\bm{d}_{i}$ to the vector $\mathbf{E}_{\text{3D}}(\bm{d}_{ij})\in\mathbb{R}^{H}$. These $k+1$ vectors $\mathbf{E}_{\text{3D}}(\bm{d}_{ij})$ are then averaged to obtain the 3D embedding for token $\bm{d}_{i}$, which is given by $\mathbf{E}_{\text{3D}}(\bm{d}_{i})=(1/k+1)\sum_{j=0}^{k}{\mathbf{E}_{\text{3D}
}(\bm{d}_{ij})}$.
The final joint representation $\mathbf{E}$ for each atom token is computed as:

$$ $\mathbf{E}=\begin{cases}\mathbf{E}_{\text{1D}},&\text{if only 1D information is present}\\ \mathbf{E}_{\text{3D}},&\text{if only 3D information is present}\\ 0.5\cdot\mathbf{E_{\text{1D}}}+0.5\cdot\mathbf{E}_{\text{3D}},&\text{if both 1 D and 3D information are present}\end{cases}$ $$

This combination captures both the sequential and spatial information of the molecule, resulting in a comprehensive representation suitable for various downstream tasks.

### 3.4 Pre-training

With the molecular tokenization approach introduced above, we can train LMs to incorporate 3D molecular structures.
Our model backbone is T5 [], a transformer-based encoder-decoder model, which forms the foundation of our 3D-MolT5. Detailed model configurations are provided in Appendix A.
For pre-training, we design two categories of tasks within a multi-task framework:
(1) self-supervised denoising task to recover masked spans, and (2) translation tasks between different modalities to further enhance the model’s capability (see ablation study in Section 5).
More details about pre-training configurations and datasets can be found in Appendix B.

##### Denoising Tasks.

Given a sequence $X=\{x_{i}\}_{i=1}^{n}$, some consecutive spans of $X$ are randomly masked by sentinel tokens, and the model learns to reconstruct these spans:

$$ $\mathcal{L}_{\mathrm{D}}=-\sum_{t=1}^{|M|}\log P(X_{M}\mid X_{\backslash M}),$ (1) $$

where $X_{M}$ are the tokens that need to be recovered/generated, $|M|$ is the number of masked tokens, and $X_{\backslash M}$ is the input $X$ with the masked spans replaced by sentinel tokens.

The denoising pre-training tasks are divided into two categories based on input modalities:
(1) 1D denoising, including denoising on SELFIES, text, and “wrapped” text.
For SELFIES, we random sample 50M molecules from the PubChem [] database and represent them as canonical SELFIES.
For text, we use both the C4 [] dataset from the general domain and full articles from PubMed Central [, ] in the biomedical domain.
The concept of “wrapped” text is derived from MolXPT [], where the molecules mentioned in the text are replaced with SELFIES.
As in  [, , ], training on such “wrapped” text is helpful as the context is full of descriptions about the molecule.
We follow the same pipeline as MolXPT to detect molecules mentioned in text using BERN2 [] on the PubMed abstracts, and then suffix them with corresponding SELFIES.
(2) 1D + 3D joint denoising, which involves denoising on summed 1D SELFIES tokens and 3D tokens, with the goal of recovering the 1D SELFIES tokens(^2^22In our work, since the 3D molecular token for each atom contains $k+1$ different items, we do not aim to predict the 3D molecular tokens for the denoising tasks, which is same for the translation tasks.).
We use the PCQM4Mv2 dataset from the OGB Large Scale Challenge [] for this task, which includes 3.37M DFT-calculated [] 3D molecular structures that are commonly regarded as ground-state 3D structures.

##### Translation Tasks.

In addition to denoising tasks, we also add translation tasks between modalities to enhance the representation learning,

$$ $\mathcal{L}_{\mathrm{T}}=-\sum_{t=1}^{|Y|}\log P(Y\mid X),$ (2) $$

where $X$ is the input sequence, such as 3D molecule tokens, and $Y$ is the target output sequence, such as the 1D text sequence.

Concretely, we incorporate three translation tasks from $X$ to $Y$ to further connect different modalities:
(1) 3D to 1D translation.
We use the same PCQM4Mv2 dataset as in 1D + 3D denoising in Section 3.4.
The input is the sequence of 3D token representations $\mathbf{E}_{\text{3D}}$ of the molecule, and the output is the corresponding 1D SELFIES.
(2) 3D molecule to text translation.
We use the pre-training split of the PubChem dataset collected by 3D-MoLM [], which includes 298K 3D molecule-text pairs from the PubChem [] database.
The input is the summed 1D and 3D tokens of the molecule, and the output is the corresponding textual description.
(3) Text to 1D molecule translation.
We use the same data as (2), but the input is the textual description of the molecule, and the output is the corresponding 1D SELFIES.

**Table 1: MAE results of computed property prediction tasks on PubChem [] and PubChemQC [] datasets. The valid answer rate is also reported as LMs may fail to generate valid numerical responses. 3D-dependent properties are colored in blue. ${\dagger}$ refers to a variant of 3D-MoLM [] that is initially pre-trained on the original PubChem text without GPT-3.5 enrichment. * represents no fine-tuning.**
| Dataset | PubChem | PubChemQC |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Weight (g/mol) | LogP | TPSA (Å^2) | Complexity | HOMO (eV) | LUMO (eV) | H-L Gap (eV) |
| Non-LM |  |  |  |  |  |  |  |
| Uni-Mol | 20.35 | 0.59 | 13.48 | 57.24 | 0.32 | 0.35 | 0.21 |
| Specialist |  |  |  |  |  |  |  |
| Llama2-7B | 22.10 (96%) | 1.45 (95%) | 15.87 (92%) | 69.74 (93%) | 1.24 (96%) | 1.04 (95%) | 0.88 (92%) |
| 2D-MoLM | 21.48 (94%) | 0.88 (96%) | 13.52 (92%) | 55.74 (94%) | 0.92 (98%) | 0.80 (96%) | 0.67 (93%) |
| 3D-MoLM${\dagger}$ | 16.18 (96%) | 0.95 (96%) | 10.26 (94%) | 49.15 (95%) | 0.45 (98%) | 0.36 (96%) | 0.41 (94%) |
| 3D-MoLM | 14.79 (95%) | 0.66 (97%) | 9.71 (93%) | 44.85 (94%) | 0.26 (97%) | 0.25 (94%) | 0.28 (94%) |
| 3D-MolT5 | 12.30 (100%) | 0.44 (100%) | 3.93 (100%) | 29.51 (100%) | 0.08 (100%) | 0.08 (100%) | 0.08 (100%) |
| Generalist |  |  |  |  |  |  |  |
| Llama2-7B* | 42.18 (82%) | 2.10 (85%) | 27.11 (84%) | 121.87 (76%) | 2.87 (70%) | 1.89 (71%) | 1.86 (70%) |
| Llama2-7B | 27.42 (92%) | 1.78 (93%) | 17.07 (90%) | 78.16 (92%) | 1.89 (90%) | 1.26 (90%) | 1.25 (91%) |
| 2D-MoLM | 20.80 (92%) | 1.36 (94%) | 12.47 (89%) | 52.70 (91%) | 1.52 (93%) | 1.13 (92%) | 1.09 (88%) |
| 3D-MoLM${\dagger}$ | 19.54 (93%) | 0.92 (92%) | 11.14 (92%) | 54.68 (90%) | 0.65 (94%) | 0.41 (92%) | 0.55 (89%) |
| 3D-MoLM | 16.58 (92%) | 0.78 (95%) | 10.90 (90%) | 45.49 (89%) | 0.35 (95%) | 0.36 (93%) | 0.32 (90%) |
| 3D-MolT5 | 14.54 (100%) | 0.61 (100%) | 6.37 (100%) | 37.59 (100%) | 0.11 (100%) | 0.11 (100%) | 0.11 (100%) |

## 4 Experiments

We evaluate 3D-MolT5 on three types of text-based molecule-related downstream tasks:
(1) molecular property prediction, including both 3D-independent properties (e.g., molecular weight, LogP) and 3D-dependent properties (e.g., HOMO-LUMO gap);
(2) 3D molecule captioning;
(3) text-based molecule generation.
All downstream data is formatted as instructions, with tasks framed as conditional text or molecule generation based on the input instructions.
The training objective is the same as the standard cross-entropy loss used in pre-training the language model.

Following 3D-MoLM [] and Mol-Instructions [],
we present performances of two variants of 3D-MolT5:
Specialist, which is fine-tuned only on a specific task;
Generalist, which is fine-tuned in a multi-task setup.
We maintain the same multi-task setup as the baselines for a fair comparison.
More details about the downstream datasets, Generalist settings, and baselines are in Appendix C.

### 4.1 Molecular Property Prediction

Following 3D-MoLM [], we assess 3D-MolT5 on two types of molecular property prediction tasks.
(1) For computed property prediction, we focus on the MAE performance by extracting the predicted numerical value of the property from the output text.
(2) For descriptive property prediction, we evaluate text similarity metrics between the predicted text and the ground truth.

#### 4.1.1 Computed Property Prediction

Setup.
We adopt three datasets to evaluate the performance of 3D-MolT5 on computed property prediction task, including QM9 [, ], PubChemQC [, ], and PubChem [].
The QM9 [, ] dataset includes over 130,000 molecules with their ground-state 3D structures obtained through DFT computations [], each molecule having fewer than nine heavy atoms.
It is widely used for quantum property prediction, including HOMO, LUMO, and HOMO-LUMO gap (H-L gap).
The PubChemQC [, ] dataset is on a larger scale, containing 3.37M molecules with more heavy atoms with their DFT-calculated [] 3D structures.
We use the same quantum properties as QM9 for PubChemQC.
The PubChem [] database also provides various molecular properties, and we follow [] to select four properties that can be inferred from 1D or 2D molecular information.

For the QM9 dataset, we use instruction data from Mol-Instructions [].
For the PubChemQC and PubChem datasets, we use instruction data constructed by 3D-MoLM [].

Baselines & Evaluation.
We compare 3D-MolT5 against three types of baseline models based on input modalities.
For 1D sequence-based models, we include Llama2-7B [], Vicuna [], Mol-Instructions [], and BioT5+ [].
For 2D graph-based models, we include 2D-MoLM [] and InstructMol [].
For the 3D structure-based model, we compare with Uni-Mol [] and 3D-MoLM [].
Note that models based on 2D graphs or 3D structures can also concurrently accept 1D sequences as input [, ].

Results.
The computed property results for the PubChem and PubChemQC datasets are shown in Table 1, and for QM9 in Table 2.
Key observations from the results include:
(1) 3D-MolT5 achieves superior performances on 3D-independent properties over all baseline methods on the PubChem [] dataset.
For molecular hydrophobicity (LogP), determined by 1D and 2D features like functional groups, molecular connectivity, and topology, 3D-MolT5 consistently surpasses the LMs trained on 1D, 2D, and 3D molecular information.
(2) 3D-MolT5 exhibits significant improvements on 3D-dependent properties.
For energy properties including HOMO, LUMO, and HOMO-LUMO gap, which are largely determined by 3D molecular structure, 3D-MolT5 shows strong performance enhancements.
For the Specialist version on the PubChemQC [] dataset, the improvements over the previous SOTA methods [, ] are 0.18 eV, 0.17 eV, and 0.13 eV, respectively.
For the Generalist version on the QM9 [] dataset, the average improvement is 0.0008 Ha.
(3) Compared to Uni-Mol [], 3D-MolT5 also shows consistent improvement.
Uni-Mol [] is specially designed for 3D molecular representation learning and pre-trained on large-scale (209M) 3D molecular data.
The superiority of 3D-MolT5 illustrates the benefit of unified 3D molecule-text modeling. Through comprehensive pre-training, 3D-MolT5 gains both structural knowledge from molecules and contextual knowledge from the biological literature, enhancing its generalization to molecular property prediction tasks.
(4) Compared to 3D-MoLM [], 3D-MolT5 also exhibits continuous enhancement.
3D-MoLM [] integrates the molecular encoder Uni-Mol [] with Llama2-7B [] as the language decoder for 3D molecule-text interpretation.
With a 3D molecule-to-text projector, the molecular encoder and LM can be aligned by training on molecule-text pairs.
The superiority of 3D-MolT5 can be attributed to multi-task pre-training and 3D molecular tokenization, which significantly improve 3D-MolT5’s ability to handle complex fine-grained 3D molecular structures and enhance cross-modal understanding.
(5) On PubChemQC and PubChem datasets, the Generalist version of 3D-MolT5 also outperforms all baselines but slightly underperforms compared to the Specialist version, likely due to the nature of multi-task training. These results demonstrate the capability of 3D-MolT5 to handle multiple tasks simultaneously as a Generalist.

**Table 2: MAE results on computed property prediction tasks on QM9 [] dataset. * means direct inference without further fine-tuning.**
| Model | HOMO (Ha) | LUMO (Ha) | H-L Gap (Ha) | Avg $\downarrow$ |
| --- | --- | --- | --- | --- |
| Generalist |  |  |  |  |
| Llama2-7B* (5-shot) | 0.7367 | 0.8641 | 0.5152 | 0.7510 |
| Vicuna-13B* (5-shot) | 0.7135 | 3.6807 | 1.5407 | 1.9783 |
| Mol-Instructions | 0.0210 | 0.0210 | 0.0203 | 0.0210 |
| BioT5+ | 0.0022 | 0.0024 | 0.0028 | 0.0025 |
| InstructMol-G | 0.0060 | 0.0070 | 0.0082 | 0.0070 |
| InstructMol-GS | 0.0048 | 0.0050 | 0.0061 | 0.0050 |
| 3D-MolT5 | 0.0017 | 0.0016 | 0.0016 | 0.0017 |

**Table 3: Results of the descriptive property prediction task on PubChem [] dataset. ${\dagger}$ refers to a variant of 3D-MoLM [] that is initially pre-trained on the original PubChem text without GPT-3.5 enrichment. * means direct inference without further fine-tuning.**
| Model | BLEU-2 | BLEU-4 | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR |
| --- | --- | --- | --- | --- | --- | --- |
| Specialist |  |  |  |  |  |  |
| Llama2-7B | 28.15 | 23.24 | 35.14 | 22.08 | 30.41 | 46.87 |
| 2D-MoLM | 30.84 | 25.09 | 38.46 | 24.22 | 33.04 | 50.92 |
| 3D-MoLM${\dagger}$ | 30.33 | 24.47 | 38.48 | 23.93 | 32.98 | 51.33 |
| 3D-MoLM | 32.00 | 26.13 | 40.13 | 25.55 | 34.64 | 52.15 |
| 3D-MolT5 | 51.24 | 43.06 | 56.06 | 40.79 | 52.48 | 56.97 |
| Generalist |  |  |  |  |  |  |
| Llama2-7B* | 25.22 | 21.16 | 31.48 | 19.21 | 25.22 | 43.17 |
| Llama2-7B | 27.68 | 22.81 | 34.73 | 21.55 | 29.91 | 46.39 |
| 2D-MoLM | 30.23 | 24.57 | 37.85 | 22.95 | 32.28 | 50.08 |
| 3D-MoLM${\dagger}$ | 29.92 | 24.44 | 38.62 | 22.83 | 32.30 | 50.81 |
| 3D-MoLM | 31.81 | 26.08 | 40.13 | 25.87 | 34.99 | 51.93 |
| 3D-MolT5 | 49.84 | 41.42 | 54.23 | 38.45 | 50.39 | 54.76 |

#### 4.1.2 Descriptive Property Prediction

Setup.
For the descriptive property prediction task, we evaluate the performance of our proposed 3D-MolT5 on the PubChem dataset, as constructed by 3D-MoLM [].
Unlike computed property prediction, this task entails generating descriptions of molecular 1D, 2D, and 3D properties in natural language, necessitating accurate and contextually relevant textual output.

Baselines & Evaluation.
We compare 3D-MolT5 against the same baselines evaluated on PubChemQC [] and PubChem [] datasets as Section 4.1.1.
Following MolT5 [], the evaluation metrics are commonly used text generation metrics, including BLEU [], ROUGE [], and METEOR [], to measure the similarity between the generated property descriptions and ground truth.

Results.
The results for the PubChem dataset are presented in Table 3.
From the table, we can have several findings:
(1)
3D-MolT5 outperforms all baseline methods. Specifically, the Specialist version shows significant improvements, with BLEU-2 and ROUGE-L scores increasing by 19.24 and 17.84, respectively. The Generalist version also shows substantial gains, with improvements of 18.03 and 15.4 in BLEU-2 and ROUGE-L, respectively.
(2) The Generalist version of 3D-MolT5 outperforms all baselines but slightly underperforms compared to the Specialist version.

**Table 4: Results for the 3D molecule captioning task on PubChem [] dataset. ${\dagger}$ refers to a variant of 3D-MoLM [] that is initially pre-trained on the original PubChem text without GPT-3.5 enrichment.**
| Model | BLEU-2 | BLEU-4 | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR |
| --- | --- | --- | --- | --- | --- | --- |
| Specialist |  |  |  |  |  |  |
| MolT5-Small | 22.53 | 15.23 | 30.44 | 13.45 | 20.30 | 23.98 |
| MolT5-Base | 24.51 | 16.61 | 32.19 | 14.04 | 21.35 | 26.10 |
| MolT5-Large | 25.87 | 17.28 | 34.07 | 16.42 | 23.41 | 28.04 |
| MoMu-Small | 22.86 | 16.01 | 30.98 | 13.65 | 20.75 | 24.35 |
| MoMu-Base | 24.74 | 16.77 | 32.45 | 14.62 | 22.09 | 27.16 |
| MoMu-Large | 26.34 | 18.01 | 34.75 | 16.86 | 24.76 | 28.73 |
| 3D-MoLM${\dagger}$ | 29.82 | 22.39 | 37.23 | 22.49 | 31.07 | 32.69 |
| 3D-MoLM | 30.32 | 22.52 | 36.84 | 22.32 | 31.23 | 33.06 |
| 3D-MolT5 | 42.05 | 34.16 | 48.13 | 33.20 | 42.33 | 44.69 |
| Generalist |  |  |  |  |  |  |
| Llama2-7B | 27.01 | 20.94 | 35.76 | 20.68 | 28.88 | 32.11 |
| 2D-MoLM | 27.15 | 21.19 | 36.02 | 20.76 | 29.12 | 32.28 |
| 3D-MoLM${\dagger}$ | 29.25 | 22.07 | 36.48 | 21.80 | 30.95 | 33.12 |
| 3D-MoLM | 28.95 | 21.63 | 36.51 | 21.26 | 30.02 | 33.55 |
| 3D-MolT5 | 43.11 | 34.98 | 48.69 | 33.54 | 42.72 | 44.82 |

### 4.2 3D Molecule Captioning and Text-based Molecule Generation

Despite the molecular property prediction tasks, we also evaluate our 3D-MolT5 on 3D molecule captioning and text-based molecule generation tasks.

#### 4.2.1 3D Molecule Captioning

Setup.
We use PubChem [] dataset for 3D molecule captioning to evaluate 3D-MolT5’s proficiency to understand the 3D molecular structure.
This dataset contains approximately 15,000 3D molecular structure-text pairs source from the PubChem database [].
Each input molecule has an optimized 3D structure, and the output is the corresponding caption that may include 3D-related descriptions.
Specifically, the molecular captions include both molecular names and descriptions to assess the model’s capability in name prediction [] and description prediction [], offering more comprehensive evaluation compared to the descriptive properties in Section 4.1.2.

Baselines & Evaluation.
The compared baselines are classified into three types based on input modalities.
For 1D sequence-based models, we include MolT5 [] and Llama2-7B [].
For 2D graph-based models, we include MoMu [] and 2D-MoLM [].
For 3D structure-based models, we compare with 3D-MoLM [].
Note that the 2D graph or 3D structure-based models may also take a 1D sequence as input simultaneously.
The evaluation metrics are the same as those in Section 4.1.2

Results.
Table 4 shows the performances for the 3D molecule captioning task.
3D-MolT5 demonstrates excellent results, surpassing all baselines.
Compared to 3D-MoLM [], 3D-MolT5 achieves an improvement of approximately 11 points in ROUGE-L and METEOR scores for both the Specialist and Generalist versions.
3D-MolT5 also exceeds baselines with 1D and 2D information, highlighting the importance of 3D structure for molecule understanding and captioning.
This further validates the efficacy of our pre-training on 1D SELFIES, 3D structure, and text with 3D molecular tokenization.

#### 4.2.2 Text-based Molecule Generation

Setup.
To further demonstrate the capability of our 3D-MolT5, we also evaluate its performance on the text-based molecule generation tasks.
We use the CheBI-20 [] dataset, which is widely used for this task [, , , , ].
The input for this task is the textual description of the molecule, and the target is to generate a 1D molecular sequence that fits the description.

Baselines & Evaluation.
The compared baselines include Llama2-7B [], GPT-3.5 [], GPT-4 [], T5 [], MolT5 [], MoMu [], MolFM [], GIT-Mol [], MolXPT [], and BioT5 [].
Following [], the evaluation metric includes BLEU [], exact match score, levenshtein distance, fingerprint similarity [, , ], FCD score [], text2mol [] score, and validity.

**Table 5: Results on text-guided molecule generation task on ChEBI-20 [] dataset.**
| Model | BLEU$\uparrow$ | Exact$\uparrow$ | Levenshtein$\downarrow$ | MACCS FTS$\uparrow$ | RDK FTS$\uparrow$ | Morgan FTS$\uparrow$ | FCD$\downarrow$ | Text2Mol$\uparrow$ | Validity$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Generalist |  |  |  |  |  |  |  |  |  |
| Llama2-7B (0-shot) | 0.104 | 0.000 | 84.18 | 0.243 | 0.119 | 0.089 | 42.01 | 0.148 | 0.631 |
| Llama2-7B (2-shot) | 0.693 | 0.022 | 36.77 | 0.808 | 0.717 | 0.609 | 4.90 | 0.149 | 0.761 |
| GPT-3.5-turbo (0-shot) | 0.489 | 0.019 | 52.13 | 0.705 | 0.462 | 0.367 | 2.05 | 0.479 | 0.802 |
| GPT-3.5-turbo (10-shot) | 0.790 | 0.139 | 24.910 | 0.847 | 0.708 | 0.624 | 0.57 | 0.571 | 0.887 |
| GPT-4-0314 (10-shot) | 0.857 | 0.280 | 17.14 | 0.903 | 0.805 | 0.739 | 0.41 | 0.593 | 0.899 |
| Specialist |  |  |  |  |  |  |  |  |  |
| T5-base | 0.762 | 0.069 | 24.950 | 0.731 | 0.605 | 0.545 | 2.48 | 0.499 | 0.660 |
| T5-large | 0.854 | 0.279 | 16.721 | 0.823 | 0.731 | 0.670 | 1.22 | 0.552 | 0.902 |
| MolT5-base | 0.769 | 0.081 | 24.458 | 0.721 | 0.588 | 0.529 | 2.18 | 0.496 | 0.772 |
| MolT5-large | 0.854 | 0.311 | 16.071 | 0.834 | 0.746 | 0.684 | 1.20 | 0.554 | 0.905 |
| MoMu-base | 0.815 | 0.183 | 20.520 | 0.847 | 0.737 | 0.678 | - | 0.580 | 0.863 |
| MolFM-base | 0.822 | 0.210 | 19.445 | 0.854 | 0.758 | 0.697 | - | 0.583 | 0.892 |
| GIT-Mol | 0.756 | 0.051 | 26.315 | 0.738 | 0.582 | 0.519 | - | - | 0.928 |
| MolXPT | - | 0.215 | - | 0.859 | 0.757 | 0.667 | 0.45 | 0.578 | 0.983 |
| BioT5 | 0.867 | 0.413 | 15.097 | 0.886 | 0.801 | 0.734 | 0.43 | 0.576 | 1.000 |
| 3D-MolT5 | 0.849 | 0.487 | 10.527 | 0.884 | 0.806 | 0.744 | 0.41 | 0.574 | 1.000 |

Results.
The results are presented in Table 5.
Our 3D-MolT5 outperforms all compared baselines on most metrics.
Notably, 3D-MolT5 achieves an exact match score of 0.487, indicating that nearly 50% of the generated molecules precisely match the ground truth molecules.
These results underscore that 3D-MolT5 has acquired extensive knowledge about molecules during pre-training, enabling it to effectively generate accurate molecular sequences based on textual descriptions.

## 5 Study

Figure: Figure 3: Ablation studies on PubChemQC [] dataset. The evaluation metric is MAE.
Refer to caption: extracted/5654605/imgs/ablation_study.png

To validate the efficacy of 3D molecular tokenization and multi-task pre-training, we conduct ablation studies focused on property prediction using the PubChemQC [] dataset, a task heavily reliant on 3D structural information.
The ablation results are shown in Figure 3.
More case studies are shown in Appendix D.

Whether 3D input truly help?
To access the impact of 3D input, we pre-train and fine-tune a variant of 3D-MolT5 excluding all the 3D structure information.
As illustrated in Figure 3, removing 3D information leads to a performance drop on 3D-dependent property prediction task.
For example, the MAE for the HOMO-LUMO gap drops from 0.0791 to 0.0968.
This indicates that integrating 3D structure information into LMs can enhance its understanding of the molecule.

Whether 3D-related pre-training help?
To demonstrate the efficacy of our 3D-related pre-training, we remove the 1D + 3D joint denoising task and translation tasks separately.
The results in Figure 3 indicate that both of them contribute to improvements in 3D-related downstream tasks, underscoring the importance of incorporating 3D information into the pre-training process.

## 6 Conclusion

In this paper, we introduce 3D-MolT5, a unified framework that integrates both 1D molecular sequences and 3D molecular structures to enhance the capabilities of language models in handling various molecular tasks.
By proposing a 3D molecular tokenization method, we can effectively map 3D structures to 3D tokens.
The combination of 1D SELFIES tokens and 3D tokens enables comprehensive representation of molecule.
Through extensive pre-training on 1D and 3D data and subsequent instruction tuning, 3D-MolT5 demonstrates superior performance in molecular property prediction, molecule captioning, and text-based molecule generation tasks, significantly outperforming existing methods.

## Appendix A Model Configuration

3D-MolT5 adopts the same architecture as T5 model [] with T5-1.1-base(^3^33[https://huggingface.co/google/t5-v1_1-base](https://huggingface.co/google/t5-v1_1-base)) configuration.
The encoder and decoder have 12 layers.
The dimensions of attention and feed-forward layers are 768 and 2048, respectively.
The number of attention heads is 12.
The size of the 1D vocabulary is 35,045, including original text tokens of T5 and additional SELFIES tokens, and the size of the 3D vocabulary is 4096.
The total number of parameters of 3D-MolT5 is 255M.
We use nanoT5(^4^44[https://github.com/PiotrNawrot/nanoT5](https://github.com/PiotrNawrot/nanoT5)) [] as our codebase.

## Appendix B Pre-training

The pre-training is done on eight NVIDIA 80GB A100 GPUs.
The total number of steps for pre-training is 400,000, with warm-up steps set to 10,000.
AdamW [] with Root Mean Square (RMC) scaling optimizer is used.
The peak learning rate is 2e-3 with cosine decay, and the minimum learning rate is 1e-5.
The maximum length for input and output is 512.
The batch size is set to 768.
As shown in Table 6, the sizes of pre-training datasets vary significantly.
To balance the data from different tasks during pre-training, we implement a batch-level balancing strategy. Each batch evenly includes data from all tasks, ensuring a more balanced and comprehensive pre-training process. For smaller datasets, such as molecule-text pairs from PubChem, we employ a round-robin strategy to repeat their usage multiple times, compensating for their limited size.
For all molecular data, we first get its canonical SMILES from the provided SMILES or 3D structure using RDKit [], and then convert it to SELFIES using selfies toolkit(^5^55[https://github.com/aspuru-guzik-group/selfies](https://github.com/aspuru-guzik-group/selfies)) [].
The resulting SELFIES are also wrapped by special tokens $\langle bom\rangle$ and $\langle eom\rangle$ to differentiate from text.

**Table 6: Statistics of pre-training datasets. Deno. refers to T5 [] denoising task; Tran. refers to translation task. For the PubChem dataset, the 3D structure is obtained using the MMFF algorithm in RDKit [] and the text enriched by GPT-3.5 [].**
| Data | Text | Molecule | Task | Size |  |
| --- | --- | --- | --- | --- | --- |
| 1D | 3D |  |  |  |  |
| PubChem SELFIES | - | $\checkmark$ | - | Deno. | 38,400,000 |
| C4-English | $\checkmark$ | - | - | Deno. | 2,210,000 |
| PubMed Central full articles | $\checkmark$ | - | - | Deno. | 38,400,000 |
| PubMed abstracts | $\checkmark$ | $\checkmark$ | - | Deno. | 33,404,528 |
| PCQM4Mv2 | - | $\checkmark$ | $\checkmark$ | Deno. & Tran. | 3,377,055 |
| PubChem molecule-text pairs | $\checkmark$ | $\checkmark$ | $\checkmark$ | Tran. | 298,861 |

## Appendix C Fine-tuning

Here we introduce more details about fine-tuning, including details about datasets and baselines.
The fine-tuning is done on a single NVIDIA 80GB A100 GPU.

Details for datasets for fine-tuning are shown in Table 7.
For all downstream datasets, we follow the same pipeline as described in Appendix B to first get the canonical SMILES from the provided SMILES or 3D structure using RDKit [], and then convert it to SELFIES wrapped by $\langle bom\rangle$ and $\langle eom\rangle$.
All reported results for 3D-MolT5 are the mean value obtained from three independent random runs.

**Table 7: Dataset statistics for donwstream fine-tuning. All the datasets are in instruction format. Small differences exist between our processed datasets and the original version, as we discard the data that can not be processed by E3FP [].**
| Dataset | Molecule | Task | Size (Train/Validation/Text) |
| --- | --- | --- | --- |
| PubChemQC | 3D | Computed Property Prediction | 2,463,404/308,024/308,248 |
| QM9 | 3D | Computed Property Prediction | 347,774/1,928/1,928 |
| PubChem | 3D | Computed Property Prediction | 46,532/3,885/7,746 |
| 3D | Descriptive Property Prediction | 59,775/4,980/9,940 |  |
| 3D | 3D Molecule Captioning | 11,955/996/1988 |  |
| CheBI-20 | 1D | Text-based Molecule Generation | 26,407/3,301/3,300 |

For PubChemQC [] and PubChem [] datasets,
we use the instruction versions built by 3D-MoLM [].
The Generalist version of 3D-MolT5 here is trained simultaneously on these two datasets with three types of tasks: computed property prediction, description property prediction, and 3D molecule captioning.
For descriptive property prediction, the descriptive text is generated by employing GPT-3.5 [] to read molecular captions and create five QA pairs for each molecule.
The reported baseline results are derived from 3D-MoLM [].
Specifically, the baseline method 2D-MoLM is a variant of 3D-MoLM [], where the 3D molecular encoder is replaced with a 2D molecular encoder.
The baseline Llama2-7B [] directly removes the 3D molecular encoder of 3D-MoLM [] and uses 1D SMILES as the molecular representation.

For QM9 [] dataset, we use its instruction version built by Mol-Instructions [].
The 3D structures are downloaded from DeepChem [].
The Generalist version for QM9 is trained on the combination of its three subsets: HOMO, LUMO, and HOMO-LUMO gap.
The reported baseline results are derived from BioT5+ [].

For CheBI-20 [] dataset, we manually convert it to instruction version.
To avoid data leakage, we exclude the molecules of the CheBI-20 test set that are also present in the PubChem 3D molecule-text pairs in pre-training. In this task, molecular names are removed from the text to prevent the model from learning a simple mapping from molecular names to 1D sequences.
The reported baseline results are mainly sourced from MolT5 [], MolReGPT [], MolFM [], GIT-Mol [], MolXPT [], and BioT5 [].

## Appendix D Case Study

The cases for computed molecular property prediction are shown in Table 8.
We can find that 3D-MolT5 can give accurate numerical predictions about the computed properties of the input molecule.
For descriptive property prediction, results in Table 9 show that 3D-MolT5 successfully answers the question about the composition of the input molecule, including the attached hexacosanoyl group and sphinganine backbone.

The case for 3D molecule captioning is shown in Table 10.
3D-MolT5 successfully predicts the molecular names, composition, pH, and functional relationship.
The case for text-based molecule generation is shown in Table 11, where 3D-MolT5 generates the molecule that exactly matches the ground truth molecule.

## Appendix E Limitations

In 3D-MolT5, the 3D structure information is only incorporated in the input, and 3D-MolT5 can not generate 3D molecular structure directly.
The molecule-text pairs for pre-training are in limited size (less than 300K).
Larger datasets with high-quality molecule-text pairs may further enhance the performance of 3D-MolT5.
We leave these constraints for future exploration.

**Table 8: Case studies for computed molecular property prediction task.**
| Molecule | Instruction | 3D-MolT5 | Ground truth |
| --- | --- | --- | --- |
|  | I would like to know the Molecular Weight of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with the numerical value only. | The Molecular Weight for the input molecule is 273.40 g/mol. | The Molecular Weight for the input molecule is 263.37 g/mol. |
| I would like to know the LogP of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with the numerical value only. | The LogP for the input molecule is 3.10. | The LogP for the input molecule is 2.80. |  |
| I need to know the Topological Polar Surface Area of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with the numerical value only. | The Topological Polar Surface Area for the input molecule is 49.30 Å^2. | The Topological Polar Surface Area for the input molecule is 49.30 Å^2. |  |
| I need to know the Complexity of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with the numerical value only. | The Complexity for the input molecule is 384.00. | The Complexity for the input molecule is 363.00. |  |
|  | I need to know the HOMO of this molecule, could you please provide it? If uncertain, provide an estimate. Respond with the numerical value only. | The HOMO for the input molecule is -5.793 eV. | The HOMO for the input molecule is -5.769 eV. |
| Please provide the LUMO value for this molecule. If uncertain, provide an estimate. Respond with the numerical value only. | The LUMO for the input molecule is 0.011 eV. | The LUMO for the input molecule is 0.054 eV. |  |
| I am interested in the HOMO-LUMO Gap of this molecule, could you tell me what it is? If uncertain, provide an estimate. Respond with the numerical value only. | The HOMO-LUMO Gap for the input molecule is 5.810 eV. | The HOMO-LUMO Gap for the input molecule is 5.823 eV. |  |
|  | Please provide the highest occupied molecular orbital (HOMO) energy of this molecule. | -0.2131. | -0.2132. |
|  | What is the lowest unoccupied molecular orbital (LUMO) energy of this molecule? | -0.0066. | -0.0064. |
|  | Can you give me the energy difference between the HOMO and LUMO orbitals of this molecule? | 0.2744. | 0.2756. |

**Table 9: Case studies for descriptive molecular property prediction task.**
| Molecule | Instruction | 3D-MolT5 | Ground truth |
| --- | --- | --- | --- |
|  | What is the composition of 1-O-(alpha-D-galactopyranosyl)-N-hexacosanoylsphinganine? | 1-O-(alpha-D-galactopyranosyl)-N-hexacosanoylsphinganine consists of a hexacosanoyl group attached to the nitrogen atom, with a galactosyl residue connected to the oxygen atom at position 1 of the sphinganine backbone. | 1-O-(alpha-D-galactopyranosyl)-N-hexacosanoylsphinganine is composed of a sphinganine backbone, a hexacosanoyl group attached to the nitrogen atom, and an alpha-D-galactopyranosyl residue at the O-1 position. |

**Table 10: Case studies for 3D molecular captioning task.**
| Molecule | Instruction | 3D-MolT5 | Ground truth |
| --- | --- | --- | --- |
|  | Describe the input molecule. | 4-methylbenzoyl-AMP(1-) is an organophosphate oxoanion obtained by deprotonation of the phosphate OH group of 4-methylbenzoyl-AMP; major species at pH 7.3. It is functionally related to an adenosine 5’-monophosphate(2-) and a 4-methylbenzoate. | 4-methylbenzoyl-AMP(1-) is an organophosphate oxoanion obtained by deprotonation of the phosphate OH group of 4-methylbenzoyl-AMP; major microspecies at pH 7.3. It is functionally related to an adenosine 5’-monophosphate(2-) and a p-toluate. |

**Table 11: Case studies for text-based molecular generation task.**
| Description | Instruction | 3D-MolT5 | Ground truth |
| --- | --- | --- | --- |
| The molecule is a member of the class of naphthoates that is 1-naphthoate substituted at positions 3 and 5 by hydroxy and methyl groups respectively; major species at pH 7.3. It has a role as a bacterial metabolite. It is a conjugate base of a 3-hydroxy-5-methyl-1-naphthoic acid. | Generate a molecule that fits the input description. | [C][C][=C][C][=C][Branch1][C][O-1][C][=C][Branch1][=Branch1][C] [=Branch1][C][=O][O][C][Ring1] [#Branch2][=C][C][=C][Ring1][=C]. | [C][C][=C][C][=C][Branch1][C][O-1][C][=C][Branch1][=Branch1][C] [=Branch1][C][=O][O][C][Ring1] [#Branch2][=C][C][=C][Ring1][=C]. |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Given that multimodal approaches are increasingly common, how does 3D-MolT5 specifically leverage 3D structural data differently or more effectively than existing fine-tuned LLMs that incorporate multimodal information?
In the model’s tokenized format for 3D structures, what considerations were made for maintaining structural fidelity across different molecule sizes and complexities? How does the approach handle conformational flexibility within molecular ensembles?
Could the model’s performance be further validated on additional downstream tasks, such as reaction prediction or drug-target interaction, to assess its versatility across a broader range of applications in molecular science?
soundness: 3
strengths: This work shows strength in its comprehensive comparison with various existing models, showcasing 3D-MolT5’s effectiveness across different approaches and highlighting its strong performance in molecular structure understanding. By integrating both molecular sequence and 3D structural information, the study addresses a crucial aspect of molecular modeling that is often underexplored, underscoring its relevance in capturing complex molecular properties.
summary: This paper introduces 3D-MolT5, a novel framework for integrating molecular sequence and 3D structural information within a unified language model. While recent language models have advanced molecular science applications, most neglect critical 3D structural data essential for accurately understanding molecular properties. To address this, 3D-MolT5 uses a unique 3D token vocabulary that captures detailed 3D substructure information, enabling it to encode molecular sequences, spatial structures, and natural language together in a cohesive architecture. Through multi-task pre-training, 3D-MolT5 develops robust cross-modal comprehension. Experimental results show strong generalization, with the model achieving nearly a 70% improvement in molecular property prediction compared to current methods.
weaknesses: While the work presents a solid approach, the concept of multimodal integration itself is not entirely new. Including a comparison with fine-tuned large language models (LLMs) that incorporate multimodal data[1] would further strengthen the evaluation, providing a clearer benchmark comparing to a boarder types of multimodal molecular representation models and other LLM as well e.g. Galactica[2].

[1] Le, Khiem, et al. "MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension." arXiv preprint arXiv:2406.06777 (2024).
[2] Taylor, Ross, et al. "Galactica: A large language model for science." arXiv preprint arXiv:2211.09085 (2022).

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: As noted in the Weaknesses section, although the proposed method achieves significant performance improvements over baseline models, it appears overly complex and includes numerous subcomponents. The necessity of each part is not sufficiently justified.
* The authors introduce a novel pretraining scheme comprising five subtasks. Are all these subtasks equally essential for optimal model performance, or could the model show similar performance using only one or two? Intuitively, the “3D molecule to text translation” task might be sufficient, as it encompasses all three modalities—text, 1D molecular structure, and 3D molecular structure. It would be valuable to explore this question further by showing the performance of 3DMol-T5 after pretraining on each individual task while excluding the remaining four.
* Several recent works [a, b] have demonstrated that language models can process spatial molecular data directly as text containing atom coordinates. Compared to the proposed E3FP-based discrete tokens, this direct coordinate representation is considerably simpler, more transparent, and requires no model modifications. Furthermore, it can serve as both input and output (for example, in spatial molecular generation tasks). Providing a comparison with this approach is important to justify the use of the E3FP-based discrete token scheme.
* The proposed alignment scheme for combining 1D and 3D molecular information involves summing their embeddings as $E=\frac{1}{2}E_{1D} + \frac{1}{2}E_{3D}$ which necessitates modifications to the language model's source code—requiring time and substantial programming knowledge to adapt it for other models. However, is this modification truly necessary? Could the E3FP tokens be interpreted as textual tokens and used directly within the text without altering the model? Intuitively, a sequence such as “$s_0, d_{01}, d_{02}, d_{03}, s_1, d_{11}, d_{12}, d_{13} \cdots$” might accomplish this. It would be useful to elaborate on this question.

Minor suggestions:
* While the QM9 dataset is suitable for demonstrating the model's capability in using 3D molecular information to predict quantum properties, I recommend that the authors consider the QMUGS dataset [c] due to the greater diversity in molecular structures and quantum properties. Notably, QMUGS includes atom-wise properties, which could provide a more comprehensive evaluation. Testing model performance on this dataset could enhance the significance of the experimental section.

a. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files, 2023

b. BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning, 2024

c. QMugs, quantum mechanical properties of drug-like molecules, 2022
soundness: 2
strengths: The paper is well-written and easy to follow, with supplementary materials providing all necessary details to reproduce the method. The concept of integrating spatial molecular fingerprints alongside molecular sequence representations is a novel contribution in the field of language models. Experimental results demonstrate a notable improvement in model performance compared to baseline models.
summary: The authors introduce a novel approach that integrates spatial molecular data by calculating a set of E3FP-based discrete tokens for each atom, then combining these tokens with corresponding SELFIES embeddings. Additionally, they propose a new pretraining scheme designed to improve the alignment and interaction between structural (1D) and spatial (3D) tokens. Finally, the authors present a thorough evaluation of the 3DMol-T5 model, which incorporates these novel methods into an encoder-decoder architecture, across various molecular tasks.
weaknesses: Despite the comprehensive benchmarking of the model across various molecular tasks, the limited ablation studies make it difficult to assess the individual contributions of each component of the proposed approach to overall model performance. Additionally, a minor drawback is the absence of non-language model baselines in the Molecular Property Prediction tasks section. Further details on these points are provided in the Questions section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why do the authors focus solely on T5 instead of exploring other LLM architectures? Could the 3D "tokenization" approach work effectively with models like Llama or Mistral?

- What advantages does the E3FP algorithm have over a 3D molecular encoder?

- How many embeddings are used for encoding SELFIES tokens and 3D tokens?

- Since the SELFIES sequence length differs from the number of atoms, how do you manage the additional SELFIES tokens?

- What are the time and memory costs for running the E3FP algorithm?
soundness: 3
strengths: - The introduction of 3D tokens for representing molecular substructures at atomic level removes the dependency on structure encoders.
- The combination of 1D and 3D embeddings enables alignment of molecular sequence and structural data at atomic level.
- The novel pre-training denoising and translation tasks enhance cross-modal training and alignment.
- The model demonstrates performance gains over existing baselines across different benchmarks, including 3D-dependent tasks.
summary: The paper introduces a unique framework, 3D-MolT5, that integrates 3D structural data and molecular sequences into a unified language model. 3D-MolT5 embeds molecular geometry directly into the language model using discrete 3D tokens generated from the E3FP algorithm. The model is pre-trained with multi-task objectives, including both denoising and translation tasks. The model achieves state-of-the-art results in tasks like molecular property prediction, molecule captioning, and text-based molecule generation.
weaknesses: - The term “tokenization” may not be accurate here, as 3D-MolT5 does not achieve unified molecule-text modeling in token space as UniMoT [1] does. UniMoT can generate molecule tokens as text, but 3D-MolT5 lacks the ability to generate 3D tokens. The token generation ability is the major difference between discrete and continuous structures, with 3D-MolT5 limited to encoding rather than generating 3D data.

  - The 3D “tokenization” in this paper functions more like a 3D molecular encoder, encoding each atom into a 3D embedding rather than generating 3D structures.
  - 3D-MolT5 lacks 3D generation abilities because it cannot map 3D embeddings back to specific atoms or structures.
  - The motivation section resembles that of UniMoT. A more detailed comparison between the two would be helpful.

- The E3FP algorithm introduces additional space and time complexity than using a well-pretrained encoder.

- The reasoning for further encoding $d_i$ with $E_{3D}(d_i)$ instead of using $d_i$ directly is unclear.

- It’s not explained why the authors chose the E3FP algorithm over a 3D molecular encoder like Uni-Mol. And the paper does not provide an ablation comparison to illustrate any advantages of E3FP.

- In Algorithm 1, it appears that $\cup$ is used to indicate concatenation; this should be stated for clarity.

[1] Juzheng Zhang, Yatao Bian, Yongqiang Chen, and Quanming Yao. Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.0086.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See above section for details.
soundness: 3
strengths: The authors describe developments for LLMs to capture textual information in molecules, as well as graph representation models to capture 3D structure topology of molecules. Limitations of the existing work is also well described. 

There are a variety of tasks evaluated both on text generation as well as structure learning, which is informative and comprehensive.
summary: The authors propose 3D-MolT5, a unified framework designed to model molecule in both sequence and 3D structure spaces. The key innovation of our approach lies in mapping fine-grained 3D substructure representations into a specialized 3D token vocabulary. This methodology facilitates the integration of sequence and structure representations in a tokenized format, enabling 3D-MolT5 to encode molecular sequences, molecular structures, and text sequences within a unified architecture. Leveraging this tokenized input strategy, we build a foundation model that unifies the sequence and structure data formats. We then conduct joint pre-training with multi-task objectives to enhance the model’s comprehension of these diverse modalities within a shared representation space. Authors also provide source code for reproducibility.
weaknesses: The paper seems to be missing the important citation from the IJCAI survey paper on Graph-based molecular representation learning: 

[IJCAI 2023] Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, Yihong Ma, Olaf Wiest, Xiangliang Zhang, Wei Wang, Chuxu Zhang, and Nitesh V. Chawla. 2023. Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 744, 6638–6646. https://doi.org/10.24963/ijcai.2023/744

While experiment results achieve competitive results, there seem to be only 2 benchmark datasets being considered. There are more diverse and difficult datasets such as USPTO. Can the authors evaluate on more datasets to make the results more conclusive?

Meta-review: This paper introduces 3D-MolT5, a novel framework that unifies molecular sequence and 3D structural information within a single language model. By mapping fine-grained 3D substructure representations into a specialized 3D token vocabulary derived from the E3FP algorithm, the model effectively integrates sequence and structure representations in a tokenized format. The authors propose a multi-task pre-training scheme, including denoising and translation tasks, to enhance cross-modal comprehension and alignment of molecular sequences, spatial structures, and textual data. 

A key strength of the paper is its innovative approach to integrating 3D structural data directly into a language model, addressing a crucial and often underexplored aspect of molecular modeling. The model removes dependency on separate structure encoders by incorporating 3D tokens at the atomic level, allowing for effective alignment between molecular sequences and 3D structures. The comprehensive evaluation across multiple tasks showcases the model's robustness and generalization capabilities, with notable improvements over current methods.

For the potential improvements, the detailed ablation studies would strengthen the model design and clarify the the individual contributions of each components. Evaluating the model on a broader range of datasets, including more diverse and complex ones like USPTO or QMUGS, would enhance the conclusiveness of the results.

Overall, I recommend the acceptance of this paper.

Decision: Accept (Poster)