UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation

## Abstract

Abstract The remarkable success of Large Language Models (LLMs) across diverse tasks has driven the research community to extend their capabilities to molecular applications. However, most molecular LLMs employ adapter-based architectures that do not treat molecule and text modalities equally and lack a supervision signal for the molecule modality. To address these issues, we introduce UniMoT , a Uni fied Mo lecule- T ext LLM adopting a tokenizer-based architecture that expands the vocabulary of LLM with molecule tokens.
Specifically, we introduce a Vector Quantization-driven tokenizer that incorporates a Q-Former to bridge the modality gap between molecule and text. This tokenizer transforms molecules into sequences of molecule tokens with causal dependency, encapsulating high-level molecular and textual information.
Equipped with this tokenizer, UniMoT can unify molecule and text modalities under a shared token representation and an autoregressive training paradigm, enabling it to interpret molecules as a foreign language and generate them as text.
Following a four-stage training scheme, UniMoT emerges as a multi-modal generalist capable of performing both molecule-to-text and text-to-molecule tasks. Extensive experiments demonstrate that UniMoT achieves state-of-the-art performance across a wide range of molecule comprehension and generation tasks.

## 1 Introduction

The incredible capabilities of Large Language Models (LLMs) [, ] have led to their widespread use as versatile tools for completing diverse real-world tasks. This success has sparked interest in Multi-modal LLMs [, ], which aim to enhance LLMs by enabling them to process multi-modal inputs and outputs. Prior research efforts [, , , , , , ] have focused on adapting LLMs to molecular tasks, resulting in the development of molecular LLMs. These molecular LLMs can analyze molecule structures [, , ], address drug-related inquiries [, ], assist in synthesis and retrosynthesis planning [], support drug design [], and more.

Figure: (a) Projection-Based Architecture.
Refer to caption: x1.png

Prevalent molecular LLMs commonly employ adapter-based architectures, adopting either a linear projection [, , ] or a Q-Former [, ] as an adapter to translate molecule features into the semantic space of LLM, as illustrated in Figure 1(a) and Figure 1(b).
Despite demonstrating initial capabilities in molecular comprehension and yielding promising results in molecule-to-text generation tasks, they still fall short in text-to-molecule generation tasks.
The critical issue within these methods is their unequal treatment of molecules and text, resulting in a lack of supervision for the molecule modality. This limitation significantly constrains model capacity and effectiveness.

Discretizing continuous molecule features into discrete molecule tokens offers a promising solution for conducting both molecule-to-text and text-to-molecule generation tasks. By treating tokens from different modalities equally, we can predict the next molecule or text token in an autoregressive manner.
However, directly discretizing molecule features poses several challenges:
(i) This approach results in long sequences, with lengths equivalent to the number of atoms in a batch. LLMs typically experience a quadratic increase in computational complexity with sequence length [].
(ii) Molecule tokens derived from molecule features lack left-to-right causal dependency, which conflicts with the unidirectional attention mechanism in LLMs.
(iii) Molecule features lack textual information, hindering effective molecule-text interactions and alignment.

To this end, we present UniMoT, a Unified Molecule-Text LLM that adopts a tokenizer-based architecture, integrating molecule comprehension and generation, as depicted in Figure 1(c).
A pivotal aspect of UniMoT’s architecture is the molecule tokenizer for transforming molecules into molecule tokens. We introduce a Vector Quantization-driven [] tokenizer, which incorporates a Q-Former [] to bridge the modality gap between molecules and text.
Specifically, we incorporate causal masks for the queries, enabling the Q-Former to generate a causal sequence of queries compatible with the unidirectional attention in LLMs.
The sequence of queries is subsequently quantized into a sequence of molecule tokens using a learnable codebook.
The molecule tokens encapsulate high-level molecular and textual information, which are then aligned with the
latent space of a generative model via an MLP adapter, enabling the generation of desired molecules.

Pretrained LLMs can integrate the molecule tokenizer by treating molecule tokens as new words and constructing a molecule vocabulary through mapping the learned codebook. We adopt the unified discrete token representation for molecules and text, coupled with the unified next-token-prediction training paradigm of LLM. This unification of representation and training paradigm enhances LLMs’ ability to understand molecule-text interactions and alignment.
UniMoT interprets molecules akin to understanding a foreign language, and generates them as if they were text. Following a four-stage training scheme, UniMoT serves as a multi-modal generalist capable of performing both molecule comprehension and generation tasks.

Our contributions can be summarized as follows:

- •
We introduce a molecule tokenizer specifically designed for LLMs, enabling the tokenization of molecules into short sequences of molecule tokens with causal dependency. These tokens encapsulate high-level molecular and textual information and can be decoded into desired molecules during inference.
- •
We present UniMoT, a unified molecule-text LLM that adopts a tokenizer-based architecture instead of traditional adapter-based architectures. UniMoT unifies the modalities of molecule and text under a shared token representation and an autoregressive training paradigm.
- •
UniMoT exhibits remarkable capabilities in multi-modal comprehension and generation. Extensive experiments demonstrate that UniMoT achieves state-of-the-art performance across a wide spectrum of molecule comprehension tasks and molecule generation tasks.

## 2 Related Works

#### Molecular Large Language Models.

The recent emergence of Vision Large Language Models (VLLMs) [, , ] has catalyzed advancements in Molecular LLMs, which encompass both single modality and multi-modality approaches.
In the single modality domain, researchers are exploring diverse molecule representations, such as 1D sequences like SMILES strings [, , ], 2D molecule graphs [, ], 3D geometric conformations [, ], and textual information from the literature [, , ].
In the multiple modalities domain, various innovative approaches are being employed. MolT5 [], a T5-based [] model, is designed for SMILES-to-text and text-to-SMILES translations. Other works, such as MoMu [], MoleculeSTM [], MolFM [], and GIT-Mol [], leverage cross-modal contrastive learning to align the representation spaces of molecules and text.
Additionally, some studies use multi-modal learning architectures to develop molecular LLMs, which often adopt adapter-based architectures. For instance, InstructMol [], GraphGPT [], and DrugChat [] employ a simple projection layer to map molecule features to LLM’s input space. MolCA [] and 3D-MoLM [] utilize a Q-Former [] to bridge the modality gap between molecules and text.
However, these methods do not treat molecule and text modalities equally and lack a supervision signal for the molecule modality, limiting model capacity and effectiveness.

#### Vector Quantization.

Vector Quantization (VQ) [] is a widely used technique in generative models. VQ-VAE [] converts an image into a set of discrete codes within a learnable discrete latent space by learning to reconstruct the original image. VQ-GAN [] enhances the generation quality by leveraging adversarial and perceptual objectives.
In the context of molecules, VQ has been effectively applied to quantize molecule features. For example, DGAE [] introduces a VQ model specifically for molecules, where molecules are encoded into discrete latent codes. Mole-BERT [] uses VQ to rethink the pre-training of GNNs for molecular tasks. IMoLD [] proposes using VQ to enhance invariant molecule representations, and VQSynergy [] demonstrates the use of VQ for drug discovery.

## 3 Method

Our objective is to leverage the reasoning and generation capabilities of LLMs to enhance the comprehension and generation of molecule and text data. To achieve this, we focus on representing these modalities uniformly within the token representation, utilizing the next-token-prediction training paradigm of LLMs.
As illustrated in Figure 2, we introduce a molecule tokenizer (Section 3.1) designed to transform molecules into molecule tokens by learning to reconstruct the input molecule. The molecule sequence can then be concatenated with the text sequence to form a multi-modal sequence, which is subsequently fed into an LLM for autoregressive pretraining (Section 3.2), as illustrated in Figure 3.
The LLM vocabulary is expanded with molecule tokens mapped from the learned codebook.
We introduce a four-stage training scheme for UniMoT (Section 3.3) comprising Causal Q-Former pretraining, molecule tokenizer pretraining, unified molecule-text pretraining, and task-specific instruction tuning.
UniMoT is capable of performing both molecular comprehension and generation tasks following the training scheme.

Figure: Figure 2: Illustration of our proposed molecule tokenizer. The tokenizer generates discrete molecule tokens, which can be fed into LLMs for downstream tasks. The generated molecule tokens can be decoded into molecules using the adapter and the SMILES decoder during inference.
Refer to caption: x4.png

### 3.1 Molecule Tokenizer for LLMs

#### Molecule Encoder.

We represent the structural information of a molecule as a graph, denoted by $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of atoms and $|\mathcal{V}|=N$ is the number of atoms.
The task of the molecule encoder is to extract molecule features that are context-aware and encompass diverse local neighborhood structural information.
By employing a molecule encoder, we obtain molecule features $\mathbf{X}\in\mathbb{R}^{N\times F}$, where each atom feature contains context-aware structural information.

#### Causal Q-Former.

We employ a Q-Former model introduced by BLIP-2 [] to generate queries $\mathbf{Z}=\{\boldsymbol{z}_{i}\}_{i=1}^{M}\in\mathbb{R}^{M\times d}$ containing high-level molecular and textual information, where $M$ represents the number of queries and $d$ denotes the dimension of queries. Specifically, we incorporate causal masks into the queries, ensuring that they only interact with preceding queries. This ensures the sequence of queries maintains a causal dependency, aligning with the requirements of LLMs operating on text sequence.
Details regarding the Causal Q-Former can be found in Appendix A.

#### Vector Quantization.

The Causal Q-Former converts molecules and text into a causal sequence of queries.
Subsequently, the causal sequence of queries $\{\boldsymbol{z}_{i}\}_{i=1}^{M}$ is quantized into a causal sequence of molecule tokens $\{s_{i}\}_{i=1}^{M}$ by identifying the closest neighbor in a learnable codebook $\mathcal{C}=\{\boldsymbol{c}_{i}\}_{i=1}^{K}$, where $K$ represents the size of the codebook. The codebook is randomly initialized and optimized during pretraining. Specifically, token $s_{i}$ is determined as follows:

$$ $s_{i}=\operatorname{argmin}_{j\in\{1,\cdots,K\}}\left\|\boldsymbol{z}_{i}- \boldsymbol{c}_{j}\right\|_{2},\quad\text{for}\quad i=1,2,\cdots,M.$ (1) $$

Intuitively, the query $\boldsymbol{z}_{i}$ is quantized to the closest neighbor $\boldsymbol{c}_{s_{i}}$ in the codebook. As the vector quantization process is non-differentiable, we adopt the straight-through estimator [] to train the Causal Q-Former by copying the gradient from the molecule tokens to the queries, as shown in Figure 2.
The resulting embeddings of molecule tokens,
denoted as $\mathbf{C}=\{\boldsymbol{c}_{s_{i}}\}_{i=1}^{M}$, are subsequently utilized for reconstructing molecules.

#### Molecule Reconstruction.

An MLP adapter $\psi$ needs to be trained to align the discrete latent space of molecule tokens with the continuous latent space of a molecular generative model for molecule reconstruction. This can be represented as $\mathbf{X}_{R}=\psi(\mathbf{C})$, where $\mathbf{X}_{R}$ denotes the embeddings for reconstruction.
To achieve alignment, we minimize the Mean Squared Error (MSE) loss between $\mathbf{X}_{R}$ and the SMILES [] embeddings $\mathbf{X}_{S}$ produced by the pretrained SMILES encoder.
Subsequently, we can reconstruct the molecule from $\mathbf{X}_{R}$ using the pretrained SMILES decoder.
The training loss of the tokenizer is expressed as follows:

$$ $\mathcal{L}_{\text{Tokenizer}}=\left\|\mathbf{X}_{R}-\mathbf{X}_{S}\right\|_{2 }^{2}+\frac{1}{M}\sum_{i=1}^{M}\left\|\operatorname{sg}\left[\boldsymbol{z}_{i }\right]-\boldsymbol{c}_{s_{i}}\right\|_{2}^{2}+\frac{\beta}{M}\sum_{i=1}^{M} \left\|\operatorname{sg}\left[\boldsymbol{c}_{s_{i}}\right]-\boldsymbol{z}_{i} \right\|_{2}^{2}.$ (2) $$

Here, the first term represents the alignment loss, the second term is a codebook loss aimed at updating the codebook embeddings, and the third term is a commitment loss that encourages the query to stay close to the chosen codebook embedding.
$\operatorname{sg}[\cdot]$ denotes the stop-gradient operator, and the hyperparameter $\beta$ is set to 0.25.

### 3.2 Unified Molecule-Text Language Model

#### Expanding Vocabulary.

Employing the molecule tokenizer, a molecule can be tokenized into a molecule sequence $\{s_{i}\}_{i=1}^{M}$ with causal dependency. The molecule sequence can be concatenated with the text sequence to form a multi-modal sequence $\{u_{i}\}_{i=1}^{L}$, where $L$ is the length of the multi-modal sequence.
To facilitate the representation of the multi-modal sequence, we construct the molecule vocabulary $\mathcal{V}^{m}=\{\boldsymbol{v}^{m}_{i}\}_{i=1}^{K}$, which maintains the order of the molecule codebook $\mathcal{C}=\{\boldsymbol{c}_{i}\}_{i=1}^{K}$. Additionally, $\mathcal{V}^{m}$ includes several special tokens such as boundary indicators, e.g., [MOL] and [/MOL], to mark the beginning and end of the molecule sequence.
Next, we merge the original text vocabulary $\mathcal{V}^{t}=\{\boldsymbol{v}^{t}_{i}\}_{i=1}^{T}$ with the molecule vocabulary $\mathcal{V}^{m}$.
The unified molecule-text vocabulary $\mathcal{V}=\{\mathcal{V}^{m},\mathcal{V}^{t}\}$ facilitates joint learning from molecules and text under a unified next-token-prediction objective.
As the vocabulary is expanded, the corresponding embeddings and prediction layers also need to be extended, with the newly introduced parameters initialized randomly.

#### Unified Molecule-text Modeling.

Figure: (a) Molecule-to-Text Autoregression.
Refer to caption: x5.png

The multi-modal sequence $\{u_{i}\}_{i=1}^{L}$ is fed into the pretrained LLM for performing multi-modal autoregression.
UniMoT adopts the general Language Modeling (LM) objective to directly maximize the log-likelihood of the data distribution:

$$ $\mathcal{L}_{\text{LM}}=-\sum_{u\in\mathcal{D}}\sum_{i\in\mathcal{I}}\log p \left(u_{i}\mid u_{1},\cdots,u_{i-1};\theta\right),$ (3) $$

where $\mathcal{D}$ represents the dataset, $\mathcal{I}$ represents the set of indices of the generation target, and $\theta$ denotes the parameters of the LLM.
The unification of representation and training paradigm for molecules and text enhances the abilities of LLMs to understand molecule-text interactions and alignment. UniMoT can interpret molecules similar to understanding a foreign language, and generate them as if they were text. We conduct autoregressive pretraining on molecule-to-text and text-to-molecule tasks to enhance the molecule comprehension and generation capabilities.

#### Molecule-to-Text Autoregression.

While structural information is embedded in molecule features and captured by the molecule tokens through the tokenizer, we also aim to incorporate
sequential information of molecules for better comprehension.
Therefore, we concatenate the molecule sequence $\{s_{i}\}_{i=1}^{M}$ with the SMILES [] sequence and a prompt to form the multi-modal input sequence $\{u_{i}\}_{i=1}^{L}$, as illustrated in Figure 3(a). The text sequence of the corresponding molecule caption is used as the generation target.

#### Text-to-Molecule Autoregression.

For molecule generation, a prompt and the molecule caption are concatenated, with a [MOL] token appended to signify the beginning of the molecule sequence, as illustrated in Figure 3(b). The molecule sequence $\{s_{i}\}_{i=1}^{M}$ produced by the tokenizer is used as the generation target. During inference, given a prompt and the molecule caption, the output molecule sequence can be decoded into the desired molecule by the pretrained adapter and SMILES decoder.

### 3.3 Training Strategy

The training strategy for UniMoT is structured across four stages. Stage-1 focuses on Causal Q-Former pretraining with tailored objectives.
In Stage-2, the molecule tokenizer is optimized using the frozen encoders and decoder.
Stage-3 integrates the tokenizer with a language model for multi-modal comprehension and generation.
Finally, Stage-4 fine-tunes UniMoT for specific tasks, aligning it with human instructions and optimizing performance for various molecular applications.
More details regarding the training process can be found in Appendix C.

#### Stage-1: Causal Q-Former Pretraining.

We connect the molecule encoder and Causal Q-Former, leveraging the pretrained MoleculeSTM molecule encoder []. The molecule encoder remains frozen while only the Causal Q-Former is updated.
Both queries and text inputs are used, while only queries serve as input in subsequent stages.
In our experiments, we utilize 16 queries.
We employ three tailored objectives for the pretraining of the Causal Q-Former: Molecule-Text Contrastive Learning (MTC), Molecule-Text Matching (MTM), and Molecule-grounded Text Generation (MTG). The details of these objectives can be found in Appendix A.

#### Stage-2: Molecule Tokenizer Pretraining.

We connect the Causal Q-Former with subsequent blocks and use the objective defined in Equation (2).
We employ the pretrained ChemFormer [] as the generative model. Specifically, we leverage the SMILES encoder and the SMILES decoder provided by ChemFormer. The molecule codebook size is set to $K=2048$.
As shown in Figure 2, we keep the molecule encoder, the SMILES encoder, and the SMILES decoder frozen, while updating the Causal Q-Former, the learnable codebook, and the adapter.

#### Stage-3: Unified Molecule-Text Pretraining.

We integrate the molecule tokenizer with the LLM using the unified vocabulary of molecule tokens and text tokens.
We employ the LM objective defined in Equation (3) to pretrain the LLM.
Pretraining involves molecule-to-text autoregression and text-to-molecule autoregression, aimed at enhancing UniMoT’s multi-modal comprehension and generation capabilities.
To enhance efficiency, we train the LLM using LoRA tuning [].

#### Stage-4: Task-Specific Instruction Tuning.

UniMoT is fine-tuned on seven comprehension and generation tasks: molecular property prediction, molecule captioning, molecule-text retrieval, caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis.
We also utilize LoRA tuning to improve efficiency.
This stage ensures UniMoT can accurately interpret and respond to human instructions, making it versatile and effective for diverse molecular tasks.

## 4 Experiments

### 4.1 Molecule Comprehension Tasks

#### Molecular Property Prediction Task.

The goal of molecular property prediction is to forecast a molecule’s intrinsic physical and chemical properties. For the classification task, we incorporate eight binary classification datasets from MoleculeNet []. Models are tasked with generating a single prediction (“yes” or “no”).
We compare UniMoT with the following baselines: KV-PLM [], AttrMask [], InfoGraph [], MolCLR [], GraphMVP [], MoleculeSTM [], and InstructMol []. The ROC-AUC (%) results on the MoleculeNet datasets are shown in Table 1. The performance of the regression task of molecular property prediction is provided in Appendix D.
Compared to traditional graph learning methods and molecular LLMs like InstructMol [], UniMoT demonstrates consistent improvements across the eight datasets, indicating its robust molecule comprehension abilities.

**Table 1: ROC-AUC (%) of molecular property prediction task (classification) on the MoleculeNet [] datasets. Bold indicates the best performance and underline indicates the second best performance.**
| Model | BBBP$\uparrow$ | Tox21$\uparrow$ | ToxCast$\uparrow$ | Sider$\uparrow$ | ClinTox$\uparrow$ | MUV$\uparrow$ | HIV$\uparrow$ | BACE$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| KV-PLM [] | 70.50 | 72.12 | 55.03 | 59.83 | 89.17 | 54.63 | 65.40 | 78.50 |
| AttrMask [] | 67.79 | 75.00 | 63.57 | 58.05 | 75.44 | 73.76 | 75.44 | 80.28 |
| InfoGraph [] | 64.84 | 76.24 | 62.68 | 59.15 | 76.51 | 72.97 | 70.20 | 77.64 |
| MolCLR [] | 67.79 | 75.55 | 64.58 | 58.66 | 84.22 | 72.76 | 75.88 | 71.14 |
| GraphMVP [] | 68.11 | 77.06 | 65.11 | 60.64 | 84.46 | 74.38 | 77.74 | 80.48 |
| MoleculeSTM [] | 69.98 | 76.91 | 65.05 | 60.96 | 92.53 | 73.40 | 76.93 | 80.77 |
| InstructMol (Vicuna-7B) [] | 70.00 | 74.67 | 64.29 | 57.80 | 91.48 | 74.62 | 68.90 | 82.30 |
| UniMoT (Llama-2-7B) | 71.37 | 76.43 | 65.78 | 59.79 | 92.89 | 75.97 | 78.49 | 83.69 |

#### Molecule Captioning Task.

The molecule captioning task involves generating a comprehensive description of a molecule. We compare UniMoT with several baselines: MolT5 [], MoMu [], InstructMol [], MolCA [], and 3D-MoLM []. BLEU [], ROUGE [], and METEOR [] are adopted as evaluation metrics. UniMoT is evaluated for molecule captioning on the PubChem [] and CheBI-20 [] datasets. Performance on the PubChem dataset is shown in Table 2, while the performance on the CheBI-20 dataset and some concrete examples are presented in Appendix D.

From Table 2, we observe that UniMoT consistently outperforms the baselines by a significant margin on the PubChem [] dataset. This task is more complex than classification or regression, providing a robust measure of the model’s molecule comprehension abilities. Notably, our proposed tokenizer-based architecture surpasses the projection-based architecture (such as InstructMol []), Q-Former-based architecture (such as MolCA [] and 3D-MoLM []), and models trained with contrastive learning strategies (such as MoMu []). The results demonstrate that the molecule tokenizer can generate molecule tokens with high-level molecular and textual information.

**Table 2: Performance (%) of molecule captioning task on the PubChem [] dataset. Bold indicates the best performance and underline indicates the second best performance.**
| Model | BLEU-2$\uparrow$ | BLEU-4$\uparrow$ | ROUGE-1$\uparrow$ | ROUGE-2$\uparrow$ | ROUGE-L$\uparrow$ | METEOR$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| MolT5-Small (T5-Small) [] | 22.5 | 15.2 | 30.4 | 13.5 | 20.3 | 24.0 |
| MolT5-Base (T5-Base) [] | 24.5 | 16.6 | 32.2 | 14.0 | 21.4 | 26.1 |
| MolT5-Large (T5-Large) [] | 25.9 | 17.3 | 34.1 | 16.4 | 23.4 | 28.0 |
| MoMu-Small (T5-Small) [] | 22.9 | 16.0 | 31.0 | 13.7 | 20.8 | 24.4 |
| MoMu-Base (T5-Base) [] | 24.7 | 16.8 | 32.5 | 14.6 | 22.1 | 27.2 |
| MoMu-Large (T5-Large) [] | 26.3 | 18.0 | 34.8 | 16.9 | 24.8 | 28.7 |
| InstructMol (Vicuna-7B) [] | 18.9 | 11.7 | 27.3 | 11.8 | 17.8 | 21.3 |
| MolCA (OPT-125M) [] | 25.9 | 17.5 | 34.4 | 16.6 | 23.9 | 28.5 |
| MolCA (OPT-1.3B) [] | 28.6 | 21.3 | 36.2 | 21.4 | 29.7 | 32.6 |
| 3D-MoLM (Llama-2-7B) [] | 30.3 | 22.5 | 36.8 | 22.3 | 31.2 | 33.1 |
| UniMoT (Llama-2-7B) | 31.3 | 23.8 | 37.5 | 23.7 | 33.6 | 34.8 |

#### Molecule-Text Retrieval Task.

The molecule-text retrieval task involves using a molecule to retrieve text (M2T) and using text to retrieve a molecule (T2M). We compare UniMoT with several baselines: Sci-BERT [], KV-PLM [], MoMu [], MoleculeSTM [], MolCA [], and 3D-MoLM []. We report the performance of retrieval using a batch of 64 random samples and the entire test set, evaluated with the metrics of Accuracy and Recall@20.
We use the checkpoint from Stage-1 of pretraining. UniMoT is evaluated on the datasets of PubChem [], PCdes [], and MoMu []. Performance on the PubChem dataset is shown in Table 3, while performances on the PCdes and MoMu datasets are presented in Appendix D. UniMoT can understand complex molecule-text interactions through the introduction of the Causal Q-Former. From Table 3, UniMoT demonstrates superior performance over the baselines on molecule-to-text retrieval. This underscores UniMoT’s capability in learning fine-grained alignment between molecules and text.

**Table 3: Performance (%) of molecule-text retrieval task on the PubChem [] dataset. Bold indicates the best performance and underline indicates the second best performance.**
| Model | Retrieval in batch | Retrieval in test set |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| M2T (%) | T2M (%) | M2T (%) | T2M (%) |  |  |  |  |  |
| Acc$\uparrow$ | R@20$\uparrow$ | Acc$\uparrow$ | R@20$\uparrow$ | Acc$\uparrow$ | R@20$\uparrow$ | Acc$\uparrow$ | R@20$\uparrow$ |  |
| Sci-BERT [] | 85.3 | 98.7 | 84.2 | 98.4 | 41.7 | 87.3 | 40.2 | 86.8 |
| KV-PLM [] | 86.1 | 98.6 | 85.2 | 98.5 | 42.8 | 88.5 | 41.7 | 87.8 |
| MoMu (Sci-BERT) [] | 87.6 | 99.2 | 86.4 | 99.4 | 47.3 | 90.8 | 48.1 | 89.9 |
| MoMu (KV-PLM) [] | 88.2 | 99.4 | 87.3 | 99.4 | 48.5 | 91.6 | 49.5 | 90.7 |
| MoleculeSTM [] | 90.5 | 99.6 | 88.6 | 99.5 | 52.7 | 92.9 | 53.2 | 92.5 |
| MolCA (OPT-1.3B) [] | 92.6 | 99.8 | 91.3 | 99.5 | 67.9 | 94.4 | 68.6 | 93.3 |
| 3D-MoLM (Llama-2-7B) [] | 93.5 | 100.0 | 92.9 | 99.6 | 69.1 | 95.9 | 70.1 | 94.9 |
| UniMoT (Llama-2-7B) | 93.6 | 100.0 | 92.7 | 99.4 | 69.5 | 96.3 | 69.8 | 94.4 |

### 4.2 Molecule Generation Tasks

**Table 4: Performance of molecule generation tasks on the Mol-Instructions [] datasets, including caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis. Bold indicates the best performance, and underline indicates the second best performance.**
| Model | Exact$\uparrow$ | BLEU$\uparrow$ | Levenshtein$\downarrow$ | RDK FTS$\uparrow$ | MACCS FTS$\uparrow$ | Morgan FTS$\uparrow$ | Validity$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Caption-guided Molecule Generation |  |  |  |  |  |  |  |
| Llama [] | 0.000 | 0.003 | 59.864 | 0.005 | 0.000 | 0.000 | 0.003 |
| Vicuna [] | 0.000 | 0.006 | 60.356 | 0.006 | 0.001 | 0.000 | 0.001 |
| Mol-Instructions [] | 0.002 | 0.345 | 41.367 | 0.231 | 0.412 | 0.147 | 1.000 |
| MolT5 [] | 0.112 | 0.546 | 38.276 | 0.400 | 0.538 | 0.295 | 0.773 |
| UniMoT | 0.237 | 0.698 | 27.782 | 0.543 | 0.651 | 0.411 | 1.000 |
| Reagent Prediction |  |  |  |  |  |  |  |
| Llama [] | 0.000 | 0.003 | 28.040 | 0.037 | 0.001 | 0.001 | 0.001 |
| Vicuna [] | 0.000 | 0.010 | 27.948 | 0.038 | 0.002 | 0.001 | 0.007 |
| Mol-Instructions [] | 0.044 | 0.224 | 23.167 | 0.237 | 0.364 | 0.213 | 1.000 |
| InstructMol [] | 0.129 | 0.610 | 19.664 | 0.444 | 0.539 | 0.400 | 1.000 |
| UniMoT | 0.167 | 0.728 | 14.588 | 0.549 | 0.621 | 0.507 | 1.000 |
| Forward Reaction Prediction |  |  |  |  |  |  |  |
| Llama [] | 0.000 | 0.020 | 42.002 | 0.001 | 0.002 | 0.001 | 0.039 |
| Vicuna [] | 0.000 | 0.057 | 41.690 | 0.007 | 0.016 | 0.006 | 0.059 |
| Mol-Instructions [] | 0.045 | 0.654 | 27.262 | 0.313 | 0.509 | 0.262 | 1.000 |
| InstructMol [] | 0.536 | 0.967 | 10.851 | 0.776 | 0.878 | 0.741 | 1.000 |
| UniMoT | 0.611 | 0.980 | 8.297 | 0.836 | 0.911 | 0.807 | 1.000 |
| Retrosynthesis |  |  |  |  |  |  |  |
| Llama [] | 0.000 | 0.036 | 46.844 | 0.018 | 0.029 | 0.017 | 0.010 |
| Vicuna [] | 0.000 | 0.057 | 46.877 | 0.025 | 0.030 | 0.021 | 0.017 |
| Mol-Instructions [] | 0.009 | 0.705 | 31.227 | 0.283 | 0.487 | 0.230 | 1.000 |
| InstructMol [] | 0.407 | 0.941 | 13.967 | 0.753 | 0.852 | 0.714 | 1.000 |
| UniMoT | 0.478 | 0.974 | 11.634 | 0.810 | 0.909 | 0.771 | 1.000 |

We employ molecule generation tasks, which encompass caption-guided molecule generation [], reagent prediction [], forward reaction prediction [], and retrosynthesis [].
Caption-guided molecule generation involves generating molecular structures based on textual descriptions. Reagent prediction entails determining suitable reagents given reactants and products. Forward reaction prediction involves predicting probable products given specific reactants and reagents. Retrosynthesis involves deconstructing a target molecule into simpler starting materials.
We compare UniMoT with the following baselines: Llama [], Vicuna [], Mol-Instructions [], and InstructMol []. The metrics used to evaluate molecule generation tasks include Exact Match, BLEU [], Levenshtein Distance [], RDKit Fingerprint Similarity [], MACCS Fingerprint Similarity [], and Morgan Fingerprint Similarity []. These metrics evaluate structural similarity between generated and target molecules, along with Validity [], which assesses the proportion of chemically valid molecules generated.
We utilize the Mol-Instructions [] datasets to evaluate the generation capabilities of UniMoT, and the results are presented in Table 4.

As the baselines generate SMILES strings and then convert them to molecules, UniMoT directly leverages the generated molecule tokens and obtains their embeddings from the learned codebook. These embeddings can be decoded to desired molecules through the pretrained adapter and SMILES decoder.
Regarding the results in Table 4, UniMoT exhibits the capability to generate valid molecules with a higher degree of similarity to the target molecules compared to the baselines. UniMoT can generate molecules as if they were text, demonstrating strong generation capabilities and providing a new perspective to molecule generation tasks.

### 4.3 Ablation Studies

#### Cross-Modal Projector.

**Table 5: Ablation study on the projector and representation form for the molecule captioning task using the PubChem [] dataset.**
| Projector | Input to LLM | BLEU-2$\uparrow$ | BLEU-4$\uparrow$ | ROUGE-1$\uparrow$ | ROUGE-2$\uparrow$ | ROUGE-L$\uparrow$ | METEOR$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Projection Layer | Molecule Emb. | 19.3 | 12.1 | 27.9 | 12.3 | 18.1 | 21.5 |
| Q-Former | Query Emb. | 28.6 | 21.3 | 36.2 | 21.4 | 29.7 | 32.6 |
| Causal Q-Former | Causal Emb. | 32.8 | 25.2 | 39.2 | 24.8 | 35.3 | 36.5 |
| Causal Q-Former | Causal Tokens | 31.3 | 23.8 | 37.5 | 23.7 | 33.6 | 34.8 |

We conducted an ablation study on the cross-modal projector, with the results on the molecule captioning task shown in Table 5. The linear projection demonstrated the worst performance, indicating that the molecule features lack textual information, thus hindering effective molecule-text interactions and alignment. Additionally, we compared the performance of a Q-Former with bidirectional self-attention to a Causal Q-Former with causal self-attention.
The results show that queries with causal dependency outperform those with bidirectional dependency. This demonstrates that input with left-to-right causal dependency aligns with the unidirectional attention mechanism in LLMs, leading to improved performance.

#### Discrete vs. Continuous Representation.

We compare the performance of continuous causal embeddings and discrete tokens quantized from causal embeddings as inputs to LLMs. As shown in Table 5, continuous embeddings demonstrate better performance than discrete tokens in understanding molecules. This result is reasonable since the quantization process causes information loss in discrete tokens.
However, we still use discrete token representation to facilitate the autoregressive training paradigm of LLMs, which supports the unification of comprehension and generation tasks. To achieve this unification, we unavoidably sacrifice some performance in comprehension tasks.

#### Model Size and Tuning Stategy.

**Table 6: Ablation study on the model size and tuning strategy for the molecule captioning task using the PubChem [] dataset.**
| Model Size | Tuning Stategy | BLEU-2$\uparrow$ | BLEU-4$\uparrow$ | ROUGE-1$\uparrow$ | ROUGE-2$\uparrow$ | ROUGE-L$\uparrow$ | METEOR$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-2-7B | LoRA Tuning | 31.3 | 23.8 | 37.5 | 23.7 | 33.6 | 34.8 |
| Llama-2-7B | Full Tuning | 32.0 | 24.6 | 38.3 | 24.3 | 34.7 | 35.6 |
| Llama-2-13B | LoRA Tuning | 31.8 | 24.3 | 38.0 | 24.1 | 34.4 | 35.3 |

We conducted a comparison of molecule captioning performance across various model sizes and tuning strategies, as illustrated in Table 6. Our findings indicate that scaling up the LLM to 13B or adopting a full tuning strategy yields only marginal improvements in performance compared to using Llama-2-7B with LoRA tuning.
While larger models and full tuning strategies might offer slight gains in performance, they come at a significant cost in terms of efficiency.
Considering the trade-off between achieving high performance and maintaining efficiency, we have chosen to utilize Llama-2-7B with LoRA tuning in our experiments. This ensures that our model remains both powerful and practical.

## 5 Conclusion

This work introduces UniMoT, an innovative advancement in the field of molecular-textual understanding and generation, which successfully unifies these two distinct modalities under a coherent framework. By introducing a Vector Quantization-driven tokenizer, UniMoT overcomes previous architectural limitations where molecule and text modalities were not treated equally. This molecule tokenizer transforms molecules into sequences of discrete tokens, embedding high-level molecular and textual information. The LLM vocabulary is expanded with molecule tokens mapped from the learned codebook.
Moreover, by employing a four-stage training scheme, UniMoT has emerged as a versatile multi-modal LLM, adept at handling both molecule-to-text and text-to-molecule tasks. Extensive empirical evaluations demonstrate that UniMoT attains state-of-the-art performance across diverse molecule comprehension and generation tasks.

## Appendix A Details of Causal Q-Former

The Q-Former operates as a query-based transformer that utilizes learnable query vectors to interact with molecule features extracted by a frozen encoder.
These queries are essential for extracting relevant information from the molecule features.
The Q-Former comprises both a molecule transformer and a text transformer, sharing self-attention layers.
The molecule transformer incorporates cross-attention layers between self-attention and feed-forward layers, while the text transformer architecture is based on BERT [].
Q-Former employs a cross-attention mechanism where the query vectors selectively attend to different aspects of the molecule features, allowing the model to capture critical details necessary for understanding and generating textual descriptions of molecular properties.

Specifically, we incorporate causal masks into the queries, ensuring that they only interact with preceding queries. This ensures the sequence of queries maintains a causal dependency, aligning with the requirements of LLMs operating on text sequence. The Causal Q-Former is illustrated in Figure 4.
We employ the Causal Q-Former to generate causal queries $\mathbf{Z}=\{\boldsymbol{z}_{i}\}_{i=1}^{M}\in\mathbb{R}^{M\times d}$ containing high-level molecular and textual information, where $M$ represents the number of queries and $d$ denotes the dimension of queries.
Next, we introduce three tailored objectives MTC, MTM, and MTG for the pretraining of the Causal Q-Former.

Figure: Figure 4: Illustration of our proposed Causal Q-Former. The Causal Q-Former provides causal queries for subsequent blocks.
Refer to caption: x7.png

Molecule-Text Contrastive Learning (MTC) aims to align molecule and text features by maximizing their mutual information. This is achieved by maximizing the molecule-text similarity of positive pairs against that of negative pairs.
We utilize the last query $\boldsymbol{z}_{M}$ of the query sequence $\{\boldsymbol{z}_{i}\}_{i=1}^{M}$ as the query representation, since the output query sequence is causal and the last query contains global information from the queries. For text representation, we use the output embedding of the [CLS] token, denoted as $\boldsymbol{y}$. The contrastive learning loss is expressed as follows:

$$ $\mathcal{L}_{\text{MTC}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp((\boldsymbol {z}_{M}^{i})^{T}\boldsymbol{y}^{i}/\tau)}{\sum_{j=1}^{B}\exp((\boldsymbol{z}_{ M}^{i})^{T}\boldsymbol{y}^{j}/\tau)}-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(( \boldsymbol{y}^{i})^{T}\boldsymbol{z}_{M}^{i}/\tau)}{\sum_{j=1}^{B}\exp(( \boldsymbol{y}^{i})^{T}\boldsymbol{z}_{M}^{j}/\tau)},$ (4) $$

where $B$ denotes the batch size, and $\tau$ represents the temperature parameter. Here, $\boldsymbol{z}_{M}^{i}$ and $\boldsymbol{y}^{i}$ refer to the $i$-th query representation and text representation in a batch, respectively.

Molecule-Text Matching (MTM) focuses on learning fine-grained alignment between molecule and text features.
As queries $\{\boldsymbol{z}_{i}\}_{i=1}^{M}$ capture both molecular and textual information through cross-attention and self-attention layers respectively, we utilize the last query $\boldsymbol{z}_{M}$ as input to a binary classifier. This classifier predicts whether a given molecule-text pair is matched or unmatched. The corresponding loss function is formulated as follows:

$$ $\mathcal{L}_{\text{MTM}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(\phi( \boldsymbol{z}_{M}\mid\mathbf{X}^{i},\boldsymbol{t}^{i}))}{\sum_{j=1}^{B}\exp( \phi(\boldsymbol{z}_{M}\mid\mathbf{X}^{i},\boldsymbol{t}^{j}))+\sum_{j=1}^{B} \exp(\phi(\boldsymbol{z}_{M}\mid\mathbf{X}^{j},\boldsymbol{t}^{i}))},$ (5) $$

where $\phi$ represents a binary classifier, and $\mathbf{X}^{i}$ and $\boldsymbol{t}^{i}$ denote the $i$-th input molecule features and input text in a batch, respectively.

Molecule-grounded Text Generation (MTG) focuses on generating textual descriptions given a molecule input. In this task, causal masks for queries are not applied since only textual output is required. However, causal masks are applied for text, allowing each text token to attend to its preceding text tokens and all queries, but not subsequent tokens.
The Language Modeling (LM) loss function is applied to model the generation of text $\boldsymbol{t}^{i}$ conditioned on the molecule input $\mathbf{X}^{i}$, formulated as:

$$ $\mathcal{L}_{\text{MTG}}=-\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{L}\log p\left(t ^{i}_{j}\mid t^{i}_{1},\cdots,t^{i}_{j-1},\mathbf{X}^{i}\right),$ (6) $$

where $t^{i}_{j}$ represents the $j$-th token in the text sequence $\boldsymbol{t}^{i}$. Here, $\mathbf{X}^{i}$ and $\boldsymbol{t}^{i}$ denote the $i$-th input molecule features and generated text in a batch, respectively.

The total loss for training the Causal Q-Former encompasses the three aforementioned objectives:

$$ $\mathcal{L}_{\text{Q-Former}}=\mathcal{L}_{\text{MTC}}+\mathcal{L}_{\text{MTM} }+\mathcal{L}_{\text{MTG}}.$ (7) $$

## Appendix B Details of Datasets

**Table 7: Summary of datasets, their types, tasks, descriptions, URLs, and licenses used for evaluating UniMoT.**
| Dataset | Type | Tasks | Description | URL | License |
| --- | --- | --- | --- | --- | --- |
| BBBP | Classification | Molecular Property Prediction | Predicts blood-brain barrier penetration ability. | [BBBP URL](https://moleculenet.org/) | CC-BY 4.0 |
| Tox21 | Classification | Molecular Property Prediction | Toxicity prediction using the Tox21 initiative data. | [Tox21 URL](https://moleculenet.org/) | Public Domain |
| ToxCast | Classification | Molecular Property Prediction | Broad toxicity prediction with various biological assays. | [ToxCast URL](https://moleculenet.org/) | Public Domain |
| Sider | Classification | Molecular Property Prediction | Predicts drug side effects. | [Sider URL](https://moleculenet.org/) | CC-BY 4.0 |
| ClinTox | Classification | Molecular Property Prediction | Clinical trial toxicity prediction. | [ClinTox URL](https://moleculenet.org/) | Public Domain |
| MUV | Classification | Molecular Property Prediction | Virtual screening for unbiased validation. | [MUV URL](https://moleculenet.org/) | CC-BY 4.0 |
| HIV | Classification | Molecular Property Prediction | Predicts anti-HIV activity of molecules. | [HIV URL](https://moleculenet.org/) | Public Domain |
| BACE | Classification | Molecular Property Prediction | Predicts inhibitors of the BACE-1 enzyme. | [BACE URL](https://moleculenet.org/) | Public Domain |
| QM9 | Regression | Molecular Property Prediction | Predicts various molecular properties such as atomization energy, dipole moment, etc. | [QM9 URL](http://quantum-machine.org/datasets/) | CC-BY 4.0 |
| PubChem | Captioning, Retrieval, Generation | Molecule Captioning, Molecule-Text Retrieval, Caption-guided Molecule Generation | Generates descriptions and retrieves text/molecules based on input molecules/text, and guides molecule generation from captions. | [PubChem URL](https://pubchem.ncbi.nlm.nih.gov/) | Public Domain |
| ChEBI-20 | Captioning | Molecule Captioning | Generates detailed descriptions of molecular structures. | [ChEBI-20 URL](https://www.ebi.ac.uk/chebi/) | CC-BY 4.0 |
| PCdes | Retrieval | Molecule-Text Retrieval | Used for evaluating accuracy in molecule-text retrieval tasks. | [PCdes URL](https://github.com/thunlp/KV-PLM) | CC-BY 4.0 |
| MoMu | Retrieval | Molecule-Text Retrieval | Dataset for molecule-text interaction and retrieval evaluation. | [MoMu URL](https://github.com/BingSu12/MoMu) | CC-BY 4.0 |
| USPTO | Generation | Reagent Prediction, Forward Reaction Prediction, Retrosynthesis | Provides data for predicting reagents, forward reaction outcomes, and retrosynthetic pathways. | [USPTO URL](https://huggingface.co/datasets/zjunlp/Mol-Instructions) | CC-BY 4.0 |

This section provides detailed information about the datasets used in evaluating the performance of UniMoT across various tasks. The datasets are utilized for molecular property prediction, molecule captioning, molecule-text retrieval, caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis task. Each dataset serves a unique purpose in assessing different capabilities of the model. We provide a comprehensive overview of datasets, including their types, associated tasks, descriptions, URLs for access, and licensing information.

We present the details of the Molecular Property Prediction Datasets below:

- •
BBBP []: The Blood-Brain Barrier Penetration dataset predicts the ability of molecules to penetrate the blood-brain barrier.
- •
Tox21 []: This dataset is part of the Toxicology in the 21st Century initiative, used for toxicity prediction.
- •
ToxCast []: Another toxicity prediction dataset with a broader range of biological assays.
- •
Sider []: Side Effect Resource database, used for predicting drug side effects.
- •
ClinTox []: Clinical Toxicity dataset for predicting clinical trial toxicity outcomes.
- •
MUV []: Maximum Unbiased Validation dataset for virtual screening.
- •
HIV []: Human Immunodeficiency Virus dataset for predicting anti-HIV activities.
- •
BACE []: Beta-Secretase 1 dataset for predicting inhibitors of the BACE-1 enzyme, relevant for Alzheimer’s research.
- •
QM9 []: The quantum mechanics properties dataset, where the objective is to predict key quantum mechanics properties of a given molecule, such as HUMO, LUMO, and the HUMO-LUMO gap.

We present the details of the Molecule Captioning Datasets below:

- •
PubChem []: A large dataset of chemical molecules used for generating textual descriptions of molecular structures.
- •
ChEBI-20 []: A subset of the Chemical Entities of Biological Interest database, provides structured and detailed descriptions of molecules.

We present the details of the Molecule-Text Retrieval Datasets below:

- •
PubChem []: Used for both molecule-to-text (M2T) and text-to-molecule (T2M) retrieval tasks.
- •
PCdes []: Another dataset for evaluating M2T and T2M retrieval accuracy.
- •
MoMu []: Dataset specifically designed for molecule-text interactions and retrieval tasks.

We present the details of the Molecule Generation Datasets below:

- •
Mol-Instructions []: These datasets include tasks such as caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis. They are used to evaluate the model’s ability to generate molecular structures based on textual descriptions and other related tasks.
- •
PubChem []: Used for caption-guided molecule generation, generating molecular structures based on textual descriptions.
- •
USPTO []: Used for reagent prediction, forward reaction prediction, and retrosynthesis, providing data for predicting reagents, reaction outcomes, and retrosynthetic pathways.

We summarize the datasets used for evaluating UniMoT in Table 7. It encompasses various types of datasets, including those for classification, regression, captioning, retrieval, and generation tasks. Each dataset is described in terms of its type, tasks it supports, a brief description of its content, its URL for access, and the license under which it is distributed. The licenses vary, with some datasets being in the public domain and others under CC-BY 4.0 license.

## Appendix C Details of Training

#### Stage-1: Causal Q-Former Pretraining.

During Stage-1, we only connect the molecule encoder and the Causal Q-Former, leaving out other blocks.
We leverage the pretrained molecule encoder from MoleculeSTM [], which has undergone extensive contrastive learning with molecule-text pairs.
We utilize the PubChem [] dataset for pretraining, keeping the molecule encoder frozen while updating only the Causal Q-Former.
Both queries and text serve as input to the Causal Q-Former, while only queries serve as input in subsequent stages.
Inspired by BLIP-2 [], we employ three tailored objectives – Molecule-Text Contrastive Learning (MTC), Molecule-Text Matching (MTM), and Molecule-grounded Text Generation (MTG) – for the pretraining of the Causal Q-Former, as detailed in Appendix A.

The dimension of molecule features is set to 300. We use 16 queries, each with a dimension of 768. The size of $\mathbf{Z}$ ($16\times 768$) is much smaller than the size of molecule features $\mathbf{X}$ (e.g., $150\times 300$). The Q-former is pretrained for 50 epochs. We adopt the AdamW optimizer with a weight decay of 0.05, and a cosine decay learning rate scheduler, with a minimal learning rate of 1e-5. The batch size is set to 64. The computational overhead for this pretraining is 20 GPU hours on 4 NVIDIA A100 GPUs.

#### Stage-2: Molecule Tokenizer Pretraining.

We connect the Causal Q-Former with the subsequent blocks and train the molecule tokenizer using the objective defined in Equation (2).
Following the approach of RetMol [], we utilize SMILES strings [] to represent molecules, and employ the pretrained ChemFormer [] as the generative model. Specifically, we leverage the SMILES encoder and SMILES decoder components provided by ChemFormer.
We utilize PubChem [] and CheBI-20 [] datasets, keeping the molecule encoder, SMILES encoder, and SMILES decoder frozen, while updating the Causal Q-Former, codebook, and adapter. Once optimized, the molecule tokenizer remains unchanged throughout the subsequent stages.

The molecule codebook size is set to $K=2048$, and the dimension of codebook embedding is 768.
The tokenizer is pretrained for 50 epochs. We adopt the AdamW optimizer with a weight decay of 0.05, and a cosine decay learning rate scheduler, with a minimal learning rate of 1e-5. The batch size is set to 64. The computational overhead for this pretraining is 40 GPU hours on 4 NVIDIA A100 GPUs.

#### Stage-3: Unified Molecule-Text Pretraining.

We connect the molecule tokenizer with the LLM and employ the LM objective defined in Equation (3) to pretrain the LLM.
We utilize Llama [] as the default LLM. To construct the unified molecule-text vocabulary, we merge 2048 molecule codes with the original text vocabulary.
Pretraining the LLM involves molecule-to-text autoregression and text-to-molecule autoregression, aimed at enhancing UniMoT’s multi-modal comprehension and generation capabilities. We utilize datasets PubChem [] and CheBI-20 [] for this purpose.
To enhance efficiency, we train the LLM using LoRA tuning [].

The multi-modal LLM is pretrained for 10 epochs. We adopt the AdamW optimizer with a weight decay of 0.05, and a cosine decay learning rate scheduler, with a minimal learning rate of 1e-5. The batch size is set to 32. The computational overhead for this pretraining is 50 GPU hours on 4 NVIDIA A100 GPUs.
To reduce CUDA memory usage, we integrate LoRA with the parameters set to $r=8$, $\alpha=32$, and dropout = 0.1. This integration is applied to the k_proj, v_proj, q_proj, and o_proj modules.

#### Stage-4: Task-Specific Instruction Tuning.

We perform instruction tuning to align UniMoT with human instructions through supervised fine-tuning on seven tasks: molecular property prediction, molecule captioning, molecule-text retrieval, caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis.
For the molecular property prediction task, we utilize the quantum mechanics properties dataset [] for regression prediction and the MoleculeNet [] datasets for property classification.
For the molecule captioning and molecule-text retrieval tasks, we employ datasets PubChem [], PCdes [], and MoMu [].
For the molecule generation tasks, we utilize the Mol-Instructions [] datasets to conduct instruction tuning.
We fine-tune UniMoT for 10 epochs on each task using the same optimizer, learning rate scheduler, and LoRA configurations as in Stage-3 pretraining.
Instruction samples for comprehension and generation tasks are shown in Table 8.

**Table 8: Instruction samples for comprehension and generation tasks: molecular property prediction, molecule captioning, molecule-text retrieval, caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis.**
| Task | Instruction |
| --- | --- |
| Molecular Property Prediction (Regression) | Instruction: Could you give me the LUMO energy value of this molecule?<br>(Optional: The SMILES sequence is: SMILES)<br>Output: 0.0576. |
| Molecular Property Prediction (Classification) | Instruction: Evaluate whether the given molecule is able to enter the blood-brain barrier.<br>(Optional: The SMILES sequence is: SMILES)<br>Output: Yes. |
| Molecule Captioning | Instruction: Could you give me a brief overview of this molecule?<br>(Optional: The SMILES sequence is: SMILES)<br>Output: The molecule is an indole phytoalexin that … |
| Molecule-Text Retrieval | Instruction: Retrieve relevant text for the given molecule.<br>(Optional: The SMILES sequence is: SMILES)<br>Output: The molecule is associated with … |
| Caption-Guided Molecule Generation | Instruction: Create a molecule with the structure as described: The molecule is a primary arylamine that …<br>Output: SMILES of the molecule. |
| Reagent Prediction | Instruction: Please provide possible reagents based on the following chemical reaction.<br><REACTANT A> <REACTANT B> ... >> <PRODUCTs><br>Output: SMILES of the reagents. |
| Forward Reaction Prediction | Instruction: With the provided reactants and reagents, propose potential products:<br><REACTANT A> <REACTANT B> ... <REAGENT A> <REAGENT B> ...<br>Output: SMILES of the products. |
| Retrosynthesis | Instruction: Please suggest potential reactants and reagents used in the synthesis of the products: <PRODUCTs><br>Output: SMILES of the reactants and reagents. |

We have summarized the detailed training hyperparameters of UniMoT in Table 9.

**Table 9: The detailed training hyperparameters of UniMoT.**
| Configuration | Q-Former Pretraining | Tokenizer Pretraining | LLM Pretraining |
| --- | --- | --- | --- |
| Molecule Encoder | MoleculeSTM | MoleculeSTM | MoleculeSTM |
| SMILES Encoder | - | ChemFormer | ChemFormer |
| SMILES Decoder | - | ChemFormer | ChemFormer |
| LLM Base | - | - | Llama-2-7B |
| Epoch | 50 | 50 | 10 |
| Optimizer | AdamW | AdamW | AdamW |
| Codebook Size | 2048 | 2048 | 2048 |
| Number of Queries | 16 | 16 | 16 |
| Query Emb. Dim. | 768 | 768 | 768 |
| Molecule Emb. Dim. | 300 | 300 | 300 |
| Batch Size | 64 | 64 | 32 |
| Minimal LR | 1e-5 | 1e-5 | 1e-5 |
| LR Scheduler | Cosine | Cosine | Cosine |
| Warm-up Steps | 1000 | 1000 | 1000 |
| Weight Decay | 0.05 | 0.05 | 0.05 |
| LoRA Config | - | - | $r=8$, $\alpha=32$, $\text{dropout}=0.1$ |
| Precision | bfloat16 | bfloat16 | bfloat16 |
| GPU Usage | 4 NVIDIA A100s | 4 NVIDIA A100s | 4 NVIDIA A100s |
| Training Time | 20 GPU hours | 40 GPU hours | 50 GPU hours |

## Appendix D Details and More Results of Experiments

#### Molecular Property Prediction Task.

Property prediction aims to anticipate a molecule’s intrinsic physical and chemical properties based on its structural or sequential characteristics. In the regression task, we conduct experiments on the quantum mechanics properties dataset QM9 [], where the objective is to predict key quantum mechanics properties of a given molecule, such as HUMO, LUMO, and the HUMO-LUMO gap. We compare UniMoT against several baselines, including Alpaca [], Baize [], Llama-2-7B [], Vicuna-13B [], Mol-Instructions [], and InstructMol []. Mean Absolute Error (MAE) serves as our evaluation metric. The performance of the regression task on the QM9 dataset is presented in Table 10. Compared to previous single-modal instruction-tuned LLMs and molecular LLMs, UniMoT exhibits further improvement on the regression task, showcasing its fundamental comprehension abilities in molecular contexts.

**Table 10: Mean Absolute Error (MAE) of molecular property prediction task (regression) on the QM9 [] dataset. Bold indicates the best performance and underline indicates the second best performance. $\Delta\epsilon$ is the HOMO-LUMO energy gap.**
| Model | HOMO$\downarrow$ | LUMO$\downarrow$ | $\Delta\epsilon\downarrow$ | AVG$\downarrow$ |
| --- | --- | --- | --- | --- |
| Alpaca (Llama-7B) [] | - | - | - | 322.109 |
| Baize (Llama-7B) [] | - | - | - | 261.343 |
| Llama-2-7B [] | 0.7367 | 0.8641 | 0.5152 | 0.7510 |
| Vicuna-13B [] | 0.7135 | 3.6807 | 1.5407 | 1.9783 |
| Mol-Instructions (Llama-7B) [] | 0.0210 | 0.0210 | 0.0203 | 0.0210 |
| InstructMol (Vicuna-7B) [] | 0.0048 | 0.0050 | 0.0061 | 0.0050 |
| UniMoT (Llama-2-7B) | 0.0042 | 0.0047 | 0.0055 | 0.0049 |

#### Molecule Captioning Task.

The molecule captioning task involves generating a comprehensive description of a molecule. For this task, we compare UniMoT with several baselines: MolT5 [], MoMu [], InstructMol [], MolCA [], and 3D-MoLM []. We adopt BLEU [], ROUGE [], and METEOR [] as the evaluation metrics. The performance of UniMoT in the molecule captioning task on the CheBI-20 [] dataset is presented in Table 11. Some concrete examples of molecule captioning task are presented in Table 12.
From the results, it is evident that UniMoT consistently outperforms the baselines by a significant margin. These results underscore the effectiveness of the molecule tokenizer in providing molecule tokens with high-level molecular and textual information, thus enhancing molecule comprehension.

**Table 11: Performance (%) of molecule captioning task on the CheBI-20 [] dataset. Bold indicates the best performance and underline indicates the second best performance.**
| Model | BLEU-2$\uparrow$ | BLEU-4$\uparrow$ | ROUGE-1$\uparrow$ | ROUGE-2$\uparrow$ | ROUGE-L$\uparrow$ | METEOR$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| T5-Small [] | 50.1 | 41.5 | 60.2 | 44.6 | 54.5 | 53.2 |
| T5-Base [] | 51.1 | 42.3 | 60.7 | 45.1 | 55.0 | 53.9 |
| T5-Large [] | 55.8 | 46.7 | 63.0 | 47.8 | 56.9 | 58.6 |
| MolT5-Small (T5-Small) [] | 51.9 | 43.6 | 62.0 | 46.9 | 56.3 | 55.1 |
| MolT5-Base (T5-Base) [] | 54.0 | 45.7 | 63.4 | 48.5 | 57.8 | 56.9 |
| MolT5-Large (T5-Large) [] | 59.4 | 50.8 | 65.4 | 51.0 | 59.4 | 61.4 |
| MoMu-Small (T5-Small) [] | 53.2 | 44.5 | - | - | 56.4 | 55.7 |
| MoMu-Base (T5-Base) [] | 54.9 | 46.2 | - | - | 57.5 | 57.6 |
| MoMu-Large (T5-Large) [] | 59.9 | 51.5 | - | - | 59.3 | 59.7 |
| InstructMol (Vicuna-7B) [] | 47.5 | 37.1 | 56.6 | 39.4 | 50.2 | 50.9 |
| MolCA (OPT-125M) [] | 61.6 | 52.9 | 67.4 | 53.3 | 61.5 | 63.9 |
| MolCA (OPT-1.3B) [] | 63.9 | 55.5 | 69.7 | 55.8 | 63.6 | 66.9 |
| UniMoT (Llama-2-7B) | 66.4 | 58.3 | 72.2 | 58.4 | 66.4 | 70.3 |

**Table 12: Examples of molecule captioning task on the ChEBI-20 dataset. We highlight in blue the text that accurately describes the molecule structures in the generated caption, ensuring alignment with the ground truth.**
| Molecule | Generated Molecule Caption | Ground Truth |
| --- | --- | --- |
|  | The molecule is an optically active form of phenylalaninate having D-configuration. It is a conjugate base of a D-phenylalanine. It is an enantiomer of a L-phenylalaninate. | The molecule is the D-enantiomer of phenylalaninate. It is a conjugate base of a D-phenylalanine. It is an enantiomer of a L-phenylalaninate. |
|  | The molecule is an ammonium ion that is the conjugate acid of 2-phenylpropylamine arising from protonation of the primary amino function; major species at pH 7.3. It has a role as a human metabolite, an Escherichia coli metabolite and a mouse metabolite. It is a conjugate acid of a 2-phenylpropylamine. | The molecule is the cation obtained by protonation of the amino group of 2-phenylethylamine. It has a role as a human metabolite and an Escherichia coli metabolite. It is a conjugate acid of a 2-phenylethylamine. |
|  | The molecule is an enamide obtained by the carboxy group of trans-cinnamic acid with the secondary amino group of (2S,5R)-1,2,5-trimethylpiperazine. It has a role as an Aspergillus metabolite. It is an alkaloid, a N-acylpiperazine, an enamide and a tertiary carboxamide. It derives from a trans-cinnamic acid. | The molecule is an enamide obtained by formal condensation of the carboxy group of trans-cinnamic acid with the secondary amino group of (2R,5R)-1,2,5-trimethylpiperazine. It has a role as an Aspergillus metabolite. It is a N-acylpiperazine, a N-alkylpiperazine, an alkaloid, an enamide and a tertiary carboxamide. It derives from a trans-cinnamic acid. |
|  | The molecule is an (omega-1)-hydroxy fatty acid ascaroside obtained by formal condensation of the alcoholic hydroxy group of (10R)-10-hydroxylauric acid with ascarylopyranose (the alpha anomer). It is a metabolite of the nematode Caenorhabditis elegans. It has a role as a Caenorhabditis elegans metabolite. It is a monocarboxylic acid and an (omega-1)-hydroxy fatty acid ascaroside. It derives from an (11R)-11-hydroxylauric acid. It is a conjugate acid of an ascr18(1-). | The molecule is an (omega-1)-hydroxy fatty acid ascaroside obtained by formal condensation of the alcoholic hydroxy group of (10R)-10-hydroxyundecanoic acid with ascarylopyranose (the alpha anomer). It is a metabolite of the nematode Caenorhabditis elegans. It is a monocarboxylic acid and an (omega-1)-hydroxy fatty acid ascaroside. It derives from a (10R)-10-hydroxyundecanoic acid. It is a conjugate acid of an ascrblue18(1-). |
|  | The molecule is a 2-oxo monocarboxylic acid that is pyruvic acid in which one of the methyl hydrogens is substituted by a 4-vinylcyclohex-2-en-1-yl group. It has a role as a plant metabolite. It derives from a pyruvic acid. It is a conjugate acid of a 4-[(1E)-4-vinylcyclohex-2-en-1-yl]pyruvate. | The molecule is a 2-oxo monocarboxylic acid that is pyruvic acid in which one of the methyl hydrogens has been replaced by a methylenecyclopropyl group. It has a role as a rat metabolite and a xenobiotic metabolite. It is a 2-oxo monocarboxylic acid, a member of cyclopropanes and an olefinic compound. It derives from a pyruvic acid. |

#### Molecule-Text Retrieval Task.

The molecule-text retrieval task involves using a molecule to retrieve text (M2T) and using text to retrieve a molecule (T2M). We compare UniMoT with several baselines: Sci-BERT [], KV-PLM [], MoMu [], MoleculeSTM [], MolCA [], and 3D-MoLM []. We report the performance of retrieval using a batch of 64 random samples and the entire test set, evaluated with the metrics of Accuracy and Recall@20. We use the checkpoint from Stage-1 of pretraining. Performance on the PCdes [] and MoMu [] datasets is shown in Table 13. UniMoT demonstrates superior performance over the baselines on molecule-text retrieval, particularly in molecule-to-text retrieval. This demonstrates that UniMoT has learned fine-grained alignment between molecules and text, and it can understand molecule-text interactions through the introduction of the Causal Q-Former.

**Table 13: Accuracy (%) of molecule-text retrieval task on the PCdes [] and MoMu [] datasets. Bold indicates the best performance and underline indicates the second best performance. We report the performance of retrieval using a batch of 64 random samples and the entire test set.**
| Model | Retrieval in batch | Retrieval in test set |  |  |
| --- | --- | --- | --- | --- |
| M2T (%) | T2M (%) | M2T (%) | T2M (%) |  |
| Sci-BERT [] | 62.6 | 61.8 | 60.7 | 60.8 |
| KV-PLM [] | 77.9 | 65.0 | 75.9 | 64.3 |
| MoMu (Sci-BERT) [] | 80.6 | 77.0 | 79.1 | 75.5 |
| MoMu (KV-PLM) [] | 81.1 | 80.2 | 80.2 | 79.0 |
| MoleculeSTM [] | 86.2 | 83.9 | 84.6 | 85.1 |
| MolCA (OPT-1.3B) [] | 91.4 | 88.4 | 90.5 | 87.6 |
| 3D-MoLM (Llama-2-7B) [] | 92.3 | 89.6 | 91.2 | 88.5 |
| UniMoT (Llama-2-7B) | 92.6 | 89.4 | 91.6 | 88.3 |

#### Molecule Generation Tasks.

Molecule generation tasks include caption-guided molecule generation, reagent prediction, forward reaction prediction, and retrosynthesis.

- •
Caption-guided molecule generation involves creating molecular structures from textual descriptions, leveraging NLP and cheminformatics to interpret and translate descriptions into chemical structures.
- •
Reagent prediction focuses on identifying suitable reagents for given reactants and desired products, optimizing synthetic routes.
- •
Forward reaction prediction forecasts probable products from specific reactants and reagents, using knowledge of chemical reactivity.
- •
Retrosynthesis deconstructs target molecules into simpler starting materials.

In molecule generation tasks, evaluating the quality of generated molecules involves several metrics that measure different aspects of similarity and validity.

- •
Exact Match checks if the generated molecule is identical to the target molecule, offering a stringent criterion for precise replication but potentially overlooking chemically similar variants.
- •
The BLEU score [], adapted from machine translation, measures the overlap of n-grams (short sequences of atoms or bonds) between generated and target molecules, thus assessing partial similarities.
- •
Levenshtein Distance [] evaluates the minimum number of edits needed to transform the generated molecule into the target, providing insight into structural changes required.
- •
RDKit [], MACCS [], and Morgan [] Fingerprint Similarities compare the generated and target molecules based on various molecular fingerprinting methods, which capture different aspects of molecular structure and properties.
- •
The Validity [] metric assesses the proportion of chemically valid molecules generated, ensuring that the output consists of plausible chemical structures.

Together, these metrics offer a comprehensive evaluation framework, balancing exact matches with structural and chemical validity.

## Appendix E Limitations

While UniMoT demonstrates considerable advancements in unifying molecule and text modalities for comprehensive understanding and generation tasks, several limitations must be acknowledged. Although UniMoT exhibits strong performance in molecule-to-text and text-to-molecule tasks, it has not been extensively tested on more complex molecule generation tasks such as molecule editing, which require precise modifications to molecular structures. Future work could explore extending UniMoT’s capabilities to handle such sophisticated molecular manipulations.

Due to the scarcity of annotated data in the molecular field, the training of UniMoT is less extensive compared to fields like computer vision. This limitation restricts the model’s ability to fully learn and generalize from diverse molecular structures and properties. In contrast, the visual domain benefits from abundant labeled datasets, allowing for more comprehensive training and better performance. Addressing this data scarcity in the molecular domain is crucial for improving UniMoT’s training effectiveness and overall capabilities.

The current empirical evaluations, though extensive, are primarily conducted on standard datasets and benchmarks; expanding the evaluation to a broader array of datasets and real-world scenarios will provide a more comprehensive understanding of the model’s robustness and generalizability.

## Appendix F Broader Impacts

The development of UniMoT, a unified model for molecule and text modalities, has significant potential to positively impact various fields. UniMoT can streamline the drug discovery process by enabling efficient molecule generation and optimization based on textual descriptions. In material science, it can aid in discovering new materials with desirable properties. Additionally, UniMoT can enhance research collaboration between chemists, biologists, and data scientists by integrating molecular and textual data, leading to comprehensive research insights and innovative solutions.

This paper does not pose any ethical concerns. The study does not involve human subjects and follows proper procedures for dataset releases. There are no potentially harmful insights, methodologies, or applications. Additionally, there are no conflicts of interest or sponsorship concerns. Discrimination, bias, and fairness issues are not applicable. Privacy and security matters have been appropriately addressed, legal compliance has been maintained, and research integrity has been upheld.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The fourth stage includes prompting and instruction following. Is this stage essential, considering the complexities of prompt generation and response parsing? If so, how does prompt engineering influence the final performance?
soundness: 3
strengths: 1. The work is well-motivated and builds on a solid foundation, given existing multimodal studies in text-image tasks. The approach to integrating molecules and text in a unified framework is reasonable and leverages known multimodal strategies.
2. The experiments cover multiple task types, showing the effectiveness of the proposed method across various molecule-related tasks. In most cases, the proposed approach outperforms established baselines.
3. The paper is clearly presented and well-organized, with figures that illustrate key concepts, making the ideas easy to follow.
summary: A key limitation in current applications of large language models (LLMs) to molecule-text tasks is the imbalance in processing modalities: text is handled directly by the encoder, while molecules require separate adapters. To address this, the authors propose UniMoT, a model that employs a Vector Quantization (VQ) tokenizer to incorporate molecules into the LLM vocabulary as discrete tokens. This tokenizer works with a Q-Former, which transforms molecular features into causal sequences, allowing the model to train on interleaved molecular and textual data with a single next-word prediction objective. A four-stage training process is then introduced. Experimental results across tasks, including molecular property prediction, molecule captioning, molecule-text retrieval, and molecule generation, show that the proposed method can prompt the understanding of text and molecules, often better than established baselines.
weaknesses: 1. Technically speaking, the proposed method shares overlaps with existing studies, such as SEED-LLaMA. While UniMoT targets molecule-text modalities, the frameworks overlap in modules like the discrete tokenizer, causal Q-Former, next-token prediction, and reconstruction during inference. This overlap raises questions about novelty.
2. Considering the complexity of the four-stage training process, the performance improvements may not fully justify the added effort. In some tasks, for example, Table 2, its performance is no better than that of baseline methods and is sometimes worse.
3. While the focus is on molecule-text, the paper needs a more comprehensive discussion of and comparison with existing multimodal research, even if primarily in text-image or multi-modality settings, such as AnyGPT and MIO.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Could the causal dependency in learnable queries negatively impact molecular understanding compared to bidirectional or other full-information modeling approaches?
* Are the models for each downstream task trained separately or together? The authors should provide more detailed introduction.
* More details on the adapter and SMILES decoder would strengthen clarity, particularly regarding their roles in aligning molecular representations and decoding.
soundness: 3
strengths: * The VQ-driven tokenizer for molecules enables the unified modeling of molecule and text in language model.
* UniMoT shows impressive state-of-the-art results across multiple understanding and generation tasks.
summary: The paper presents UniMoT, a unified molecule-text language model that leverages discrete token representations to integrate molecular and text. UniMoT uses a tokenizer-driven framework, where a VQ-based tokenizer, combined with a causal Q-Former, encodes molecules as discrete tokens. These discrete tokens are aligned with text tokens in a shared vocabulary, enabling UniMoT to operate in an autoregressive training paradigm across both modalities.
weaknesses: * The core components, such as the Q-Former and VQ-based tokenizer, appear to be direct adaptations from CV, which may limit the paper’s originality within the molecular modeling domain.
* The discrete tokenization approach could lead to information loss, and the model’s effectiveness over directly training with combined molecule SMILES and text is not convincingly demonstrated.
* The multi-stage training process is complex.
* The evaluation on the Mol-Instructions benchmark lacks some important baselines, like DARK[1].

References

[1] Liu, Jinzhe et al. “DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMs.” ArXiv abs/2406.18535 (2024).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Suggestions:
* Additional MoleculeNet experiments with encoder-only chemical language models, such as ChemBERTa [1], PubChemDeBERTa [2], would  show whether LLMs are needed for simple regression/classification tasks in the chemical domain.
* Provide statistical significance tests' results to explore whether the performance gain over baseline models is statistically significant.
* Why are MolT5, MoMu, MolCA absent from Table 1? For me, it makes sense to compare UniMoT with them on MoleculeNet since they are also chemical language models.
* Add more experiments with other recent state-of-the-art chemical language models: GIT-MOL [3], nach0 [4], and Text+Chem T5 [5]. nach0 and GIT-MOL also report the MoleculeNet results as well as performance on molecule generation and molecule captioning  
* Why don't you adopt SELFIES and experiment with SMILES only?
* Line 046: for a reader not familiar with the domain, it would be good have a brief description of what is SMILES and provide some examples.
* As far as I know, MoleculeNet mostly covers small datasets. How stable are results across different runs?

Typos:
* Line 159: $F$ is not introduced earlier.


References:
[1] Chithrananda, Seyone, Gabriel Grand, and Bharath Ramsundar. "ChemBERTa: large-scale self-supervised pretraining for molecular property prediction." arXiv preprint arXiv:2010.09885 (2020).
[2] Schuh, Maximilian G., Davide Boldini, and Stephan A. Sieber. "TwinBooster: Synergising Large Language Models with Barlow Twins and Gradient Boosting for Enhanced Molecular Property Prediction." arXiv preprint arXiv:2401.04478 (2024).
[3] Liu, Pengfei, et al. "Git-mol: A multi-modal large language model for molecular science with graph, image, and text." Computers in biology and medicine 171 (2024): 108073.
[4] Livne, Micha, et al. "nach0: Multimodal natural and chemical languages foundation model." Chemical Science 15.22 (2024): 8380-8389.
[5] Christofidellis, Dimitrios, et al. "Unifying molecular and textual representations via multi-task language modelling." International Conference on Machine Learning. PMLR, 2023.
soundness: 3
strengths: * The paper proposes a novel methodology for chemical Large Language Model (LLM) pretraining which addressed multiple problems related to cross-modal text and molecule pretraining, including long sequence lengths and the lack of left-to-right causal dependency in molecule features.
* Extensive evaluation on MoleculeNet highlights the effectiveness of the proposed pre-training technique on non-generative molecular property prediction tasks.
* The experimental results on molecular retrieval and generation tasks show that the proposed UniMoT models performs on par or better than the selected baselines.
* The methodological decisions are experimentally supported with a thorough ablation study.
summary: The paper presents UniMoT, a Unified Molecule-Text Large Language Model that addresses limitations in existing molecular LLMs by employing a tokenizer-based architecture that treats molecule and text modalities equally. By introducing a Vector Quantization-driven tokenizer, UniMoT effectively bridges the gap between these modalities, enabling it to interpret and generate molecules as text. Extensive experiments show that UniMoT achieves state-of-the-art performance in various molecular comprehension and generation tasks.
weaknesses: * From the paper, it it unclear if the improvement over baseline methods is statistically significant.
* On molecule generation task, UniMot is mostly compared against general-purpose LLMs which were not specifically designed for chemistry-related tasks. The comparison against more domain-specific methods, such as nach0, GIT-MOL, and Text+Chem T5 could strenghten the paper claims.
* Lack of comparison with chemical language models (nach0, GIT-MOL, and Text+Chem T5, MolT5 as well as encoder-only ChemBERTa and PubChemDeBERTa) on MoleculeNet benchmark and molecule-text retrieval task.
* Lack of ablation study for training objectives. For instance, Equation 2 has 3 terms, what would happen if you remove some of them? How much would the performance drop if the codebook is removed and the model is forced to generate SMILES directly?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: See above sections for details.
soundness: 2
strengths: The background and motivation for utilizing LLMs for the learning process is well described, with relevant references. 

There are different informative experiment tasks evaluated including molecular property prediction and molecule captioning. Ablation studies are also informative for the different generation tasks.
summary: The authors introduce UniMoT, a Unified Molecule-Text LLM adopting a tokenizer-based architecture that expands the vocabulary of LLM with molecule tokens. Specifically, we introduce a Vector Quantization-driven tokenizer that incorporates a Q-Former to bridge the modality gap between molecule and text. This tokenizer transforms molecules into sequences of molecule tokens with causal dependency, encapsulating high-level molecular and textual information. Equipped with this tokenizer, UniMoT can unify molecule and text modalities under a shared token representation and an autoregressive training paradigm. It can interpret molecules as a foreign language and generate them as text. Following a four-stage training scheme, UniMoT emerges as a multimodal generalist capable of performing both molecule-to-text and text-to-molecule tasks.
weaknesses: There is not much information about how the design choice of the author’s model is representing the natural graph molecular structure of the molecule itself (e.g., atom and bond structure). This is a recent survey paper that is missing in the cited work that the authors should include. The LLM itself is known to not as comprehensively capture graph structure.  

[IJCAI 2023] Graph-based molecular representation learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 744, 6638–6646. https://doi.org/10.24963/ijcai.2023/744

The experiment benchmarks need to include more GNN based representation models, and experiment results of the model do not seem conclusive for the molecular property prediction task, perhaps indicating that there needs to be more explicit learning of graph topological structure.

Meta-review: The paper introduces UniMoT, a Unified Molecule-Text Large Language Model (LLM) designed to improve how LLMs handle both molecular and textual data. The central claim is that by using a novel tokenizer-based architecture, UniMoT can treat molecules and text equally within the LLM framework. The paper clearly outlines the limitations of existing adapter-based architectures and justifies the need for a unified approach. The paper is generally well-written and organized, making it relatively easy to follow. Some reviewers point out that individual components are adapted from other domains (like computer vision), raising questions about the overall novelty. The quantization process introduces some information loss, leading to a dip in performance on certain molecule comprehension tasks compared to specialized models. The multiple-stage training scheme is somewhat complex, which could be a barrier to wider adoption. A more extensive comparison to relevant domain-specific models would strengthen the paper.

Decision: Reject