Mol2Lang-VLM: Vision- and Text-Guided Generative Pre-trained Language Models for Advancing Molecule Captioning through Multimodal Fusion (2024)

Abstract: This paper introduces Mol2Lang-VLM, an enhanced method for refining generative pre-trained language models for molecule captioning using multimodal features to achieve more accurate caption generation. Our approach leverages the encoder and decoder blocks of the Transformer-based architecture by introducing third sub-layers into both. Specifically, we insert sub-layers in the encoder to fuse features from SELFIES strings and molecular images, while the decoder fuses features from SMILES strings and their corresponding descriptions. Moreover, cross multi-head attention is employed instead of common multi-head attention to enable the decoder to attend to the encoder’s output, thereby integrating the encoded contextual information for better and more accurate caption generation. Performance evaluation on the CheBI-20 and L+M-24 benchmark datasets demonstrates Mol2Lang-VLM’s superiority, achieving higher accuracy and quality in caption generation compared to existing methods. Our code and pre-processed data are available at https://github.com/nhattruongpham/mollang-bridge/tree/mol2lang/.

heading: 1 Introduction
text: In the field of cheminformatics, molecule captioning plays a crucial role in helping researchers by automatically generating captions for molecular structures. The accuracy and quality of these captions are vital as they directly impact the understanding of chemical information and scientific discoveries. Traditional techniques primarily rely on unimodal data, often focusing only on textual representations like SMILES (Simplified Molecular Input Line Entry System) (Weininger, 1988) strings or SELFIES (Self-referencing Embedded Strings) (Krenn et al., 2020) strings. Although these methods have shown satisfactory results, their dependence on a single modality limits the richness and accuracy of the
generated captions. The rise of multimodal data, which uses information from different sources, presents an opportunity for significant advancements in molecule captioning. Multimodal approaches integrate various forms of molecule, enabling a more comprehensive understanding of molecular characteristics. However, effectively utilizing multimodal data in generative models is challenging and requires advanced techniques to integrate and improve the models effectively.
In this paper, we introduce an enhanced methodology, named Mol2Lang-VLM, to improve generative models in molecule captioning by utilizing multimodal features. Our approach integrates SELFIES strings and high-level features from molecular images in the encoder, while incorporating SMILES features and corresponding descriptions in the decoder. This multimodal integration allows the model to have a deeper understanding of chemical structures within the generative model, which is further refined during the decoder stage.

heading: 2 Related Work

heading: 2.1 Unimodal Language Models
text: MolT5 (Edwards et al., 2022) involves translating molecular structures into natural language using a text-to-text transfer transformer (T5) (Raffel et al., 2020) model. This model leverages the robust linguistic capabilities of T5 to understand and generate descriptions of molecular structures accurately. BioT5 (Pei et al., 2023) extends the capabilities of T5 to integrate chemical knowledge and natural language associations into biological contexts. BioT5 employs SELFIES for representing small molecules, as it offers considerable advantages over SMILES. Specifically, SELFIES ensures a more reliable and error-resistant molecular representation, thereby avoiding the problem of invalid structures that frequently occur with SMILES. This
98
model improves the cross-modal understanding between biological texts and chemical data. While MolT5 and BioT5 are encoder-decoder language models, MolXPT (Liu et al., 2023) utilizes a generative pre-trained Transformer (GPT) (Radford and Narasimhan, 2018) which is a decoder-only language model by introducing a generative pretraining approach by wrapping molecular structures within descriptive texts. MolXPT leverages both text and SMILES sequences for molecular modeling. It wraps SMILES sequences with text, allowing them to influence each other. Specifically, it detects molecule names in text sequences and replaces them with corresponding SMILES representations. ChemBERTa (Chithrananda et al., 2020) is an encoder-only language model that utilizes the RoBERTa (Liu et al., 2019) model that focuses on molecular representation learning and property prediction.
SwinOCSR (Xu et al., 2022) uses Swin Transformer (Liu et al., 2021) architecture for end-to-end optical chemical structure recognition of molecular images. This model can effectively recognize and describe chemical structures from images, providing a significant improvement in the accuracy of vision-language tasks in cheminformatics.
While the aforementioned models have made significant contributions to molecule captioning, their reliance on unimodal data restricts their potential for advancements. By not harnessing the power of multimodal data, these models encounter limitations in terms of information richness, completeness, contextual understanding, generalization, and interpretability.

heading: 2.2 Multimodal Language Models
text: GIT-Mol (Liu et al., 2024) introduces a multimodal large language model that integrates graph, image, and text data to enhance molecular science applications. This model leverages the strengths of different data modalities to provide comprehensive and accurate molecular descriptions. Besides that, MoMu (Su et al., 2022) associates molecular graphs with natural language, providing a sophisticated multimodal foundation model. This model enhances the interpretability and accuracy of molecular captions by integrating graph representations of molecules with their textual descriptions.
While the use of multimodal data is feasible, the aforementioned models face certain issues. These models necessitate significant computational resources and large dataset training. Additionally,
scaling up multimodal models can pose challenges.

heading: 3 Methodology

heading: 3.1 Generative Language Model
text: We use the T5 (Raffel et al., 2020) architecture as our generative language model. The process begins with the tokenization of the SELFIES string, resulting in token embeddings Xt ∈ RLenc×dt . Here, Lenc represents the length of the encoder input, while dt denotes the dimensionality of the feature vectors. The encoder comprises a sequence of N encoding layers, with each layer consisting of a Multi-head Self-Attention (MSA) (Vaswani et al., 2017) mechanism (Eq. 1) and a Feed-Forward Network (FFN) (Eq. 2). Following each sub-layer, there is a residual connection that precedes layer normalization (LN). Unlike the original Transformer (Vaswani et al., 2017), T5 incorporates relative position embeddings (Shaw et al., 2018), which are added to the respective logits during the computation of attention weights and shared across all layers in the model.
Zenc ′
l = LN(MSA(Z enc l−1) + Z enc l−1) (1)
Zencl = LN(FFN(Z enc′ l ) + Z enc′ l ) (2)
In parallel with the encoder, the decoder similarly consists of N layers. It exhibits three distinctive aspects compared to the encoder: First, target sequences, which are molecular captions, are tokenized into embeddings Yt ∈ RLdec×dt , where Ldec is the length of the target sequence. They are shifted right by one token to ensure that the groundtruth token is used as the input to predict the next token. Second, a Masked Multi-head Self-Attention (MMSA) (Vaswani et al., 2017) is utilized to ensure auto-regressive generation, maintaining a strict leftto-right processing order (Eq. 3). Third, a Cross Multi-head Attention (CMA) (Vaswani et al., 2017) layer is employed, which enables the decoder to attend to the encoder’s output, thereby integrating the encoded contextual information (Eq. 4). Analogous to the encoder, the decoder includes a FFN in each layer (Eq. 5).
Zdec ′
l = LN(MMSA(Z dec l−1) + Z dec l−1) (3)
Zdec ′′ l = LN(CMA(Z dec′ l , Z enc N , Z enc N ) + Z dec′ l )
(4) Zdecl = LN(FFN(Z dec′′ l ) + Z dec′′ l ) (5)

heading: 3.2 Vision- and Text-Guided Fusion
text: Inspired by VG-GPLMs (Yu et al., 2021), we integrate a third sub-layer into both the encoder and decoder of the language model. In the encoder, we insert text-vision fusion at the end of the encoder to learn the cross-modality between the SELFIES string and the molecular image. In the decoder, we replace the last FFN with text-text fusion to capture the relationship between the corresponding caption and the SMILES string. In both fusion processes, we utilize the CMA mechanism to learn the correlation between the two sets of features. The overview of the architecture is exhibited in Figure 1.
In the fusion process of the encoder, the embeddings of SELFIES, denoted as Zenct , are linearly projected to the query Qenc (Eq. 6), while the embeddings of the image, denoted as Zencv , are linearly projected to the key Kenc (Eq. 7) and the value V enc (Eq. 8). These projections are performed before feeding them to the CMA mechanism, which generates the output Oenc (Eq. 9).
Qenc = Zenct W enc q (6)
Kenc = Zencv W enc k (7) V enc = Zencv W enc v (8)
Oenc = CMA(Qenc,Kenc, V enc) (9)
The fusion process of the decoder occurs after the CMA between the decoder’s embeddings and the
output embeddings of the encoder, resulting in Zdect . In this fusion process, Zdect are linearly projected to the query Qdec (Eq. 10), while the embeddings of SMILES, denoted as Zdecs , are also projected to the key Kdec (Eq. 11) and the value V dec (Eq. 12). Subsequently, CMA is applied to generate Odec (Eq. 13). The features of SMILES help enhance the overall effectiveness of the features, enabling more effective generation of the desired output.
Qdec = Zdect W dec q (10)
Kdec = Zdecs W dec k (11) V dec = Zdecs W dec v (12)
Odec = CMA(Qdec,Kdec, V dec) (13)
At each fusion, the output is concatenated with the initial embeddings to produce Zenc ′ t and Z dec′ t (Eq. 14 and 15).
Zenc ′
t = (Z enc t ⊕Oenc)W encc (14)
Zdec ′
t = (Z dec t ⊕Odec)W decc (15)
Finally, forget gates, denoted as F enc and F dec, are applied to filter out noisy and redundant information introduced during the interactions (Eq. 16 and 17), then point-wise multiplication is applied on Oenc and Odec to produce Oenc ′ and Odec ′ (Eq. 18).
F enc = σ((Zenct ⊕Oenc)W encf ) (16)
F dec = σ((Zdect ⊕Odec)W decf ) (17)
Oenc ′ = F enc ⊗Oenc, Odec′ = F dec ⊗Odec (18)

heading: 4 Implementation Details

heading: 4.1 Architectures
text: We employ BioT5 (Pei et al., 2023) as our generative language model, which uses the T5-base version. The model consists of 252 million parameters and has a configuration that includes an embedding dimensionality of 768. It is composed of 12 layers in both the encoder and decoder. The input tokens and output tokens are limited to a maximum length of 512.
To extract visual features from molecular images, we utilize the encoder of SwinOCSR (Xu et al., 2022) which employs the Swin Transformer (Liu et al., 2021) architecture, uses Swin-L version. The encoder has a total of 194 million parameters. By inputting images with the size of 224 × 224, the encoder generates feature embeddings with a length of 49 and a hidden dimensionality of 1536.
To extract features from SMILES representations, we use ChemBERTa (Chithrananda et al., 2020), which is built upon the RoBERTa-base architecture with a total of 44 million parameters. The input tokens for ChemBERTa are also limited to a length of 512.
To compute the cross-modality attention in textvision fusion of the encoder, as well as text-text fusion of the decoder, all features are linearly projected to a gated dimensionality of 256. The textvision fusion is then integrated at the last two layers of the encoder (the 11th and 12th layers). Concurrently, text-text fusion is incorporated into the initial two layers of the decoder (the 1st and 2nd layers).

heading: 4.2 Datasets
text: L+M-24: The L+M-24 dataset, first introduced from Language + Molecules Workshop @ ACL 2024 (Edwards et al., 2024), is designed to highlight three key benefits of natural language in molecule design: compositionality, functionality, and abstraction. It contains over 160, 560 moleculedescription pairs, which are divided into 80%/20% for train/validation splits.
CheBI-20: The CheBI-20 dataset is widely used in molecular description tasks. It was first introduced in the Text2Mol (Edwards et al., 2021). This dataset contains 33, 010 molecule-description
pairs, which are split into 80%/10%/10% for train/validation/test sets.
Since the aforementioned datasets currently lack SELFIES strings and molecular images, we employ selfies 1 and RDKit 2 package to generate this additional data. We use the prompting template of the molecule captioning task from BioT5 (Pei et al., 2023) to fine-tune the model.

heading: 4.3 Configurations
text: Training: During the training process, we utilize a batch size of 64. To optimize the model, we employ the AdamW (Loshchilov and Hutter, 2019) optimizer. The learning rate scheduler follows a cosine annealing strategy, with a base learning rate of 3e− 5. The warming-up steps for the learning rate scheduler are set to 1 epoch to gradually adjust the learning rate.
Inference: To ensure a fair comparison when evaluating the model, we employ greedy decoding for generating molecular captions by setting the number of beam search to 1, with the decoder starting token as <pad>, and the end of sentence token as 
. Furthermore, post-processing is also applied to skip all special tokens.

heading: 5 Results and Discussion
text: Table 1 presents the performance comparison of Mol2Lang-VLM with all baseline models, such as MolT5-Small, MolT5-Base, MolT5-Large, and BioT5 on the L+M-24 dataset. We used several performance evaluation metrics to evaluate these models, including BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR. Notably, Mol2Lang-VLM outperforms all three baseline models in almost all metrics, with BLEU-2, BLEU4, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR values of 77.7, 56.3, 78.6, 59.1, 56.5, and 74.1, respectively. Although Mol2Lang-VLM achieves a lower METEOR of 0.2 compared to MolT5-Large, its number of parameters is approximately 1.5 times lower than MolT5-Large, indicating that the model can learn more efficiently. Compared to BioT5, Mol2Lang-VLM achieves better performance in terms of BLEU-2, BLEU4, ROUGE-1, and METEOR, with slightly lower scores in ROUGE-2 and ROUGE-L, demonstrating that it generally outperforms BioT5.
We also evaluate our proposed method, along
1https://github.com/aspuru-guzik-group/selfies 2https://github.com/rdkit/rdkit
with all three baseline models and BioT5, on the CheBI-20 dataset. Table 2 displays the performance comparison in terms of BLEU-2, BLEU4, ROUGE-1, ROUGE-2, ROUGE-L, METEOR, and Text2Mol metrics. Interestingly, Mol2LangVLM achieves the second-best performance in all metrics, while BioT5 excels on this dataset. This might be acceptable because, in some cases, the fused information may not provide significant additional context or may even introduce noise, making it challenging for the model to effectively utilize the fused embeddings.
Moreover, we conduct an ablation analysis to evaluate Mol2Lang-VLM with and without employing the forget gate. Table 3 compares the performance of these two strategies. Mol2LangVLM with the forget gate outperforms the version without it across most metrics, including BLEU-2, BLEU-4, ROUGE-2, and METEOR. The presence of the forget gate mechanism contributes to enhanced caption quality in terms of accuracy and relevance, showcasing the effectiveness of incorporating this mechanism in the model architecture for improved captioning outcomes.

heading: 6 Conclusion
text: This paper introduced Mol2Lang-VLM, a visionand text-guided generative pre-trained language model designed to enhance molecule captioning performance through multimodal fusion. Our proposed approach achieved comparative results in terms of BLEU, ROUGE, METEOR, and Text2Mol metrics, demonstrating its effectiveness in generating accurate and meaningful captions for molecular structures. The findings highlight the potential of Mol2Lang-VLM in advancing molecule captioning tasks. Future research can explore alternative fusion methods, fine-tuning strategies, and the generalization of the model to other tasks. Additionally, integrating Mol2Lang-VLM with downstream applications and enhancing interpretability can further enhance its practical utility in the field of cheminformatics.

heading: Acknowledgements
text: This work was supported by the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (RS-2024-00344752). This research was supported by the Department of Integrative Biotechnology, Sungkyunkwan Univer-
sity (SKKU) and the BK21 FOUR Project. This work was supported by the Korea Bio Data Station (K-BDS) with computing resources including technical support.

heading: Future Work
text: Future research can explore alternative fusion methods, fine-tuning strategies, and the generalization of the model to other tasks. Additionally, integrating Mol2Lang-VLM with downstream applications and enhancing interpretability can further enhance its practical utility in the field of cheminformatics