MolTC: Towards Molecular Relational Modeling In Language Models (2024)

Abstract: Molecular Relational Learning (MRL), aiming to understand interactions between molecular pairs, plays a pivotal role in advancing biochemical research. Recently, the adoption of large language models (LLMs), known for their vast knowledge repositories and advanced logical inference capabilities, has emerged as a promising way for efficient and effective MRL. Despite their potential, these methods predominantly rely on textual data, thus not fully harnessing the wealth of structural information inherent in molecular graphs. Moreover, the absence of a unified framework exacerbates the issue of insufficient data exploitation, as it hinders the sharing of interaction mechanisms learned across various datasets. To address these challenges, this work proposes a novel LLM-based multi-modal framework for Molecular inTeraction modeling following Chain-of-Thought (CoT) theory, termed MolTC, which effectively integrate graphical information of two molecules in pair. To train MolTC efficiently, we introduce a Multi-hierarchical CoT theory to refine its training paradigm, and conduct a comprehensive Molecular Interactive Instructions dataset for the development of biochemical LLMs involving MRL. Our experiments, conducted across twelve datasets involving over 4,000,000 molecular pairs, exhibit the superiority of our method over current GNNand LLM-based baselines. Our code is available at https://github.com/MangoKiller/MolTC.

heading: 1 Introduction
text: Molecular Relational Learning (MRL) (Lee et al., 2023a), aiming to understand interactions between molecular pairs, has gained significant interest due to its wide range of applications (Roden et al., 2020). For example, Drug-Drug Interactions
*Equal contribution. †Corresponding author. Xiang Wang is also affiliated with Institute of Dataspace, Hefei Comprehensive National Science Center.
(DDIs) are critical in pharmacology and drug development (Lin et al., 2020), while solute-solvent interactions (SSIs) are fundamental in solution chemistry and the design of chemical processes (Varghese and Mushrif, 2019; Chung et al., 2022). However, the exhaustive experimental validation of these interactions is notoriously time-consuming and costly. In response, adopting large language models (LLMs) (Brown et al., 2020; Taylor et al., 2022), known for their vast knowledge repositories and advanced logical inference capabilities, has emerged as an efficient and effective alternative for MRL (Park et al., 2022; Jha et al., 2022a).
Despite their promise, a primary concern of current LLM-based paradigm is the insufficient data exploitation. Specifically, they predominantly rely on the textual data such as SMILES (Simplified Molecular Input Line Entry System) and property descriptions, thus not fully harnessing the wealth of structural information inherent in molecular graphs (Sagawa and Kojima, 2023), as indicated in Figure 1 (a). Current studies have indicated that it is challenging for LLMs to fully understand the complex graphs based solely on textual data, hence, it’s crucial to explicitly model these structures given their significance in MRL (Park et al., 2022).
Compounding this concern is the absence of a unified framework for LLM-based MRL (Livne et al., 2023; Pei et al., 2023). Concretely, this absence impedes the sharing and integration of interaction mechanisms learned across various datasets, leading to a fragmentation in collective insights. Especially, it poses a catastrophic challenge for tasks with a limited number of labeled pairs (Chung et al., 2022), where LLMs often struggle with due to the high risk of overfitting, as illustrated in Figure 1 (b). Worse still, such limited datasets are prevalent in MRL since the experimental acquisition is often constrained by high costs (Lee et al., 2023a).
To overcome these limitations, in this work, we propose MolTC, a unified multi-modal frame-
1943
work for Molecular inTeraction modeling following the Chain-of-thought theory (Wei et al., 2022). As depicted in Figure 1 (c), MolTC employs the Graph Neural Networks (GNNs) (Kipf and Welling, 2017), known for their proficiency in graph modeling, to explicitly gather graphical information of molecular pairs, and integrates them into the input space of LLMs by two meticulously crafted projectors. In response to empirical findings that LLMs may confuse two input molecules in pair, MolTC incorporates the molecules’ SMILES information to reinforce the concept of molecular order.
More importantly, a two-pronged approach is developed to train MolTC efficiently:
(1) Training Paradigm Refinement: As shown in Figure 1 (c), we introduce a Multi-hierarchical CoT theory to guide the training paradigm of MolTC. Concretely, the broad-grained CoT guides the pretraining stage to identify individual molecular properties before predicting interactions, ensuring an acute awareness of each molecule’s unique attribute. For quantitative interaction tasks, which are challenging for LLMs, a fine-grained CoT enables the fine-tuning stage to initially predict a range, and then progressively refining it to a precise value.
(2) Dataset Foundation Construction: In sight of the absence of a comprehensive MRL datasets for biochemical LLMs, we construct a Molecular inTeractive instructions dataset, termed MoTinstruction. Specifically, we first conduct twelve well-established MRL datasets across various domain, and source their detailed molecular prop-
erties from authoritative biochemical databases. Based on this, we meticulously compile these properties and empirically determine their optimal instructions. These process ensures that MoTinstructions can not only enhance the performance of our MolTC, but also contribute to the development of other biochemical LLMs involving MRL.
Our contributions can be summarized as follows:
• We identify the issue of insufficient data exploitation in current LLM-based MRL, and take the first attempt to develop a unified multi-modal framework for LLM-based MRL, named MolTC. • We introduce the multi-hierarchical CoT theory to enhance the MolTC’s training process, especially for quantitative interaction tasks. • We construct MoT-instructions, the first comprehensive instruction dataset in MRL domain, to enhance the development of biochemical LLMs involving MRL. • Our experiments, across over 4,000,000 molecular pairs in various domains such as DDI and SSI, demonstrate the superiority of our method over current GNN and LLM-based baselines.

heading: 2 Methodology
text: In this section, we detail our MolTC, which harnesses the power of LLMs for comprehending molecular interactions. We begin with the introduction of model framework in Section 2.1. Taking a step further, the training paradigm guided by the principle of Multi-hierarchical CoT is outlined in Section 2.2. Moreover, the dynamic parameter
sharing strategy tailored for MolTC and our developed datasets, MoT-instructions, are elaborated in Section 2.3 and 2.4, respectively.

heading: 2.1 Framework of MolTC
text: Here we introduce four key components of MolTC’s framework: Graph Encoder, Representation Projector, SIMLES Injector, and the backbone LLM. The specific instantiation details can be found in the experimental section and the appendix.
Graph Encoder. The first step of extracting interactions is to precisely encode the molecular graphs. In sight of this, we utilize two GNN-based encoders to capture the embedding of the given molecular pairs, leveraging the GNN’s robust capability in aggregating structural information. More formally, let Ga = {Va, Ea} and Gb = {Vb, Eb} denote the input pair, where V, E represent atomic nodes and the chemical bonds, respectively. The two graph encoders fenc1 and fenc2 perform aggregating to obtain the atomic embedding:
Ha = [h 1 a,h 2 a, . . . ,h |Va| a ] = fenc1(Ga),
Hb = [h 1 b ,h 2 b , . . . ,h |Vb| b ] = fenc2(Gb),
(1)
where hia and h i b denote to the embedding of the ith atom in molecule Ga and Gb; Va and Vb represent the number of nodes.
Representation Projector. After acquiring molecular pair representations Ha and Hb, the next step is to map them into the backbone LLM’s hidden space using Projectors fpro1 and fpro2. These projectors serve as pivotal connectors, translating Ha and Hb into LLM-comprehensible encodings Ma and Mb. Drawing inspiration from the state-of-theart vision-language models, we instantiate fpro1 and fpro2 by Querying Transformers (Q-Formers) (Li et al., 2023a; Dai et al.). More formally,
Ma = [m 1 a,m 2 a, . . . ,m q a] = fpro1(Ha), Mb = [m 1 b ,m 2 b , . . . ,m q b ] = fpro2(Hb), (2)
where q denotes the number of learnable query tokens of Q-Former’s transformer.
In detail, our Projectors, based on the BERT architecture, incorporate an additional cross-attention module positioned between the self-attention and feed-forward modules. This instantiation offers two key benefits. Firstly, it supports seamless integration with conventional BERT-based text encoders, allowing fpro1 and fpro2 pre-training with
extensive molecular graph-text pairs. Secondly, it maintains compatibility with various input dimensions d, and allows adjustments in the size of learnable query tokens to align with the LLM’s token embedding size. These advantages lay a solid foundation for the thorough interaction of two molecules during the LLM’s inference process. Future work will also explore more projector designs, such as streamlining it through specially tailored MLPs (Yang et al., 2023).
SMILES Tokenization. When directly analyzing the representations Ma and Mb with LLMs, our experiments suggest a potential confusion by LLMs in distinguishing the properties of each molecule in a pair. This observation naturally inspires us to integrate textual information of the molecules to strengthen the concept of their sequential order. Here MolTC employs SMILES due to its ubiquity and specificity. Additionally, SMILES serves as a conduit, linking the task-specific prompts with the corresponding biochemical knowledge stored within the LLM. Therefore, we directly input the SMILES of both molecules into the backbone LLM, utilizing the inherent encoder to acquire their tokens Sa and Sb.
Backbone LLM. MolTC leverages Galactica, a decoder-only transformer built on the OPT framework, as its base LLM. Pretrained on an extensive collection of scientific literature, Galactica demonstrates exceptional proficiency in biochemistry knowledge. This expertise, particularly in parsing molecular sequences such as SMILES and SELFIES strings, enables Galactica to adeptly capture the properties crucial for molecular interactions. Specifically, the goal of MolTC is to harness Galactica’s advanced inferential skills to interpret the contextual interactions between two molecular sets of token collections, {Ma,Sa} and {Mb,Sb}. More formally, we denote an integrated prompt sequence as follows:
X = {P,Ma,Sa,Mb,Sb} = [x1,x2, ...,xl] s.t. P ∼ Pr, (3)
where l is the integrated input length, P denotes the task-specific prompt, and Pr represents a collection of various manually designed prompts, each tailored for the molecular interaction task r. The generation process adopts a causal mask to generate a response encapsulating key interactive properties
with length T :
X̂ = [x̂1, x̂2, ..., x̂T ]. (4)
Utilizing Galactica’s autoregressive framework, the training objective involves regressing the target response based on the input prompt X. Specifically, the output for i-th token x̂i, is computed based on its preceding tokens as follows for t ∈ (1, T ):
p ( X̂[1:t]|X ) = t∏
i=1
p ( x̂i|X, X̂[1:i−1] ) . (5)

heading: 2.2 Training Paradigm of MolTC
text: In this part, we elaborate the training paradigm of MolTC, including pretraining and fine-tuning processes, which is guided by the principle of Multihierarchical CoT, as shown in Figure 2.

heading: 2.2.1 Broad-grained CoT Guided Pretraining
text: Given the challenge of directly understanding complex interactions between two input molecules in pair, the broad-grained CoT guides MolTC to initially identify individual molecular properties. By thoroughly understanding each molecule’s characteristics, MolTC establishes a solid foundation for accurately predicting their interactions. Specifically, in the pretraining stage, the prompt is uniformly designed as follows:
Prompt for Pretraining Stage
Input Prompt
<SMILES1>, <GraEmb1>, the front is the first molecule, followed by the second molecule: <SMILES2>. <GraEmb2>. Please provide the biochemical properties of the two molecules one by one.
Target Response The properties of the first molecule are [Property1], and the properties of the second molecule are [Property2].
This prompt design enable MolTC to delineate key properties of two molecules sequentially. Based on it, MolTC utilize the generation loss of the backbone LLM to train Graph Encoders, fenc1 and fenc2, as well as the Representation Projectors, fpro1 and fpro2. Notably, during this phase, the backbone LLM remains frozen.
Dataset Construction for Pretraining. To ensure backbone LLM can understand the individual characteristics of each molecule, it is pivotal to prepare
a comprehensive dataset comprising molecule pairs and their corresponding biochemical properties. To this end, (1) we first conduct an extensive survey of various authoritative biochemical database such as PubChem1 and Drugbank (Kim et al., 2023), and collect a large amount of molecule-textual properties pairs; (2) then, recognizing the variability in annotation quality within this dataset, we augment and enrich molecular descriptions that were less extensively annotated; (3) subsequently, to simulate diverse molecular interactions, we generated molecular pairs by randomly grouping two distinct molecules from the above database. This random pairing facilitates a broad spectrum of molecular combinations, exposing the pretraining stage to diverse interaction scenarios, thus naturally enhancing the generalizability of our MolTC.

heading: 2.2.2 Fine-grained CoT Guided Fine-tuning
text: During the fine-tuning phase, MolTC is trained to enable the backbone LLM to generate interaction properties based on the properties of individual molecules it initially identifies. To this end, prompts in the fine-tuning stage should be crafted for specific downstream task. For example, in DDI tasks, we construct the following prompt:
1https://pubchem.ncbi.nlm.nih.gov
Prompt for DDI Tasks (Fine-tuning)
Input Prompt
<SMILES1>, <GraEmb1>, the front is the first molecule, followed by the second molecule: <SMILES2>. <GraEmb2>. What are the side effects of these two drugs?
Target Response The property of the first molecule is [Property1], while the property of the second molecule is [Property2]. Hence, the first drug molecule may increase the photosensitizing activities of the second drug molecule.
Despite the effectiveness of this prompt design, LLMs face notable challenges in quantitative analysis, especially in complex molecular interaction contexts such as SSI and chromophore-solvent interaction (CSI). Our experiments in Section 3 highlight this difficulty, demonstrating that LLMs tend to exhibit indecision regarding the quantitative values in their outputs. To address this, a finegrained CoT concept is introduced to refine the training paradigm. Specifically, the backbone LLM is guided to initially suggest a range for the target numerical value, then progressively refining it to a precise value. Take a meticulously prompt for SSI tasks as an example:
Prompt for SSI Tasks (Fine-tuning)
Input Prompt
<SMILES1>, <GraEmb1>, the front is the first molecule, followed by the second molecule: <SMILES2>. <GraEmb2>. What is the solvation Gibbs free energy of this pair of molecules?
Target Response The property of the first molecule is [Property1], while the property of the second molecule is [Property2]. Hence, the solvation Gibbs free energy of these two molecules is above 3.0 and below 3.5, so the accurate value is 3.24791.
This step-wise refinement process fosters a more accurate and reliable resolution of numericallyintensive challenges. Based on these prompts, in the fine-tuning stage, the parameters in backbone
LLM are updated through Low-Rank Adaptation (LoRA) (Hu et al., 2021) strategy, known for its efficiency in tailoring the LLM to the requirements of downstream tasks and minimal memory demands in storing gradients. Meanwhile, to ensure that other modules are optimally adjusted to suit the specifics of the downstream tasks, Graph Encoders fenc1 and fenc2, as well as Representation Projectors fpro1 and fpro2 are trained following the generation loss of the backbone LLM.

heading: 2.3 Dynamic Parameter Sharing Strategy
text: To implement the above training paradigm effectively, we introduce a novel parameter-sharing strategy, inspired by key biochemical insights:
(1) The Importance of Role-Playing: A molecule’s role in an interaction crucially influences the outcome. For example, in SSI scenario like the waterethanol pair, utilizing water and ethanol as solvents, respectively, yields different energy releases (Reichardt, 2021). Sometimes, a reversal of roles can even result in the absence of interaction.
(2) The Importance of Input Order: In certain molecular pairs, the sequence of introducing molecules significantly impacts the interactions. For instance, the order of drug introduction can lead to varying therapeutic effects.
(3) The Importance of Role and Order-Specific Feature Extraction: The role and input order of molecules determine the relevance of their structural features. For example, a chemical group in a solute-solvent pair may be crucial for the release of Gibbs free energy when in the solute, but less so in the solvent (Reichardt, 2021; J et al., 2022).
These insights inspire MolTC to adaptively prioritize distinct key information, creating unique tokens for the same molecule based on its role and order. To enable this nuanced learning while also capitalizing on the shared aspects of molecular learning, we introduce the following parametersharing strategy, as shown in Figure 2:
(1) The GNN-based Encoders fenc1 and fenc2, which focus on extracting molecular graph structures, share parameters during both pretraining and fine-tuning stages to enhance learning efficiency.
(2) The Qformer-based Projectors fpro1 and fpro2, tasked with aligning molecular structures to semantic information, share parameters during pretraining stage to promote generalization and robustness. However, in the fine-tuning stage, we cease sharing
to allow customized semantic mappings tailored to the varying roles and orders.
In summary, this strategy is tailored to balance the need for role and order-based distinctively learning with the efficiency gained from commonalities across molecular pairs.

heading: 2.4 Construction of MoT-instructions
text: Given the absence of a comprehensive instruction datasets tailored for LLM-based MRL, we aim to develop a molecular interactive instructions dataset, termed MoT-instructions. This dataset is designed to fulfill several key criteria: (1) it should include extensive molecular pairs capable of interaction, covering a broad spectrum of domains, (2) it should detail important biochemical properties of each molecule within these pairs, and (3) it should elaborate the resultant properties from molecular interactions. Specifically, MoT-instructions are constructed through a three-step process as follows.
(1) We begin by aggregating twelve representative molecular interaction datasets across various widely recognized biochemical tasks, such as DDI, SSI, and CSI. Following this, we engage in a systematic search for textual descriptions of the biochemical properties of each molecule involved in these interactions. Specifically, we source this information from authoritative biochemical databases such as DrugBank and PubChem.
(2) The next critical step is the experimental determination of the optimal instructions. Specifically, for all molecular pairs in step (1), we first deconstruct the lengthy molecular properties into a series of questions and answers, a format more comprehensible to LLMs (Taylor et al., 2022). The granularity of this deconstruction is decided based on the performance of our MolTC. For more challenging quantitative tasks, instructions guided by fine-grained CoT are required to provide a numerical range before specifying a concrete value. Given the vast number of possible correct ranges, exhaustive testing is impractical. Therefore, we initially determine the optimal range for a small subset of datasets using a grid search, guided by the predictive performance of MolTC. Subsequently, we derive statistics, such as mean and standard deviation, from these datasets to establish a relationship between statistics and optimal ranges. Finally, for other datasets, we determine their optimal range based on this established rule.
(3) The final step in our dataset construction in-
volved filtering out pairs that lacked sufficient information on molecule properties or interaction data. Specifically, partial properties of a molecular pair are often missing in some datasets. To maximize the utilization of information from these datasets, we consider extracting each property within them as a separate dataset. This approach allows us to naturally omit missing values without wasting other information present in the molecular pair.

heading: 3 Experiment
text: In this section, we aim to answer the following research questions:
• RQ1: Is MolTC capable of generating the interactive property, involving the qualitative knowledge, of the given molecular pair? • RQ2: Does MolTC have the ability to generate the interactive property, involving the quantitative property, for a given molecular pair? • RQ3: What is the impact of the proposed strategies, such as the CoT enhancement strategy and SMILES injection strategy, on the inference process of our MolTC?

heading: 3.1 Experimental Setting
text: We evaluate MolTC on twelve well-established downstream molecule interaction tasks involving qualitative and quantitative analysis. Here we provide a brief overview of our experimental setup. Detailed descriptions are presented in the appendix.
Datasets. We employ 12 datasets across various domains such as DDI, SSI, and CSI. Specifically, we collect Drugbank (Version 5.0.3), ZhangDDI (Zhang et al., 2017), ChChMiner (Zitnik et al., 2018), DeepDDI (Ryu et al., 2018), TWOSIDES (Tatonetti et al., 2012), Chromophore (Joung et al., 2020), MNSol (Marenich et al., 2020), CompSol (Moine et al., 2017), Abraham (Grubbs et al., 2010), CombiSolv (Vermeire and Green, 2021), FreeSolv (Mobley and Guthrie, 2014) and CombiSolv-QM (Vermeire and Green, 2021).
Baselines. For a comprehensive evaluation, we conduct various baseline methods encompassing distinct categories such as methods based on: GNNs, DL models other than GNN, and LLMs. Specifically, For DDI task, we employ GoGNN (Wang et al., 2020), MHCADDI (Deac et al., 2019), DeepDDI (Ryu et al., 2018), SSI-DDI, CGIB (Lee et al., 2023a), CMRL (Lee et al., 2023b), MDF-SADDI (Lin et al., 2022), DSN-DDI (Li et al., 2023c)
as the backbone. For SSI and CSI tasks, we utilize D-MPNN (Vermeire and Green, 2021), SolvBert (Yu et al., 2023a), SMD (Meng et al., 2023), CGIB (Lee et al., 2023a), CIGIN (Pathak et al., 2020), GEM (Fang et al., 2022), GOVER (Rong et al., 2020), Uni-Mol (Zhou et al., 2023) as the backbone. Furthermore, all downstream tasks adopt LLM-based methods, such as Galactica (Taylor et al., 2022), Chem T5 (Christofidellis et al., 2023), MolT5 (Edwards et al., 2022) and MolCA (Liu et al., 2023a) as the backbone.
Metrics. For qualitative tasks, we employ prediction Accuracy and AUC-ROC (Area Under the Receiver Operating Characteristic curve) as comparative metrics, while for quantitative tasks, MAE (Mean Absolute Error) and RMSE (Root Mean Square Error) are utilized as the standards.

heading: 3.2 Qualitative Prediction Performance (RQ1)
text: Table 1 exhibits the performance in qualitative interactive tasks. Due to page width limitations, only a subset of the results is presented, with additional results detailed in the appendix. From Table 1, we deduce the following observations:
Obs.1: MolTC consistently outshines its counterparts in qualitative interaction predictions, While GNN-based methods demonstrate commendable performance, maintaining over 90% accuracy across numerous datasets, MolTC transcends these figures in every evaluated scenario. For instance, it marks a notable 1.05% improvement in accuracy on the drugback dataset, a feat attributable to the synergy between the LLMs’ reasoning faculties and the GNNs’ proficiency in graph modeling.
Obs.2: The variability of MolTC’s outcomes, as indicated by the standard deviation, is consistently minimal in comparison to other models, On average, the standard deviation for MolTC is 35.41% lower than GNN-based models and 46.86% lower than LLM-based models. The precision in MolTC’s performance is largely attributed to the training paradigm enhanced by the multihierarchical CoT, which ensures a meticulous and accurate inference process.

heading: 3.3 Quantitative Prediction Performance (RQ2)
text: Table 2 shows the performance in a subset of quantitative tasks, with an exhaustive set of results detailed in the appendix. The datasets offer fourdimensional molecular information, comprising atom type, chirality tag, bond type, and bond direction. Key observations from Table 2 include:
Obs.3: MolTC continues to lead in quantitative analysis tasks, an area typically challenging for LLMs. Despite the strong baseline set by CGIB, characterized by low MAE and RMSE across datasets, MolTC outperforms it in every metric. For instance, it achieves a 23.98% reduction in RMSE on the CombiSolv dataset relative to CGIB. This underscores the advantage of adeptly leveraging the interaction between SMILE representations
and molecular graph structures. Obs.4: LLM-based models, in general, exhibit subpar performance in quantitative tasks compared to traditional DL-based models, attributed to their inadequacy in sharing and transferring learned molecular interaction insights across datasets and the absence of CoT-guided inference.

heading: 3.4 Ablation Study (RQ3)
text: Table 3 presents an ablation study aimed at dissecting the influence of SMILE auxiliary analysis and the optimized training paradigms based on Broad-grained and Fine-grained CoT. For the CSI dataset, properties such as the maximum absorption wavelength (Absorption), maximum emission wavelength (Emission), and excited state lifetime (Lifetime) are denoted as Abs., Emis., and Life., respectively. Key observations are as follows:
Obs.5: The three studied ablations exhibit significant influence on the results. For example, the collective impact of these three ablations registers an average drop of 12.77%, affirming the substantial enhancement imparted by the proposed strategies. Obs.6: The most pronounced effect is observed with the ablation of the Fine-grained CoT paradigm, which incurs an average accuracy decrement of 18.82%. This underscores the pivotal role of guiding the LLM to deduce a numerical range, a strategy particularly beneficial for quantitative analysis tasks, typically a challenging domain for LLMs. Obs.7: The least pronounced, yet significant, impact stems from the optimization of the Broadgrained CoT training paradigm, with an average accuracy reduction of 4.35%. Its importance is particularly underscored for molecular pairs involving larger and more complex molecules, where directly predicting interactive property by LLMs is arduous.

heading: 4 Discussion
text: Potential Data Leakage Risk. LLM might have memorized the prediction target during pretraining using a vast amount of text corpus. Hence, we acknowledge that LLM-based methods for biochemistry tasks such as MolT5 (Edwards et al., 2022), MolCA (Liu et al., 2023a) and our MolTC may face potential data leakage risks due to the memory during the pre-training process of their base LLMs. However, we claim that ablation experiments can mitigate the influence of data leakage, demonstrating the effectiveness of methods despite using potentially contaminated LLMs. For instance, in our
ablation experiments, the backbone LLMs are fed with SMILES strings and textual descriptions of molecules solely, and their performance is much worse than that of original methods. This stark contrast leads to two analytical observations:
• Firstly, note that these text inputs, highly recognizable and scattered throughout the LLM’s training data, could provide all the necessary information to identify a molecule. Hence, if there is significant data leakage, the LLM could easily recall from memory and make reasonably accurate predictions based on these inputs. Based on this, the poor performance of these ablation experiments can, to some extent, counter the concern of data leakage.
• Secondly, since these ablation experiments and the original experiments utilized the exact same backbone LLM, the significant differences in their outcomes indicate that the original method’s design is highly effective, regardless of whether the backbone LLM is subject to data leakage.
In summary, while these ablation experiments can not completely eliminate the impact of data leakage, they do mitigate it. Furthermore, it is worth noting that while the risk of data leakage exists, it does not deter researchers from continuously exploring and optimizing LLM-based frameworks for biochemical tasks. In the future, by constructing new biochemical datasets through biochemical experiments, which do not exist in the LLM’s training data, researchers can eliminate the data leakage issue in these frameworks thoroughly.

heading: 5 Conclusion
text: This work focuses on molecule rationale learning, which plays a pivotal role in predicting molecular interactions. Specifically, we introduce a novel, unified LLM-based framework for predicting molecular interactive properties, termed MolTC. To efficiently train it, we propose a multi-tiered CoT principle to guide the training paradigm. Experiments conducted across twelve varied datasets demonstrate the superiority of our method over the current GNN and LLM-based baselines. This breakthrough sets a new standard for integrating multimodal data in LLM-based MRL.
Limitations
While this research has undergone extensive testing across a diverse array of datasets covering various
domains, it does have certain limitations. Specifically, the study has not been subjected to datasets comprising exceptionally large molecules, which represent extreme cases. Furthermore, the methodologies employed in this research have not yet been adapted or evaluated in contexts requiring few-shot or zero-shot learning scenarios. Future endeavors will focus on expanding the scope of this study to encompass these areas.
Ethics Statement
This work is primarily foundational in molecular relational learning, focusing on the development of a unified LLM-based paradigm. Its primary aim is to contribute to the academic community by enhancing the understanding and implementation of the molecular relational modeling process. We do not foresee any direct, immediate, or negative societal impacts stemming from the outcomes of our research.

heading: Acknowledgement
text: This research is supported by the National Science and Technology Major Project (2023ZD0121102), National Natural Science Foundation of China (92270114).

heading: A Related Work
text: Since exhaustive experimental validation of the molecule interactions is notoriously timeconsuming and costly (Lee et al., 2023a), more recently, adopting LLM has emerged as a promising alternative for efficient and effective molecular relational learning, which are known for their vast knowledge repositories and advanced logical inference capabilities. Compared to the prediction of single-molecule properties (Liu et al., 2024b; Li et al., 2024; Liu et al., 2023b; Fang et al., 2023d; Guo et al., 2023a), the field of molecular interaction prediction is still in its early stages. For instance,
• Protein-protein interactions (PPI). In this context, proteins are represented as residue contact graphs, also known as amino acid graphs, where each node is a residue (Park et al., 2022; Jha et al., 2022a,b,b). Notably, Jha et al. (2022b) leverages the superior encoding capabilities of the biochemical LLMs, where the input to the LLM is the protein sequence, and the output is a feature vector for each amino acid in the sequence. This output is then used as node features in the residue contact graph to enhance the prediction of PPI tasks.
• DDI. In this context, Pei et al. (2023) focuses on enriching cross-modal integration in biology with chemical knowledge and natural language associations, achieving significant results in multiple drug-target interaction prediction tasks. Recently, few-shot drug pair synergy prediction is gradually gaining attention and entering the spotlight(Li et al., 2023b).
• Chemical reactions. For understanding chemical reactions (Chen et al., 2023; Guo et al., 2023b), Shi et al. (2023) selects in-context reaction examples with varying confidence scores closest to the target reaction query, encouraging large models to understand the relationships between these reactions. Sagawa and Kojima (2023) focuses on optimizing low-sample organic
chemical applications by pretraining them with extensive compound libraries and fine-tuning with smaller in-house datasets for specific tasks. Livne et al. (2023) introduces a new foundational model, nach0, capable of solving various chemical and biological tasks, including molecular synthesis.
More recently, ReactXT (Liu et al., 2024a) also focuses on modeling molecular interactions using chemical reactions. However, it is unable to leverage CoT for more robust information extraction. Meanwhile, the Mol-instructions dataset has been introduced (Fang et al., 2023c) for effectively training biochemistry LLMs, that can be considered as complementary to our MoT-instruction.

heading: B Experiments
text: Here, we provide a detailed experimental setup along with additional results. It is important to note that for aspects such as dataset division and hyperparameter configurations in baselines, we followed the settings established by CGIB (Lee et al., 2023a). Moreover, all settings can be found in our code https://github.com/MangoKiller/MolTC.
B.1 Datasets We employ 12 datasets across various domains such as DDI, SSI and CSI.
Drugbank (version 5.0.3). This dataset consists of 1704 drugs, 191400 drug pairs, and defines 86 distinct DDI event types. Essential drug information, including DrugBank ID, drug name, molecular SMILES, and target. provided. ZhangDDI (Zhang et al., 2017). It contains 548 drugs and 48,548 pairwise interaction data and multiple types of similarity information about these drug pairs. ChChMiner (Zitnik et al., 2018). It contains 1,322 drugs and 48,514 labeled DDIs, obtained through drug labels andscientific publications. DeepDDI (Ryu et al., 2018). It contains 192,284 labeled DDIs and their detailed side-effect information, which is extracted from Drugbank. TWOSIDES (Tatonetti et al., 2012). It collects 555 drugs and their 3,576,513 pairwise interactions involving 1318 interaction types from TWOSIDES. Chromophore (Joung et al., 2020). It contains 20,236 combinations of 7,016 chromophores and 365 solvents which are given in the SMILES string format. All optical properties are based on scientific publications and unreliable experimental
results are excluded after examination of absorption and emission spectra. In this dataset, we measure our model performance on predicting maximum absorption wavelength (Absorption), maximum emission wavelength (Emission) and excited state lifetime (Lifetime) properties which are important parameters for the design of chromophores for specific applications. We delete the NaN values to create each dataset which is not reported in the original scientific publications. Moreover, for Lifetime data, we use log normalized target value since the target value of the dataset is highly skewed inducing training instability.
MNSol (Marenich et al., 2020). It contains 3,037 experimental free energies of solvation or transfer energies of 790 unique solutes and 92 solvents.In this work, we consider 2,275 combinations of 372 unique solutes and 86 solvents following previous work.
FreeSolv (Mobley and Guthrie, 2014). It provides 643 experimental and calculated hydration free energy of small molecules in water. In this work, we consider 560 experimental results following previous work.
CompSol (Moine et al., 2017). This dataset is proposed to show how solvation energies are influenced by hydrogen-bonding association effects. We consider 3,548 combinations of 442 unique solutes and 259 solvents in the dataset following previous work.
Abraham (Grubbs et al., 2010). This dataset is a collection of data published by the Abraham research group at College London. We consider 6,091 combinations of 1,038 unique solutes and 122 solvents following previous work.
CombiSolv (Vermeire and Green, 2021). It contains all the data of MNSol, FreeSolv, CompSol, and Abraham, resulting in 10,145 combinations of 1,368 solutes and 291 solvents.
CombiSolv-QM (Vermeire and Green, 2021). It is generated with 1 million combinations of 284 commonly used solvents and 11,029 solutes. Those 1 million data points are randomly selected from all possible solvent–solute combinations. Solvents and solutes with elements H,B,C,N,O, F, P, S, Cl, Br and I are included with a solute molar mass ranging from 2.02 g/mol to 1776.89 g/mol.
B.2 Baselines
We use both specific task conventional deep learning models and current biochemical LLMs as the baselines. Specifically, for qualitative tasks:
GoGNN (Wang et al., 2020). It extracts features from structured entity graphs and entity interaction graphs in a hierarchical manner. We also propose a dual attention mechanism that enables the model to preserve the importance of neighbors in both levels of the graph. MHCADDI (Deac et al., 2019). A gated information transfer neural network is used to control the extraction of substructures and then interact based on an attention mechanism. DeepDDI (Ryu et al., 2018). First, the structural similarity profile is calculated between the two input drugs and other drugs, and then prediction is completed based on the deep neural network. SSI-DDI (Nyamabo et al., 2021). It uses a 4-layer GAT network to extract substructures at different levels, and completes the final prediction based on the co-attention mechanism. CGIB (Lee et al., 2023a). Based on the graph conditional information bottleneck theory, conditional subgraphs are extracted to complete the interaction between molecules. CMRL (Lee et al., 2023b). It detects the core substructure that is causally related to chemical reactions by introducing a novel conditional intervention framework whose intervention is conditioned on the paired molecule. MDF-SA-DDI (Lin et al., 2022). It predicts interaction based on multi-source drug fusion, multisource feature fusion and transformer self-attention mechanism. DSN-DDI (Li et al., 2023c). It employs local and global representation learning modules iteratively and learns drug substructures from the single drug ‘intra-view’) and the drug pair (‘inter-view’) simultaneously.
For quantitative task, we employ the following baselines: D-MPNN (Vermeire and Green, 2021). It employs a transfer learning approach to predict solvation free energies, integrating quantum calculation fundamentals with the heightened accuracy of experimental measurements through two new databases, CombiSolv-QM and CombiSolv-Exp. SolvBert (Yu et al., 2023a). It interprets solute and solvent interactions through their combined
SMILES representation. After pre-training by unsupervised learning with a substantial computational solvation free energy database, SolvBERT is adaptable to predict experimental solvation free energy or solubility by fine-tuning on specific databases. SMD (Meng et al., 2023). It utilizes the quantum charge density of a solute and a continuum representation of the solvent. It breaks down solvation free energy into two components: bulk electrostatic contribution, treated through a self-consistent reaction field using IEF-PCM, and a cavity-dispersionsolvent-structure term, accounting for short-range interactions in the solvation shell based on atomic surface areas with geometry-dependent constants. CIGIN (Pathak et al., 2020). It adopts an end-toend framework consisting of three essential phases: message passing, interaction, and prediction. In the final phase, these stages are leveraged to predict
solvation free energies. GEM (Fang et al., 2022). It exhibits a uniquely designed geometry-based graph neural network architecture, complemented by several dedicated selfsupervised learning strategies at the geometry level. That aims to acquire comprehensive molecular geometry knowledge for accurate prediction of molecular properties. GOVER (Rong et al., 2020). It captures rich structural information from extensive unlabeled molecular data through self-supervised tasks, employing a flexible Transformer-style architecture integrated with Message Passing Networks. This allows GROVER to be trained efficiently on largescale datasets without supervision, addressing data scarcity and bias challenges. Uni-Mol (Zhou et al., 2023). It incorporates two pre-trained models featuring the SE(3) Transformer architecture: a molecular model pre-trained on 209 million molecular conformations and a pocket model pre-trained on 3 million candidate protein pocket data. Additionally, Uni-Mol integrates various fine-tuning strategies to effectively apply these pre-trained models across diverse downstream tasks.
B.3 Modules
In our experiments, the two graph encoder are instantiated by the five-layer GINE (Hu et al., 2019). We conduct 2 million molecules from the ZINC15 (Sterling and Irwin, 2015) dataset to pretrain them by contrastive learning following (Liu et al., 2023a). Similarly, two projector are initialized with the encoder-only transformer, Sci-BERT, which is pretrained on scientific publications (Beltagy et al., 2019), while its cross-attention modules are randomly initialized. More detailed pretraining process of our Q-Formors follows the training process
in (Liu et al., 2023a), such as there are 8 query tokens in Q-Formers (Nq = 8). Note that for LLMbased baselines, we fine-tune the backbone LLMs on task-specific datasets for fair comparison. Their prediction is considered accurate only if the outputs include words or numbers that correctly depict the interaction in question, without presenting any that describe alternative interactions.
B.4 Training Epochs During the fine-tuning phase, the number of epochs varies for different tasks. For example, for the DDI task, we typically fine-tune for 100 epochs. For SSI datasets with more than 3000 molecular pairs, we initially fine-tune on the CombiSolv-QM (Vermeire and Green, 2021) dataset for 100 epochs, followed by an additional 30 epochs on their respective datasets. For SSI datasets with fewer than 3000 molecular pairs, this number is adjusted to 20. Furthermore, both the fine-tuning and pre-training phases employ the same configuration for the optimizer and learning rate scheduler, as detailed in the following section.
B.5 Training Strategy We employ the AdamW optimizer (Loshchilov and Hutter, 2017) with a weight decay set at 0.05. Our learning rate strategy utilizes a combination of linear warm-up and cosine decay, optimizing the training process by initially increasing the learning rate to promote faster convergence, and then gradually decreasing it according to a cosine curve to finetune the model parameters. LoRA is implemented using the Open Delta library (Ding et al., 2022), and the PEFT library (Mangrulkar et al., 2022). LoRA’s rank r is set to 16, while LoRA is applied to Galactica’s modules of [q_proj, v_proj, out_proj, fc1, fc2] following (Liu et al., 2023a). This configuration yields a LoRA adapter with 12M parameters which constitutes merely 0.94% of the parametersin the Galactica1.3B.
B.6 More Experimental Results Table 4 presents the experimental results not shown in the main text due to length constraints. Note that the three datasets in the CSI domain are all derived by splitting the Chromophore dataset. As discussed in Section 3.3, for a fair comparison, we limited the input features to four-dimensional molecular information, comprising atom type, chirality tag, bond type, and bond direction. Given the difficulty of convergence for some DL-based baselines under
this setting, we only showcased the performance of the LLM-based baselines. Meanwhile, considering that our SSI tasks are firstly fine-tuned on the CombiSolv-QM dataset, we present the comprehensive results of this dataset, as shown in Table 5. Observations from Table 4 and 5 are largely consistent with those in the main experimental section. That is, across all tasks, our MolTC outperforms the LLM-based baseline methods in a large margin.

heading: C Future Work
text: In this paper, we introduce a novel unified framework, leveraging LLM technology to predict molecular interactive properties. The future development directions of this project are twofold. First, we aim to adopt advanced graph encoding techniques to enhance the comprehension of molecular graphs (Gao et al., 2023; Wang et al., 2023a; Yu et al., 2023b, 2024; Gao et al., 2024). Secondly, in light of the extensive input information introduced in multi-molecular input scenarios, we plan to employ techniques such as graph interpretability (Ying et al., 2019; Luo et al., 2020; Zheng et al., 2023; Fang et al., 2024) and sparsification (Wang et al., 2023b,c) to eliminate information redundancy and resist out-of-distribution (Fang et al., 2023a,b).

heading: Future Work
text: The future development directions of this project are twofold. First, we aim to adopt advanced graph encoding techniques to enhance the comprehension of molecular graphs (Gao et al. , 2023; Wang et al. , 2023a; Yu et al. , 2023b, 2024; Gao et al. , 2024). Secondly, in light of the extensive input information introduced in multi-molecular input scenarios, we plan to employ techniques such as graph interpretability (Ying et al. , 2019; Luo et al. , 2020; Zheng et al. , 2023; Fang et al. , 2024) and sparsification (Wang et al. , 2023b,c) to eliminate information redundancy and resist out-of-distribution (Fang et al. , 2023a,b)