Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation

## Abstract

Abstract High-throughput reaction condition (RC) screening is fundamental to chemical synthesis. However, current RC screening suffers from laborious and costly trial-and-error workflows. Traditional computer-aided synthesis planning (CASP) tools fail to find suitable RCs due to data sparsity and inadequate reaction representations. Nowadays, large language models (LLMs) are capable of tackling chemistry-related problems, such as molecule design, and chemical logic Q&A tasks. However, LLMs have not yet achieved accurate predictions of chemical reaction conditions. Here, we present MM-RCR, a text-augmented multimodal LLM that learns a unified reaction representation from SMILES, reaction graphs, and textual corpus for chemical reaction recommendation (RCR). To train MM-RCR, we construct 1.2 million pair-wised Q&A instruction datasets. Our experimental results demonstrate that MM-RCR achieves state-of-the-art performance on two open benchmark datasets and exhibits strong generalization capabilities on out-of-domain (OOD) and High-Throughput Experimentation (HTE) datasets. MM-RCR has the potential to accelerate high-throughput condition screening in chemical synthesis.

## 1 Introduction

Chemical synthesis is a crucial step for the discovery of transformative molecules in multiple fields, including drug design, materials, renewable energy, etc.
In chemical synthesis, reaction conditions are usually optimized to maximize the yield of each target molecule or minimize the cost of the corresponding process [, ]. Despite significant advancements in chemical synthesis over the past few decades, discovering suitable reaction conditions from the extensive substrates combined with high-dimensional conditions renders exhaustive experimental impractical. []. Chemists have focused on building reliable and convenient computer-aided synthesis planning (CASP) tools to facilitate chemical synthesis [, , ]. However, few efforts have been made to solve the problem of reaction condition screening due to the low sparsity of chemical data, and the lack of effective reaction representation [, ]. In summary, to realize efficient synthesis in chemistry, there is an urgent need to realize high-efficiency reaction condition recommendations.

Nowadays, the emergency of generative pre-trained transformer-based large language models (LLMs), typified by GPT-4, has sparked significant interest in the field of AI for chemistry [, , , ].
Prtrained with massive chemical reaction data including molecular simplified molecular-input line-entry system (SMILES)  [] and chemistry literature in natural language, LLMs are endowed with fundamental chemical knowledge through text-to-text generation. However, for tasks that demand a precise understanding of molecular SMILES representations, such as retrosynthesis and chemical condition recommendation, LLMs have exhibited less competitive performance compared to traditional methods [, ]. Further, these text-to-text models cannot fully exploit the advantages of molecular structure data and fall short in understanding reaction mechanisms []. To address these challenges, chemical reaction condition recommendation necessitates LLMs to possess additional chemical comprehension representation beyond textual data to understand effectively and reason over chemical processes.

Multimodal large language models (MM-LLMs) have been proven to achieve higher accuracy and perform more effectively in a wide range of applications [, , ]. Considering that, in addition to SMILES strings, there are various types of data in the field of chemistry, such as molecular graphs and external textual corpus of reaction []. By synergizing the strengths of multiple modalities of chemical data, we enhance the capabilities of LLMs to understand complex chemical processes [].
However, there is currently no widely adopted multimodal prediction model specifically tailored for chemical reaction condition recommendation. Hence, it is imperative to develop an effective prediction model that can incorporate different chemical data into LLMs to achieve a more comprehensive understanding of reaction processes, facilitating the task of chemical reaction condition recommendation.

In view that molecules can be expressed as sequences, and reactions are described as natural language, e.g. text corpus, MM-LLMs can be a potential solution due to the following advantages: (i) pre-trained with extensive reaction data, foundational LLMs can learn relationships between molecules in reactions, thereby acquiring chemical knowledge akin to the learning process of chemists []; (ii) via learning the joint representation of chemical reactions from different modalities, including graphs, SMILES, and corpus, LLMs might be empowered the capability of understanding the mechanism of reactions, which facilitates the task of RCR. To this end, we fine-tune general-purpose LLMs with domain-specific reaction data for RCR. Specifically, we present MM-RCR, a multimodal LLM that jointly learns from the SMILES, graphs, and textual corpus of reactions.
The contributions of this work can be summarized as follows:

- 1.
We propose a multimodal LLM, a.k.a. MM-RCR, designed to learn a unified reaction representation from SMILES, graphs, and textual corpus of reactions for condition recommendation tasks. We further develop two distinct types of prediction modules, a classification module, and a generation module for MM-RCR to enhance its compatibility with different chemical reaction condition predictions.
- 2.
We design text-augmented instruction prompts to construct a 1.2 million pair-wised Q&A dataset for training. We propose the Perceiver module for modality alignment, which utilizes latent queries to align graphs and SMILES tokens with text-related tokens.
- 3.
Through experimental validation on benchmark datasets, MM-RCR achieves competitive results comparable to state-of-the-art models. Furthermore, MM-RCR exhibits strong generalization capabilities on out-of-domain (OOD) and high-throughput experimentation (HTE) datasets.

## 2 Related Work

In chemical synthesis, reaction conditions are usually developed and optimized to maximize the yield of each target molecule or minimize the cost of the corresponding process [, ]. High-throughput reaction condition (RC) screening, as an important tool in synthesizing molecules, exerts an important influence on chemical synthesis. However, discovering suitable reaction conditions from the extensive matrix of substrates combined with the high-dimensional reaction conditions renders exhaustive experimental impractical. []. For decades, chemists have focused on building reliable and convenient computer-aided synthesis planning (CASP) tools to facilitate chemical synthesis [, ]. For instance, Coley et al. built a multiway classification model based on a two-step graph convolutional network (GCN) for the reaction prediction task [, ].
Due to the effectiveness of a simplified molecular-input line-entry system (SMILES) [], as strings of a context-free, Nam et al. proposed the first sequence-to-sequence model for forward prediction using the SMILES representations of molecules []. Inspired by attention-based transformer model [], Schwaller et al. proposed molecular transformers [, ], which were applied in forward prediction and reaction condition recommendation (RCR) tasks [, ].

Figure: Figure 2: Architecture of MM-RCR. MM-RCR processes task-specific questions constructed by text-augmented multimodal instruction prompts and generates answers. Specifically, it takes three modalities of data as inputs: text (a textual corpus of reactions and question prompts), molecular SMILES, and reaction graphs. Two distinct types of prediction modules, a classification module, and a generation module are proposed to predict chemical reaction conditions.
Refer to caption: x2.png

Chemical reaction condition recommendation tasks aim to recommend catalysts, reagents, solvents, or other conditions for a specific reaction. The exploration of a suitable condition is crucial for the realization of CASP, as it dictates the expected outcomes, including reaction yields and rates []. Gao et al. developed a neural network model to predict the chemical context as well as the temperature for any particular organic reaction []; Maser et al. proposed a machine-learned ranking model to predict the set of conditions used in a reaction as a binary vector []; Wang et al. proposed Parrot, a powerful and interpretable Transformer-based model for the prediction of reaction condition []; In the meantime, in order to enhance the representation of reactions, Qian et al. [] designed TextReact, which introduced relevant corpus retrieved from literature to enhance the molecular representation of the reaction based on SMILES. Nevertheless, these methods rely on manual feature selection by experts’ knowledge and lack a general prediction model with powerful reaction representation.

Nowadays, the emergency of generative pre-trained transformer-based large language models (LLMs), typified by GPT-4, has triggered keen interest in leveraging such techniques to tackle chemistry challenges [, ]. Several works focus on chemical agents for the exploration of chemical conditions [, ].
However, for tasks demanding a precise understanding of molecular SMILES representation, such as reaction prediction, and retrosynthesis, LLMs exhibited a less competitive performance than traditional machine learning baselines []. Partially, the reason is that, without an in-depth understanding of the SMILES strings, and the reaction process that transforms reactants into products, it will be difficult for LLMs to generate accurate responses.

Besides SMILES strings, there are various types of data such as molecule graphs and the reactions’ external textual corpus in the chemistry synthesis field.
By synergizing the strengths of multiple modalities, multimodal large language models (MM-LLMs) can achieve higher accuracy, and perform more effectively in a wide range of applications [, , , , , ].

## 3 Methods

### 3.1 Problem Setup

For a task of reaction condition recommendation, we define the $X$ as the input for the chemical reaction $R$, $T$ as the reaction corpus, $G$ as the graph representations of reactions, and the output $Y$ as a list of reaction conditions including the catalyst, solvent, and reagent. Thus, we define prediction model $\mathcal{F}$, i.e., $Y=\mathcal{F}(X,G,T)$.

In this paper, we incorporate three types of data for the training of model $\mathcal{F}$:

- 1.
SMILES of a reaction $X$: each example in the training set is presented by chemical SMILES, i.e., “CC(C)O.O=C(n1ccnc1)nccnc1 $>>$ CC(C)OC(=O)n1ccnc1”.
- 2.
Graphs of reaction $G$: each SMILES representation of the reactants and the product is encoded using a graph neural network (GNN). All compounds are integrated to generate a comprehensive reaction representation.
- 3.
An unlabeled reaction corpus: a paragraph describing a chemical reaction, e.g., “To a solution of CDI (2 g, 12.33 mmol), in DCM (25 mL) was added isopropyl alcohol (0.95 mL, 12.33 mmol) at 0° C.”.

### 3.2 Model Structure

Here we first describe the MM-RCR, a multimodal LLM designed for reaction condition recommendation (RCR). An overview of MM-RCR is provided in Figure. 2. MM-RCR responds to task-specific questions constructed by instruction prompts such as “please recommend a catalyst of this reaction: Reactant1.Reactant2$\gg$Product”, and generates answers about reaction conditions. MM-RCR takes three modalities of data as inputs, including text (a textural corpus of reaction and question prompts), molecular SMILES, and graphs of reactions. We employ both transformer-based reaction encoder and GCN models to jointly learn reaction representations from SMILES. Subsequently, the modality projection transforms the graph and SMILES embeddings into language tokens compatible with LLM space. These learnable tokens, defined as reaction tokens, along with tokens of question prompts, are then input into the LLM to predict chemical reaction conditions. Note that, we develop two distinct types of prediction modules, a classification and a generation prediction module to enhance its compatibility with different chemical reaction conditions.

#### 3.2.1 Construction of Text-Augmented Instruction Prompts

Instruction prompt datasets refer to format structured or unstructured data as natural language instructions so that LLMs can respond properly [, ]. Compared to creating language instruction datasets for fine-tuning LLMs, constructing multimodal instruction datasets requires a thorough understanding of domain-specific tasks. Recent advancements indicate that the other data modalities, such as images, and graphs, can be transformed as the prefix of prompts thereby facilitating effective reasoning based on inputs [, , ].

Toward reaction condition recommendation task in chemical synthesis, we design a tailored instruction prompts system for better cross-modality alignment and instruction tuning (Figure. 3). Compared to instruction prompts for natural language instruction tunning (Figure. 3(a)), we introduce augmented text tokens and multimodal tokens into instruction prompts (Figure. 3(b)). In particular, given a reaction, we collect corpus (<Corpus>), a paragraph that is similar to this reaction, and its SMILES (<Reaction SMILES>) to construct high-quality Q&A datasets. Question templates such as ‘please predict the optimal conditions’ are generated by GPT-4 autonomously using prompt engineering; reaction embeddings (<SMILES> and <Graph>) are inserted into instruction prompts. The expected answer for each question is the combination of chemical conditions, such as ‘Cl.ClCCl’. It is important to note that, to maintain the diversity of instruction datasets, we randomly generate 2,000 question templates using GPT-4 for each pair-wised Q&A. In a word, we encode all representations from different modalities into a unified language space, which facilitates the generation of responses by LLMs.

Figure: Figure 3: Instruction of text-augmented prompts. (a) Traditional instruction prompts for natural language instruction tunning; (b) Our proposed text-augmented multimodal instruction Q&A prompts.
Refer to caption: x3.png

#### 3.2.2 Encoder and Decoder

Given a reaction $R$, we adapt a pioneering transformer-based encoder, Parrot [] to produce the reaction embeddings ${\mathbf{X}}_{R}\in\mathbb{R}^{N\times C}$. Here, $N$ and $C$ indicate the length of text tokens and embedding channels, respectively. During training the encoder computes a contextual vector representation of the reactions by performing self-attention on the masked canonicalized SMILES string of molecules. We denote reaction embeddings as SMILES embedding in the following section.

In the meantime, we leverage a GNN [] to model the relationship between atoms in molecules. We denote directed and labeled multi-graphs as $G=(\mathcal{V},\mathcal{E},\mathcal{R})$ with nodes (atom entities), $v_{i}\in\mathcal{V}$ and labeled edges (atom relations) $\left(v_{i},r,v_{j}\right)\in\mathcal{E}$, where $r\in\mathcal{R}$ is a relation type. GNN can be understood as special cases of a simple differentiable message-passing framework:

$$ $h_{i}^{(l+1)}=\sigma\left(\sum_{m\in\mathcal{M}_{i}}g_{m}\left(h_{i}^{(l)},h_{ j}^{(l)}\right)\right)$ (1) $$

where $h_{i}^{(l)}\in\mathbb{R}^{d^{(l)}}$ is the hidden state of node $v_{i}$ in the $l$-th layer of the neural network, with $d^{(l)}$ being the dimensionality of this layer’s representations. Incoming messages of the form $g_{m}(\cdot,\cdot)$ are accumulated and passed through an element-wise activation function $\sigma(\cdot)$, such as the $\operatorname{ReLU}(\cdot)=\max(0,\cdot),\mathcal{M}_{i}$ denotes the set of incoming messages for node $v_{i}$ and is often chosen to be identical to the set of incoming edges. $g_{m}(\cdot,\cdot)$ is typically chosen to be a (message-specific) neural network-like function or simply a linear transformation $g_{m}\left(h_{i},h_{j}\right)=Wh_{j}$ with a weight matrix $W$.
Motivated by this architecture, GCNN [] proposed a refined propagation model for the forward-pass update of an entity or node:

$$ $h_{i}^{(l+1)}=\sigma\left(\sum_{r\in\mathcal{R}}\sum_{j\in\mathcal{N}_{i}^{r}} \frac{1}{c_{i,r}}W_{r}^{(l)}h_{j}^{(l)}+W_{0}^{(l)}h_{i}^{(l)}\right)$ (2) $$

where $\mathcal{N}_{i}^{r}$ denotes the set of neighbor indices of node $i$ under relation $r\in\mathcal{R}.c_{i,r}$ is a problem-specific normalization constant that can either be learned or chosen in advance (such as $c_{i,r}=\left|\mathcal{N}_{i}^{r}\right|$ ).

We develop two distinct types of prediction modules, a classification module and a generation module for MM-RCR to enhance its compatibility with different chemical reaction conditions. Prediction modules are used to generate probability distributions over potential tokens, we define two types of loss for this:

$$ $\text{Prediction}:\begin{cases}(1)\,X,G,T\xrightarrow{(classifer)}(c_{i}, \widehat{c}_{i}):\mathcal{L}=\sum_{i\in I}CrossEntropyLoss\left(c_{i},\hat{c}_ {i}\right)\\ (2)\,X,G,T\xrightarrow{(generate)}(C,\widehat{C}):\mathcal{L}=-\sum_{l=1}^{L} \sum_{v=1}^{V}y_{l}^{v}\log P_{\theta}(y_{l}^{v}\mid y_{<l},(x,g,t))\end{cases}$ (3) $$

where $classifer$ refers to classification head, $I$ is the chemical context condition number, $c_{i}$ is the predicted label of the $i$-th condition, $\widehat{c}_{i}$ is the ground truth label of the $i$-th condition; $generate$ refers to generation head, $C$ and $\widehat{C}$ are the combination of predicted and the ground truth conditions, respectively. $L$ is the sequence length, $V$ is the vocabulary size. $y_{l}$ is the one-hot encoded target token at position $l$, $y_{l}^{v}$ is the $v$-th element of the one-hot encoded target token at position $l$; $y_{<l}$ represents all previous tokens before position $l$; $(x,g,t)$ is the input context tokens representing SMILES, graphs, and corpus.

#### 3.2.3 Modality Projection

For the reaction condition recommendation task, the representation of the reaction is extracted by encoders (see in section  3.2.2), and the text representation is tokenized by LLMs. However, fusing two types of representation introduces inductive biases issues [, ]. To effectively fuse representations from multiple modalities, we propose projection modules, the Perceiver [], for modality alignment (Figure 2). This module employs latent tokens to align graphs and SMILES embeddings with text-related tokens extracted from question prompts and a text-augmented corpus.
During training, we employ two Transformer-based Perceivers as projectors. Although these modules share an identical model architecture, they are distinguished by their unique weights Consequently, learnable tokens contain highlighted reaction cues that are most related to the text tokens. We show the pseudo-code for modality projection in Appendix. C.

## 4 Experiments and Results

### 4.1 Data

We curate two large datasets, named USPTO-Condition and USPTO_500MT_Condition for evaluation. Data volumes are presented in Table. 7. The visualization of data distribution is depicted in Figure. 5. As depicted in Table. 1, for the USPTO-Condition dataset, five conditions categories are separated by commas in order. For the USPTO_500MT_Condition dataset, all conditions are combined by dot as strings. The detailed data description can be seen in Appendix. B.

**Table 1: Data description of USPTO-Condition and USPTO_500MT_Condition.**
| Dataset | Condition label | Prediction type | Training set |
| --- | --- | --- | --- |
| USPTO-Condition | [Zn],C1CCOC1,O,CO,[Cl-].[NH4+] | classification | 546,728 |
| USPTO_500MT_Condition | CO.[Na+].CC(=O)O.[BH3-]C#N | generation | 88,410 |

### 4.2 Experiment Setup

In our work, the reaction encoder is implemented based on Wang et al. []. A pre-trained graph model proposed by [] encodes the molecules in the reaction. We utilize LLaMA-2 [] as a text decoder. Each reaction has the corresponding corpus, a paragraph describing a chemical reaction with an average length of 190 tokens.
During the training process, we fix the weight parameters of GCN, reaction encoder, and LLaMA-2. The modality projection and condition prediction layer is trainable. The detailed training setting can be seen in Appendix. A.

### 4.3 Performance Comparison

We assess the performance of our proposed MM-RCR for reaction condition recommendation. The top-$N$ accuracy of condition recommendation on the combined test datasets of USPTO-Condition and USPTO_500MT_Condition are presented in Table. 2 and Table. 3, respectively. Compared methods include RCR [], Reaction GCNN [], TextReact [], and Reagent Transformer [], and the details of the baselines are present in Appendix. D.

**Table 2: Results of reaction condition recommendation on USPTO-Condition dataset. The best performance is in bold.**
| Model |  | Top-$k$ Accuracy (%) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Catalyst |  | Solvent 1 |  | Solvent 2 |  | Reagent 1 |  | Reagent 2 |  |  |  |  |  |  |  |  |  |  |  |
|  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  |
| RCR |  | 91.6 | 94.1 | 95.2 |  | 48.3 | 64.4 | 70.2 |  | 81.4 | 83.4 | 84.6 |  | 48.2 | 64.4 | 70.8 |  | 76.5 | 84.1 | 86.4 |
| Parrot |  | 89.9 | 96.4 | 97.7 |  | 35.2 | 60.9 | 72.2 |  | 81.2 | 93.7 | 96.7 |  | 40.4 | 62.3 | 71.7 |  | 80.6 | 90.6 | 93.6 |
| TextReacts |  | 63.3 | 74.6 | 78.1 |  | 59.5 | 73.2 | 78.5 |  | 38.4 | 49.3 | 55.2 |  | 51.5 | 66 | 72.2 |  | 44.2 | 57.4 | 63.6 |
| MM-RCRs |  | 92.8 | 98.6 | 99.3 |  | 54.7 | 76.5 | 84.9 |  | 81.9 | 94.8 | 97.6 |  | 53.4 | 75.9 | 83.9 |  | 78.6 | 93.2 | 96.2 |

Figure: Table 3: Results of reaction condition recommendation on USPTO_500MT_Condition dataset. The best performance is in bold.

For the USPTO-Condition dataset, we calculate top-$k$ accuracy with a strict matching policy. As depicted in Table. 2, TextReacts refers that we utilize similar text [] paired with the corresponding reaction for training. To avoid label leak issues, we do not use gold text mentioned in his work for training or testing. MM-RCRs refers that we use a similar corpus paired with each reaction as input to construct Q&A instruction datasets for training. Thanks to the work of Qian et al., we can retrieve the most similar corpus for each reaction from the literature or patents using their pre-trained model.

From the results, we observe that due to the low data sparsity of catalysts in the USPTO-Condition dataset (Figure. 8), all compared methods perform well, with the top-1 accuracy of the catalyst almost exceeding 90%. For solvent prediction, MM-RCR outperforms the other methods, with top-1 accuracy of 54.7% (solvent 1) and 81.9% (solvent 2), respectively. The overall top-1 accuracy of MM-RCR is 34.1% higher than that of the Parrot model. We conclude that our proposed MM-RCR exhibits strong capabilities of reaction representation, akin to the learning process of chemists [].

Unlike the USPTO-Condition dataset which includes three types of chemical condition data–catalysts, solvents, and reagents–the USPTO_500MT_Condition dataset categorizes all conditions as ‘reagents’. Thus, we ask LLM to generate answers directly as the sequence-to-sequence generation instead of using a condition classification head. The performance of comparative methods on the USPTO_500MT_Condition dataset is shown in Table. 3. The visualization of performance is shown in Appendix Figure. 7. We examine top-1, top-3, top-5 and top-10 predictive results. Notably, we can see that MM-RCR demonstrates the most favorable performance on the USPTO_500MT_Condition dataset, where achieves 25.9% top-1 accuracy when compared with other baseline methods such as Parrot (13.8%), Reagent Transformer (17.5%), and Reaction GCNN (16.1%). Since all SMILES conditions in the USPTO_500MT_Condition dataset are concatenated with dots, they present challenges due to the long token sequences. However, MM-RCR, pre-trained on a vast natural language corpus, effectively manages and accurately generates these long tokens. We also visualize the predicted results on the USPTO-Condition in Appendix. D.

### 4.4 Ablation Study

#### 4.4.1 Model Structure

In MM-RCR, SMILES strings provide a textual representation of molecular structures, concisely encoding vital connectivity and stereochemistry details. Structural graphs of molecules offer a topological view of molecules in two-dimensional space, where atoms are nodes and bonds are edges. The textual corpus introduces a natural language context into the model to enhance the chemical interpretation capability of LLMs.

**Table 4: Performance evaluation of MM-RCR under different combinations of mono-domain data on the USPTO-Condition Dataset.**
| SMILES | Graph | Corpus | Top-$k$ Accuracy (%) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Catalyst |  | Solvent 1 |  | Solvent 2 |  | Reagent 1 |  | Reagent 2 |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  |  |  |
| ✓ | ✗ | ✗ |  | 90.3 | 97.5 | 98.7 |  | 37.1 | 64.5 | 75.7 |  | 80.8 | 92.9 | 96.8 |  | 37.1 | 63.5 | 74.7 |  | 73.7 | 89.9 | 94.1 |
| ✗ | ✗ | ✓ |  | 87.1 | 87.4 | 87.8 |  | 14.1 | 26.1 | 44.9 |  | 80.7 | 88.1 | 92 |  | 26.0 | 32.1 | 37.3 |  | 75.1 | 76.6 | 77.9 |
| ✓ | ✗ | ✓ |  | 92.6 | 98.5 | 99.3 |  | 54.0 | 76.0 | 84.4 |  | 81.8 | 94.7 | 97.6 |  | 52.8 | 75.4 | 83.3 |  | 78.6 | 93.1 | 96.1 |
| ✓ | ✓ | ✓ |  | 92.7 | 98.6 | 99.2 |  | 54.6 | 76.4 | 84.8 |  | 81.8 | 94.8 | 97.6 |  | 53.4 | 75.8 | 83.9 |  | 78.7 | 93.2 | 96.2 |

First, to examine the effect of different modalities on the performance of MM-RCR, we evaluate the performance under the different combinations of mono-domain data including SMILES, graph, and corpus on the USPTO-Condition dataset. As indicated in Table. 4, the results show that MM-RCR benefits from combining chemical mono-domain data. The performance is reported as top-k accuracy for various prediction tasks, including catalysts, solvents, and reagents. From the results we can see that, the model enhanced with SMILES representation (the first line) performs better than the model trained on the only corpus (the third line), with a 3.2% higher accuracy.

Next, we investigate the impact of combining different modalities of chemical data. The results indicate that the model trained with both SMILES and corpus data outperforms the model trained solely on SMILES representations, with top-1 accuracy of 54.0% of solvent 1, and 52.8% of reagent 1, respectively. By integrating a corpus into the model already trained with SMILES representation, we achieve improvements of 2.6% and 16.9% in the prediction accuracy of the catalyst and solvent 1, respectively. The reason is that incorporating additional corpus data into the model trained on SMILES representations provides LLMs with a more comprehensive understanding of chemical reactions, thereby enhancing their ability to address chemical synthesis tasks. Further, we observe that by introducing graph representations into the model, we achieve an additional average improvement of 1% in performance. The smaller improvement observed with the graph representation can be attributed to the pre-trained graph model’s development on a connectivity dataset rather than on chemical data. Consequently, the model is adept at learning the relationships among various connections rather than specific chemical interactions.

In a word, experimental results substantiate that integrating different modalities of chemical data including SMILES, graphs, and natural corpus, presents an effective representation of reactions, which is effective for RCR scenarios.

#### 4.4.2 Modality Projection

By leveraging the strengths of multiple modalities, multimodal LLMs can achieve higher accuracy in a wide range of applications. However, aligning representations among different modalities remains a challenging task. In our proposed MM-RCR, we employ the Perceiver module [] to integrate molecular SMILES tokens and graphs tokens into text-related language space, where text tokens are augmented by the reaction corpus, as illustrated in Figure 2. This modality projection module maps the embeddings of reactions to a latent vector and enhances this representation using a Transformer tower. Consequently, learnable queries contain highlighted reaction contents that are most related to the text tokens. We compared three typical methods for modality projection, including Perceiver [], Reprogramming [], and MLP.

**Table 5: Performance evaluation of MM-RCR under different modality projections, the best performance are in bold.**
| Projection<br>Layer |  | Top-$k$ Accuracy (%) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Catalyst |  | Solvent 1 |  | Solvent 2 |  | Reagent 1 |  | Reagent 2 |  |  |  |  |  |  |  |  |  |  |  |
|  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  | 1 | 3 | 5 |  |
| MLP |  | 90.9 | 97.8 | 98.9 |  | 51.1 | 73.3 | 82.2. |  | 81.1 | 93.9 | 97.1 |  | 47,4 | 71.0 | 79.9 |  | 77.0 | 91.7 | 95.2 |
| Reprogramming |  | 92.1 | 98.3 | 99.1 |  | 52.8 | 75.1 | 83.7 |  | 81.3 | 94.3 | 97.4 |  | 50.2 | 73.5 | 81.9 |  | 77.7 | 92.5 | 95.7 |
| Perceiver |  | 92.7 | 98.6 | 99.2 |  | 54.6 | 76.4 | 84.8 |  | 81.8 | 94.8 | 97.6 |  | 53.4 | 75.8 | 83.9 |  | 78.7 | 93.2 | 96.2 |

As depicted in Table. 5, the Perceiver module achieves significant gains in the prediction of all categories. Compared with MM-RCR (with Reprogramming), MM-RCR (with Perceiver) can be further enhanced and attains peak performance in all predicted categories with 7.2% significant gain. Specifically, For the solvent 1 prediction, a hard case, the Perceiver module stands out with a top-1 accuracy of 54.6%, significantly surpassing MLP (51.1%) and Reprogramming (52.8%). Its ability to consistently achieve high accuracy in both top-1 and top-k evaluations suggests a robust and versatile approach for reaction condition recommendation.

### 4.5 Geralization Performance

In order to validate the out-of-domain performance of MM-RCR, we employ MM-RCR trained on the USPTO_500MT_Condition to test on the USPTO-Condition. The evaluation strategy includes three specific training conditions: reagents, catalysts, and solvents. We adopt a metric of partial matched accuracy to illustrate the generalization capability of MM-RCR. The idea is that if the predicted results match the substitutable part of the ground truth. The evaluation strategy includes three specific training conditions: reagents, catalysts, and solvents. Table. 6 reports the top-1 partial match accuracy for each condition prediction.

Figure: Table 6: The top-1 partial matched accuracy of MM-RCR under OOD setting.

For the reagent and solvent prediction, MM-RCR achieves a top-1 partial matched accuracy of 67.1% and 58.1%, respectively. This relatively high accuracy indicates that solvents and reagents have more consistent characteristics that the model can learn effectively from USPTO_500MT_Condition and apply to USPTO-Condition. In contrast, The model’s performance in predicting catalysts demonstrates a lower top-1 partial match accuracy at 89.9%.

In summary, our MM-RCR can successfully distinguish reagents from the combination of all conditions in a reaction, as it learns the relationships between reaction conditions effectively. Additionally, training MM-RCR on USPTO-Condition, a larger chemical reaction dataset, further enhances its ability to learn reaction representations. This enables MM-RCR to perform well even under significant disparities in the chemical space of the datasets, allowing it to capture crucial information effectively.

### 4.6 Zero-Shot Prediction on High-Throughput Experimentation Reaction

Discovering effective reaction conditions precisely for high-throughput reaction condition screening is very important, as it has the potential to release chemists from laborious and costly trial-and-error workflows.
Thus, we evaluate our proposed MM-RCR on the high-throughput reaction datasets, aiming to recommend conditions that yield high-product outputs. Recently, Pd-catalysed C–H direct functionalization has earned increasing interest in pharmaceutical development for its ability to generate molecule complexity without the need for pre-functionalized starting material []. Thus, We select imidazole C–H arylation reaction for evaluation. Imidazole C–H arylation dataset is extracted from the work proposed by Shields et al. in 2021 [], where the substrate scope contains 8 imidazoles and 8 aryl bromides associated with conditions including ligands, bases, and solvents.

Figure: Figure 4: Boxplot of the performance for ligand recommendation on C-H arlyation reaction.
Refer to caption: x4.png

Catalysts are vital compounds in chemical reactions, as they play a crucial role in determining both reactivity and yield. The catalyst used in imidazole C–H arylation comprises a metal (Pd) and ligands. Thus, we evaluate the performance of ligand recommendations. First, we ensure that reaction data of imidazole C–H functionalization is excluded from the test set of the USPTO-Condition dataset to prevent data leakage issues. MM-RCR recommends a ligand under a pre-defined solvent-base combination of conditions. As shown in Figure. 4, we randomly select six cases for performance evaluation. The referenced bases, solvents, and ligands can be found in the reaction formula, which has been annotated by ‘B’,‘S’, ‘L’. For example, in Figure. 4(a), under the combination of CsOAc and DMAc, MM-RCR identifies the XPhos ligand, which results in a higher yield.

We also analyze the recommendation performance between diverse ligands for each base-solvent combination. We can observe that, for 15 of the 16 base-solvent combinations, the recommended ligand performs best in terms of the median value of reaction yields, suggesting that MM-RCR can recommend ligands with higher yields. Moreover, we can conclude that the capability of MM-RCR to recommend suitable conditions for chemical reactions has the potential to accelerate high-throughput reaction condition screening in the future.

## 5 Conclusion and Limitations

Conclusions In this paper, we present a multimodal LLM, a.k.a. MM-RCR for chemical reaction condition recommendation.
Trained with 1.2 million pair-wised Q&A instruction datasets that integrate with multimodal reaction representations and corpus in natural language, MM-RCR effectively answers questions regarding reaction conditions through either a classification head or sequence generation. MM-RCR achieves competitive results with state-of-the-art models via experimental validation. Additionally, MM-RCR exhibits strong generalization abilities on OOD and HTE datasets.

Limitations Further, we will focus on optimizing data representation with full fine-tuning training strategies to improve its performance across various chemical reaction tasks in future work.

## Acknowledgement

This work was supported by the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and the Fundamental Research Funds for the Central Universities. The computations in this paper were run on the AI for Science Platform supported by the Artificial Intelligence Institute at Shanghai Jiao Tong University.

## Appendix

## Appendix A Training settings

To realize peak efficiency within our MM-RCR model, we carefully design the training phases. This section offers a comprehensive summary of the training settings and the hyperparameter values. Through the detailed orchestration of these parameters, we ensure that MM-RCR is capable of fully leveraging its capabilities in the application contexts.

- •
Optional Settings: There are alternatives for modification in the MM-RCR framework, such as the replacement of the Perceiver-based modality projection layer with other architectures like Reprogramming and MLP.
- •
Reaction Condition Recommendation task: Within the framework, the model takes the 32-layer LLaMA-2-7b as the LLM backbone. Besides, we utilize a pre-trained SMILES-to-text retriever proposed by Qian et al. [] and extract the most similar unpaired corpus as the reaction text. Meanwhile, we introduce Parrot, a Bert-like model to encode the reaction SMILES. We leverage R-GCN [] to encode the molecules in the reaction, and the combination of reactant and product embeddings is considered as the reaction representation. In the training process, the encoders in all modalities are frozen. After the alignment of the representation space, the SMILES- and the graph-based tokens have a length of 128 and 3, respectively. Additionally, the model employs the OneCycleLR as the learning rate schedular, initializing the learning rate as 3e-5. The batch size is set to 16, with less than 6 epochs 48 hours in training. The GPU configuration is 8 $\times$ 80G A800.

## Appendix B Data Description

We curate two large datasets, named as USPTO-Condition and USPTO_500MT_Condition, with the data volumes presented in Table. 7. Both datasets are split with the ratio of train:validation:test=8:1:1 in our work. For USPTO-Condition dataset, all molecules including reactants, products, and conditions are collected in canonical SMILES. Each reaction entry contains five condition labels, including one catalyst, two solvents, two reagents, and an additional “none” category is introduced to illustrate that the reaction does not require this type of reaction condition []. The visualization of data distribution is depicted in Figure. 5 (left). From Figure. 5 we can see that this dataset covers a vast variety of reaction types, characterized by a substantial proportion of heteroatom alkylation, arylation, and acylation reactions, while C-C formation reactions are less included. We also introduce the corpus of reaction descriptions proposed by Qian et al. [] into the USPTO-Condition dataset. Each reaction is associated with a corpus of reaction descriptions. It should be noted that the corpus will not be utilized directly for training. Instead, we employ the corpus as an input for the pre-trained retrieval module proposed by []. This approach allows us to obtain similar embeddings necessary for the multimodal representation learning of our MM-RCR, and avoid data leaking issues. For USPTO_500MT_Condition datasets, it collects Top-500 types of reactions from the USPTO-MIT datasets  [], in which the top-100 types of reactions make up 59% of the entire dataset, which can be seen in Figure. 5 (right). In order to calculate the predicted accuracy on the USPTO_500MT_Condition dataset, it is necessary to separate all reagents in an appropriate manner. However, separating reagents using the dot as a delimiter is challenging, as compounds like [Na+].[OH-] constitutes a single reagent and cannot be split.
Besides, to have a comprehensive knowledge of the datasets, we do sparsity analyses. We calculate the non-empty count and density of every condition in the USPTO-Condition dataset, which is presented in Table. 8. From the table, we can see that some conditions, such as ‘Catalyst’, ‘Solvent 2’, and ‘Reagent 2’ show a high extent of sparsity, with a non-empty density of fewer than 30%. For the USPTO_500MT_Condition, as it only covers the condition of non-split reagents, all of the reaction entries have their corresponding non-empty condition label.

Furthermore, we make an investigation on the condition categories in the USPTO-Condition and USPTO_500MT_Condition dataset, which is illustrated in Figure. 6. The visualization of the most common chemical contexts of the regents, catalysts, and solvents in USPTO-Condition, and separate reagents in USPTO_500MT_Condition is depicted in Figure. 6 (A-D), respectively. From the figures, we learn that reaction conditions have a property of diversity and imbalance. Besides, we count categories of every condition, as is presented in Figure. 6 (E). Reagents in both datasets consist of more than 200 categories, which highlights the difficulty of the reaction condition recommendation task. Additionally, we prove that reagents in the USPTO_500MT_Condition dataset follow the power-law distribution, which indicates the condition keeps the long-tail feature in distribution and a small number of categories account for the majority of the data size.

**Table 7: Data volume of USPTO-Condition and USPTO_500MT_Condition datasets.**
| Dataset | Training set | Validation set | Testing set |
| --- | --- | --- | --- |
| USPTO-Condition | 546728 | 68341 | 68341 |
| USPTO_500MT_Condition | 88410 | 9778 | 10828 |

Figure: Figure 5: Left: The reaction distribution of USPTO-Condition. Right: The reaction distribution of USPTO_500MT_Condition.
Refer to caption: x5.png

**Table 8: Sparsity analysis of the USPTO-Condition dataset.**
| USPTO-Condition | Catalyst | Solvent 1 | Solvent 2 | Reagent 1 | Reagent 2 |
| --- | --- | --- | --- | --- | --- |
| Non-empty count | 89,756 | 673,634 | 130,326 | 504,169 | 170,752 |
| Non-empty density | 13% | 99% | 19% | 74% | 25% |

Figure: Figure 6: Distribution of types of reactions in the USPTO-Condition and USPTO_500MT_Condition. (A-D) The bar charts of the fifteen most common reagents, catalysts, and solvents in the USPTO-Condition and reagents in the USPTO_500MT_Condition, respectively, where the shallow color presents the decimal-scale proportion and the deep color presents the log-scale count. (E) The bar charts of the total category count of the conditions illustrated in (A-D). (F) Power law fitting of the reagent distribution in the USPTO_500MT_Condition, where the shallow points show the probability density and the deep dashed-line shows the ideal power-law fitting, respectively.
Refer to caption: x6.png

## Appendix C Details of Modality Alignment

For the reaction condition recommendation task, the representation of the reaction is extracted by encoders (see in section  3.2.2), and the text representation is tokenized by LLMs. However, fusing two types of representation introduces inductive biases issues [, ]. To effectively fuse representations from multiple modalities, we propose the use of a projection module, the Perceiver [], for modality alignment (Figure 2). This module employs latent queries to align graph and SMILES tokens with text-related tokens, such as question prompts and a text-augmented corpus. We show the pseudo-code for modality projection in Algorithm. 1.

Figure: Algorithm 1 Pseudo code for modality projection.

## Appendix D Model performance

A chemical reaction can be represented as the transformation of a sequence of characters (reactants, conditions) into another sequence (products), with compounds connected by special characters, such as ‘>>’. This structure makes sequence-to-sequence models, such as the Transformer, well-suited for predictive modeling of reaction representation [, ]. However, existing SMILES-based Transformer models for reaction representation encounter limitations in various aspects, particularly with respect to atom permutations and the interpretability of reaction mechanisms.
Consequently, our proposed MM-RCR fuses data from diverse sources including corpus, SMILES and graphs of molecules to present a comprehensive view of the reaction.
We assess the performance of our proposed MM-RCR and the aforementioned baseline methods for reaction condition recommendation. The top-$N$ accuracy of condition recommendation on the combined test datasets of USPTO-Condition and USPTO_500MT_Condition are presented in Table. 2 and Table. 3, respectively. We introduce several comparative methods to illustrate the performance of MM-RCR.

- 1.
RCR []. This method proposes a reaction fingerprint to represent the difference between the product and reactant fingerprints.
- 2.
Reaction GCNN []. This method proposes a machine-learned ranking model to predict the set of conditions used in a reaction as a binary vector.
- 3.
Parrot []. This method leverages the attention-based model architecture to encode the reaction and design a training methodology specifically to enhance the reaction center.
- 4.
TextReact []. It aims to enhance the molecular representation of the reaction by introducing relevant corpus retrieved from literature into sequence-to-sequence Transformers.
- 5.
Reagent Transformer []. This method leverages Molecular Transformer, [] a state-of-the-art model to tackle the task of reagent prediction.

To have a comprehensive overview of the recommendation performance, we visualize the prediction results of USPTO-Condition and USPTO_500MT_Condition datasets, as described in Table. 2,  3. Specifically, we draw radar charts of our model and other competitive models, which are presented in Figure. 7. For the USPTO-Condition dataset, we reproduce Parrot, RCR, and TextReact. Then, we plot the top-3 predicting accuracy of different conditions (catalyst, solvent 1, solvent 2, reagent 1, and reagent 2), as is depicted in Figure. 7 (left). For the USPTO_500MT_Condition dataset, we recommend reagents in SMILES sequence and take Parrot, Reagent Transformer, and Reaction GCNN as comparative methods. For more intuition, we visualize top-1, 3, 5, and 10 exactly matched accuracy in log scale, which is shown in Figure. 7 (right). From the charts, we can see that our model covers the largest area of the performance circle in both datasets, indicating that MM-RCR markedly outperforms other competitive models.

Figure: Figure 7: Left: Radar plot of top-3 predition accuracy of conditions on the USPTO-Condition dataset. The classification performance consists of comparative methods such as Parrot, RCR, TextReact, and our methods with similar corpus. Right: Radar chart of log-scale accuracy of reagents in the USPTO_500MT_Condition dataset.
Refer to caption: x7.png

### D.1 Ablation study on modality

Besides, we visualize the results of ablation study on modality on the USPTO-Condition dataset, which can be seen in Table. 4. Specifically, we categorize the conditions of the USPTO-Condition into two types: more complex and less complex. According to the data sparsity, reagent 1 and solvent 1 are considered more complex, while catalyst, reagent 2, and solvent 2 are considered less complex. Then, the investigation on the effectiveness of modalities comprising similar corpus, SMILES, graph is depicted in Figure. 8. From the results, we can see that compared with the model with multiple modalities, the model with single one modality degrades dramatically. Moreover, MM-RCR with three modalities combined achieves the best performance, which demonstrates the vital importance of capturing the reaction representations from different dimensions.

Figure: Figure 8: Bar charts demonstrating the ablation study of modalities including similar corpus, SMILES and graph. The classification performance is assessed on the conditions in the USPTO-Condition dataset, which are split into two groups according to data sparsity.
Refer to caption: x8.png

### D.2 Case Study

In this section, we select four cross-coupling reactions from USPO-Condition datasets for performance validation. We visualize the predicted results in Figure. 10. As depicted in Figure 10, the reaction centers and leaving groups are highlighted in different colors. For C–N cross-coupling reactions (the first and the third row), MM-RCR can predict all conditions precisely. For C–C bond formation and Formylation reactions (the second and the fourth row), MM-RCR fails to predict Ethyl Acetate (the second case) and THF (the fourth case). The reason why MM-RCR is less effective for these reactions is that the data volume of C–C bond formation reactions in the USPTO-Condition dataset is only 5%, as shown in Figure 5. This limited representation constrains the model’s ability to learn the patterns associated with C–C bond formation reactions. Consequently, MM-RCR lacks sufficient training examples to capture and generalize the underlying reaction mechanisms accurately. The scarcity of diverse and representative data hampers its effectiveness, leading to a lower precision in predicting these types of reactions.

Figure: Figure 9: Visualization of recommended conditions on four reactions. We select four Suzuki–Miyaura cross-coupling reactions to present the performance of condition recommendation. The reaction centers and leaving groups are highlighted in different colors.
Refer to caption: x9.png

Further, we visualize the predicted results on OOD datasets in Figure. 10. We select two reaction cases for analysis. In case 1, Toluene is not predicted by MM-RCR. In case 2, 1,4-Dioxane and 1-(diphenylphosphaneyl)cyclopenta-2,4-dien-1-ide are predicted. However, it is confirmed that Toluene and 1,4-Dioxane are common solvents, and 1-(diphenylphosphaneyl)cyclopenta-2,4-dien-1-ide is frequently used as a ligand. Therefore, we do not categorize these as failed cases because the model successfully predicts all the reagents in the labels and avoids predicting other conditions.

Figure: Figure 10: Visualization of recommended conditions on two reactions. In case 1, Toluene was not predicted by MM-RCR. In case 2, 1,4-Dioxane and 1-(diphenylphosphaneyl)cyclopenta-2,4-dien-1-ide were predicted. However, it is confirmed that Toluene and 1,4-Dioxane are common solvents, and 1-(diphenylphosphaneyl)cyclopenta-2,4-dien-1-ide is frequently used as a ligand. Therefore, we do not categorize these as failed cases because the model successfully predicts all the reagents in the labels and avoids predicting other conditions.
Refer to caption: x10.png

Figure: Figure 11: Boxplot of the performance for ligand recommendation (1).
Refer to caption: x11.png

Figure: Figure 12: Boxplot of the performance for ligand recommendation (2).
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: In Section 4.4, the ablation study highlights performance variations across different data types (SMILES, Graph, Corpus). However, it would be helpful to specify how the model utilizes each modality to enhance predictions.
In Section 4.6, the term "top-1 partial match accuracy" in the results table requires clarification. The distinction between partial and complete matches, especially regarding reagent and solvent predictions, is not clearly defined.
soundness: 2
strengths: Chemma-RC captures the complexity of chemical reactions more comprehensively by combining multiple modalities, including SMILES, graph structures, and textual corpora. This integration may enhance the model's understanding and predictive capabilities regarding reaction conditions. Chemma-RC is trained on a dataset of 1.2 million question-and-answer pairs, significantly enhancing the model's learning efficacy and accuracy, especially in high-throughput experiments. By incorporating text-augmented instruction prompts, Chemma-RC effectively guides the model in learning and generating reaction conditions.
summary: The paper introduces Chemma-RC, a multimodal large language model designed for recommending chemical reaction conditions. It integrates SMILES, reaction graphs, and textual descriptions to improve prediction accuracy, supported by a dataset of 1.2 million question-and-answer pairs. Experimental evaluations on benchmark datasets demonstrate its effectiveness in various contexts. The Perceiver module enhances alignment between data modalities, contributing to the model's strong performance.
weaknesses: Integrating multiple modalities enhances the model's abilities but also complicates it, potentially leading to longer training durations and increased debugging challenges.
The model depends on extensive, high-quality multimodal data for training, which may be difficult to obtain, particularly in data-scarce environments.
The complexity of the model can hinder its interpretability, making it challenging to trace the rationale behind specific predictions, a crucial aspect in chemistry.
The model's high complexity may result in overfitting, especially when training datasets are small or noisy.
Although effective with existing multimodal data, the model might struggle to adapt to new data types or domains without adequate training examples.
The model’s success is closely tied to the quality of its training data; poor or biased data can lead to inaccurate reaction condition predictions.
Developing and training such a sophisticated model may require significant computational power and time, which could be a challenge for some research groups.
The model's performance may differ across various chemical conditions or experimental contexts, restricting its overall transferability.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What are the results of the reaction condition recommendation (RCR) when using the USPTO dataset employed by the TextReact model, and how do the results differ when the dataset is partitioned according to RS (random split) and TS (time split)?

2. In Abstract: The term "unified reaction representation" might need clarification. It would be helpful to specify how the SMILES, reaction graphs, and textual corpus are integrated into this unified representation.

3. In Introduction (Lines 38-39): The problem of "low sparsity of chemical data" is mentioned but could be better explained. It’s unclear if this refers to limited diversity in reaction data or the low availability of high-quality reaction data.

4. Figure 1 - Model architecture description: The diagram of Chemma-RC's architecture (Figure 1) is complex, and the explanation in the text doesn’t fully clarify how each component (e.g., modality projection and condition prediction) interacts within the model. A step-by-step walkthrough could enhance understanding.

5. In section 3.2.1 - Text-Augmented Instruction Prompts: The description of how prompts are constructed might benefit from more examples, especially regarding "question templates" and their variety. It is unclear how "2,000 question templates" differ in structure or content and how these templates influence model training.

6. In section 4.3 - Performance Comparison (Tables 2 and 3): The term "strict matching policy" for top-k accuracy might need clarification. It’s unclear what criteria are used for strict matching—whether it’s an exact match of predicted and true labels or some other criteria.

7. Ablation Study (Section 4.4): While this section reports performance changes with various data types (SMILES, Graph, Corpus), it would be beneficial to clarify how the model interprets these individually. For example, what specific information from each modality improves the prediction?

8. In section 4.6 - Zero-Shot Prediction on High-Throughput Experimentation: The phrase "top-1 partial match accuracy" in the results table needs explanation. It is unclear what is considered a partial match versus a complete match, particularly for reagent and solvent prediction.

9. In Conclusion and Limitations: The paper mentions plans to "optimize data representation with full fine-tuning training strategies" but doesn’t clarify why full fine-tuning was not initially used. It would be helpful to understand the limitations of the current training strategy and how it impacts model performance.

10. In Appendix B - Data Description: In the dataset descriptions, some technical terms like "sparsity analysis" might be confusing. A brief explanation of "non-empty count" and "density" could aid in understanding the data distribution.
soundness: 2
strengths: 1. Large-scale training data: Chemma-RC utilizes a dataset comprising 1.2 million pairs of question-and-answer instructions during training. The substantial size of this dataset may contribute to improved learning efficacy and accuracy of the model, enhancing its performance on high-throughput experimental data.

2. Generalization capability: The paper indicates that Chemma-RC performs well on out-of-domain (OOD) and high-throughput experimentation (HTE) datasets, demonstrating its robustness and adaptability across different data distributions.
summary: The paper titled “Text-Augmented Multimodal Large Language Models for Chemical Reaction Condition Recommendation” introduces Chemma-RC, a novel multimodal large language model designed for chemical reaction condition recommendation (RCR). It leverages SMILES, reaction graphs, and textual corpus to improve the accuracy of RCR predictions and builds a 1.2 million pairwise Q&A dataset to train the model. Experimental results on USPTO benchmark datasets indicate that Chemma-RC achieves strong performance in both in-domain and out-of-domain settings, with a proposed Perceiver module for modality alignment.
weaknesses: 1. While the paper presents Chemma-RC as a novel multimodal model, the experimental results lack sufficient evidence that its core techniques—such as modality alignment and instruction tuning—offer a distinct advantage over simpler models like T-Rex and TextReact. To address this, I recommend that the authors include specific comparisons to highlight the unique contributions of Chemma-RC. For example, broadening the set of baseline models beyond T-Rex and TextReact could provide a more comprehensive view of Chemma-RC’s competitive edge.

2. While the manuscript addresses certain aspects of generalization (e.g., OOD and HTE results), it would benefit from additional evaluations to comprehensively demonstrate Chemma-RC’s versatility in chemical reaction modeling. For instance, models like TextReact illustrate their flexibility by performing multiple tasks, including reaction condition prediction and retrosynthesis. To parallel these capabilities, I suggest that the authors add experiments to assess Chemma-RC on diverse predictive tasks, such as reaction condition recommendation (RCR) using the USPTO dataset, as in TextReact. Additionally, it would be valuable to report how Chemma-RC performs under different dataset splits, such as random split (RS) and time split (TS), to further highlight its robustness across varied conditions.

3. The construction and training of Chemma-RC, as a complex multimodal model, may demand substantial computational resources and time, potentially creating challenges for research teams with limited access to high-performance computing. To make the model more accessible, I suggest the authors provide details on the computational requirements (e.g., training time, hardware specifications) and consider exploring ways to reduce these demands, such as model simplifications or alternative training strategies. This would offer clearer guidance for teams looking to replicate or adapt Chemma-RC within resource constraints.

4. While integrating multiple modalities indeed enhances the model's capabilities, it also increases its complexity, potentially leading to extended training times and additional debugging challenges. To assist others in implementing or building upon this work, I suggest that the authors discuss any specific challenges they encountered during the model’s training and debugging process due to its complexity. Insights into how these issues were addressed, including strategies or tools used to streamline debugging, would be valuable additions to the community and provide practical guidance for future work.

5. The model’s dependence on a large volume of high-quality multimodal data may limit its applicability in fields with limited data availability. I recommend that the authors discuss how Chemma-RC might perform with smaller datasets and explore potential strategies for adapting it to data-scarce domains. For example, they could consider approaches such as transfer learning, data augmentation, or semi-supervised learning to enhance the model's applicability where data is less abundant. Including these insights would make the model’s utility more broadly applicable to real-world scenarios.

6. Given the model's complexity, there is an inherent risk of overfitting, particularly in cases of limited or noisy training data. I suggest that the authors provide details on any regularization techniques or strategies they employed to mitigate overfitting. Additionally, discussing how they validated the model’s performance on unseen data—such as through cross-validation or using an independent test set—would help clarify the model’s robustness and reliability. These insights would be valuable for understanding the model's generalization capabilities in varied data environments.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: * What is the specific purpose of introducing the new dataset? Aside from integrating available data through templating, what novel contributions does this dataset offer? According to the paper, templates were generated using the GPT-4 model. How was the correctness of these generated templates verified?
* The purpose of using reaction graph embeddings remains unclear based on the current ablation study. Integrating graph embeddings requires additional effort including language model’s source code modification, yet Table 4 shows only a slight improvement in model performance with their usage. It appears that relying solely on textual corpora and SMILES representations might be sufficient. It may be beneficial to include an additional ablation study demonstrating cases where graph embeddings significantly enhance performance.
* One of the main contributions of the paper is the integration of multiple data representations and sources within a unified paradigm, positioning the proposed model closer to multi-task, multi-domain foundational models rather than single-task models, mainly presented in the Experimental section. To strengthen the significance of the results, it would be beneficial to include comparisons with current state-of-the-art chemical language models that address reaction planning tasks in their scopes, such as [a, b].

a. BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning, 2024

b. nach0: multimodal natural and chemical languages foundation model, 2024
soundness: 3
strengths: The application of language models to chemical reaction planning represents a highly valuable direction for research. While the incorporation of additional data modalities into language models is not entirely novel, it is still an important approach that has the potential to enhance current methodologies by utilizing a broader range of available data. Moreover, the demonstrated ability of the Chemma-RC model to handle out-of-distribution data adds further significance to this work.  Experimental results and ablation study sections are comprehensive.
summary: The authors present a novel Chemma-RC model designed for chemical reaction recommendation and prediction tasks. This model is trained on multiple data representations, including SMILES notations, reaction graphs, and chemical reaction recommendation corpora. Additionally, the authors introduce an instruction dataset comprising data from various sources, which has been augmented with textual templates. They further demonstrate that the Chemma-RC model can be easily fine-tuned on out-of-distribution data, enabling it to generalize to new types of chemical reactions.
weaknesses: While the paper is generally well-written, certain sections lack clarity and sufficient justification for specific decisions made in the study. Additional details on these issues can be found in the Question section.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. In line 66: How can one determine that the LLMs are pre-trained on extensive reaction data?

2. Line 80: The term "Perceiver" appears without context. Please add a citation and provide an introduction.

3. In line 84/4.5: What distinguishes the USPTO 500MT Condition dataset from the USPTO-Condition dataset, and what causes their distributional differences? Are there any statistical results indicating that these tasks are out-of-distribution?

4. Figure 1 needs improvement. The boxes for data and model are indistinguishable, or there are no boxes, making it difficult for readers to determine where to begin.

5. Tables should be self-explanatory. In Table 1, what does the condition label mean?

6. In lines 300-303: Is there a specific reason why the authors freeze the parameters of all models and only fine-tune the projectors?
soundness: 2
strengths: **Originality**: The paper focuses on an interesting problem with a reasonable solution. This is a useful and new case demonstrating the effectiveness of multimodal LLMs in drug discovery.

**Quality**: The proposed method seems promising.

**Clarity**: The paper includes figures to aid readability, with a well-illustrated method. The experimental setups and baselines are properly introduced.

**Significance**: The paper shows promising contributions to reaction condition recommendation.
summary: The paper focuses on chemical reaction condition recommendation using LLMs. It proposes incorporating different data modalities, such as reaction formulas, graph-structured molecules, and reaction descriptions, into a single multimodal large language model (LLM) called Chemma-RC, fine-tuned through the Llama 2 model. The model is compared to previous baselines and shows promise in generating useful reaction conditions to accelerate high-throughput condition screening in chemical synthesis. Overall, the proposed method is interesting.
weaknesses: 1. Reaction condition recommendation is not a novel task in the era of LLMs. The paper needs a more thorough discussion and demonstration of why research focus should shift to LLM-based reaction condition recommendation. Do LLMs truly provide higher accuracy? Or do LLMs offer more interpretability through text generation? Has any work benchmarked the performance of LLMs in reaction condition recommendation?

2. Why are there two types of condition prediction modules mentioned in line 76? There is no motivation provided for them. LLMs are more naturally suited for generative tasks rather than prediction; why is prediction necessary here?

3. The ablation studies should be improved. In Table 4, consider adding a study that enables SMILES and graphs but disables the corpus (i.e., disabling the LLM part of the model).

4. There is a lack of detailed description regarding the process of constructing the question-answering dataset. Do the authors ensure that there is no data leakage? How was the data collected and deduplicated?

Meta-review: The work introduces a multimodal LLM-based model for predicting chemical reaction conditions and yield.

Two reviewers voted for accepting the work, while two other voted for rejection. Among the strengths, reviewers (WdLV and 3FKb) noted the extensive instruction tuning dataset, very encouraging (as noted by GYuz, though it is important to note it was demonstrated on one dataset) OOD results, and contributing to the important field of using LLMs in the context of chemical reactivity (GYuz).

Among weaknesses, reviewers noted overall complexity of the solution (WdLV, 3FKb, GYuz), and asked for additional baselines (3FKb). During the rebuttal phase, Authors have added requested baselines such as nach0, with the proposed models achieving significant gains.

Another significant issue, though not raised specifically by any reviewer, is novelty. The paper is technically advanced and nontrivial to execute, but conceptually rather straightforward application of a large language model with fused (using a Perceiver) modality information. As such it is not the best target for ICLR, and is better suited for a more specialized and application oriented venue. Relatedly, the paper shows small improvements in Table 2 while large improvements in Table 5 (AC is not sure what is the difference between Table 2 and Table 5 evaluation setting) compared to a closely related TextReact, and is not compared to TextReact on USPTO 500MT Condition. From these results it is not yet possible to fully understand the improvement on top of TextReact. This issue is further excarberated by the fact that (to the best knowledge of AC) the ablations are not done by removing modality at train time but merely at test time.

To summarize, despite the strengths of the paper, its limited novelty and complexity rank the paper slightly below the bar for acceptance. I hope the comments will be helpful in improving your work and thank you for submitting to ICLR.

Decision: Reject