ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining (2024)

Abstract: Molecule-text modeling, which aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge, is an emerging research direction. Beyond single molecules, studying reaction-text modeling holds promise for helping the synthesis of new materials and drugs. However, previous works mostly neglect reaction-text modeling: they primarily focus on modeling individual molecule-text pairs or learning chemical reactions without texts in context. Additionally, one key task of reaction-text modeling – experimental procedure prediction – is less explored due to the absence of an open-source dataset. The task is to predict step-by-step actions of conducting chemical experiments and is crucial to automating chemical synthesis. To resolve the challenges above, we propose a new pretraining method, ReactXT, for reaction-text modeling, and a new dataset, OpenExp, for experimental procedure prediction. Specifically, ReactXT features three types of input contexts to incrementally pretrain LMs. Each of the three input contexts corresponds to a pretraining task to improve the text-based understanding of either reactions or single molecules. ReactXT demonstrates consistent improvements in experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. Our code is available at https: //github.com/syr-cn/ReactXT.

heading: 1 Introduction
text: Multi-modal large language models (LMs) have recently attracted extensive research attention. Remarkably, in the vision-language domain, LMs enhanced with visual encoders show impressive
*Equal contribution. †Corresponding author. Xiang Wang is also affiliated with Institute of Dataspace, Hefei Comprehensive National Science Center.
results in visual question-answering and image captioning (Liu et al., 2023a; Li et al., 2023). Inspired by their successes, molecule-text modeling (MTM) becomes an emerging research field (Liu et al., 2023b; Zeng et al., 2022; Su et al., 2022), aiming to build the natural language interface for molecular tasks, including text-guided molecule generation, molecule captioning, and molecule-text retrieval (Edwards et al., 2022; Liu et al., 2022).
Building upon these MTM works, we study reaction-text modeling (RTM), aiming to improve LMs’ performance on reaction-relevant tasks. Chemical reactions, involving the transformation of reactants into products, are fundamental to advancing drug discovery and material science (Schwaller et al., 2022). Revisiting prior works, we identify key research gaps in both the learning paradigm and the evaluation benchmark for RTM:
• Learning Paradigm. Most prior works either focus on generating the textual description of a single molecule (cf. Figure 1a) (Liu et al., 2023b; Edwards et al., 2022; Su et al., 2022), or apply LMs for chemical reaction prediction without including the textual descriptions of molecules/reactions in the context (cf. Figure 1b) (Christofidellis et al., 2023; Fang et al., 2023; Born and Manica, 2023). Such methods overlook the potential knowledge in textual descriptions to improve performance. Pioneer works (Shi et al., 2023; Guo et al., 2023) include labels of molecular roles and experimental conditions when prompting ChatGPT, but achieve suboptimal performances for being limited to prompt engineering.
• Evaluation Benchmark. An open-source dataset for experimental procedure prediction is notably missing. As illustrated in Figure 2, experimental procedure prediction aims to deduce the
5353
step-by-step actions for experimental execution through interpreting chemical reactions (Vaucher et al., 2021), which has a significant value for automating chemical synthesis processes (Vaucher et al., 2020; Zeng et al., 2023). This task aligns well with our focus on RTM, requiring an understanding of chemical reactions and a textual interface to articulate experimental steps. Unfortunately, the absence of public datasets hinders further research and development in this area.
Addressing the identified research gaps, we propose Reaction-Contextualized Molecule-Text Pretraining (ReactXT), aiming to improve the text-based understanding of chemical reactions and molecules. Further, we construct an opensource dataset for experimental procedure prediction (OpenExp), serving as a key benchmark to evaluate RTM methods. Below, we elaborate on their details.
ReactXT aims to improve the learning paradigm of RTM by introducing three types of input contexts, each of which corresponds to a pretraining task to improve LMs’ understanding of chemical
reactions or individual molecules. As Figure 1d depicts, the forward reaction context is crafted to learn the chemical connections among molecules involved in the same reaction. These connections are grounded on chemical reaction principles, such as the conservation laws (Atkins and Jones, 2007). Building on this molecular interplay, we hypothesize that understanding other molecules in the same reaction and their descriptions can help predict the current molecule and its textual description. ReactXT encourages LMs to harness these intermolecule relationships to improve their ability to generate molecular descriptions in reactions and, in turn, deepen their understanding of chemical reaction principles. Further, a backward reaction context is introduced to support retrosynthesis tasks (cf. Section 3.1). Finally, as Figure 1c illustrates, ReactXT includes the random molecule context, cultivating the LMs’ understanding of individual molecules outside their reactions.
OpenExp features 274, 439 pairs of chemical reactions and their corresponding step-by-step instructions of experimental procedures. This dataset, compiled from the USPTO-Applications (Lowe,
2017) and ORD (Kearnes et al., 2021) databases, will be released under the CC-BY-SA license. To ensure data quality, we have conducted careful data preprocessing. Further, we invite human experts to evaluate the dataset quality. Out of 100 randomly chosen samples, 50 samples could be directly used without any human intervention, and 90 samples required only minor modifications for experimental execution (cf. Figure 5).
Our contributions can be summarized as follows:
• We propose ReactXT, a method that incorporates three types of input contexts to incrementally pretrain an LM. These contexts are tailored to enhance LMs’ understanding of chemical reactions and individual molecules.
• We curate an open-source experimental procedure prediction dataset OpenExp, a new benchmark for automating chemical synthesis research.
• ReactXT achieves state-of-the-art performances for experimental procedure prediction on the OpenExp dataset, highlighting its superior RTM ability. It also outperforms baselines by 3.2% for molecule captioning on the PubChem324k dataset. ReactXT has competitive performances for retrosynthesis, and we are refining it to surpass the current state-of-the-art method.

heading: 2 Related Works
text: Molecule-Text Modeling (MTM). MTM aims to jointly model molecules and texts to address text-related molecular tasks (Edwards et al., 2022, 2021). Molecules can be represented by 1D sequences of SMILES (Weininger, 1988) and SELFIES (Krenn et al., 2020), making it feasible to pretrain unified LMs on mixed 1D sequences of texts and molecules (Taylor et al., 2022; Edwards et al., 2022; Chithrananda et al., 2020; Zeng et al., 2022). Further, these LMs can be aligned to human preference via instruction tuning (Christofidellis et al., 2023; Fang et al., 2023). In parallel to 1D LMs, multi-modal methods are also studied, using graph neural networks (GNNs) (Hu et al., 2020; Liu et al., 2023c) to encode 2D molecular graphs. Notably, CLIP-style (Radford et al., 2021) crossmodal contrastive learning and BLIP2-style (Li et al., 2023) cross-modal projector are both investigated to facilitate molecule-text retrieval (Su et al., 2022; Liu et al., 2022), and molecule-to-text generation (Liu et al., 2023b; Li et al., 2024), respectively.
Recently, MolTC (Fang et al., 2024) is also proposed to model molecular interactions using chain of thoughts. However, prior works mainly focus on individual molecules rather than chemical reactions. To bridge the gap, ReactXT explores reaction-text modeling, facilitating reaction-relevant tasks with a text interface and textual knowledge.
Experimental Procedure Prediction. Synthesizing complex compounds requires detailed planning of synthetic pathways and intermediate steps, a process that is both labor-intensive and complex. Machine learning (ML) can potentially automate the process by predicting experimental procedures. Prior works have explored predicting reaction conditions (e.g., catalyst and solvent) (Gao et al., 2018) and sequences of synthesis steps (Vaucher et al., 2021) by reading chemical reactions. Given known experimental procedures, ML is also explored to empower chemical lab robots (Burger et al., 2020), and automated lab pipelines (Coley et al., 2019; Nicolaou et al., 2020). Notably, tool-augmented GPT4 (OpenAI, 2023) is explored to plan and execute known chemical experiments (Boiko et al., 2023). Unlike prior works, our OpenExp dataset is the first open-source dataset to facilitate the procedure prediction of unseen chemical experiments.
Retrosynthesis and Chemical Reaction Prediction. Given a chemical reaction, retrosynthesis is to predict reactants from products and reaction prediction is to predict products from reactants (Schwaller et al., 2022). They can be formalized as sequence-to-sequence translation represented by SMILES strings (Liu et al., 2017; Irwin et al., 2022; Zhong et al., 2022; Tetko et al., 2020; Ucak et al., 2022). Concurrently, 2D molecular graphs are explored for reaction prediction: selection-based methods focus on classifying the most suitable reaction templates (Chen and Jung, 2021; Dai et al., 2019); and graph-based generative models directly synthesize target molecules (Shi et al., 2020; Sacha et al., 2021; Yan et al., 2020). However, the methods above leverage only reactions without texts. While notably two pioneer works apply ChatGPT for reaction prediction (Shi et al., 2023; Bran et al., 2023), their performances are limited to exploring only prompt engineering.

heading: 3 ReactXT: Reaction-Contextualized Molecule-Text Pretraining
text: ReactXT consists of two key components: 1) the method of creating input contexts to incrementally
pretrain an LM, and 2) a balanced sampling strategy for the reaction contexts. We begin by introducing our multi-modal LM backbone, then proceed to elaborate on ReactXT’s two components.
Multi-Modal Language Model Backbone. Molecules can be represented by their 1D SMILES or 2D molecular graphs (Wells, 2012). We employ MolCA (Liu et al., 2023b) as our primary LM backbone to effectively harness both the 1D and 2D molecular modalities. Specifically, MolCA incorporates a GNN encoder (You et al., 2020) for encoding 2D molecular graphs. This GNN’s output then is mapped to an LM’s (i.e., Galactica; Taylor et al. (2022)) input space via a cross-modal projector, thereby enabling the LM to perceive 2D molecular graphs. Both the cross-modal projector and the GNN have been pretrained for molecule-text alignment (Li et al., 2023). MolCA shows promising performances when finetuned for molecule captioning and IUPAC name prediction.

heading: 3.1 Creating Input Contexts
text: Addressing the core challenges of LMs hinges on the careful selection of the input data. As shown in
Table 1, ReactXT incorporates three types of input contexts to incrementally pretrain LMs: forward reaction context, backward reaction context, and random molecule context. These contexts are tailored for a text-based understanding of chemical reactions and individual molecules:
• Forward Reaction Context. As Figure 3 illustrates, the forward reaction context labels molecules according to their roles – Reactant, Catalyst, Solvent, and Product – in the reaction, and arranges them in this specific sequential order. Note, not every reaction has a Catalyst or Solvent. For each molecule, we append its 2D molecular graph embeddings (e.g., <Mol1>; Liu et al. (2023b)) after its SMILES to enhance the LM’s understanding of molecular structures; and append molecular descriptions (e.g., $DESC1) following the 2D molecular graph embeddings to align molecules with texts.
• Backward Reaction Context. Similar to the forward context but with the order of molecular roles reversed, this context aims to combat the Reversal Curse (Berglund et al., 2023) of LMs:
LMs trained on “A is B” fail to generalize to “B is A”. The reversal generalization is crucial because downstream applications include backward retrosynthesis (Schwaller et al., 2022).
• Random Molecule Context. Introduced to ensure LMs retain the capability to describe individual molecules outside chemical reactions.
Context Length. In each input context, we use up to k molecules and their descriptions, where k is a hyperparameter. For reactions with over k molecules, we apply weighted molecule sampling, as explained in Section 3.2.
Molecule Descriptions. One crucial component of the input contexts is the molecule description, whose quality and comprehensiveness are vital for molecule-text alignment. We collect molecular descriptions and properties from multiple sources, encompassing three types of content:
• Molecule Patent Abstracts. We source patent abstracts from PubChem’s Patent View*. These abstracts typically describe molecular structures, properties, or applications, but may also include irrelevant information if the molecule is merely mentioned in passing rather than being the central subject. Despite the noise, patent abstracts are indispensable for RTM: they cover ∼95% molecules in our employed reaction databases (Lowe, 2017; Kearnes et al., 2021). In contrast, the molecule-text datasets (Liu et al., 2022, 2023b) derived from PubChem’s description section only cover ∼1% of these molecules.
• Computed and Experimental Properties. We retrieve these numerical properties from PubChem, aiming to enhance the understanding of molecular structures through predictive learning. Certain properties are also helpful for reaction prediction. For example, knowing the solubility helps determine concentrations when preparing solutions; the knowledge of melting and boiling points helps identify the states of matter at given temperatures. Table 2 shows an example of a patent abstract and computed/experimental properties. Table 14 includes detailed statistics of our collected molecule properties.
• PubChem Descriptions. Following (Liu et al., 2022, 2023b), we employ molecular descriptions from PubChem. Due to their limited
*https://pubchem.ncbi.nlm.nih.gov/docs/ patents
coverage (∼1%) for molecules in reaction databases (Lowe, 2017; Kearnes et al., 2021), we incorporate them exclusively for the random molecule context.
Autoregressive Language Modeling for Interleaved Molecule-Text Sequences. Given the input contexts above of interleaved molecules and texts, we apply language modeling loss to incrementally pretrain the LM, molecule encoder, and projector. We compute loss only for text tokens, excluding 2D molecular graph embeddings.

heading: 3.2 Balanced Sampling of Reaction Contexts
text: Figure 4 reveals a skewed distribution of molecules in chemical reactions (the red bars), with a small group of molecules appearing far more frequently than others. To address this imbalance, we develop a sampling strategy that promotes a fairer representation of molecules across reactions. This method reduces the dominance of commonly occurring molecules by adjusting 1) the sampling weight of each reaction r: W (r), and 2) the sampling weight of each molecule m within a chosen reaction r: W (m|r), based on the equations below:
W (r) = ∑ m∈r 1/Count(m)∑
r′∈R ∑ m∈r 1/Count(m) , (1)
W (m|r) = 1/Count(m)∑ m′∈r 1/Count(m′) , (2)
where R denotes the dataset of chemical reactions; Count(m) denotes molecule m’s count in R.
Equation (1) sets a reaction’s sampling weight inversely to the total occurrences of its molecules, favoring reactions with rare molecules; Equation (2) boosts the weights of rarer molecules within a
given reaction. These weights are then applied for weighted random sampling without replacement (Efraimidis and Spirakis, 2006). The blue bars in Figure 4 present the sampling frequency of molecules after adjustment, showing a flatter distribution. Implementation details are in Appendix B.

heading: 4 OpenExp: An Open-Source Dataset for Experimental Procedure Prediction
text: Here we briefly introduce OpenExp’s curation process and defer the details to Appendix A.1. OpenExp is sourced from chemical reaction databases of USPTO-Applications (Lowe, 2017) and ORD (Kearnes et al., 2021). As illustrated in Figure 2, these databases include chemical reactions and the corresponding unstructured descriptions of experimental procedures. To convert these unstructured descriptions into structured action sequences, we first run the pragraph2action model from (Christofidellis et al., 2023), and then conduct preprocessing following (Vaucher et al., 2021). The preprocessing is to remove low-quality data, eliminate duplicates, and construct molecule mapping between reactions and experimental procedures. Specific preprocessing steps are summarized in Table 3. An example is shown in Table 11.
As shown in Table 4, the final OpenExp dataset includes 274k reaction-procedure pairs. It is randomly divided into train/valid/test sets by the 8:1:1 ratio. Compared to the prior work (Vaucher et al., 2021), which is closed-source for using the com-
mercial Pistachio database†, we open-source this dataset to assist future research.
To obtain insights on dataset quality, we invite two graduate students in chemistry to rate the alignment between the action sequences and their original descriptions, on a scale from 1 (lowest) to 5 (highest), as depicted in Figure 5. Briefly, of the total 250 samples evaluated, 126 (≥ 50%) action sequences have at most 1 error (scores above 4), and 181 (≥ 50%) action sequences have at most 2 errors (scores above 3). Our closer inspection shows that the one error in score-4 samples is usually a typo of material/action name, or a discrepancy of numerical value, and does not impede the overall execution. See Appendix C.3.2 for details.

heading: 5 Experiment
text: We empirically evaluate ReactXT across three downstream tasks, including experimental procedural prediction, molecule captioning, and retrosynthesis. Further, we include ablation studies showcasing the contributions of individual components. To ensure the significance of our experimental, we include statistical tests results in Appendix C.2.

heading: 5.1 Experimental Setting
text: ReactXT is initialized by the stage-2 checkpoint of MolCA1.3B (Liu et al., 2023b), if not specially noted. It is then pretrained using our proposed method, and subsequently finetuned for each downstream dataset separately. The context length k is 4. We employ full-parameter tuning for pretraining and finetuning. More details are in Appendix B.
ReactXT’s Pretraining Dataset. Our pretrain dataset includes PubChem324k’s pretrain subset (Liu et al., 2023b), which includes 298k molecule-text pairs, and 1.11 million chemical reactions from the USPTO-Applications (Lowe, 2017) and ORD (Kearnes et al., 2021) databases. For molecules in reactions, we obtain their patent abstracts and molecular properties following Section 3.1. To prevent information leakage, we have
†https://www.nextmovesoftware.com/pistachio
excluded 54k reactions that appear in the valid/test sets of the downstream datasets (i.e., OpenExp, USPTO-50K (Schneider et al., 2016)) from the initial collection of 1.16 million reactions. See Appendix A.2 for more details.
Baselines. We compare ReactXT with the state-of-the-art LMs in science domain, including Galactica (Taylor et al., 2022), MolT5 (Edwards et al., 2022), TextChemT5 (Christofidellis et al., 2023), and MolCA (Liu et al., 2023b). For retrosynthesis and forward reaction prediction tasks, we also compare with task-specific LMs: RSMILES (Zhong et al., 2022), AT (Tetko et al., 2020), MEGAN (Sacha et al., 2021), and Chemformer (Irwin et al., 2022). For captioning, we additionally compare against MoMu (Su et al., 2022).

heading: 5.2 Experimental Procedure Prediction
text: Following (Vaucher et al., 2021), we employ the following evaluation metrics: Validity, which checks the syntactical correctness of the action sequence; machine-translation metrics BLUE (Papineni et al., 2002) and ROUGE (Lin, 2004); and the normalized Levenshtein similarity (Levenshtein et al., 1966). Specifically, 90%LEV denotes the proportion of predictions with a normalized Levenshtein score larger than 0.9. The three naive baselines based on random sampling and nearest neighbor are borrowed from (Vaucher et al., 2021). See Appendix B for details.
Table 5 presents the performances. We can observe that ReactXT consistently outperforms baselines across all metrics. Specifically, it surpasses baselines by 2.2% for BLEU-2 and 3.3% for 75%LEV, demonstrating ReactXT’s effectiveness for text-based reaction understanding.

heading: 5.3 Molecule Captioning
text: To evaluate ReactXT’s ability to understand single-molecules, we present its performances of molecule captioning on the PubChem324k (Liu et al., 2023b) and CheBI-20 (Edwards et al., 2022) datasets. We report metrics of BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005).
Table 6 presents the captioning performances. We can observe that ReactXT consistently outperforms the baselines. Specifically, ReactXT shows improvements of 3.2% BLEU-2 and 2.3% ROUGE2 scores on PubChem324k, and 1.7% ROUGE-2 on CheBI-20. These improvements underscore the
effectiveness of our pretraining method for enhancing understanding of individual molecules.

heading: 5.4 Retrosynthesis
text: Retrosynthesis is to predict the reactant molecules given the product molecules. For this task, we employ the evaluation metrics of top-k accuracy, which measures the percentage of exact match to the ground truth in the top-k predictions. Following (Zhong et al., 2022), we conduct selfsupervised pretraining on the USPTO-full(Dai et al., 2019) dataset and use the root-aligned augmentations of SMILES during training and testing. Additionally, we report performances of testing without these augmentations.
Table 7 presents the results. ReactXT outperforms R-SMILES across all metrics when testing with augmentations. Notably, the improvement in top-1 accuracy is particularly significant, achieving a 2.3% increase over the second best value. Regardless of whether test set data augmentation is applied, ReactXT achieves better top-k accuracies than MolT5-Large, which is also a multimodal LM. These performance improvements stem from ReactXT’s use of reactions for pretraining, rather than individual molecules.

heading: 5.5 Ablation Study
text: In this section, we conduct ablation studies to show the impact of different pretrain data types and backbone LMs in our method.
Pretrain Data Type. We ablate the key components of ReactXT, using the baseline of MolCA, Galac1.3B without incremental pretraining. Table 8
presents the results. Specifically, we compare three variants of ReactXT: 1) pretraining with solely the random molecule contexts using the same pretrain dataset; 2) pretraining with forward and backward reaction contexts without the random molecule context; and 3) applying uniform sampling on reaction contexts instead of balanced sampling.
We can observe that 1) ReactXT’s full model shows the best performance, showing its performance is the integrated contribution of all components; 2) applying random molecule contexts alone improves upon the baseline, underscoring the valuable textual knowledge from our meticulously crafted pretraining dataset; 3) incorporating reaction contexts yields better results than random molecule contexts, highlighting the benefits of learning reaction knowledge during pretraining; and 4) balanced sampling improves the performance upon uniform sampling.
Backbone LMs. We conduct ablation studies on the backbone LMs. This study involves three different molecular-text LMs: 1) MolCA, which represents molecules using both 1D SMILES and 2D graphs, based on a decoder-only architecture; 2) Galactica, which represents molecules using 1D SMILES, based on a decoder-only architecture; and 3) MolT5, which represents molecules using 1D SMILES, based on an encoder-decoder architecture. The experimental results are presented in Table 9. We can observe that the ReactXT pretraining scheme achieves consistent performance improvements, regardless of the backbone language model used.

heading: 6 Conclusion and Future Works
text: In this work, we explore reaction-text modeling to empower reaction-relevant tasks with textual interfaces and knowledge. We present ReactXT, a pretraining method to learn chemical reactions within the context of the corresponding molecular textual descriptions. Additionally, we propose a new dataset OpenExp to support open-source research for experimental procedure prediction. ReactXT establishes the best performances across tasks of experimental procedure prediction and molecule captioning. It presents competitive performances for retrosynthesis.
In future work, we plan to apply LMs to learn the interactions among large molecules (e.g., proteins and nucleic acids), or introduce molecules’ dynamics and 3D spatial structures for better molecule-
language understanding (Luo et al., 2023).

heading: Limitations
text: In this and also the previous work (Vaucher et al., 2021), the evaluation for experimental procedure prediction is constrained to the comparison between the predictions and the reference action sequences. While improving this metric does reflect the improvement in experimental design, it should be acknowledged that the evaluation of real-world chemical experiments is preferred for the developed models in future. For this purpose, the methods on automated chemistry pipelines (Boiko et al., 2023; Coley et al., 2019; Nicolaou et al., 2020) can be potentially considered.
Another limitation or future direction is improving the action space defined in our proposed OpenExp dataset, aiming to cover a wider range of chemical experiments. For example, the action of ‘Purify’ is absent; and the action of ‘Concentration’ can be refined into operations such as ‘Evaporation’ and ‘Pressurize’ for clearer instructions of chemical experiments.

heading: Potential Ethics Impact
text: In this study, the proposed method and dataset focus on chemical reactions and molecules, and include no human subjects. Consequently, we believe this study presents no direct ethical concerns. However, the inclusion of LMs in our study does raise potential issues, as LMs can be misused to produce incorrect or biased information. Therefore, the ethical implications of our work align with those common to LM research, emphasizing the need for responsible use and application of LMs.

heading: Acknowledgement
text: This research is supported by the National Science and Technology Major Project (2023ZD0121102), National Natural Science Foundation of China (92270114). This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-010-SGIL), the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207) and the Google Cloud Research Credits program with the award (Q4MJ-YH1K-3MVX-FP6Q). This research is supported by NExT Research Center.

heading: A Dataset Details

heading: A.1 Collection and Preprocessing of OpenExp
text: OpenExp is compiled from the raw data from the two following sources:
• USPTO-Applications (Lowe, 2017). This dataset comprises records of 1.94 million reactions and their corresponding applications from the United States Patent and Trademark Office (USPTO) published between 2001 and September 2016. We download the raw XML files from the Figshare website ‡. For each reaction in this dataset, we extract its key information from four elements: <productList>, which contains the products of the reaction; <reactantList>, detailing the reactants; <spectatorList>, encompassing the catalysts and solvents; and <dl:paragraphText>, which provides a textual description of the experimental procedures.
• Open Reaction Database (Kearnes et al., 2021). The ORD § dataset contains over 2 million chemical reactions, which include detailed records of reaction conditions and experimental procedures. It includes data from the USPTO applications (2001-2016 Sep), USPTO-granted patents (1976- 2016 Sep), and experimental records from chemical literature.
Paragraph2Action. As illustrated in Figure 2, these databases include chemical reactions and the corresponding unstructured descriptions of experimental procedures. The unstructured nature of these descriptions poses a significant challenge to 1) automate chemical synthesis with robots (Vaucher et al., 2020; Burger et al., 2020); and 2) apply ML methods to predict experimental procedures of unseen reactions. To address this, the task of paragraph2action (Vaucher et al., 2020; Zeng et al., 2023) is proposed, aiming to convert unstructured experimental procedure descriptions into structured, step-by-step instructions with pre-defined actions. In this study, we leverage the action space defined by (Vaucher et al., 2020, 2021), and the pragraph2action model released by (Christofidellis et al., 2023).
Preprocessing. Following (Vaucher et al., 2021), we conduct preprocessing after the paragraph2action conversion, The preprocessing has
‡https://figshare.com/articles/dataset/ Chemical_reactions_from_US_patents_1976-Sep2016_ /5104873?file=8664370
§https://open-reaction-database.org
two purposes: 1) extracting the important entities (i.e., molecules) in experimental procedures and mapping all molecules to their precursors in the chemical reaction; 2) applying a rule-based filtration to improve the dataset quality. Our preprocessing strategy is inspired by (Vaucher et al., 2020), augmented with additional 2 steps: perplexity filtering and similar action aggregation. The complete preprocessing steps are listed below:
• Perplexity Filtering. To ensure the quality of the above translation step, we compute a perplexity score for each output and exclude samples with a score larger than 1.0. These perplexity scores are calculated using the TextChemT5 model.
• Entity Recognition. We extract all the molecules (either by name or SMILES) from the action sequences using the source codes of (Vaucher et al., 2020). Then, we conduct string matching of IUPAC names between the extracted molecules and those in the chemical reactions. STOUT (Rajan et al., 2021) and PubChemPy¶ are used for the translation between IUPAC names and SMILES. If any molecule cannot be matched with its counterpart in the chemical reactions, we consider the reaction data invalid and remove it from the dataset. However, we permit the inclusion of certain common substances, such as common organic solvents, in every reaction. The names and SMILES expressions of the 134 common substances are included in our code. After entity recognition, we assign each entity a unique ID and update the experimental procedures by replacing the entity mentions with the corresponding entity IDs.
¶https://github.com/mcs07/PubChemPy
• Common Substance Renaming. We standardized the nomenclature for common substances that are known by multiple names (e.g., water may also be referred to as H2O, pure water, water (aq.), etc.) to improve the dataset’s precision. Using PubChemPy, we align the different names to their standardized SMILES representations, allowing us to identify when different terms refer to the same molecule by comparing their SMILES expressions.
• Similar Action Aggregation. If two adjacent operations are highly similar (e.g., STIR and STIR for 5 min), they are merged together.
• Ensuring Single Product. This dataset focuses on the preparation of a single material, hence we remove reactions that yield multiple products.
• Action Filtering. We remove action sequences that have fewer than five actions or contain invalid actions.
• Reaction Deduplication. We remove the duplicated reactions from the dataset.
Table 12 presents the number of samples removed at each preprocessing step. Further, Table 11 provides an example from the final OpenExp dataset, we can observe that it encompasses:
• Structured, step-by-step instructions of experimental procedures;
• All molecules in the reaction and their roles (i.e., reactant, solvent, catalyst, product).
• The mapping between the recognized entities (i.e., molecules) and their IDs.
• The original unstructured experimental procedures.
Discussion on License. The ORD database is accessible under the CC-BY-SA license, and the USPTO-Applications dataset is available under the CC0 license. We have used codes from TextChemT5 (Christofidellis et al., 2023) and Paragraph2Actions (Vaucher et al., 2021), which are
both licensed under the MIT license. Therefore, we will release OpenExp under the CC-BY-SA license to comply with the most restrictive license of these resources. This license permits content distribution and sharing, provided the same license is applied.
Human Evaluation. We invite two PhD students majoring in chemistry to evaluate the quality of the OpenExp dataset. Specifically, 250 data points are randomly sampled from the dataset, and assigned to the evaluators according to the following rules: 1) the first 50 data points are assigned to both volunteers simultaneously to verify the consistency of their evaluations; 2) the remaining 200 data points are then evenly assigned to the two evaluators. Under this allocation rule, each evaluator is responsible for 150 data points. Tthe evaluators are then asked to rate the quality of each data point on a scale from 1 (lowest) to 5 (highest). Our instructions to the evaluators are shown below:
Instructions to human evaluators.
We are curating a dataset partially generated by an AI model and want to seek feedback on its quality from human experts. During the evaluation process, we will provide both machine language sequences (the machine-generated operational sequences of experimental actions) and the corresponding natural language sequences (descriptions of experimental procedures in their original free texts). You should rate these samples based on how well the operational sequences align with the original descriptions. Please use a rating scale of 1 (low alignment) to 5 (high alignment). Molecular skeletal formulas are provided as images for reference during evaluation. All original data for this dataset come from the United States Patent and Trademark Office (USPTO), ensuring the viability of the reactions.
The following are the detailed scoring guidelines, with a maximum score of 5: • 5: The machine-generated action sequence in-
cludes no errors in capturing key details of the original experimental procedure, including actions, materials, and numerical values. • 4: The machine-generated action sequence includes at most one (nerr ≤ 1) error or omission related to actions, materials, or numerical values. • 3: The machine-generated action sequence includes at most two (nerr ≤ 2) errors or omissions related to actions, materials, or numerical values. • 2: The machine-generated action sequence includes at most four (nerr ≤ 4) errors or omissions related to actions, materials, or numerical values. • 1: The machine-generated action sequence includes more than four (nerr > 4) errors or omissions related to actions, materials, or numerical values.
Figure 5 presents the human evaluation results. Statistics of these 250 data points and the entire
dataset can be found in Figure 7 and Table 13. We can observe that the distribution of the sampled data points closely resembles that of the entire dataset, suggesting that the human evaluation results can reflect the overall quality of the OpenExp dataset.
Based on the 50 shared data points, we calculate the score differences in scores for the same samples (i.e., , the score given by evaluator 1 minus the score given by evaluator 2). The results are presented in Figure 6. We can observe the exact alignment in ratings for 40% of the samples (20 out of 50), and a marginal score difference (±1) is recorded for 54% of the samples (27 out of 50). Discrepancies of two or more scores are exceedingly rare, occurring in just 6% of the samples (3 out of 50). Some examples of human evaluated data points are in Appendix C.3.2.

heading: A.2 Collection and Preprocessing of ReactXT’s Pretraining Dataset
text: In Section 3, we collect and compile a dataset to incrementally pretrain an LM for improved understanding of chemical reactions and individual molecules. Here we elaborate on the details of this dataset, which includes the following contents:
• A total of 1,162,551 chemical reactions;
• Patent abstracts and computed/experimental properties of 1,254,157 molecules, which are all
from the chemical reactions.
We extract chemical reactions from ORD and USPTO datasets. Then, we source patent abstracts from PubChem’s Patent View|| and obtain molecular properties using the PubChem’s PubView API**. For each molecule, the abstract text derives from the abstracts of patent documents where the molecule is mentioned, and its properties include both computational and experimental ones. Table 14 shows a complete list of these properties.
In Table 15, we compare the statistics of our pretraining dataset with that of PubChem324k. We can observe that ReactXT’s pretraining dataset includes more molecules and additionally includes chemical reactions.
To prevent information leakage, we exclude a total of 54,403 reactions that appear in the validation and test sets of the downstream datasets (i.e., OpenExp and USPTO-50K (Schneider et al., 2016)) from the pretraining dataset. The remaining 1,108,148 reactions are used for pretraining.
Discussion on License. The ORD database is accessible under the CC-BY-SA license, and the USPTO-Applications dataset is available under the CC0 license. The patent abstracts from PubChem are provided by Google Patent††, which is released under the CC-BY-4.0 license. To comply with the strictest license terms, we will release our dataset under the CC-BY-SA license.
Additionally, we have utilized textual descriptions, computed properties, and experimental properties from the PubChem website for pretraining. Given that this data is aggregated from various sources by PubChem, determining a single appropriate license is challenging. To support future
||pubchem.ncbi.nlm.nih.gov/docs/patents **pubchem.ncbi.nlm.nih.gov/docs/pug-view ††patents.google.com
research while avoiding licensing complexities, we will provide the scripts for downloading and preprocessing this data, rather than distributing the data directly.

heading: B Experimental Details

heading: B.1 Hyperparameters
text: Here we detail the hyperparameters for ReactXT’s pretraining and finetuning across three downstream tasks. Due to the prohibitive costs associated with training large LMs, finetuning on downstream datasets is limited to a single run.
ReactXT Pretrain. The pretraining stage of ReactXT has 5 million steps, with the number of molecules per reaction being k = 4. Following MolCA’s (Liu et al., 2023b) experimental setup, we employ a Q-former with 8 query tokens. We use AdamW as the optimizer, with a weight decay set to 0.05. The optimizer’s peak learning rate is set to 1× 10−4, scheduled by linear warmup with cosine decay. The warmup has 1000 steps and starts at a learning rate of 1× 10−6.
Experimental Procedure Prediction. We fully finetune all the baseline methods and ReactXT for 20 epochs, with a batch size of 32. The optimizer and learning rate settings are consistent with the pretraining phase.
Retrosynthesis. Following (Zhong et al., 2022), we sample 20 root-aligned augmentations for the training and testing subsets. Before finetuning on USPTO-50K, We first conduct 2 epochs of masked self-supervised pretraining for MolT5 and ReactXT on the USPTO-full dataset (Dai et al., 2019), following the pretraining strategy of R-SMILES (Zhong et al., 2022). During finetuning, we train MolT5 for 20 epochs and ReactXT for 5 epochs on the augmented training set using a batch size of 32. We then average the model’s parameters on the last several tuning steps as the final checkpoint for testing. During testing, we conduct a beam search with a beam size of 20 for both models and return the top ten results as the model’s predictions. The beam size (20) and the number of results (10) are following the experiment of R-SMILES (Zhong et al., 2022). The optimizer and learning rate settings are kept consistent with the pretraining phase.
Molecule Captioning. On both datasets, we full finetune MolCA and ReactXT 20 epochs, with a batch size of 32. The optimizer and learning rate settings are consistent with the pretraining phase.

heading: B.2 Other Implementation Details
text: Baselines. We briefly introduce the baselines:
• Galactica (Taylor et al., 2022). Galactica is a scientific language model which is pretrained on 2 million compounds from PubChem. It has a decent understanding of SMILES formulas.
• MolT5 (Edwards et al., 2022). MolT5 is developed based on the T5 model. Its training corpora include both natural language and SMILES data, making it suitable for both molecule captioning and text-based molecular generation tasks.
• TextChemT5 (Christofidellis et al., 2023). TextChemT5 is a T5-based multi-domain LM, which is tuned on various text-molecule tasks.
• MolCA (Liu et al., 2023b). MolCA is a multimodal language model finetuned on Galactica. It includes both graph encoder and LM, where a Querying Transformer is applied to align their latent spaces.
• AT (Tetko et al., 2020). AT trains transformers with data augmentation for retrosynthesis. The data augmentation is achieved by rearranging the order of characters in SMILES strings in both the training and test sets.
• MEGAN (Sacha et al., 2021). MEGAN represents chemical reactions as a sequence of graph edits and performs retrosynthesis by sequentially modifying the target molecule.
• MoMu (Su et al., 2022). Momu contrastively pretrains a GNN and an LM with paired molecular graph-text data, and can be adapted to retrieval and generation tasks.
• Chemformer (Irwin et al., 2022). Chemformer is a Transformer-based molecule LM that is selfsupervised pretrained on a SMILES corpus. It can be applied to both generation and property prediction tasks.
• Random, among all reactions (Vaucher et al., 2021). Randomly pick an action sequence from the training set.
• Random, compatible pattern (Vaucher et al., 2021). Randomly pick an action sequence from the training subset of reactions that have the same number of molecules as the current reaction.
• Nearest Neighbor (Vaucher et al., 2021). Pick the action sequence from the training set with the reaction most similar to the current one, as determined by reaction fingerprints (Schwaller et al., 2019).

heading: C More Experimental Results

heading: C.1 Ablation Study
text: Table 16 presents an ablation study examining the impact of input contexts on molecule captioning. The removal of the random molecule context results in diminished captioning performance. This observation can be attributed to two factors: 1) including the PubChem324k dataset, which is used for creating random molecule contexts, is important to maintain molecule captioning performance;
and 2) without random molecule contexts, the LM becomes overly dependent on reaction contexts, compromising its capability to accurately caption individual molecules. This finding underscores the significance of incorporating random molecule contexts in training.

heading: C.2 Statistical Analysis
text: We carry out statistical tests on the experimental results to demonstrate that ReactXT achieves a significant performance improvement compared to the baseline models. For most metrics (such as BLEU, ROUGE, METEOR), we employ the T-test; for Top-k accuracy, where calculating the standard deviation was challenging, we use a 2-proportion Z-test instead.
The results of the statistical tests are presented in Tables 17 to 20. We bold p-values that are smaller than 0.05. From these tables, it can be observed that our method achieves statistically significant improvements across all metrics within the tasks of experimental procedure prediction (Table 17) and molecule captioning (Tables 18 and 19). As for the retrosynthesis task (Table 20), our method demonstrates statistically significant enhancements in both Top1 and Top3 accuracies. These observations collectively demonstrate the effectiveness of our proposed pretraining method.
Field Value
Reactant $1$: OCCCCCCCc1ccccc1 $2$: C#CC(=O)O $4$: c1ccccc1
Catalyst $3$: Cc1ccc(S(=O)(=O)O)cc1
Product $-1$: C#CC(=O)OCCCCCCCc1ccccc1
Source A mixture of 0.5 g of 7-phenylheptanol, 0.27 g of propiolic acid, 0.005 g of p-toluenesulfonic acid and 25 ml of benzene was refluxed with stirring for six hours while water formed was removed by a Dean-Stark water separator. After the reaction was completed, the reaction solution was washed successively with a 5% aqueous sodium bicarbonate solution and a saturated sodium chloride solution, and dried over anhydrous magnesium sulfate. After removal of the solvent under reduced pressure, the obtained residue was subjected to silicagel column chromatography to yield 0.368 g of 7-phenylheptyl propiolate (compound 3).

heading: C.3 Case Studies and Error Analysis

heading: C.3.1 Experimental Procedure Prediction
text: In this section, we present case studies from the experimental procedure prediction task to inform future research. We include examples of accurate predictions (see Table 21), inaccurate predictions (see Tables 22), and predictions that are different from the annotations but may also work (see Table 23 and Table 24). Our selection criteria prioritizes the accuracy of action sequences and the correct identification of primary materials, while overlooking specifics like material quantities and temperatures. All the examples are from the test set of OpenExp.
Table 21 displays two examples where experimental procedures are accurately predicted, showing close alignment between predicted and annotated actions, albeit with slight variances in material quantities and experiment times. These cases highlight the capability of LMs to predict experimental procedures, suggesting a path toward automating chemical synthesis.
Table 22 displays two failed examples of experimental procedure prediction. The predicted action sequences significantly deviate from the annotated sequences, making them impractical. Additionally, we can observe one common error of repetition, with the same or similar actions being duplicated.
Tables 23 and Table 24 showcase three examples where the predictions, while different from the annotations, could still be viable. In Example 5, as an alternative to the annotated ’EXTRACT with ethyl acetate’, the model proposes a series of actions (‘COLLECT LAYER’, ‘WASH with ethyl acetate’, ‘DRY SOLUTION’, and ‘FILTER’), serving a similar function. In Example 6, instead of the specified ’SET TEMPERATURE’ and ’STIR’, the model recommends ‘STIR for 1h at 0 °C’, serving the same purpose. In Example 7, the model suggests adding components (‘ADD $4$’, ‘ADD $5$’, ‘ADD $6$’) sequentially rather than making a single solution as annotated, which could also be effective.

heading: C.3.2 Human Evaluation of OpenExp
text: In this section, we present case studies from human evaluations on the OpenExp dataset. Samples rated from 5 to 1 by human evaluators are included, as shown in Tables 25 to Tables 29. All samples are from the 250 human evaluated data points (see Appendix A.1). It can be observed that samples
with two or fewer errors may only have minor flaws, such as typol errors or incorrect numerical values.
Field Value
Reactant $1$: Nc1ccc(C(=O)N[C@H](CO)Cc2ccccc2)c(/C=C/c2ccccc2)c1 $3$: CC(=O)OC(C)=O
Solvent $2$: C1CCOC1
Product $-1$: CC(=O)Nc1ccc(C(=O)N[C@H](CO)Cc2ccccc2)c(/C=C/c2ccccc2)c1
Source 1 g (2.7 mmol) of (S)-4-amino-2(E-2-phenylethen-1-yl)-N-(3-phenylpropan-1-ol-2-yl)benzamide (intermediate 43f) was suspended in 50 ml of tetrahydrofuran and mixed with 0.25 ml (2.7 mmol) of acetic anhydride at 100° C. The mixture was stirred for 16 h. The reaction was then concentrated under reduced pressure and the residue was recrystallized from ethanol. 0.78 g (71%) of the product was obtained.
Field Value
Reactant $1$: COc1ccc(-c2cccc(CC(=O)O)c2)cc1C(C)=NO $3$: FC(F)(F)c1ccc(CBr)cc1 $4$: [H-].[Na+] $5$: Cl
SOLVENT $2$: C1CCOC1
PRODUCT $-1$: COc1ccc(-c2cccc(CC(=O)O)c2)cc1C(C)=NOCc1ccc(C(F)(F)F)cc1
Source 9 mg of [3’-(1-Hydroxyiminoethyl)-4’-methoxybiphenyl-3-yl]acetic acid was dissolved in 0.2 mg of tetrahydrofuran, then 25 mg of 4-trifluoromethylbenzyl bromide and 10 mg of sodium hydride were added thereto, and the mixture was stirred at room temperature for 3 days. The reaction mixture was neutralized with 1N hydrochloric acid and then extracted with ethyl acetate. The organic layer was concentrated, and the residue was purified by reverse-phase high performance liquid chromatography to give 5.21 mg of the title compound.
Field Value
Reactant $1$: CNC(=O)c1cn(CCCCc2ccc(N)nn2)nn1 $2$: O=C(O)Cc1cc(Br)ccn1 $4$: CCCP1(=O)OP(=O)(CCC)OP(=O)(CCC)O1 $6$: CCN(C(C)C)C(C)C
Solvent $3$: CN(C)C=O $5$: CCOC(C)=O $7$: O
Product $-1$: CNC(=O)c1cn(CCCCc2ccc(NC(=O)Cc3cc(Br)ccn3)nn2)nn1
Source To a solution of 1-(4-(6-aminopyridazin-3-yl)butyl)-N-methyl-1H-1,2,3-triazole-4-carboxamide (2.0 g, 7.2 mmol) and 2-(4-bromopyridin-2-yl)acetic acid (1.6 g, 7.2 mmol) in DMF (5 ml) were added T3P (50 wt. % in EtOAc, 9.2 g, 14 mmol) and DIEA (2.7 g, 22 mmol). The mixture was stirred at RT for 3 h, then water (50 mL) was added, the mixture was stirred for 30 min, and precipitate was isolated by filtration to give the title compound as a yellow solid (1 g, 40%). MS (ES+) C19H21BrN8O2 requires: 473. found: 474 [M+H]+.

heading: Future Work
text: In this section, we present case studies from the experimental procedure prediction task to inform future research. We include examples of accurate predictions (see Table 21), inaccurate predictions (see Tables 22), and predictions that are different from the annotations but may also work (see Table 23 and Table 24). Our selection criteria prioritizes the accuracy of action sequences and the correct identification of primary materials, while overlooking specifics like material quantities and temperatures. All the examples are from the test set of OpenExp. Table 21 displays two examples where experimental procedures are accurately predicted, showing close alignment between predicted and annotated actions, albeit with slight variances in material quantities and experiment times. These cases highlight the capability of LMs to predict experimental procedures, suggesting a path toward automating chemical synthesis. Table 22 displays two failed examples of experimental procedure prediction. The predicted action sequences significantly deviate from the annotated sequences, making them impractical. Additionally, we can observe one common error of repetition, with the same or similar actions being duplicated. Tables 23 and Table 24 showcase three examples where the predictions, while different from the annotations, could still be viable. In Example 5, as an alternative to the annotated ’EXTRACT with ethyl acetate’, the model proposes a series of actions (‘COLLECT LAYER’, ‘WASH with ethyl acetate’, ‘DRY SOLUTION’, and ‘FILTER’), serving a similar function. In Example 6, instead of the specified ’SET TEMPERATURE’ and ’STIR’, the model recommends ‘STIR for 1h at 0 °C’, serving the same purpose. In Example 7, the model suggests adding components (‘ADD $4$’, ‘ADD $5$’, ‘ADD $6$’) sequentially rather than making a single solution as annotated, which could also be effective