Predictive Chemistry Augmented with Text Retrieval
Authors: Yujie Qian
Zhening Li
Zhengkai Tu
Connor W. Coley
Regina Barzilay
Venue: EMNLP 2023 Main (2023)
Keywords: information retrieval
chemistry

Abstract: This paper focuses on using natural language descriptions to enhance predictive models in the chemistry field. Conventionally, chemoinformatics models are trained with extensive structured data manually extracted from the literature. In this paper, we introduce TextReact, a novel method that directly augments predictive chemistry with texts retrieved from the literature. TextReact retrieves text descriptions relevant for a given chemical reaction, and then aligns them with the molecular representation of the reaction. This alignment is enhanced via an auxiliary masked LM objective incorporated in the predictor training. We empirically validate the framework on two chemistry tasks: reaction condition recommendation and one-step retrosynthesis.  By leveraging text retrieval, TextReact significantly outperforms state-of-the-art chemoinformatics models trained solely on molecular data.

Predictive Chemistry Augmented with Text Retrieval
Yujie Qian t
Zhening Li Zhengkai Tu Connor W. Coley Regina Barzilay
'Computer Science and Artificial Intelligence Lab, MIT Department of Electrical Engineering and Computer Science, MIT
$Department of Chemical Engineering, MIT Cyujieq, regina)écsail. httpssaaalliii...........e (z1i11010, ztu, ccoley edu
Abstract
2021). At the same time, scientific literature pro- vides several cues about potential catalysts (see highlighted excerpts). While on their own these paragraphs might not provide a comprehensive am- swer, they can guide molecular predictor. This intuition motivates our approach to aggregating tex- tual sources with molecular representations when reasoning about chemistry tasks. Specifically, we aim align the representations of the reaction and its corresponding text description, enabling the model to operate in the combined space. This design not only enables the model to retrieve read- ily available information from the literature but also enhances its generalization capacity for new chemical contexts.
This paper focuses using natural language descriptions to enhance predictive models in the chemistry field. Conventionally, chemoin- formatics models are trained with extensive structured data manually extracted from the lit- erature. In this paper, we introduce TextReact, a novel method that directly augments predic- tive chemistry with texts retrieved from the lit- erature. TextReact retrieves text descriptions relevant for given chemical reaction, and then aligns them with the molecular representation of the reaction. This alignment is enhanced via an auxiliary masked LM objective incorporated the predictor training. We empirically val- idate the framework on two chemistry tasks: reaction condition recommendation and one- step retrosynthesis. By leveraging text retrieval, TextReact significantly outperforms state-of- the-art chemoinformatics models trained solely molecular data.
We propose TextReact, a novel predictive chem- istry framework augmented with text retrieval. TextReact comprises two modules a SMILES- to-text retriever' that maps an input reaction to cor- responding text descriptions, and a text-augmented predictor that fuses the input reaction with the trieved texts. The model learns the relation be- tween a chemical reaction and text via an auxiliary masked LM objective incorporated in the predictor training. Furthermore, improve generalization to unseen reaction classes, we simulate novel inputs by eliminating from the training data the closest textual descriptions for given reactions.
1 Introduction
In this paper, we propose method for leveraging automatically retrieved textual knowledge to im- prove the predictive capacity chemistry models. These chemoinformatics models are utilized for a wide range of tasks, from analyzing properties of individual molecules to capturing their interactions in chemical reactions (Yang et al., 2019; Segler al., 2018; Coley et al., 2018). However, stan- dard approaches make these predictions operating solely on molecular encodings. Despite significant advances neural molecular representations, the accuracy of these models still offers room for im- provement. We hypothesize that their performance can be further enhanced using relevant information from the scientific literature.
In addition to condition recommendation, TextReact can readily applied to other chem- istry tasks. In our experiments, we also consider one-step retrosynthesis (Coley et al., 2017), the task of predicting reactants used to synthesize a target molecule (see Figure 2). By leveraging rel- evant text, TextReact achieves substantial perfor- mance improvement compared to the state-of-the- chemoinformatics models trained on reaction data alone. For instance, for condition recommen-
For instance, consider the task of finding a cat- alyst for the chemical reaction shown Figure 1. This prediction proves to be challenging when con- sidering the reaction components alone (Gao et al.,
SMMIES,, i.c. Simplified Molecular-Ipput Line-Entry Sys- tem, is specification in the form of a line notation for de- scribing chemical structures (Weininger, 1988).

---

Input: mracion
Reactant
Product
Text Retrioval
dione 10:12 Aminn 3 pheryteropyl 2,4|3H, dlone dissolved McOH benzaldelyde AcOH dropt added -... NaBHBCN added
salutien (S) tort butty 41 tomylberzyy
Augment
(1-Inaphthalen- weapttaallee---teellllleeee beeettylureidoobuyleaaabboo dichloroethane mi) was added N-Denaylamins acetic (10 drops) and Na(DAc]SBH. mixure was salurated aqueous solution sodium bicarbonate.
Model
portion .... crude product is dissoived in DCMMMON (4:1) and purified.. to afford the desired product.
Output: reaction condition
Catalyst
Solvent
Reagent
CIL.COn
CH:OH
NaHCOr
Na/BH/CNI
Figure I: Predictive chemistry augmented with text retrieval. For example, given the task of reaction condition recommendation, we retrieve texts relevant to the input reaction to provide additional evidence for the model's prediction. The two retrieved texts describe similar reactions to the input and share similar conditions. For visualization purposes, mark the catalyst, solvent, and reagent in red, green, and blue in the retrieved texts. However, we do not assume that the text corpus contains such structured data.
dation, TextReact increases the top-1 prediction accuracy by 58.4%, while the improvement in one- step retrosynthesis
Input:
Output:
Model
13.6-15.7%. The improve- ment is consistent under both random and time- based splits of the datasets, validating the efficacy of our retrieval augmentation approach in general- izing to new task instances. Our code and data are publicly available at https://g't thub. com/ tho mas0809/textreac"
Ni
Figure One-step retrosynthesis, another predictive chemistry task studied in this paper.
Retrieval-Augmented NLP There have been several studies on augmenting NLP models with retrieval. The idea is to retrieve relevant documents from a corpus and use them as additional context to the model. Earlier works on open-domain question answering adopted standalone retrievers to iden- tify supporting paragraphs for the given questions (Chen et al., 2017: Lee al., 2019; Karpukhin et al., 2020). Guu et al. (2020) jointly trained an end-to-end retriever with language model, and Lewis et al. (2020b) extended the idea to general sequence-to seueence-to-sequence generation.
2 Related Work
Multimodal Retrieval This field studies retrieval algorithms when the query and target are differ- ent modalities, as exemplified by image-text re- trieval (Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al.. 2014: Karpathy et al., 2014; Faghri et al., 201 8). Previous research learns mul- timodal embeddings in image and text using tech- niques ranging from kernel methods to more pressive neural networks. This line also extends to other modalities such as video (Miech et al., 2020) and audio (Aytar et al., 2017).
Our work implements similar idea of retrieval augmentation. However, we focus on tasks in the chemistry domain (see Figures I and 2), with the goal of enhancing chemistry prediction models with natural language descriptions retrieved from the literature.
Our retrieval method closely relates to CLIP (Radford et al., 2021), a multimodal model that leverages contrastive learning to align images with their corresponding natural language descriptions. Moving to the chemistry domain, Edwards et al. (2021) proposed Tex12Mol to retrieve molecules using natural language queries. Our SMILES-tO- retriever operates in the opposite direction, i.e.,
Natural Language Grounding Previous studies have demonstrated how natural language can be harnessed as a grounding mechanism to supervise tasks in various domains. For example, in com- puter vision, natural language descriptions have been used improve fine-grained classification of bird images (He and Peng, 2017: Liang et al., 2020) and to regularize the learning of visual representa- tions (Andreas et al., 2018; Mu et al., 2020). In rein-
we use the aligned embedding spaces to retrieve a text description given a reaction query. We then use the retrieved text to enhance chemistry prediction.

---

Reactant - Product
chemistry input
Q
Catalyst
Text-Augmented Predictor
Reaction Text Corpus
Solvent
neighbory
text
Reagent
SMILES-to-text
input
Retriever
Figure 3: Overview of TextReact as applied to reaction condition recommendation. The retriever searches for texts relevant to the chemistry input, which are then used to augment the input the predictor.
forcement learning and robotics, natural language instructions transformed into executable actions of the agent (Branavan al., 2009; Narasimhan al., 2015; Ichter et al., 2022) and enable adapt- ability to new tasks (Hill et al., 2020).
(2) An unlabeled text corpus of chemical reactions
T {tz, ,...., Mr where cach ty is paragraph describing a chemical reaction. They can be easily obtained from the text of journal articles and patents using a model or heuristics to determine that they contain reac- tion description.
In the chemistry domain, our work comple- ments ongoing efforts in the automatic extraction of structured reaction data from the chemistry litera- ture (Lowe, 2012; Swain and Cole, 2016; Guo et al., 2021: Qian et al., 2023a.b; Wilary and Cole, 2021; He et al., 2021; Nguyen et al., 2020; Vaucher et al., 2020; Zhai et al., 2019). While these works focus on creating structured databases used for training molecular models, we focus on directly leverag- ing unstructured natural language descriptions to further augment their predictions.
More generally, other predictive chemistry tasks can use the same problem setup while changing the definitions of X and Y accordingly. For instance, in one-step retrosynthesis, X is product molecule and is list of potential reactants.
3.2 TextReact Framework
We propose a novel framework TextReact to aug- ment chemistry prediction with text retrieval. As illustrated Figure 3, the TextReact framework comprises two major components: a SMILES -S- text retriever (Section 3.2.1) and a text-augmented predictor (Section 3.2.2). The retriever searches the unlabeled corpus for texts relevant the particular chemistry input. The predictor leverages both the chemistry input and the retrieved texts generate the prediction.
3 Method
3.1 Problem Setup
For concreteness, consider the task of reaction con- dition recommendation, where the input X chemical reaction and the output is list reac- tion conditions, including the catalyst, solvent, and reagent. We aim to train a machine learning model F to generate the prediction, i.e., Y =(()). The model F is typically trained on a set of labeled training data Dunin (2;, vi), 1...., N).
3.2.1 SMILES-To-Ter Retriever
The goal the retriever is to locate relevant texts from an unlabeled corpus based on a given chem- istry input. To accomplish this task, we devise a SMILES ttttttt retriever, leveraging the widely used dual encoder architecture employed in docu- ment retrieval (Lee et al., 2019; Karpukhin et al., 2020) and image-text retrieval (Radford al., 2021). The model consists of two parts: chem- istry encoder for processing the input molecule or reaction represented as a SMILES string, and a text encoder for encoding the text descriptions. For the
In this paper, incorporate two additional re- sources for the training of model F:
(1) Each example in the training set is paired with text paragraph, i.e. (x4, ye. ye,t.) tz), where t; de- scribes the corresponding chemical reaction. Many reaction databases are curated from the chemistry literature and provide text references for their reaction data.2
FExamples of such text references are available in the appendix.

---

In this paper, we train the retriever separately from the predictor with standalone objective. While joint optimization of the retriever and the predictor is possible (Guu et al., 2020; Lewis et al., 2020b), requires significantly more computation as it involves itcrative index rebuilding and retrieval during training. We choose standalone retriever that has exhibited strong performance in identify- ing relevant texts in experiments (see Table 2).
Text Encoder
35
Chemistry Encoder
45
3.2.2 Text-Augmented Predictor
Figure SMILES-tO-Be retriever trained with con- trastive learning.
The goal of this component is merge chemistry and natural language to produce accurate predie- tions. Our text-augmented predictor designed as an encoder-decoder model, with the encoder handling both the chemistry input and retrieved texts, and the decoder generating the correspond- ing chemistry output.
chemistry encoder, we use Transformer to encode the SMILES string into a latent vector. As for the text encoder, we employ Transformer pre-trained on scientific text (Beltagy et al., 2019) to encode each paragraph into latent vector. To enable ef- ficient retrieval using nearest neighbor search, train the model align the two latent spaces.
The encoder's input consists of the concatenation of the chemistry input (i.e., the SMILES string) and the retrieved texts. To structure the input, we prepend a [CLS] token at the beginning and use [SEP] tokens to separate the SMILES and the texts. This yields the following input format:
The retriever is trained with contrastive learn- ing, as illustrated in Figure 4. Given a batch of SMILES strings and their corresponding para- graphs (2x, 4)), = I, n.... we first compute their encodings.
[CLS] SMILES [SEP] (0) TEXT, (1) TEXT [SEP]
x;-- ChemEnc(z;), t; TextEnc(tt). (I)
TEXT,, TEXT are the nearest neighbors re- trieved by the SMILES-to-text retriever, which serve as additional input to augment chemistry pre- diction. The number of the appended nearest neigh- bors k is hyperparameter.
The similarity between SMILES and text para- graph defined by the dot product of their encod- ings,
S1 - x tj-
(2)
The decoder architecture is tailored to the spe- cific predictive chemistry task. For reaction condi- tion recommendation, lished in prior research (Gao et al., 2018), which generates the reaction conditions in specific or- der: catalyst, solvent 1, solvent 2, reagent 1, and reagent 2. For one-step retrosynthesis, the decoder either directly generates the SMILES strings of the reactants (i.c., template-free approach). or predicts the reaction template first and uses chem- informatics software to derive the reactants (i.e., template-based approach).
For each SMILES string 24, its paired paragraph 4 is a positive example, and the other paragraphs within the same batch are considered negative ex- amples. We further randomly sample n paragraphs from the unlabeled corpus as additional negative examples for each batch, denoted as in I...., f2n- The training objective is to maximize the log- likelihood of matching the SMILES strings with positive text paragraphs, i.e..
adopt the approach estab-
exp(S..)
Lyet
Σ log Ej=1...m (S.g) i=
(3)
The predictor is trained via supervised learning to maximize p(y: x4, Ti), where T; is the set of retrieved texts. To encourage the predictor to lever- age the information both chemistry and text, we add an auxiliary masked language model (MLM)
After training, we pre-compute all text encod- ings and compile them into an index to support efficient retrieval. When retrieving relevant texts for a given SMILES string, we compute its en- coding with the chemistry encoder and then run maximum inner product search (Mussmann and Er- mon, 2016; 6; Johnson et al., 2019) to find its nearest neighbors in the index.
'Depending on whether to use an explicit set of reaction templates, existing one-step retrosynthesis models can be cat- egorized into template-based and template-free approaches. See Scction 5.2 and Appendix A.2 for further discussion.

---

loss. Specifically, we randomly mask out portions of the input, either the SMILES string or the text, and add prediction head on top of the last layer of the encoder to predict the masked tokens. In- spired by previous research on language model pre- training (Devlin et al., 2019; Lewis et al., 2020a; Joshi et al., 2020; Du et al., 2022), the masking is performed by repeatedly sampling spans of length drawn from Poisson distribution with À = 3, until the total number of masked tokens reaches 15% of the original input. With the MLM loss, the model learns to infer the missing part of a SMILES string from its nearest neighbor texts, and vice versa, thus encouraging it to learn the correspondence between chemistry and text. The masking also encourages the predictor not to rely on single source of input, similar to dropout, thus improving its robustness. Finally, the training objective of the predictor is
Each paragraph in the corpus provides a descrip- tion of the synthesis procedure of chemical reac- tion. The corpus consists of 2.9 million paragraphs, with an average length of 190 tokens.* While the original dataset includes structured reaction data extracted from each paragraph, including reactants, products, and reaction conditions, our model does not rely on this extracted data but instead learns directly from the unlabeled text.
The SMILES-to-text retriever is implemented with Tevatron (Gao et al., 2022). The chemistry encoder is initialized with Chem- BERTa (Chithrananda et al., 2020) (pre-trained on the SMILES strings from a molecule database) and the text encoder is initialized with SciBERT (Belt- agy et al., 2019) (pre-trained on scientific text from the literature). We finetune a separate retriever on the training set of each chemistry task. The in- put to the chemistry encoder is slightly different in each task. For reaction condition recommendation, the input is the reactants and product of reaction, while for one-step retrosynthesis, the input is only the product. both cases, the chemistry input is represented as a SMILES string.
Implementation
Ippeed = Σ log P(U 2s, Ti) + A (4)
where A is a hyperparameter controlling the weight of the MLM loss LMLM.
To improve the generalizability of the predictor, we employ a dynamic sampling training strategy that addresses the following important distinction between the training and testing of the predictor. During training, paired with each chemistry input, providing a de- scription of the corresponding reaction. However, in testing, it is not guaranteed to retrieve the gold text. Moreover, for novel chemistry inputs lack- ing a corresponding text description in the exist- ing corpus, the predictor can only use information from the nearest neighbors, which may describe similar but not identical reactions. To address this gap, employ a random sampling policy during training to simulate novel chemistry inputs that are not present in the corpus. With probability a, the chemistry input is augmented with i random neighbors from its top-K nearest neighbors, where K >> 1t and the top-K neighbors are expected to cover reactions similar to the chemistry input. In the remaining cases, the model is given the gold text along with the top-(k 1) neighbors (exclud- ing the gold text) returned by the retriever. During inference, the examples are always augmented with the top-k nearest neighbors.
The text-augmented predictor employs a pre- trained SciBERT (Beltagy et al., 2019) as the en- coder. We concatenate the input SMILES string with three neighboring text paragraphs (k 3) by default, using [CLS] and [SEP] tokens as de- scribed in Section 3.2.2. During training, we set the random sampling ratio to a = 0.8 for reaction condition recommendation and a = 0.2 for one- step retrosynthesis, and the cut-off K is set to 10. We analyze the effect of the hyperparameters in Section 5.3. During inference, we use beam search to derive the top predictions. More implementation details can be found in Appendix C.
have access to the gold text
Evaluation For each chemistry task, we evaluate TextReact under two setups. The first setup is the random split, where the dataset is randomly divided into saaiinng/vaiddaiioneestnngggg This is a commonly used setup in previous chemistry research (Gao et al., 2018; Coley et al., 2017). The second and more challenging setup is the time split, where the dataset is split based on the patent year. We train the model with historical data from older patents
"We use the tokenizer SciBERT: https: htttgi////ggngaaae co a wwwalleeeessciiii 'As the original Tevatron toolkit does not support different encoders and tokenizers for queries and passages, we make modifications to accommodate these requirements.
4 Experimental Setup
Text Corpus We construct a text corpus from USPTO patent data processed by Lowe (2012).

---

RCR
RCR
RetroSyn (RS)
RetroSyn
details can be found in Appendix B.
(RS)
(TS)
(TS)
We implement four baselines, none of which use additional text input:
Baselines
Train
546,728
565,575
40,008
38,631
Valid
68,341
63,015
5,001
5,624
Test
68,341
54,820
5,007
5,761
Reaction fingerprint (rxnfp) LSTM, repro- duction of the method proposed by Gao al, (2018). The reaction fingerprint calculated as the difference between the product and re- actant fingerprints, which further encoded by a two-layer neural network, and an LSTM decodes the reaction conditions.
Table I: Statistics of datasets for reaction condition recommendation (RCR) and one-step retrosynthesis (ReumSyn). RS: random split; TS: time split.
R@1
R@3
@
10
RCR (RS)
70.9
91.6
97.0
Reaction fingerprint (rxnfp) retrieval, which uses the conditions of the most similar reac- tions in the training as the prediction. Sim- ilar reactions are determined based on the L-2 distance of reaction fingerprints. This baseline examines the performance of pure retrieval method on this task.
RCR (TS)
75.1
91.8
96.2
RetroSyn (RS)
60.0
82.5
92.5
RetroSyn (TS)
58.0
81.4
91.5
Table 2: Performance of SMIILES-to-tex retriever trained on each dataset. We report the Recall @ 11,,100) when retrieving from the full corpus. Scores are in %.
Transformer, the most important baseline we are comparing with, which uses the same ar- chitecture our predictor. This baseline rep- resents the state-of-the-art model that only takes chemistry input.
and test its performance on the data from newer patents. Due the substantial differences between reactions new patents and previous ones, it be- comes more challenging for a model to generalize effectively under time split. During testing, we compare retrieving from the full corpus (including newer patents) and retrieving only from the years used for training. Table shows the statistics of the datasets for the tasks, which are elaborated in the next section.
ChemBERTa (Chithrananda et al., 2020). This baseline is the same as the Transformer base- line except that the encoder is pretrained on external SMILES data. The purpose of this baseline is to demonstrate the impact of such pretraining.
5 Experiments
Results Table shows our experimental results for reaction condition recommendation. TextReact substantially outperforms standard chemistry mod- els (Transformer and ChemBERTa), which are trained on reaction data without text. This signif- icant improvement can be attributed to the strong performance of our retriever (shown in Table 2). When the retriever successfully identifies the gold text, TextReact effectively utilizes this information to make predictions. Otherwise, TextReact can also benefit from retrieving texts of similar reactions, as we will demonstrate Scction 5.3.?
5.1 Reaction Condition Recommendation
Data We follow the setup of previous research (Gao et al., 2018) to construct reaction condition datasets from the USPTO data (Lowe, 2012). The reactions with at most one catalyst, two solvents, and two reagents kept, and reactions with con- ditions that occurred fewer than 100 times cluded. Each reaction is associated with text paragraph (gold) from the patent. However, only utilize the gold text during the training of the retriever and the predictor, and do not use for validation or testing.
TextReact generalizes to more challenging time split. As reactions from the same patent are often similar, it may be easy for the model to in- fer the reaction conditions when similar reactions
We create two splits of the dataset: RCR (RS) for random split, and RCR (TS) for time split. More
"Gao et al. constructed their datasets from Reaxys, which are not publicly available. We use the preprocessing script of Parrot (https: //gs:///itte thub. com//wangrr822//Parroc to process the public USPTO data.
Another noteworthy observation is that the rxufp retrieval bascline performs comparably to Transformer, suggesting that similar reactions often share similar reaction conditions, vali- dating the efficacy of retrieval-hasce methods.

---

RCR (RS)
RCR (TS)
Top-1 1
Top-3
Top-10 10
15p-15
Top-1
Top-3
Top-10 10
Top-15
rxnfp LSTM (Gao et al., 2018)
20.5
30.7
41.7
45.3
13.2
26.2
40.7
45.4
rxnfp retrieval
27.2
37.5
47.9
51.1
7.8
15.2
27.3
31.5
Transformer
30.0
43.8
56.7
60.5
18.7
31.8
47.6
52.7
ChemBERTa TextReact
30.3
44.7
58.0
62.0
18.7
31.9
47.6
52.8
88.4
93.9
96.0
96.5
83.9
90.9
93.9
94.6
Table 3: Evaluation results for reaction condition recommendation (RCR). RS: random split; TS: time split. Scores are accuracy in %.
RetroSyn (RS)
RetroSyn (TS)
Top-1
Top-3
Top-5
Top-10
Top-1
Top-3
Top-5
Top-10
Template-free models G2G (Shi et al., 2020)
48.9
67.6
72.5
75.5
Transformer (Lin et al., 2020)
43.1
64.6
71.8
78.7
Dual-TF (Sun al., 2020)
53.6
70.7
74.6
77.0
Transformeryf
45.9
63.2
68.5
75.2
31.4
46.6
51.6
57.2
TextReacty
59.5
72.6
75.9
80.0
51.0
64.1
68.1
72.9
Template-hased models
LocalRetro (Chen and Jung. 2021)
53,4
77.5
85.9
92.4
O-GNN (Zhu et al., 2023)
54.1
77.7
86.0
92.5
Transformer
52.5
72.8
79.7
86.2
43.6
65.6
73.2
82.1
TextReact;s
68.2
83.7
88.1
92.5
68.7
84.5
88.8
92.8
Table Evaluation results for one-step retrosynthesis. RS: random split; TS: time split. Scores are accuracy %.
are present in the training set. Unsurprisingly, all baselines perform worse under the time split, high- lighting the inherent difficulty of achieving such generalization. In TextReact, despite being trained only on historical data, the retriever retains high accuracy in retrieving the corresponding paragraph for reactions in the testing set (see Table 2). Thus, TextReact retains strong overall performance by leveraging the retrieved text.
the matched examples to train the retriever, whereas the predictor is trained with the full dataset.
Template-free & Template-based Models We implement TextReact in two settings: (1) TextReacty is a template-free model that uses a Transformer decoder to generate the SMILES strings of reactants. (2) TextReact, is a template- based model that follows the formulation of LocalRetro (Chen and Jung, 2021). Specifically, adopt the set of reaction templates extracted from the training data, and predict which template applicable product molecule and which atom or bond is the reaction center. For each atom, we represent it using the corresponding hidden state from the last layer of the Transformer encoder and predict probability distribution over the reaction templates using linear head. For each bond, we concatenate the representations of its two atoms and employ another linear head to predict the tem- plate.
5.2 One-Step Retrosynthesis
Data We use the popular USPTO-50K dataset (Coley et al., 2017) for our one-step retrosynthe- sis experiment. We also create two data splits: RetroSyn (RS), the original random split of the dataset, and RetroSyn (TS), the time split.
Since the dataset was constructed from the same USPTO data as our text corpus, we match the exam- ples in the dataset with text paragraphs in the cor- pus. However, due to differences in preprocessing, not all examples could be matched. 8 We use only
"Out of the 40,008 examples in the training set, 31,391 are
matched.

---

RCR (RS)
RCR (TS)
RetroSyn (RS)
RetroSya (TS)
Top-1 Acturacy
60
60
35
35
30
30
lcmplate-free
lemplate-bused
Vemplate-fivee
lemplate-bused
Transformer
TextReaci (full corpus)
TextReact (gold-removed)
TextReact (TS corpus)
Figure 5: Comparison of TextReact's performance under three settings: (1) Retrieving from the full corpus, (2)
Retrieving from the corpus with gold texts of testing examples removed, and (3) Retrieving only from the historical corpus under time split.
Baselines We implement baselines Transformery and Transformeræ for template-free and template- based settings, respectively. They closely follow the output format of TextReact but do not incor- porate retrieved text additional input. We also report the published results of previous research (Shi et al., 2020; Lin al., 2020; Sun et al., 2020; Chen and Jung, 2021; ; Zhu et al., 2023). All basc- lines are trained on the same data as TextReact, but without the text input.
RetroSyn (RS)
gold-nmuved
full
removed
Tog- Acsusacy
TextRmete
Figure 6: Performance of TextReact with respect the random sampling ratio during training.
Results Table shows the results one-step ret- rosynthesis. Similar to the RCR task, TextReact has demonstrated strong performance by leverag- ing text retrieval. On the RetroSyn (RS) dataset, our baseline Transformer models perform com- parably to previous models under both template- free and template-based settings. Upon integrat- ing augmentation, TextReact, and TextReactæ, advance the top-1 accuracy by 13.6% and 15.7%. respectively, affirming the advantage of retrieval augmentation for this task. Even under time-split scenario, TextReact maintains a high accuracy, un- derscoring its proficiency in both retrieval and final prediction stages.
Transformer baseline significantly. The improve- ment is consistent under both the random split and time split, albeit smaller compared with retriev- ing from the full corpus. However, we note that under the time split, if the model only allowed to retrieve from the historical corpus (the patents for the training data, referred to as "TS corpus"), TextReact can hardly outperform the baseline. The reason behind this could be that the reactions in patents sufficiently different from those in historical patents, placing predictive barrier for such -ut-of-distribution generalization.
TextReact's robust generalization performance is enabled by the sampling strategy we adopted during training. Figure 6 illustrates the impact of the random sampling ratio a. During training, the predictor is given the gold text paragraph with prob- ability 1 a, and randomly sampled paragraphs with a probability a. The results reveal that the choice of a plays an important role in the model, both when retrieving from the full corpus and the gold-removed corpus. In the RCR (RS) dataset, the full corpus evaluation prefers smaller a, while the gold-removed evaluation prefers larger a. Since controls the probability of using gold input in training, larger aligns better with the gold- removed setting. We set a 0.8 for the RCR experiments for a good balance between the two settings. In RetroSyn (RS), however, the model
5.3 Analysis
First, we demonstrate in Figure 5 that TextReact exhibits generalization capabilities novel reac- tions not present within the text corpus. While we have illustrated the accurate retrieval of text descriptions from the corpus and their effective utilization for predictions, is important to note that gold text for the target reaction may not al- ways be available within the corpus. To further assess the model's performance in a chal- lenging scenario, we remove the gold texts of all testing examples from the corpus (referred to as "gold-removed" in Figure 5). In both RCR and RetroSyn, TextReact continues to outperform the

---

RCR (RS)
RetroSyn (RS)
Conclusion
full
g.r.
full
g.r.
This paper presents TextReact, a novel method that augments predictive chemistry with text retrieval. We employ information retrieval techniques to iden- tify relevant text descriptions for a given chem- istry input from an unlabeled corpus, and supply the retrieved text as additional evidence for chem- istry prediction. In two chemistry tasks, TextReact demonstrates strong performance when retrieving from the full corpus, and maintains a significant improvement when retrieving from a harder corpus that excludes the gold texts.
TextReact
88.4
47.2
59.5
47.7
MMILES-only
30.0
30.0
45.9
45.9
lext-only
81.7
38.3
23.0
9.0
no MLM
87.7
47.0
51.0
43.5
no pretrain
83.5
43.8
48.1
40.3
sep. neighbors
80.7
45.2
49.4
45.3
Table 5: Ablation study of TextReact. We evaluate TextReacty for the RetroSyn task. (full: full corpus; g.r.: gold-removed)
Our results highlight the promising potential of incorporating text retrieval methods in the field of chemistry. As chemically similar reactions have similar conditions and outcomes, effectively trieving and grounding textual knowledge from patents and publications into the chemistry space can significantly enhance the predictive power of computational models.
favors much smaller We hypothesize that this difference due the varying helpfulness of the retrieved texts in different tasks. The texts appear to be more helpful for condition recommendation than retrosynthesis, perhaps due to the existence of common reaction conditions that can be reused across many different reactions of the same type.
Limitations
Table 5 presents ablation study. First, TextReact performs significantly better than the model that uses only the input SMILES or retrieved text, illustrating that TextReact effectively inte- grates both chemistry and text inputs to generate its predictions. Second, we observe a significant increase in accuracy for TextReact compared to the model trained without the MLM loss. This demon- strates the effectiveness of the auxiliary MLM ob- jective in enhancing the model's learning of the correspondence between chemistry and text inputs. addition, TextReact benefits from the SciBERT checkpoint, which has been pretrained on scien- tific text. While TextReact concatenates the top-k: neighbors together to make predictions, we com- pare with a variant that separates the neighbors and ensembles their predictions, similar to RAG (Lewis al., 2020b). This variant yields worse performance (the last row of Table 5), suggesting the benefits of jointly encoding the neighbors.
We acknowledge two limitations this paper. First, our experiments focused on two representative chemistry tasks, but we believe that the proposed method can be applied to other tasks and domains that would benefit from the knowledge in the lit- erature. Second, employed simplified imple- mentation of the retriever and predictor models demonstrate the effectiveness retrieval augmenta- tion in chemistry. There is significant room for fur- ther improvements, such as using more advanced pretrained models and exploring joint training of the models.
Acknowledgements
The authors thank Wenhao Gao, Xiaoqi Sun, Luyu Gao, and Vincent Fan for helpful discussion and feedback. This work was supported by the NSF Ex- peditions grant (award 1918839: Collaborative Re- search: Understanding the World Through Code), the Machine Learning for Pharmaceutical Discov- ery and Synthesis (MLPDS) consortium, the Ab- dul Latif Jameel Clinic for Machine Learning in Health, the DTRA Discovery of Medical Counter- measures Against New and Emerging (DOMANE) threats program, the DARPA Accelerated Molecu- lar Discovery program, the NSF AI Institute CCF- 21 2665, the NSF Award 21 2134795, the GIST-MIT Research Collaboration grant, MIT-DSTA Singa- pore collaboration, and MIT-IBM Watson AI Lab.
Additional analyses in Appendix D reveal sev- eral key findings: (1) TextReact achieves superior performance on the RCR task using only 10% of the training data compared to the Transformer base- line; (2) TextReact performs better when the re- trieved texts describe reactions that bear closer sim- ilarity to the input reaction; and (3) TextReact ben- efits from jointly modeling the input reaction and retrieved neighboring texts.

---

References
the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI-HIT 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume (Long Papers), pages 2166-2179.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding. Jiezhong Qiu, Zhilin Yang. and Jie Tang. 2022. GL.M: general language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022. pages 320-335.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2017. See. hear, and read: Deep aligned representations. arXiv preprint arXiv:1706.0000 00932.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, SMNNL--JCCLL 2019, pages 3613-3618.
Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2Mol: Cross-modal molecule retrieval with nat- ural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, pages
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: improving visual- semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, page 12.
S. R. K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the 47th Annual Meeting of the Association for Com- putational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, ACL 2009, pages 82-90.
Hanyu Gao, Thomas J. Struble, Connor W. Coley, Yu- ran Wang, William H. Green, and Klavs F, Jensen. 2018. Using machine learning to predict suitable con- ditions for organic reactions. ACS Central Science, 4(1 (111::465-1476
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Volume 1: Long Papers. pages 1870-1879.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Tevatron: An efficient and flexible toolkit for dense retrieval. arXiv preprint arXiv:2203. 05765.
Shuan Chen and Yousung Jung. 2021. Deep retrosyn- thetic reaction prediction using local reactivity and global attention. JACS Au, (10):1612-1622)
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Papers), pages 3816-3830.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. 2020. ChemBERTa: Large-scale self- supervised pretraining for molecular property predic- tion. arXiv preprint 20x0::2000009885.
Jiang Guo, A Santiago Ibanez-Lopez, Hanyu Gao, Vic- tor Quach, Connor W Coley, Klavs Jensen, and Regina Barzilay. 2021. Automated chemical reaction extraction from scientific literature. Journal of Chem- ical Information and Modeling, 2235522035
Connor W Coley, William H Green, and Klavs F Jensen. 2018. Machine learning in computer-aided syn- thesis planning. Accounts of Chemical Research, 128111228-1
Connor W. Coley, Luke Rogers, William H. Green, and Klavs F. Jensen. 2017, Computer-assister retrosyn- thesis based on molecular similarity. ACS Central Science, 3(12):1237-12)
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Ming- Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, volume 119, pages 3929-3938.
Hanjun Dai, Chengtao Li, Connor W. Coley, Bo Dai, and Le Song. 2019. Retrosynthesis prediction with conditional graph logic network. In Advances in Neural Information Processing Systems 32, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages -880.
Jiayuan He, Dat Quoe Nguyen, Saber A. Akhondi. Christian Druckenbrodt, Camilo Thorne, Ralph Hoes- sel, Zubair Afzal, Zenan Zhai, Biaoyan Fang, Hiyori Yoshikawa, Ameer Albahem, Lawrence Cavedon, Trevor Cohn, Timothy Baldwin, and Karin Verspoor. 2021. Chemu 2020: Natural language processing methods are effective for information extraction from chemical patents. Frontiers in Research Metrics and Analytics, 6:654438.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

Xiangteng He and Yuxin Peng. 2017. Fine-gre Fine-grained im- age classification via combining vision and language. In 2017 IEFF Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 7332-7340.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke 7ettlemoyer. 2020a. BART: denoising sequence-to-seqqenec pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. ACL 2020. pages 7871-7880.
Felix Hill, Andrew K. Lampinen, Rosalia Schneider, Stephen Clark, Matthew M. Botvinick, James L. Mc- Clelland, and Adam Santoro. 2020. Environmen- tal drivers of systematicity and generalization in a
situated agent. In 8th International Conference on Learning Representations, ICLR 2020.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-auu -argreeed generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, NeurIPS 2020.
Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho. Julian Ibarz, Alex Irpan, Eric Jang. Ryan Julian, Dmitry Kalashnikov, Sergey Levine. Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang- Huei Lee, Yuheng Kuang. Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. 2022. Do as I can, not as say: Grounding language in robotic affor- dances. In Conference on Robot Learning, CoRL 2022, volume 205, pages 287-318.
Weixin Liang, James Zou, and Zhou Yu. 2020. ALICE: active learning with contrastive natural language ex- planations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing, EMNLP 2020, pages 4380-4391.
Kangjic Lin, Youjun Xu, Jianfeng Pei, and Luhua Lai. 2020. Automatic retrosynthetic route planning using template-free models. Chemical Science, 11:3355- 3364.
Bowen Liu, Bharath Ramsundar, Prasad Kawthekar, Jade Shi, Joseph Gomes, Quang Luu Nguyen, Stephen Ho, Jack Sloane, Paul Wender, and Vijay Pande. 2017. Retrosynthetic reaction prediction us- ing neural sequennc--o--equuee models. ACS Cen- tral Science, 3(10): 101-
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transuctions on Big Data, 7(3):535-547.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer and Omer Levy. 2020. Span- BERT: Improving pre- tra-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Daniel Mark Lowe. 2012. Extraction of chemical struc- tures and reactions from the literature. Ph.D. thesis, University of Cambridge.
Andrej Karpathy, Armand Joulin, and Li Fei-Fei. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Informa- tion Processing Systems 2014, pages 1889-1897.
Michael R. Maser, Alexander Y. Cui, Serim Ryou, Travis J. DeLano, Yisong Yue, and Sarah E. Reisman. 2021. Multilabel classification models for the predic- tion of cross-coupling reaction conditions. Journal of Chemical Information and Modeling, 616))1166116.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages
Antoine Miech, Jean-Baptisn Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representa- tions from uncurated instructional videos. In 2020 IEFF//VF Conference on Computer Vision and Put- tern Recognition, CVPR 2020, pages 9876-9886.
Youngchun Kwon, Sun Kim, Youn-Suk Choi, and Seokho Kang. 2022. Generative modeling to pre- diet multiple suitable conditions for chemical reac- tions. Journal of Chemical Information and Model- ing. 62(23):5952-53) -5560
Jesse Mu, Percy Liang, and Noah D. Goodman. 2020. Shaping visual representations with language for few- shot classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, pages 4823-4830.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Volume 1: Long Papers, pages 6086-6096.
Stephen Mussmann and Stefano Ermon. 2016. Learning and inference via maximum inner product search. In Proceedings of the 33nd International Conference Machine Learning, ICML 2016, volume 48, pages 2587-2596.

---

Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015. Language understanding for text- based games using deep reinforcement learning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 1-11.
Richard Socher, Andrej Karpathy, Quoe V. Le, Christo- pher D. Manning. and Andrew Y. Ng. 2014. Grounded compositional semanties for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207- 218.
Dat Quoe Nguyen, Zenan Zhai, Hiyori Yoshikawa, Biaoyan Fang. Christian Druckenbroct, Camilo Thomee, Ralph Hoessel, Saber A. Akhondi, Trevor Cohn, Timothy Baldwin, and Karin Verspoor. 2020. Chemu: Named entity recognition and event extrac- tion of chemical reactions from patents. In Advances in Information Retrieval- - 42nd European Conference on IR Research, ECIR 2020, volume 12036, pages 572-579.
Vignesh Ram Somnath, Charlotte Bunne, Connor W. Coley, Andreas Krause, and Regina Barzilay. 2021. Learning graph models for retrosynthesis prediction. In Advances in Neural Information Processing Sys- tems, NeurlPS 2021, pages
Ruoxi Sun, Hanjun Dai, Li Li, Steven Kearnes, and Bo Dai. 2020. Energy- based view of retrosynthesis. arXiv preprint arXiv:2007.13437
Yujie Qian, Jiang Guo, Zhengkai Tu, Connor W. Co- ley, and Regina Barzilay. 2023a. RxnScribe: A se- quence generation model for reaction diagram pars- ing. Journal of Chemical Information and Modeling, 63314434400-
Matthew C Swain and Jacqueline M Cole. 2016. Chem- DataExtractor: toolkit for automated extraction of chemical information from the scientific litera- ture. Journal of Chemical Information and Modeling, 46(10):1894-11)
Yujie Qian, Jiang Guo. Zhengkai Tu, Zhening Li, Connor W. Coley, and Regina Barzilay. 2023b. MolScribe: Robust molecular structure recognition with image-to-grapp generation. Journal of Chemi- cal Information and Modeling, 63(7):1925-2 1934.
Alain C Vaucher, Federico Zipoli, Joppe Geluykens, Vishnu H Nair, Philippe Schwaller, and Teodoro Laino. 2020. Automated extraction of chemical syn- thesis actions from experimental procedures. Nature Communications, 1 116)):361..
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, vol- ume 139, pages
Eric Walker, Joshua Kammeraad, Jonathan Goetz, Michael T. Robo, Ambuj Tewari, and Paul M. Zim- merman. 2019. Learning to predict reaction con- ditions: Relationships between solvent, molecular structure, and catalyst. Journal of Chemical Informa- tion and Modeling. 59(9):36453365,
David Weininger. 1988. SMILES, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of Chemical Infor- mation and Computer Sciences, 28( 1-36.
Serim Ryou, Michael R Maser, Alexander Y Cui, Travis J DeLaano, Yisong Yue, and Sarah E Reisman. 2020. Graph neural networks for the prediction of substrate-specifi organic reaction conditions. arXiv preprint 200705500.
Jason Weston, Samy Bengio, and Nicolas Usunier. 2010. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81:21-35.
Marwin H. S. Segler and Mark P. Waller. 2017. Neural- symbolic machine learning for retrosynthesis and reaction prediction. Chemistry-A European Journal, 23(25):5966-5971
Damian M Wilary and Jacqueline M Cole. 2021. Re- actiondiateettractor: tool for automated extrac- tion of information from chemical reaction schemes. Journal of Chemical Information and Modeling, 6 4124449992444
Marwin HS Segler, Mike Preuss, and Mark P Waller. 2018. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 5556769))600 610.
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. 2020. A graph to graphs framework for retrosynthesis prediction. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, volume 119, pages $818-8821
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of oftee-------r art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020- Demos, pages 38- 45.
Richard Socher and Li Fei-Fei. 2010. Connecting modalities: Semi- supervisd segmentation and an- notation of images using unaligned text corpora. In The Twenty-Thin IEEE Conference Computer Vi- sion and Puttern Recognition, CVPR 2010, pages 966-973.

---

Kevin Yang, Kyle Swanson, Wengong Jin, Connor Co- ley, Philipp Eiden, Hua Gao, Angel Buzman-Pereez Timothy Hopper, Brian Kelley, Miriam Mathea, An- drew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. 2019. Analyzing learned molecular representations for property predic- tion. Journal of Chemical Information and Modeling, 59(8):3370-333 -3388.
Reaction
Nos
SMILES
pccieci http://..ooocccccc//oeeeeeooeoee CCNccec22cc1
To a solution of Teyekgronytmethhnmmne (154 mg. 1.665 mmo)) methanol (10 mL) was added @-chlonber (100 mg. 0.842 mmmo) and few drops acatic acid The reecion stired for minutes then
Gold Text
1,885 mmoo) added. reacton eyanaborohydride days al rt. concentrated vacuo, and quenched with salurated solution sodium bicarbonste.
Zenan Zhai, Dat Quoe Nguyen, Saber A. Akhondi, Camilo Thoree, Christian Druckenbrodt, Trevor Cohn, Michelle Gregory, and Karin Verspoor. 2019. Improving chemical named entity recognition in patents with contextualized word embeddings. In Proceedings the 18th BioNLP Workshop and
Reaction
SMILES
accloccen1 (3 (-) BC02cccc22
(-5 (100
Shared Task, MOONL@@ACI 2019, pages 328-338.
Proparation (8) tert butyl 4-11 (bonayaarii/mmmsshyy/rrsss
[1-tsphithaken 1oy.l/l-/////yy hywpe//lllidddoluulyyloo hywluuiddoouulyyataalllll (5)
Jinhua Zhu, Kehan Wu, Bohan Wang, Yingce Xia, Sh-
To solution of (9) fort butyl 4 (1 :4 tommberry))3:
dichiorocthanee wrr.cc/puuykcattammtttt 4, (4.9 8.8 mmal) mmali,
Iaapthalenn h-yllttt/kuracceeooo
ufang Xie, Qi Meng, Lijun Wu, Tao Qin, Wengang Zhou, Houqiang Li, and Yan Liu. 2023. O-GNN: incorporating ring priors into molecular modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023.
Gold Text
added bebaazylamiin (10 draps) Na(OAc]BBH (3.7 mmol). reaction mixiure stired
C.
mixuure was
cooled to room tomponature and quonched with saturated
aqueous soluton of sodium bicarbonate.
Figure 7: Example reactions and their corresponding gold texts the USPTO data. Reactants and products are marked in yellow and orange while the catalyst, solvent, and reagent are displayed red, green, and blue, respectively.
A Background
A.1 Reaction Condition Recommendation
Reaction condition recommendation the task of suggesting reaction conditions, such as catalysts, solvents, and reagents, for a chemical reaction. Gao et al. 22018) proposed machine learning model for this task, where the reaction is represented as the difference between the product and the reactant fin- gerprint vectors, indicating the change of substruc- tures during the reaction, and the reaction condi- tions are sequentially predicted by the model. Later research studied different methods to represent the reaction, such as MACCS key fingerprints (Walker et al., 2019) and graph neural networks (Ryou et al., 2020), and alternative machine learning formula- tions, such as multilabel classification (Maser et al., 2021) and variational inference (Kwon et al., 2022). In this work, we follow Gao al. formulation but use the more advanced Transformer architecture.
more ambitious goal: predicting the precursors di- rectly without relying on a fixed set of reaction tem- plates. Such approaches are usually implemented as mouph-to-gaaphe or eequence-to-sequeee tion models (Liu et al., 2017; Dai et al., 2019; Shi et al., 2020; Lin al., 2020; Somnath et al., 2021), and achieve comparable performance to template- based approaches on common benchmarks. In this work, we apply TextReact to both template-based and template-free approaches.
genera-
B Datasets
Figure 7 shows two example reactions and their corresponding gold texts in the USPTO corpus.
Reaction Condition Recommendation The dataset is constructed from the public USPTO data (Lowe, 2012).2 9 We create two splits:
A.2 One-Step Retrosynthesis
Another important task predictive chemistry is one-step retrosynthesis, which aims to propose re- action precursors (reactants) for target molecules (products). Prior methods can be broadly classified into two categories: template-based and template- free. Template-busee approaches classification models to predict the reaction template that can be applied the target (Segler and Waller, 2017: Chen and Jung, 2021), and subsequently employ chemin- formatics software to derive the precursors based on the template. Template-frr approaches have a
RCR (RS), where the reactions are randomly split into tio of 80%/10%//10%
with ra-
RCR (TS), where perform time split of the dataset. The reactions collected from patents before 2015 are categorized as the
6084/m9 Sigshare.510483
v1

---

training set, reactions from 2015 as validation, and reactions from 2016 as testing.
RCR (RS)
RetroSyn (RS)
0.6
Top-1 accuracy
One-Step Retrosynthesis We adopt the USPTO 50K dataset (Coley et al., 2017)"0, which is com- monly used in previous works, and create two splits:
0.4
0.5
RetroSyn (RS), i.e. random split, which the original split of the dataset.
1020
1/6 training data
% training data
TexiReac
TextReact
Transformer
(full corpus)
(golb-renoval)
RetroSyn (TS), i.e. time split, where merge the original training, validation, and testing sets and re-split based on the patent year. Data before 2012 are used for training, data from 2012 and 2013 are used for validattoo, and data from 2014 and 2015 are used for testing.
Figure 8: Top-I accuracies of TextReact and the Trans- former baseline vs. amount of training data used, percentage of the total available training data. For RCR, TextReact exhibits strong performance even in low-resource scenarios. For RetroSyn, we evaluate TextReacty in this figure.
Our datasets will be publicly available to foster future research in this direction.
One-Step Retrosynthesis The implementation and hyperparameters for the one-step retrosynthe- sis experiments largely resemble those of the reac- tion condition recommendation experiments, with few differences. The retriever is trained for 400 epochs due to the smaller size of the RetroSyn (i.e.. USPTO 50K) dataset. Each query (the SMILES string of the product molecule) has maximum length of 128.
C Implementation Details
Our experiments are implemented with the Hug- ging Face Transformers (Wolf et al., 2020), Py- Torch Lightning"! Tevatron (Gao et al., 2022), and Faiss (Johnson et al., 2019) libraries.
Reaction Condition Recommendation The SMILES-to-text retriever consists of chemistry encoder and a text encoder. We train the retriever by contrastive learning, where each batch contains 512 queries (chemistry inputs), and each query is associated with one positive paragraph and one ran- dom sampled negative paragraph from the corpus. Both query and paragraph have maximum length of 256. We train the retriever for 50 epochs using learning rate of le (with 10% warmup and linear decay).
For the predictor, the template-free TextReacty and template- TextReacty, architectures are slightly different. While both employ a SciBERT encoder, TextReacty uses a 6-layer Transformer de- coder to generate the output SMILES string, and TextReactæ predicts the reaction template using lin- heads atom and bond representations. In both scenarios, the predictor is trained for 200 epochs. The random sampling ratio in training also set differently (a = 0.2) as mentioned Section
The predictor is trained for 20 epochs using batch size of 128 and a learning rate of 1c (with 2% warmup and cosine decay). The weight of the MLM loss is A 0.1. We apply data aug- mentation by generating the SMILES strings with a random order during training. By default, we append the chemistry input with k = 3 neighbor- ing text paragraphs, and maximum length of the encoder input is 512. The input SMILES string and text are tokenized with the same SciBERT to- kenizer. For the experiments with Ao 1, 4, 5, we set the maximum length to 256, 384, 768, 1024, respectively.
D Further Analysis
To investigate the effect that the amount of train- ing data has on model performance, we trained TextReact in both the RCR and RetroSyn on 10%, 20%, 50%, and 100% of the available training data under the random split. The same was done with the Transformer baseline model for comparison. The models are evaluated on the full test dataset and the top-1 accuracies As expected, both TextReact and the Transformer baseline see a noticeable improvement in perfor- mance as the amount of training data is increased. In the RCR domain, the full-corpus accuracy and
plotted in Figure 8.
1htt tetttp::/gtthhn thub. wtw. htmm//coleygroppppeeettrrr/////e /main/data/USPTO.. htttps: / http://wwwwpptcm http://wwpwwohlltcooo inghninggtninngggg aairrg...ii/i ai ai/index.. html

---

of the reactions corresponding each pair of re- trieved texts (right panel of Figure 9). This mea- sures how different the retrieved neighbors are from each other. The graph shows that TextReact's per- formance does not depend much on how similar the retrieved neighbors are to cach other, whereas the performance of TextReact (sep. neighbors) drops when the retrieved neighbors are more different from each other. When separately encoding the neighbors, the model does know how much should trust each neighbor. On the other hand, TextReact does not suffer from the same perfor- mance drop, likely due to the attentions between the retrieved texts, which allow the model to better integrate the information from the neighbors. This demonstrates the benefits of concatenating the in- put reaction and all the retrieved texts together and feeding them into single Transformer encoder.
Top- accurney
400
Input-neighbor distance
Distance between neighbors
TextResct
TextReact (sep. neightsns)
Figure 9: Top-1 accuracies of TextReact and TextReact (sep. neighbors) RCR (RS) when retrieving from the full corpus, plotted against the average input-ncighbe distance and the average distance between neighbors.
RCR (RS)
RetroSyn (RS)
Top-1 Accuracy
160
68.4
05.6
85.4
35.3
68.5
77.8
40.4
972
17.0
17.0
15.7
47.7
40.5
Figure 10 analyzes the number of text para- graphs h that we retrieve for each example. the RCR (RS) dataset, there is a notable improve- ment from k: = to = 3, suggesting the benefits of retrieving more neighbors as additional context. In the RetroSyn (RS) dataset, the trend is similar but the improvement is smaller. However, the per- formance decreases when k: 3 for both datasets. Therefore, set l for our main experiments.
2
3
3
5
corpus gomdvoo
Figure 10: Performance TextReact with respect to the number of neighbors k retrieved for each input. We evaluate TextReacty for RetroSyn.
gold-removed accuracy of TextReact when trained on 10% of the available training data are 67.7% and 35.0%, respectively, both of which are higher than the accuracy of the baseline as trained on the full training set (30.0%). This shows that our model is capable of achieving better performance with only 10% much training data for reaction condition recommendation.
To study the effect that the retrieved texts have on model performance, we plotted the top-1 accu- racy of TextReact and TextReact (sep. neighbors) with respect to the average neigh-ne distance, defined as the average L2 distance between the fin- gerprint of the input reaction and the fingerprints of the reactions corresponding the retrieved texts (left panel of Figure 9). This distance measures the difference between the input reaction and the reactions described by the retrieved texts. The plot shows that model predictions are more accurate when the retrieved texts correspond to reactions more similar to the input reaction, thus confirming that TextReact effectively incorporates information from the retrieved texts when making predictions.
We also plotted the accuracy with respect to the average distance between neighbors, defined as the average L2 distance between the fingerprints


Reviews:
[{"review_id": "aUpPgoL8XW", "reviewer": "Reviewer_EhYp", "summary": "", "questions": "A. On a high-level, I wonder how useful the text augmentation is in practice? (With my limited background in chemistry), I would assume that these models are mainly used to predict previously unknown outcomes. Yet anything described in the literature can be hardly considered unknown. As you already explored a setting where the \"gold texts\" are removed, I assume the advantage then stems from finding the known effects of e.g. components in the literature?\n\nB. Are there really no available baselines that employ text retrieval, or perform the same tasks with text only?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their valuable comments. Below, we provide answers to the reviewer's questions.\n\n**Q1. The influence of SMILES input vs. retrieved text input.**\n\nThank you to the reviewer for this valuable suggestion! In response, we have conducted an additional ablation study using a model that solely utilizes the retrieved text as input. The results are presented below, displaying the Top-1 accuracy in %. (RCR: reaction condition recommendation. RetroSyn: retrosynthesis.)\n\n| |   RCR (RS)  | RetroSyn (RS) |\n|-| :---------: | :------------: |\n| TextReact |   88.4   |   59.5   |\n| Text-only |  81.7  |  23.0  |\n| SMILES-only | 30.0 | 45.9 |\n\nWe observe that the text-only model performs less effectively than TextReact on both tasks, illustrating that TextReact effectively utilizes both chemistry and text inputs to generate its predictions. We also note an intriguing distinction when comparing the text-only and SMILES-only models. The former demonstrates better performance in RCR, likely due to the explicit textual expression of reaction conditions in retrieved text. Conversely, the latter outperforms in RetroSyn, showing that text information offers less utility in this context.\n\n**Q2. How useful is text augmentation in practice? I would assume these models are mainly used to predict previously unknown outcomes.**\n\nAs the reviewer pointed out, it is important to evaluate predictive chemistry models on previously unknown outcomes, i.e., those not covered in existing literature. We have two ways to validate that TextReact generalizes in such scenarios: evaluation on unknown reactions (“gold-removed” setup in Figure 5), and a time-based split (Table 3, 4 and Figure 5). These results demonstrate that our model does not just extract answers from text, but leverages similar reaction data to enhance its prediction.\n\nThe generalization capability of TextReact can be attributed to two reasons: (1) chemically similar reactions have similar conditions and outcomes; (2) effective grounding enabled by jointly modeling text and chemistry formulas. In practice, TextReact can predict unknown outcomes by retrieving similar reactions as additional evidence. Moreover, as new publications emerge, TextReact can seamlessly retrieve raw texts from these new publications to improve the prediction on newly-discovered reactions (as demonstrated by our time-split experiments).\n\nWe will definitely include more discussions about how TextReact could be used in practice and its benefits, as suggested by the reviewer.\n\n**Q3. Are there really no baselines that employ text retrieval, or perform the same tasks with text only?**\n\nChemistry prediction tasks, like reaction condition recommendation and one-step retrosynthesis, have traditionally been studied through pure chemistry-based models, e.g., encoding molecules and reactions with neural networks to facilitate predictions. To the best of our knowledge, there is no previous work leveraging text retrieval for these specific tasks. Similarly, we are not aware of text-only models for these tasks as the input is a chemical reaction, not text. However, we trained another model with retrieved text only as additional ablation study, as detailed in our response to Q1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments provide strong support for their claim, achieving more than twice as high accuracy compared to the best baseline ChemBERTa.\n\nThe paper further considers two different types of train/test splits (random and time-based). The advantage of the proposed method is confirmed in both scenarios.\n\nClaims are further supported through careful ablations, such as testing a setting where the gold texts are removed and a setting where only the historical corpus is used in the time split setting. The method still shows a solid advantage over a Transformer baseline (which was ~on par with ChemBERTa in other experiments).", "reasons_to_reject": "The paper could be further improved by analyzing the influence of SMILES input vs. retrieved text input. Given the comparison with the literature that only uses SMILES input, it seems that retrieved text provides an extremely rich source of information. It leads to the question how a model would perform only based on retrieved text, if feasible.\n\nThe paper would benefit from a discussion of how useful such models are in practice (see Question).\n", "questions_to_authors": "A. On a high-level, I wonder how useful the text augmentation is in practice? (With my limited background in chemistry), I would assume that these models are mainly used to predict previously unknown outcomes. Yet anything described in the literature can be hardly considered unknown. As you already explored a setting where the \"gold texts\" are removed, I assume the advantage then stems from finding the known effects of e.g. components in the literature?\n\nB. Are there really no available baselines that employ text retrieval, or perform the same tasks with text only?", "paper_topic_and_main_contributions": "The paper proposes to make use of textual side information in machine learning models for predicting the outcome of chemical reactions. Textual side information is retrieved from chemical literature on the basis of the specifications of the chemical species (SMILES). This is done by learning a joint embedding space and kNN retrieval in the spirit of the popular image/text model CLIP. Then, both the original SMILES description and the retrieved natural language data are used together to make a prediction.\n\nThe main claim is that this natural-language-augmentation improves the performance on various chemical prediction tasks, such as condition recommendation and one-step retrosynthesis.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- define the acronym SMILES\n- Introduction \"we simulate novel inputs by eliminating from the training data the closest textual descriptions for given reactions.\" could already be more clear about whether this is training strategy or an evaluation strategy."}}, {"review_id": "8wFPGhHLj7", "reviewer": "Reviewer_zvNB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their valuable comments. Below, we provide answers to the reviewer's questions.\n\n**Q1. The paper does not study any additional methods but bi-encoders.**\n\nOur method is designed to address a multimodal retrieval task, i.e., retrieving texts related to a given chemical reaction. Bi-encoder (termed as dual encoder in our paper) is a commonly adopted model in this context. As discussed in our related work section, our approach is similar to CLIP, a bi-encoder capable of retrieval between images and texts. \n\nRegarding the two methods mentioned by the reviewer,\n- Cross-encoders are not ideally suited for retrieval tasks; instead, they are usually used for reranking a limited number of candidates. As a cross-encoder must jointly encode the query and each document, it is computationally intractable for large corpora. In contrast, bi-encoders are better suited as the encodings of documents can be pre-computed separately. Given the strong performance achieved by our bi-encoder retriever (see Table 2), we chose not to train another cross-encoder to rerank the retrieved candidates.\n- Sparse retrievers such as BM25 excel in text-to-text retrieval tasks, but cannot be trivially applied to our cross-modal SMILES-to-text retrieval task. It is because the two modalities do not have a shared vocabulary. Sparse retrievers rely on token matching, a simple and effective inductive bias for measuring similarity in pure text domains. However, when the query is a SMILES string and the document is natural language text, token matching does not work well because they do not share the same vocabulary. Thus, we did not adopt a sparse retriever.\n\n**Q2. It is unclear how the model performs in more difficult molecular reactions where there can be one, many, or no catalysts or solvents.**\n\nWe would like to clarify that our reaction condition recommendation task has considered reactions with one, many or no catalysts or solvents. Specifically, we follow the exact setup of previous research (Gao et al., 2018) to construct a reaction dataset with at most one catalyst, two solvents, and two reagents (section 5.1), which covers reactions without catalyst, solvent, or reagent. The maximum number of catalysts/solvents/reagents is determined based on the majority of the USPTO reaction data (the same as Gao et al., 2018). Our predictor generates catalysts/solvents/reagents in a sequence, and is able to handle a varied number of conditions. For example, when a reaction doesn’t have a catalyst, the predictor generates a “None” for that component.\n\n**Q3. Retrieval metrics are missing such as recall@100**\n\nOur retrieval metrics are reported in Table 2, including recall@{1,3,10}. However, recall@100, as specifically mentioned by the reviewer, is not included here since we do not seek to retrieve that many texts for a given reaction. Our main experiments use 3 retrieved paragraphs as additional input. As shown in the additional analysis in Appendix D Figure 10, we have conducted experiments using at most 5 retrieved paragraphs, and retrieving more texts is not always helpful.\n\n**Q4. Missing reference**\n\nWe thank the reviewer for pointing out the missing reference to Text2Mol, which retrieves molecules using natural language queries. Our retriever operates in the opposite direction, retrieving natural language descriptions using molecules, and we further use the retrieved text to enhance chemistry prediction tasks. Both papers contribute to the application of NLP techniques in chemistry. We will ensure to cite Text2Mol in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. TextReact allows for accurate prediction of reaction properties with minimal overhead.\n2. This work will likely inspire novel research into retrieval augmented reaction prediction. ", "reasons_to_reject": "1. The paper does not study any additional methods but bi-encoders. The contribution would significantly be improved by integrating a cross-encoder or sparse retriever like BM25.\n2. It is unclear how the model performs in more difficult molecular reactions where there can be one, many or no catalysts or solvents\n3. Retrieval metrics are missing such as recall@100", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces TextReact, a retrieval-based solution that predicts the solvents, reagents, and catalysts of chemical reactions by retrieving relevant text portions from a Patent dataset. \n\nTextReact formulates the task as a bi-encoder-based retrieval from text given a multi-model molecular representation. To Motivate their work the authors generate a novel chemical retrieval benchmark and showcase the utility of their model. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries"}, "extra_text": {}}, {"review_id": "wSedisjIVh", "reviewer": "Reviewer_3bHy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their valuable comments. Below, we provide answers to the reviewer's questions.\n\n**Q1. State-of-the-art models were not compared in the experiments.**\n\nFirst, we would like to thank the reviewer for pointing out additional models that were absent from our current results. Table 3 for reaction condition recommendation has covered state-of-the-art models.  We will extend Table 4 to include additional models, and also cite and discuss relevant papers. The following table shows the expanded results on RetrySyn (RS), i.e., the USPTO 50K dataset.\n\n| | Template-Free | Top-1 | Top-3 | Top-5 | Top-10 |\n|--|--|--|--|--|--|\n| G2G (Shi et al., 2020) | Yes | 48.9 | 67.6 | 72.5 | 75.5 |\n| Transformer (Lin et al., 2020) | Yes | 43.1 | 64.6 | 71.8 | 68.7 |\n| Transformers (ours) | Yes | 45.9 | 63.2 | 68.5 | 75.2 |\n| Dual-TF | Yes | 53.6 | 70.7 | 74.6 | 77.0 |\n| LocalRetro | No | 53.4 | 77.5 | 85.9 | 92.4 |\n| O-GNN | No | 54.1 | 77.7 | 86.0 | 92.5 |\n| TextReact | Yes | 59.5 | 72.6 | 75.9 | 80.0 |\n\nWe can see that TextReact, as reported in the paper, outperforms state-of-the-art template-free models, and also outperforms state-of-the-art template-based models in terms of top-1 accuracy. Template-based models which were excluded from the current paper adopt a different pipeline. They first predict the reaction templates that can be applied to synthesize the product, and then use rule-based cheminformatics tools to compute the reactant SMILES (see Appendix A). These models can achieve better accuracy but rely on additional resources, such as reaction templates and cheminformatics tools. Besides, they are fundamentally incapable of generalizing to new reaction templates [1] [2]. \n\nThe main purpose of our paper is to demonstrate how text retrieval can augment different chemistry prediction tasks, rather than aiming to surpass the state-of-the-art on one particular task. Thus we did not include template-based models in our previous experiment (see lines 458-470).\nOur ongoing work is investigating retrieval augmentation within a template-based approach as well, and we will include further discussion in the paper. Nevertheless, we remain confident that our experiments have adequately demonstrated the advantages of integrating text retrieval into chemistry tasks. We hope our response has addressed the reviewer's concerns.\n\nFinally, we will release our code and data to enable reproduction of the experiments and to support future research.\n\n[1] Lin, Kangjie, et al. \"Automatic retrosynthetic route planning using template-free models.\" Chemical science (2020).\n\n[2] Tu, Zhengkai, et al. \"Predictive chemistry: machine learning for reaction deployment, reaction development, and reaction discovery.\" Chemical Science (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow and is nicely read in most parts.\n\n2. The proposed retriever may benefit other practitioners and researchers in the field of chemistry if it is open-sourced.", "reasons_to_reject": "1. The effectiveness of the proposed model has not been sufficiently demonstrated because state-of-the-art models were not compared in the experiments. In the one-step retrosynthesis task, while several previous models exhibit superior performance than the performance of the proposed model, they were not considered in the conducted experiments (Table 4). Please review previous studies [1,2] and this link [3] to ensure that essential baselines have not been overlooked.\n\nIf the authors feel that it is unfair to compare the previous best models I have mentioned with the proposed model (i.e., TextReact), please reply during the author response period. I will change the initial score if the responses resolve my concerns.\n\n[1] Zhu et al., O -GNN: incorporating ring priors into molecular modeling, ICLR 2023\n\n[2] Seidl et al., Improving few-and zero-shot reaction template prediction using modern hopfield networks, Journal of chemical information and modeling, 2022\n\n[3] https://paperswithcode.com/sota/single-step-retrosynthesis-on-uspto-50k", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new retrieval-augmented approach in the chemistry domain. Specifically, the proposed model retrieves relevant paragraphs to a given input SMILES string from a patent corpus, USPTO. A cross-encoder then encodes textual and chemical data and a decoder predicts the target SMILES string. The proposed model is evaluated on two downstream tasks, reaction condition recommendation and one-step retrosynthesis, and outperforms simple single-modal baselines that use only SMILES strings as features.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors describe a methodology, TextReact, for including retrieved relevant textual information with respect to a particular chemistry input (e.g., molecular encodings)for predicting chemical reaction tasks (e.g., reaction condition recommendation, one-step retrosynthesis) using machine learned models . Specifically, they use a SMILES-to-text retriever to retrieve relevant text, which is then also encoded (with the chemistry input) to be used as input to a machine learned model -- in this case an encoder-decoder model. Experiments are conducted  using a patent corpus as the auxiliary text source for the reaction condition recommendation and one-step retrosynthesis tasks, showing strong performance over existing baseline methods and a transformer network without text augmentation.

== Quality == The reviewers agreed that the paper was well-written overall, that the methodological approach is well-motivated in considering recent NLP advances, and that the experimental results follow a rigorous procedure and demonstrate strong performance. In combination with the primary results, the the ablation studies further confirm the results, making for a sufficiently convincing case for TextReact as a useful method. The two concerns in this regard were that reviewer EhYp had a question regarding 'retrieved text-only' and reviewer 3bHy pointed out that the experiments aren't universally SotA for both tasks depending on the configuration. However, the authors rebutted this well -- some of these results and discussion which should likely be included in the final version of the paper as they are valid questions.

== Clarity == As previously stated, the paper is well-written overall, especially considering that this isn't a chemistry knowledgeable community (in general). In particular, the figures are effective in giving an overview and providing a scaffolding for the structure of the document. The one comment made is that some qualitative analysis would be useful to given an intuitive perspective regarding characterizing the utility of the text-augmentation in specific settings.

== Originality == As far as the reviewers and I can tell, this is the first use of (text) retrieval-augmented chemistry related predictions. It is a relatively simple augmentation approach shown to work well in practice for two important tasks.

== Significance = First of all, the empirical results are quite promising and convincing -- validating the method and making chemistry contributions in some cases. As RAG-style systems have gained increasing popularity, it isn't unreasonable to believe that this may also be able to be integrated into general LLMs and this work code be a basis for such integration. Overall, this is a good case of applying NLP technology to other potentially impactful use cases. As the code is promised to be released, it is also lowering the barrier for others to continue work in this direction.
Decision: Accept-Main