SlowGuess/NeurIPS_2023_OCR huggingface dataset:
radical mechanistic reactions. In particular, we adopt the pioneering text-based reaction predictor, Molecular Transformer [28], which utilizes a bidirectional encoder and autoregressive decoder with a fully connected network for generating probability distributions over possible tokens. The pre-trained Molecular Transformers were trained using different variations of the USPTO dataset [16]. During training the encoder computes a contextual vector representation of the reactants by performing self-attention on the masked and randomly augmented (non-canonicalized) SMILES string of the reactant molecules. The decoder then uses the encoder output and the right-shifted SMILES string of the products to autoregressively generate the product tokens. Since the radical reactions in RMechDB are not labeled with reactants and reagents, we used the model which was pre-trained using the USPTO/MIT_mixed dataset [16; 48; 49].
4.5 Fine-tuning Using RMechDB
Molecular Transformer enables the fine-tuning of pre-trained models for downstream tasks like radical reaction prediction. In our approach, we utilize pre-trained models and conduct reactant-to-product sequence translation. During the fine-tuning process, our only augmentation technique involved rearranging the reactant molecules within the SMILES string. Specifically, for each reaction containing  $N$  reactant molecules, we employed  $N$  SMILES strings with reactants randomly reordered. We removed all the atom mappings and arrow codes from the RMechDB training data and fine-tuned the model using the augmented training set of the RMechDB.
Table 4 shows the performance of the text-based prediction for both pre-trained and fine-tuned versions of the Molecular Transformer. We also include detailed information on training and fine-tuning parameters, as well as tokenization statistics.
5 Results and Discussion
5.1 Performance on RMechDB
We assess the performance of the two-step prediction method, comprising reactive sites identification and plausibility ranking. The top  $N$  accuracy of the reactive sites identification on the combined test datasets of RMechDB are presented in Table 2. We observe that GNN models outperform the method based on the atom descriptor (predefined feature extraction). This behavior is expected as the atom descriptor is limited to a certain radius around the atom (in this case the radius is set to three). However, the number of GNN layers can be optimized to construct the most informative atom representations. The advantage of GNNs is more evident for the atmospheric test data where there are usually more molecules present in the context of the reaction.
The second model of the two-step prediction is the Siamese architecture that ranks the reactions based on their chemical plausibility. Four different architectures were used for different reaction representations: Pre-defined feature vectors from [20], reactionFP, DRFP, and the rxnfp. The results of the top  $N$  accuracy for the combined test sets of RMechDB are shown in Table 3. The table shows that DRFP outperforms the reactionFP and the feature extraction methods. We believe the reason is
mainly because of the different nature and underlying chemistry of the radical mechanistic reaction and the USPTO reaction, which was used to pre-train the rxnfp.
Table 2: The performance of different methods for reactive sites identification. Each number represents the percentage of reactions for which both reactive atoms are identified within the top  $N$  predictions.
Method	Top2	Top3	Top5	Top10
Atom Descriptor	75.1	81.5	89.3	96.7
GNN	76.9	83.6	92.1	97.9
To predict the outcome of a mechanistic radical reaction using the two-step method, we must perform both reactive sites identification and reaction ranking. Given the performance of each individual predictor, the best combination would be the GNN and DRFP. Therefore in Table 4, we use this combination to compare the performance of the other two methods with the two-step prediction.
Table 3: The performance of different methods for plausibility ranking. Each number represents the percentage of reactions for which the correct mechanism is predicted in the top  $N$  plausible reactions.
Method	Top1	Top3	Top5	Top10
Pre-defined Feature Vectors	73.1	79.2	88.3	96.3
reactionFP	AP	74.6	82.3	90.2	97.8
Morgan2	74.3	81.9	90.0	97.3
TT	74.3	82.4	90.0	97.8
DRFP	78.6	90.2	95.1	100.0
rxnfp (pretrained)	75.9	86.2	94.3	97.9
As it is shown in Table 4, the contrastive learning approach yields the most accurate prediction across all the metrics. In terms of inference time, the contrastive learning approach is faster than the two-step prediction as it only consists of one neural network. This becomes crucial in the case of pathway search where we exponentially expand the tree of the mechanistic pathways by predictions. However, the advantage of using the two-step method becomes more evident when the size of the reactant molecules increases. In Figure 3, we show the performance of all three methods based on the number of heavy atoms on the reactant sides. As expected, the prediction of both the contrastive method and text-based models becomes more faulty when big molecules react. This can be explained using the fact that the reactive sites identification part of the two-step method is mainly dependent on a local neighborhood of atoms and is not significantly affected by the size of the molecules.
The text-based models are outperformed by the other two graph-based methods. Although they offer faster inference times, they yield less accurate predictions. This poor performance is mainly because Molecular Transformer model is trained on the USPTO/MIT_mixed dataset. Therefore, it learned to predict the only major product of overall transformations that are mostly polar. On the contrary, RMechDB data are balanced, mechanistic, and involve radical species. Align with this low performance, according to Table 4, fine-tuning on the relatively small RMechDB dataset results in a slight decrease in the performance. This can be attributed to the dissimilarities between RMechDB and the USPTO dataset. RMechDB comprises intermediate products of transformations and includes radical reactions with compounds not found in the USPTO dataset. During the tokenization process of the RMechDB reactions, new tokens emerge that have not been extracted from the USPTO dataset.
We also provided a separate performance of the predictors with respect to different radical reaction categories provided in RMechDB.
5.2 Pathway Search
After successfully predicting the outcomes of mechanistic radical reactions, we can chain these predictions to construct mechanistic pathways. Starting from a set of reactant molecules, we perform a series of predictions by using each of the top  $N$  predicted products as the reactants for the next prediction. This would form a tree structure with the starting reactants as the root. This tree can be expanded to a desired depth, representing numerous mechanistic pathways leading to different
Table 4: top  $N$  accuracy of all the three reaction prediction models on Core and Atmospheric test sets of RMechDB. For the text-based models, a prediction is considered to be correct, when at least one of the non-spectator product molecules is predicted correctly. For the text-based models, we use (p) and (f) for pre-trained, and fine-tuned models respectively.
Model	Variant	Core	Atmospheric	Time
top N	top N
1	2	5	10	1	2	5	10
Two-step	Best Combination	62.4	71.9	93.2	97.2	60.4	70.9	91.6	96.3	1.38
Contrastive	Atom Descriptor	62.9	73.8	94.2	96.5	61.0	73.6	93.0	94.4	0.08
RxnHypergraph	64.3	74.1	95.1	97.4	62.1	74.8	94.1	95.9	1.45
Text-based	Pretrained	58.2	64.3	84.2	91.0	58.0	67.3	82.6	91.0	1.30
Fine-tuned	57.7	64.0	83.9	90.4	57.1	66.8	82.2	90.3	1.30
Figure 3: Left: The number of recovered reactions in the top 5 for different methods and different classes of RMechDB reactions. Right: The recovery rate of different methods with respect to the size of the reactants.
products. These pathways are crucial for identifying synthetic routes, exploring intermediate products, and aiding in mass spectrometry analysis with broad applications to various fields, including drug design and drug degradation.
We create a dataset consisting of 100 radical reactants paired with target molecules that are expected to be formed from the reaction. Each pair is accompanied by a specified depth, indicating the length of the mechanistic chain reactions required to reach the target. We generate this dataset by simulating atmospheric conditions, taking inspiration from the atmospheric reactions observed in the RMechDB dataset and atmospheric literature [50]. The reactants in our dataset primarily consist of Isoprene, a prevalent atmospheric compound, along with other atmospheric molecules like radical Oxygen and Hydroxyl radical. To find these targets, we expand and search the tree of the mechanistic pathways by employing a breadth-first search algorithm [51]. To expand the tree for each of the 100 reactants, we use the top 10 predictions at each step (i.e., the breadth of 10 to expand the tree), and we expand the tree up to the given depth. Given the average inference time of the predictive models presented in Table 4, we used the fastest method with interpretable predictions which is the contrastive model with atom descriptor. Upon running this pathway search for these 100 pathways, we observed a significant recovery rate of  $60\%$  meaning that for the  $60\%$  of the reactants, the given target was found within the expanded tree. These pathways along with their targets and detailed information on the pathway search are presented in the Appendix.
5.3 RMechRP: Online Reaction Predictor
We developed RMechRP as an online tool for mechanistic radical reaction prediction. RMechRP, which stands for Radical Mechanistic Reaction Predictor, is accessible at http://deeprxn.ics.uci.edu/rmechrp/. It offers two interfaces: (1) Single-step Predictor; and (2) Pathway Predictor. These interfaces are equipped with the highest performing combination of the two-step prediction
models and the contrastive learning model respectively to offer the best blend of interpretability, accuracy, and fast inference. The Single-Step Predictor interface, accessible at http://deeprxn. ics.uci.edu/rmechrp/singlestep, provides users with the capability to input a set of reactant molecules. By specifying a few parameters, such as the number of reactive atoms, the system utilizes the best combination of the two-step prediction model described above to identify all possible radical mechanistic steps. Then all the predicted reactions will be ranked and displayed with side information, including arrow codes, SMIRKS, reactive orbitals, and plausibility scores. The Pathway Predictor interface, accessible at http://deeprxn.ics.uci.edu/rmechrp/pathway, is designed for searching a specific target molecule(s) within the tree of mechanistic pathways rooted at the input reactants. Users have the option to input a set of reactants and specify a set of target molecules, which can be provided in the form of molecular structures or molecular masses. Parameters such as the depth and breadth of the mechanistic pathway tree can be configured. The system then employs the contrastive learning model described above to expand and explore the mechanistic pathway tree. Upon finding the target molecule within the expanded tree, the system will present synthetic pathways from the initial reactants to the target molecule or molecules. More details on both interfaces and how to use them are presented in the Appendix.
We developed and compared multiple radical reaction predictors operating at the mechanistic level using the RMechDB database of radical elementary reaction as the source of training and evaluation data. We show that the contrastive learning approach incorporated with the graph representation of chemical structures would yield the highest accuracy. The models that merely build upon the text representation of chemical structures require a larger dataset to achieve the same level of accuracy. We show that our approach to reaction prediction is capable of providing chemical interpretability down to the interaction of molecular orbitals. It also offers pathway interpretability by breaking down an overall transformation into balanced pathways of elementary steps. Considering these chemically balanced pathways would identify interesting byproducts that can potentially give rise to pathways that are usually ignored in representing overall transformation. Additionally, we developed online interfaces for performing single-step and pathway prediction of radical reactions. Our radical reaction predictor is publicly available through an online interface at http://deeprxn.ics.uci.edu/rmechrp.
Finally, it is important to identify the limitations of the current method and note that the radical reaction predictor can be further extended in several aspects. First, although RMechDB provides high-quality mechanistic reaction data, it only includes 5500 mechanistic reactions. Developing purely data-driven machine learning models such as LLMs, with minimum dependency on physical and chemical information requires much larger datasets. Additionally, RMechDB is focused on textbook reactions and atmospheric reactions. This focus may restrict the applicability of the proposed predictor to other types of radical reactions, such as polymerization processes. In the context of pathway search and the expansion of the search tree, we employed a straightforward breadth-first search algorithm. This algorithm allows us to expand and search the entire tree of pathways, however, the exponentially growing computations would impede the search process. An interesting improvement could involve the adoption of more sophisticated search methods, like those mentioned in [52], which are not only faster but also capable of exploring a significant portion of the search space.
[1] Aaron Mood, Mohammadamin Tavakoli, Eugene Gutman, Dora Kadish, Pierre Baldi, and David L Van Vranken. Methyl anion affinities of the canonical organic functional groups. The Journal of Organic Chemistry, 85(6):4096-4102, 2020.
[2] Dora Kadish, Aaron D Mood, Mohammadamin Tavakoli, Eugene S Gutman, Pierre Baldi, and David L Van Vranken. Methyl cation affinities of canonical organic functional groups. The Journal of Organic Chemistry, 2021.
[3] J. Chen and P. Baldi. Synthesis explorer: Organic chemistry tutorial system for multi-step synthesis design and reaction prediction. Journal of Chemical Education, 85(12):1699-1703, 2008.
In this appendix, we provide a comprehensive description of the experimental details and environments in which the experiments were conducted. Additionally, we present detailed information and data pertaining to the pathway search. Furthermore, we offer an explanation of the various interfaces of the RMechRP software, which serves as the pioneering online radical reaction predictor. Each section in this appendix corresponds to the section with the same title in the main article. Finally, all the experiments are conducted using a single NVidia Titan X GPU.
7.1 Reactive Sites Identification
For the Atom Fingerprint model, we constructed a fingerprint of length 800 for each atom. This fingerprint includes 700 graph topological features explained in [20] and 85 atomic features including a one-hot vector for atom type, and chemical features of the atoms such as valance and electronegativity. The graph topological features are extracted using a neighborhood of size three. The extracted fingerprints are fed into a fully connected model with an output layer for binary classification. For the GNN model, we used the atomic feature for the initial representations of atoms. The model consists of four GNN layers with an output layer for binary classification.
Combining both training sets presented in RMechDB [25], we extracted over 51000 atoms to train each of the models above. Both models are evaluated using a combination of two test sets in RMechDB and the top  $N$  accuracy of models is reported in Table 2 of the main article. Table 5 represents the parameters used for training the models.
Table 5: The parameters used for training the models for reactive sites identification.
Model	Batch Size	Num Layers	Layers Dim	Act	Reg	Num Att Heads
Atom Fingerprint	32	3	512-256-1	GELU	L2(5e-5)	-
GNN	32	4	64-64-64-1	ReLU	Dropout (0.3)	6
7.2 Plausibility Ranking
For the plausibility ranking experiments, we used the following four methods for representing chemical reactions:
Feature Extraction: We use the same features explained in [20] which results in extracting a vector of length 3200 for each reaction.
reactionfp: We use the RDKit [53] implementation of reactionfp [33]. For all three fingerprint types (Atom Pair, Morgan2, and Topological Torsions), we use a fingerprint of size 2048, with a bit ratio of 0.2. We considered nonagent molecules with a weight of 0.4 and agent molecules with a weight of 1.0.
DRFP: We use the DRFP fingerprint [37] with a size of 2048 with a min and max radius of zero and four, while including the hydrogen atoms and rings.
rxnfp: We use the default tokenizer and pretrained model for the rxnfp [38] which results in fingerprints of length 256.
For training, we use a combination of both training sets in RMechDB. For each sample of the training data (productive reaction), we generate (at most) 40 negative samples (unproductive reactions) by randomly sampling molecular orbitals other than the reactive MOs  $(m_1^*, m_2^*)$ . This results in a data set of over 185000 pairs of productive and unproductive reactions. To train the plausibility rankers for each method, we use the parameters explained in Table 6.
7.3 Atom Pairs and Atom Descriptor
For the contrastive learning method using atom descriptors, we use the same atomic feature and graph topological features above to represent one single atom. Specifically, for the graph topological features, we use the neighborhood of size one. These features plus the atomic features result in
Table 6: The parameters used for training the models for the plausibility ranking.
Model	Batch Size	Num Layers	Layers Dim	Act	Reg
Feature Extraction	32	3	512-256-1	GELU	Dropout (0.5)
reactionfp	32	3	400-200-1	GELU	Dropout (0.5)
DRFP	32	3	400-200-1	GELU	Dropout (0.5)
rxnfp	64	2	128-1	GELU	Dropout (0.5)
a vector of length 140 for atom representation. Using these vectors, we train a contrastive model depicted in Figure 2 (left) of the main article. The objective function to train this contrastive model is as follows:
$$
\mathcal {L} = 1 - \sigma \left(\left[ f \left(a _ {1} ^ {*}\right) \times g \left(a _ {2} ^ {*}\right) \right] - \left[ f \left(a _ {1} ^ {\prime}\right) \times g \left(a _ {2} ^ {\prime}\right) \right]\right) \tag {7}
$$
$$
\operatorname {S c o r e} = \sigma ([ f (a _ {1}) \times g (a _ {2}) ]) \tag {8}
$$
Where  $a_1^*$  and  $a_2^*$  are the atoms of the reactive MOs  $m_1^*$  and  $m_2^*$ , while  $a_i'$  are randomly chosen atoms. Both  $f$  and  $g$  functions are characterized by a fully connected neural network. The first reactive atoms in both productive and unproductive reactions are fed through the same network  $f$ , and similarly, the second reactive atoms are fed through the same network  $g$ . The outputs of both  $f$  and  $g$  are single real-valued numbers, which, when multiplied together, yield a score for the respective reaction. These scores are then utilized to construct the objective function, aiming to maximize the score of the productive reaction compared to the unproductive reactions using the same reactant set. Once the model is trained, the normalized score (Equation 8) can be interpreted as the reactivity of an atom pair.
We use a combination of both training sets in RMechDB to train  $f$  and  $g$ . For each productive reaction, we form unproductive reactions by considering at most 15 samples of  $(a_1', a_2')$ ,  $(a_1', a_2')$ , and  $(a_1', a_2')$ . This negative sampling results in a dataset of over 200000 pairs of productive and unproductive atom pairs. We use this training dataset to minimize the objective function 7.
Both  $f$  and  $g$  have similar architectures that consist of three fully connected layers with a GELU activation function and a dropout with a rate of 0.5 applied to all layers. The dimensions of the layers are 128, 64, 1.
7.4 Rxn-Hypergraph
We use the Rxn-Hypergraph to replace form atom descriptors that are extracted automatically for minimizing the objective function 7. After processing the Rxn-hypergraph for N layers, the generated atom descriptors are used in the same setting above for the same minimization objective. Here in Table 7 we describe the parameters we use for training the Rxn-Hypergraph.
7.5 Text Representation and Sequence to Sequence Models
In order to develop a text-based radical reaction predictor, we utilize the pre-trained molecular transformer which was trained using the USPTO/MIT_mixed dataset. We also used the tokenizer developed by the molecular transformer. This tokenizer yields 523 distinct tokens for the USPTO/MIT_mixed dataset. There are nine tokens from the RMechDB dataset that do not match the 573 tokens of the USPTO. Therefore, we used the "unknown token" to represent these nine tokens.
For fine-tuning the pre-trained model, we used the combination of both RMechDB training sets. We fine-tune the model using a simple data augmentation described in Section 4.5 for 10 epochs. Finally, for the evaluation of the text-based models, we considered all the generated "unknown token" as correct tokens.
Table 7: The parameters used for training the Rxn-Hypergraph for the contrastive model.
Batch Size 32	Num Layers 5	Layers Dim all 64	Act GELU	Reg L2(5e-5)	Num Att Heads 6	Learning Rate 0.001
7.6 Pathway Search
In the Pathway Search section, we conducted an experiment involving the execution of the pathway search for a set of specific reactants. Each of these reactants was associated with a desired target molecule, which was expected to be found within the mechanistic pathway tree. Additionally, a set of distinct parameters such as the context, depth, and breadth is associated with each reactant.
To provide detailed information and facilitate reproducibility, we have included supplementary materials accompanying this work. The full list of these reactants used for the pathway search is accessible at http://deeprxn.ics.uci.edu/rmechrp/pathway. This list contains the reactants, corresponding targets, the provided context (if any), and the anticipated depth at which the target molecule is expected to appear within the mechanistic pathway tree. For each entry of this list, we include a visualization of the identified pathways leading to the specified target molecules. It presents the discovered pathways that were found during the experiment. These materials serve to provide comprehensive insights into the pathway search process and its outcomes, enabling readers to reproduce and further explore the obtained results.
7.7 RMechRP: Online Reaction Predictor
In addition to the methods and results presented in the main article, we have developed an online web server that enables users to utilize the trained models for predicting the outcomes of mechanistic radical reactions with the highest levels of interpretability of the outcome. RMechRP (Radical Mechanistic Reaction Predictor) accessible via the anonymized link http://deeprxn.ics.uci.edu/rmechrp.RMechRP offers two interfaces: Single-step prediction and Pathway search.
Single-Step Predictor predicts the outcome of a mechanistic reaction with a single transition state. Users have the option to either input the reactants in written form or draw them using a drawing tool provided on the web server. Additionally, users can specify the reaction conditions, with the current option being standard temperature and pressure. The number of reactive molecular orbitals (MOs) to be considered can also be specified by the user.