# Learning Hierarchical Representations for Explainable Chemical Reaction Prediction
## 4 Experiments
### Results of Reaction Classification  
Table 2 shows the comparison results on the Schneider 50k and the USPTO 1k TPL datasets for reaction classification. Two state-of-the-art methods are compared: Reaction BERT [9] and DRFP [11]. The Reaction BERT applies a Transformer-based model to learn chemical reaction representations and uses a BERT classifier to distinguish different reactions. The DRFP method maps a chemical reaction SMILES into a binary representation by hashing the symmetric difference of the two circular molecular n-gram sets generated from the molecules on the left and right of the reaction arrow, respectively. From the point of view of machine learning, representations of DRFT are hand-crafted features relying on domain prior to any statistical learning. The results of BERT and DRFT in Table 2 are reported in their original paper where CEN and MCC of Reaction BERT on Schneider 50k dataset are not reported. We follow the same experimental settings as the two methods and report the results of our model. Our method outperforms the compared methods. The performance improvement between ours and the backbone Reaction BERT verifies the effectiveness of the proposed coarse-fine level contrastive-pretraining mechanism. Both ours and Reaction Bert are better than DRFT on all three metrics, which demonstrates that representations learned from data can characterize unobvious but useful information on chemical reaction SMILES for the classification task. Table 3 shows the comparison results of the USP-few dataset for reaction classification. The compared KV-PLM [12] establishes the connections between molecule structures and biomedical text to represent chemical reactions. Our method can achieve comparable results with the KV-PLM. Similar to KV-PLM, our method leverages external information (i.e., the coarse-level reaction annotations) to model the coarse- and fine-level reaction representation for capturing both inter-class discriminative and intra-class descriptive information. The phenomenon that both KV-PLM and ours significantly outperform the backbone Reaction BERT method reveals the positive effect of external knowledge on describing molecule structures. To further study the classification performance of our model on chemical reaction representation, a further analysis was conducted on misclassified samples in the Schneider 50k dataset. Specifically, we countered the true negative and false negative samples on the Schneider 50k dataset, and calculated their proportions in each category, as shown in \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Datasets** & \multicolumn{3}{c}{**Schneider 50k**} & \multicolumn{3}{c}{**USPTO 1k TPL**} \\ \hline Metrics & Accuracy & CEN & MCC & Accuracy & CEN & MCC \\ \hline Reaction BERT & 0.985 & - & - & 0.989 & 0.006 & 0.989 \\ DRFP & 0.956 & 0.053 & 0.955 & 0.977 & 0.011 & 0.977 \\ Ours & 0.988 & 0.011 & 0.987 & 0.991 & 0.005 & 0.990 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Reaction classification results on the Schneider 50k and the USPTO 1k TPL datasets. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **Reaction BERT** & **KV-PLM** & **Ours** \\ \hline Marco F1 & 0.790 & 0.856 & 0.873 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Reaction classification results on the USP-few dataset. Figure 4. It can be found that the misclassified samples are not evenly distributed across all categories, but rather, show certain regular characteristics. The false positive proportion in the \"2.6.1 Ester Schotten-Baumann\" category is the highest, indicating that more samples were incorrectly classified into this category. One possible reason is that the features of this category are more dispersed and therefore the model assigns this category to a larger area in the sample space, making it difficult to accurately express this type of chemical reaction. The true negative rate is highest in the \"1.8.5 Thioether synthesis\" category, indicating that many misclassified samples come from this category.