# Learning Hierarchical Representations for Explainable Chemical Reaction Prediction
## 4 Experiments
### Ablation Study  
In order to evaluate the rationality of each proposed mechanism in our method, we perform an ablation study on the USP-few dataset. The mechanisms contain Coarse-level Sequence Encoding (CSE), Fine-level Sequence Encoding (FSE), probabilistic Data Augmentation (DA), the first data augmentation strategy (DA #1, i.e., the rotation operation), the  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Methods**} & \multicolumn{2}{c}{**Yield BERT**} & \multicolumn{2}{c}{**DRFP**} & \multicolumn{2}{c}{**Ours (w/o Pretraining)**} & \multicolumn{2}{c}{**Ours**} \\ \hline Splits & Random & Time & Random & Time & Random & Time & Random & Time \\ \hline Gram scale & 0.117 & 0.095 & 0.130 & - & 0.125 & 0.097 & 0.129 & 0.099 \\ Sub-gram scale & 0.195 & 0.142 & 0.197 & - & 0.198 & 0.146 & 0.200 & 0.147 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison results of R2 scores on the USPTO yield dataset.  
Figure 6: Kendall rank correlation matrix between coarse- and fine-level coefficients under various distance metrics.  
second data augmentation strategy (DA #2, i.e., the branch chain moving operation), and the third data augmentation (DA #3, i.e., intra fine-level classes switching for coarse-level augmentation), and the pretraining on the Schneider 50k dataset.  
Table 5 shows the experimental results.  
\begin{tabular}{c c c c c c c c c} \hline \hline
**\#** & **CSE** & **FSE** & **Probabilistic DA** & **DA \#1** & **DA \#2** & **DA \#3** & **Pretraining** & **Macro F1** \\ \hline
1 & - & - & - & - & - & - & - & 0.790 \\
2 & - & ✓ & ✓ & ✓ & ✓ & - & ✓ & 0.802 \\
3 & ✓ & ✓ & - & - & - & - & ✓ & 0.787 \\
4 & ✓ & ✓ & - & ✓ & ✓ & ✓ & ✓ & 0.835 \\
5 & ✓ & ✓ & ✓ & - & ✓ & ✓ & ✓ & 0.823 \\
6 & ✓ & ✓ & ✓ & ✓ & - & ✓ & ✓ & 0.867 \\
7 & ✓ & ✓ & ✓ & ✓ & ✓ & - & ✓ & 0.872 \\
8 & - & ✓ & ✓ & ✓ & ✓ & - & - & 0.793 \\
9 & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & 0.873 \\ \hline \hline \end{tabular}  
* Our full model in the 9-th row achieves the highest F1-score, which demonstrates that all the designed parts in our model help to discover discriminative and descriptive information for chemical reaction representation.
* The 1 st row in the table is our backbone model, i.e., the reaction BERT model.
* The model of the 2 nd row which removes the CSE module and the DA #3 for coarse-level augmentation is thus disabled. The improvement of the performance might mainly come from the data augmentation strategies. Compared with the 9 th row, the result also suggests the great importance and usefulness of our coarse-level representation learning.
* We banned all the data augmentation methods and the contrastive learning in the 3 rd row and observe that the result is even worse than the backbone models. It might be because the scale of Schneider 50k is too small to train a model to generalize well and lead to the negative transfer learning phenomenon. Therefore, it is important for the effectiveness of our model to conduct contrastive learning and data augmentation.
* The 4 th row indicates that randomly conducting data augmentation achieves a lower F1-score than augmenting data conditioned on the atoms' importance in the reaction. The performance difference shows the superiority of our probabilistic DA mechanism.
* The 5 th, 6 th, and 7 th rows show the results of disabling the 3 DA mechanisms, respectively. Comparing the results with the result of our full model in the 9 th row, we can conclude that the DA #1 is the most effective method and the effectiveness of the DA #3 is limited.
* The 8 th row indicates that the pretraining procedure is removed, and the coarse-level related mechanisms are thus removed without any coarse-level annotations being available. Compared with the 2 nd and 3 rd rows, the results further emphasized the necessity of our data augmentation and contrastive learning.