# Language Models for Predicting Organic Synthesis Procedures
## 7 Discussion
The _FLAN-T5-base-CC_ model performs best and produces satisfactory results, given the strict nature of the metric. It should be noted that not all procedures contain workup actions, so this metric can only be used when the ground truth and the predicted instance both have at least one step (about 75%-80% of our testing subset). To measure whether it is challenging to predict the action steps without parameters, we can exclude the solvents and materials to perform similar calculations. We observe that the scores are considerably higher: _FLAN-T5-base-CC_ (75.36%), _molT5-base-PP_ (78.84%), and _molT5-large-PP_ (78.23%). Such results once again reiterate the fact that predicting solvents or inorganic compounds for these actions is challenging since the options can be flexible. However, it also highlights the difficulty of measuring the scores, since it can be difficult to be certain whether an incorrect prediction was made or the degree to which some solvent is not suitable for an operation. * **Average reaction temperature error (RTE). ** In order to analyze the accuracy of reaction temperature prediction, we measure the average reaction temperature error. This is performed by identifying the temperature point designated by the Stir action. If several options are possible, the highest (in absolute terms) is identified. Temperatures below zero are rare, but possible. The highest temperature point is chosen because it is assumed to be the temperature that the reaction mixture must reach. The problem of simply picking temperature points randomly from the reaction procedure is that the _Stir_ action might be mentioned in the beginning with 25 \({}^{\circ}\)C as the temperature point and for a short duration (to add relevant reactants, for example), only to later be mentioned again with a temperature point of 140 \({}^{\circ}\)C for the actual synthesis. Also, _SetTemperature_ is an action that is used only to control the temperature of the reaction mixture, without any other implicit actions. It is often employed in the workup stage to adjust the temperature back to room temperature or to specific points (\(-40\)\({}^{\circ}\)C or \(-78\)\({}^{\circ}\)C), especially during crystallization processes. The action _Reflux_ is the third option for designating the reaction temperature, but it is not explicitly written, and the reaction should be conducted at the boiling point of the solvent medium. Due to the reasons above, our experiment considered only the Stir action, which pertains to about 70% of the reactions in our testing dataset. The \(RTE\) is calculated using Formula (6), where \(n\) is the number of samples, \(t_{p}\) is the predicted temperature, and \(t_{g}\) is the ground truth. \[RTE=\frac{1}{n}\sum\lvert t_{p}-t_{g}\rvert\] (6) The average temperature of reactions was around 45 \({}^{\circ}\)C and the range of temperatures was from \(-100\)\({}^{\circ}\)C to 300 \({}^{\circ}\)C. The reaction temperature average error for the top models was as follows: _FLAN-T5-base-CC_ (22.51), _molT5-base-PP_ (24.54), and _molT5-large-PP_ (19.04). The _BART-base-CC_ (18.39) model scored best, and among the tested cases, it had an average error of 5 \({}^{\circ}\)C in 54% of cases, which is a high degree of accuracy. Acceptable temperature ranges may vary based on the procedure's end goal and the compounds'stability. In our case, we can observe that there is room for improvement, but we must recognize that the source training procedures do not necessarily present the optimal temperature points for the reaction. It is at least assumed that the temperature points are good enough for the reaction to take place in the first place. This means that while temperature predictions may be too high or too low, it does not mean that the reaction will not occur. There might be cases where an optimal temperature point is predicted, and there also might be cases where the suggestions are incompatible with the compounds or the mechanisms of synthesis. * **Average synthesis duration error (SDE).