# Language Models for Predicting Organic Synthesis Procedures
## 6 Results  
The experiments involved training and testing with the dataset (as outlined in Section 4) utilizing the LLM-based ML approaches described in Section 5. The models were evaluated by comparing the actual values from the testing dataset with the predicted values. The BLEU score metric was utilized. The BLEU score [76], originally developed to assess the quality of machine translation, is now an important metric in various NLP tasks for evaluating the quality of generated text. It provides a quantitative assessment of the similarity between the produced text and the reference text and is used for reaction procedure evaluation [8]. We also provide the ROUGE-L metric (Recall-Oriented Understudy for Gisting Evaluation--Longest Common Subsequence) [77]. These metrics are frequently utilized to assess the quality of generated text in comparison to reference texts. The BLEU score is derived by computing the geometric mean of adjusted n-gram (\(N=4\)) precision (\(P_{n}\)) along with a brevity penalty (\(BP\)) to address the challenge of shorter generated texts in comparison to the reference:  
\[BLEU=BP\cdot\exp\left(\frac{\sum\log(P_{n})}{N}\right) \tag{1}\]  
where \(N\) represents the highest order of n-grams being analyzed, while the brevity penalty (\(BP\)) is determined as follows:  
\[BP= \begin{cases}1&\text{if }c>r\\ e^{(1-r/c)}&\text{if }c\leq r\end{cases}  \tag{2}\]  
where \(r\) represents the combined length of the reference texts, while \(c\) represents the combined length of the created texts. The ROUGE-L score evaluates the longest common subsequence (LCS) between the generated text and the reference text, considering both recall and precision. The ROUGE-L score is computed in the following manner:  
\[ROUGE-L=\frac{\left(1+\beta^{2}\right)\cdot\text{LCS Precision}\cdot\text{ LCS Recall}}{\beta^{2}\cdot\text{LCS Precision}\,+\text{ LCS Recall}} \tag{3}\]  
The weighting factor \(\beta=1\) adjusts the importance of precision compared to recall. LCS Precision is the proportion of the length of the LCS to the length of the generated text, whereas LCS Recall is the proportion of the length of the LCS to the length of the reference text. We also compute the normalized Levenshtein similarity [78] between each pair of references and predicted answers, assessing similarity by accounting for the minimum edits needed to change one string into the other. This similarity is then evaluated against predefined thresholds (50% and 75%) to calculate the proportion of predictions reaching or surpassing each level of similarity. For example, the 75% threshold score indicatesthe proportion of sentence pairs with a normalized Levenshtein similarity of at least 0.75 or more. Random and majority methods were chosen to determine a starting point for the trained models. The random method selects a random output from the training dataset, whereas the majority method produces the most frequently occurring sentence in the training dataset. Random and majority baselines achieve BLEU scores of 18.50 and 31.87, respectively. We also established a benchmark using the k-nearest neighbor algorithm (with \(k=1\)) as the reference point due to its simplicity, effectiveness, and suitability for chemical tasks. To vectorize the reactions and the input, we have utilized DRFP fingerprints, which are designed for chemical reactions. The methods yield a BLEU score of 25.32 with the Euclidean distance [79]. All test results are subsequently compared against these benchmarks to assess the performance of the fine-tuned models. If results exceed these baseline levels, it will indicate that the trained models have enough predictive capacity to be considered as possible solutions for our issue. The results are presented in Table 4 of the testing subset for BLEU, ROUGE-L, normalized Levenshtein similarity match (with 50% and 75% thresholds), and BLEU for a manually checked gold standard subset of 1000 instances (column name \"BLEU g1k\").