# Language Models for Predicting Organic Synthesis Procedures
## 7 Discussion  
Compared to the random and majority baseline scores (random--31.87, majority--18.50 BLEU), all evaluated model types with varying sizes and tokenizers demonstrate superior performance and predictive ability. In nearly all cases, models with a greater \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Name** & **BLEU** & **ROUGE-L** & **50\% Match** & **75\% Match** & **BLEU g1k** \\ \hline seq2 seq\_1000 & 39.61 & 51.15 & 30.99\% & 2.67\% & 41.16 \\ seq2 seq\_3000 & **43.81** & 53.93 & 39.06\% & 4.03\% & **45.29** \\ seq2 seq\_8000 & 43.45 & 53.92 & 38.77\% & 4.09\% & 44.42 \\ \hline BART-XS-PC & 42.26 & 52.48 & 34.39\% & 3.10\% & 43.38 \\ BART-XS-CC & 43.99 & 53.98 & 38.28\% & 4.03\% & 45.32 \\ BART-S-PC & 42.88 & 53.77 & 37.92\% & 4.08\% & 44.08 \\ BART-SCC & 44.50 & 54.79 & 40.28\% & 4.88\% & 45.86 \\ BART-M-PC & 43.57 & 54.02 & 38.56\% & 4.48\% & 45.43 \\ BART-M-CC & 45.21 & 55.00 & 40.58\% & 5.27\% & 46.70 \\ Bart-base-PP & 45.12 & 55.36 & 42.27\% & 5.46\% & 46.05 \\ Bart-base-PC & 44.24 & 54.54 & 39.76\% & 4.58\% & 45.88 \\ Bart-base-CC & **45.91** & 56.00 & 43.06\% & 6.46\% & **47.06** \\ \hline T5-small-PP & **44.91** & 53.76 & 38.05\% & 3.79\% & 45.41 \\ T5-small-PC & 44.79 & 54.47 & 39.63\% & 4.78\% & 46.50 \\ T5-small-CC & 43.74 & 54.34 & 38.93\% & 4.92\% & 45.40 \\ T5-base-PP & 44.63 & 54.14 & 39.12\% & 3.82\% & 46.21 \\ T5-base-PC & 44.69 & 54.37 & 39.16\% & 5.42\% & **46.52** \\ T5-base-CC & 43.39 & 53.87 & 37.59\% & 4.61\% & 45.17 \\ \hline FLAN-T5-small-PP & 44.93 & 54.80 & 40.92\% & 4.20\% & 46.01 \\ FLAN-T5-small-PC & 45.05 & 54.32 & 39.66\% & 4.83\% & 45.91 \\ FLAN-T5-small-CC & 43.47 & 54.05 & 38.60\% & 4.34\% & 45.13 \\ FLAN-T5-base-PP & 45.62 & 55.11 & 41.72\% & 4.52\% & 46.76 \\ FLAN-T5-base-PC & 45.37 & 54.77 & 39.88\% & 6.21\% & 46.08 \\ FLAN-T5-base-CC & **46.43** & 56.02 & 42.74\% & **7.91\%** & **47.70** \\ \hline molT5-small-PP & 44.54 & 53.61 & 37.80\% & 3.65\% & 45.77 \\ molT5-base-PP & 46.38 & 55.51 & 43.30\% & 4.70\% & 47.79 \\ molT5-large-PP & **47.75** & **56.74** & **45.73\%** & 6.46\% & **48.52** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results of the testing dataset. number of parameters, which are base-type models, demonstrate better performance. Models with a greater number of parameters should be able to interpret complicated input patterns and produce more precise predictions. The T5 model is an exception in this instance, while the other models have very comparable scores. The _T5-base-CC_ always produces the lowest scores within its category, unlike other types where the _CC_ (custom tokenizer and a randomly initialized model) configuration is typically better. The result might be related to the T5's unique architecture and hyperparameters, which may limit effective training with customized tokenizers. The results of classic transformer neural networks used in our text-to-text model closely match those of other smaller models such as _FLAN-T5-small_ and _molT5-small_. The vocabulary size impacts the performance of the simple transformer model, with the best scores achieved using a vocabulary of 3000. However, the overall scores are not distinctive. When comparing BLEU and BLEU g1k scores, it is apparent that all BLEU g1k scores are higher due to the correct inputs provided to the model, as the gold standard testing dataset has fewer errors compared to the basic testing dataset. When key reactants are missing, the predictions may be significantly different from the ground truth, which is reflected in the score. This implies that the model's performance in real-world scenarios could be slightly better than what the BLEU scores indicate. When comparing the _PP_ (pre-trained tokenizer and pre-trained model) with the _PC_ (pre-trained tokenizer and randomly initiated model), we find either very similar performance or a reduction in scores. We believe that the pre-trained model's exposure to a vast amount of information during its first training phase enhances its capacity to interpret and analyze new information.