# Language Models for Predicting Organic Synthesis Procedures
## 7 Discussion
In contrast, a model that starts with a randomly initialized state lacks this fundamental knowledge, which could restrict its ability to efficiently adjust to and understand the input. The _CC_ (custom tokenizer and randomly initiated model) is usually the best option and achieves better scores than other configurations. Interestingly, a fine-tuned tokenizer on specific data can have a very positive effect and allow models to achieve better results than pre-trained models. Models that are pre-trained (T5, FLAN-T5, and BART) on a variety of texts that do not include domain-specific information are not particularly useful for our task. However, molT5 is pre-trained on chemical data and demonstrably performs well for our task, achieving top scores in two out of five metrics. Three models stand out in terms of metric scores: _mo1T5-large-PP_, _molT5-base-PP_, and _FLAN-T5-base-CC_. No single model stood out as the best because none excelled in all metrics. The _molT5-large-PP_ model performs best in terms of BLEU (47.75), ROUGE-L (56.74), 50% Levenshtein match (45.73%), and BLEU g1k (48.52) metrics. The highest scores of the 75% Levenshtein match (7.91%) metric are achieved by the _FLAN-T5-base-CC_ model (pre-trained molT5 model was fine-tuned). The _molT5-base-PP_ configuration is not able to achieve a top score; however, it is close to the models mentioned earlier and is one of the few models that also achieves a BLEU g1k score over 47. Observing the pattern that models with more parameters and the _CC_ configuration perform better, we have also trained and tested the _FLAN-T5-large_ (783M parameters) and _BART-large_ (406M parameters) models using the setup as mentioned above. The _BART-large-CC_ model did not converge despite different hyper-parameter setups and achieved a BLEU score of 21.11, which is below the baseline scores. The _FLAN-T5-large-CC_ model achieved a BLEU score of 41.81 with the testing dataset and was most likely overfit, since the result is lower than most tested methods. Poor performance can be explained by the fact that these models are significantly bigger, and the training dataset is not large enough to train models effectively. In previous works, our group has also conducted experiments with OpenAI's fine-tuned GPT-3.5-turbo using a single type of reaction (about 1000 instances). The performance of such large LLMs is yet to be seen when using the entire _PRP-931k_ dataset, but we have found that a trained _FLAN-T5-base_ model was more effective for this specific chemistry-related task in our previous works. This suggests that large language models can present challenges and provide limited results in fine-tuning a computationally expensive task (text-to-text). Metric scores alone do not provide a definitive indication of the best model, the quality of generated procedures, or the criteria for assessing future methodologies. We compared the top three models using additional metrics and analyzed the results to develop an in-depth assessment. The following metrics are tailored to each task and are only used to obtain a better understanding of the results generated by various models:  
* **Ancillary compound coverage (ACC). ** Procedures frequently require supplementary materials in addition to the core reactants, which are commonly known as ancillary compounds. The _Add_ and _MakeSolution_ actions can introduce additional materials into the reaction, such as solvents, inorganic substances, or reagents. The _ACC_ metric is determined by compiling a list of all ancillary compounds involved in a reaction (from the _Add_ and _MakeSolution_ actions) and comparing how many of them are found in the predicted procedure. The _ACC_ is expressed in Formula (4), where \(A_{p}\) is the set of ancillary compounds in the predicted procedure and \(A_{gt}\) is the set of ancillary compounds in the ground truth procedure.