# Language Models for Predicting Organic Synthesis Procedures
## 5 ML Approaches  
Our study has examined the following generative ML approaches to assess their effectiveness in predicting steps and parameters of synthesis methods:  
**Seq2 seq transformer model-based approach. ** The transformer model uses an encoder-decoder architecture adapted from the classic transformer framework [56]. For the implementation, the OpenNMT-py library [57], designed for efficient neural machine translation and sequence-to-sequence modeling tasks, was utilized. The models are not pre-trained, and we have utilized the _\"TransformerBaseSharedEmbeddings\"_ model configuration and architecture from the OpenNMT library, which has about 45 million parameters. SentencePiece tokenizers have been trained for the vectorization of both input and output [58]. To investigate the effect of vocabulary size on the model's performance, we experimented with several transformer models, labeled seq2 seq_1000, seq2 seq_3000, and seq2 seq_8000, corresponding to vocabulary sizes of 1000, 3000, and 8000, respectively. The models were trained with a learning rate of 0.001 and a batch size of 1024, which are used for neural machine translation [59]. **BART model-based approach. ** The BART model, developed by Facebook AI [60], represents a significant advancement in transformer-based architecture. BART integrates attributes of both encoder and decoder mechanisms from the transformer architecture. It utilizes the Byte-Pair Encoding (BPE) tokenizer [61], which segments text into a set of subwords or tokens. Each token is then transformed into detailed context vectors, the length of which varies in accordance with the model's configuration. BART by Facebook has been trained on various natural language datasets, such as SQuAD [62], NMLI [63], and ELI5 [64]. The model employs a 512-length context vector and has 139 million parameters (BART-basemodel) in total. The fine-tuned tokenizer is created by fine-tuning the BART tokenizer on our data. Compared to T5 and FLAN-T5, which have smaller versions of the model, Facebook AI BART does not. Three more configurations were constructed to investigate the effects of model size on performance. Extra-small (BART-XS), small (BART-S), and medium (BART-M) sized models with sizes of 18, 24, and 40 million parameters, respectively, were defined. The only difference in terms of testing is that there are no pre-trained versions of these configurations. The architecture parameters for each can be found in the project's repository ([[https://github.com/Mantas-it/Chem_Procedure_Prediction](https://github.com/Mantas-it/Chem_Procedure_Prediction)]([https://github.com/Mantas-it/Chem_Procedure_Prediction](https://github.com/Mantas-it/Chem_Procedure_Prediction)), accessed on 1 March 2024). **T5 model-based approach. ** T5 was developed by Google AI [65] and it is a transformer-based model, primarily designed for natural language processing tasks. T5 (Text-to-Text Transfer Transformer) differs from other models by redefining all NLP tasks as a text-to-text problem, treating both the input and output as text sequences, which facilitates the training process for various tasks. This model utilizes a distinctive method of pre-training on a vast corpus using a self-supervised objective, which involves predicting the masked text span to improve its comprehension and creation abilities. T5 is available in various sizes, ranging from small to extra-large, providing versatility in usage based on available resources and project requirements. The T5 model utilizes the SentencePiece tokenizer to segment words into subwords or tokens. Each token is then transformed into context vectors of different lengths based on the scale of the model. The model itself was pre-trained on datasets, such as GLUE [66], SuperGLUE [67], SQuAD, CoLA [68], and CB [69]. The model we use employs a 512-length context vector and has 60 million parameters (T5-small) or 223 million parameters (T5-base). **FLAN-T5 model-based approach.