# Language Models for Predicting Organic Synthesis Procedures
## 7 Discussion
\[ACC=\frac{\left|A_{p}\cap A_{gt}\right|}{\left|A_{gt}\right|}\cdot 100\%\] (4) For example, if the ground truth instance has THF, HBTU, and DIPEA added to the procedure, while the predicted materials are THF, HBTU, and NaOH, the coverage score would be 66.7% because THF and HBTU were correctly selected out of the three in the ground truth procedure. The metric aims to assess how well the models predict relevant materials that should be added to the reaction mixture, because often the choice of the components suggests a specific reaction mechanism. However, it was difficult to assess how the score should be impacted if the predicted procedure contained other materials than the ground truth (NaOH in the example); therefore, no penalty was given in such cases. Also, the action _Add_ appears not just at the beginning but may be present at various stages of the workup process. This means that if the predicted procedure chooses different methods for workup, which may not be incorrect, the ACC score will be lower. In about 3-4% of cases, where no _Add_ or _MakeSolution_ actions are present, this metric is not applicable. We calculate an average _ACC_ score using the testing dataset (size: 20,000 instances). The scores were as follows: _molT5-large-PP_--41.08%, _molT5-base-PP_--31.61%, and _FLAN-T5-base-CC_--36.66%. Surprisingly, the model _BART-base-CC_, which had the lowest metrics out of the top four models, achieved the second-highest score of 40.37%. Other smaller models had scores of about 28-32%, while most base models had scores of 33-38%. Notably, _molT5-base-PP_ has achieved the highest score with our gold standard dataset but scored low with _ACC_, leading us to believe that it excels in capturing and applying the foundational chemical knowledge encoded in its training data for other steps, but it may not be as adept at incorporating ancillary compounds into its predictions. Alternatively, it may simply predict alternative agents or solvents, which may be more suitable in some cases. Generally, the scores of about 40% are within the expected range for the task in its current state. One factor that contributes to lower scores is the uncertainty with which certain agents could be chosen. For example, in one case, the base for the reaction is Cs\({}_{2}\)CO\({}_{3}\) (ground truth), and models predict K\({}_{2}\)CO\({}_{3}\), which is an acceptable prediction considering that Na\({}_{2}\)CO\({}_{3}\) could possibly be used. Similar cases are present in organic chemistry, where a set of substances may be suitable in one case, especially solvents, and the ACC metric does not account for that. * **Workup action and solvent coverage (WASC). ** Workup actions are a set of procedures designed to isolate the desired product and alter its state in order to typically yield a solid product. Actions that require solvents for these manipulations are _Extract_, _Quench_, _Recrystallize_, _DrySolution_, and _Purify_. To assess the model's ability to accurately produce specific steps and determine the workup action and solvent coverage (WASC) score, compile a list of all the mentioned action names and solvents in a reaction. Then, compare this list with the predicted procedure to identify the matches. For example, if the ground truth instance has \"Extract with ethyl acetate, DrySolution over MgSO\({}_{4}\)\", while the predicted procedure has \"Extract with ethyl acetate, DrySolution over Na\({}_{2}\)SO\({}_{4}\)\", the coverage score would be 50% because only the extraction was correctly predicted with its parameter (solvent). The WASC score is expressed in Formula (5), where \(W_{p}\) is the set of actions with compounds in the predicted procedure and \(W_{gt}\) is the set of actions with compounds in the ground truth procedure. \[WASC=\frac{\left|W_{p}\cap W_{gt}\right|}{\left|W_{gt}\right|}.100\% \tag{5}\]  
The top three models achieve scores of around 30%: _FLAN-T5-base-CC_ (34.00%, highest overall), _molT5-large-PP_ (32.89%), and _molT5-base-PP_ (31.52%). The scores for all the smaller models are around 28%.