dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments This work is part of the PhD thesis of  M. Filippi. G. Sessa  and  A. Risplendente  are acknowledged for technical assistance. The authors express their gratitude to  F. Casetta  and an anonymous reviewer for their constructive reviews, which have greatly contributed to enhancing the quality of the text. A special thanks is also extended to  G. Shellnutt  for his meticulous editorial handling. This work received support from the  Italian Ministry for University and Research (MUR)  through the project " Dipartimenti di Eccellenza   2023-27 ".
In this work, we explore how existing datasets of quantum chemical properties can be repurposed to build data-efficient downstream machine learning models, with a particular focus on predicting the activation energy of hydrogen atom transfer (HAT) reactions. Starting from a valence bond (VB) analysis of a generic HAT process, a set of informative descriptors is identified. Next, a surrogate neural network model is constructed to predict an informative representation, based on the identified VB descriptors, with the help of a publicly available dataset of (pre-computed) quantum chemical properties of organic radicals. We demonstrate that coupling the resulting on-the-fly informative representation to a secondary machine-learning model for activation energy prediction outperforms various predictive model architectures starting from conventional machine-learning inputs by a wide margin, at no additional computational cost.
Please place the appropriate symbol next to the author's name and include a \footnotetext entry in the correct place in the list. Entry Specification (SMILES)  1  strings can be trained to predict the main product of chemical reactions with roughly 90% accuracy on the popular USPTO benchmark dataset. [2] [3] [4]  For more specialized reactivity tasks (the prediction of activation energies/reaction rates, yields, enantiomeric excess etc. ), however, data tend to be scarce, and often number in the hundreds rather than in the hundreds of thousands. A straightforward application of the same type of data-hungry model architectures is generally not feasible in these cases, as training these models from scratch means that they must learn how to properly interpret a molecular structure. One potential approach to improve the data efficiency of machine learning models is introducing domain knowledge about the underlying physics/quantum chemistry into the model.
Models starting from physically meaningful input representations, i. e. , representations based on quantum mechanical (QM) reactivity descriptors, have been demonstrated to reach reasonable accuracy for datasets consisting of only a couple hundred or a few thousand data points,  [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]  and they have been reported to be significantly more generalizable, i. e. , they are significantly more accurate in out-of-sample predictions, than conventional, structure/graph-based, analogs. 7, 8, 15  pensive -since they typically involve an elaborate workflow consisting of conformer generation, geometry optimization and finally a single-point density functional theory (DFT) calculation for each unique molecule -limiting their applicability in broad virtual screening campaigns.
For example, in their proof of principle study, Guan et al. 20  generated a dataset of selected QM properties for 130k organic molecules, extracted from the ChEMBL  23  and Pistachio 24 databases. A potential strategy to circumvent the expensive QM descriptors calculations altogether would be to use pre-existing datasets. In recent years, various general-purpose public datasets of computed QM properties of molecules have been released. [25] [26] [27] [28] [29] [30] [31]  While these datasets have found dual use as benchmarking targets,  [32] [33] [34]  as well as training data for generative models,  [35] [36] [37]  their repurposing for (indirectly related) downstream predictive tasks, e. g. , the prediction of chemical reactivity, has remained largely unexplored up to this point. Here, we will demonstrate that such a data repurposing strategy can be readily applied to efficiently learn activation energies for hydrogen atom transfer (HAT) reactions.
Starting from a generic description of HAT reactivity within the Valence Bond perspective, we determine a set of qualitative characteristics that can be expected to capture the main trends for this type of reactivity. Based on this analysis, we identify the recently released dataset of QM properties for 40,000 closed-shell organic molecules and 200,000 radical analogs by Paton and co-workers, BDE-db,  29  as a useful source of informative descriptors and build a surrogate model able to infer these descriptors on the fly across a broad chemical reaction space based on a simple SMILES string input. 29  The QM representation emerging from this surrogate model is subsequently demonstrated to outperform traditional structurebased representations for various machine learning model architectures on a challenging, in-house generated dataset of diverse hydrogen abstraction reactions by a large margin.
Most importantly, models based on the developed surrogate QM representation are extremely data efficient, approaching convergence for training dataset sizes of only a couple hundred points. Finally, we demonstrate the practical usefulness of the developed representation by considering various real-world applications. HAT reactivity plays a central role in various sub-fields of chemistry. For example, HAT reactions constitute the starting point of many oxidation pathways in polymer, atmospheric and combustion chemistry. [38] [39] [40] [41] [42] [43]  Additionally, they are key transformations giving rise to the breakdown of active pharmaceutical ingredients, so their kinetic and thermodynamic properties facilitate the prediction of the stability of these compounds,  44, 45  as well as the formation of potentially hazardous metabolites.
46  Finally, HAT reactions also play a central role in various common synthesis strategies, e. g. in several flavors of C-H functionalization and photocatalysis, and a lot of effort has been devoted to gaining insights into the factors that affect their rates and yields. [47] [48] [49] [50]  As a result, several small datasets have been produced for this reaction class in recent years, each with distinct reaction scopes and conditions, depending on the specific application for which they were designed. 38, 46, [50] [51] [52] [53] [54] [55] [56] [57]  Because of their heterogeneity, these datasets cannot be combined easily. As will be demonstrated below, with our extremely data-efficient model architectures, we are able to meaningfully learn from these small datasets, and generate accurate predictions across the chemical spaces associated with the respective application domains, at both an unprecedented speed and an extremely low computational cost.
In essence, the proposed strategy rests on making the overall machine learning model architecture modular, with a core informative representation prediction part, followed by a fairly linear and extremely data-efficient downstream reactivity model part, that can be easily retrained on/finetuned to the specific application or dataset at hand. Overall, we believe that the presented work provides a blueprint for how -starting from physics/chemistry-inspired mechanistic insights -useful (existing) datasets of QM properties can be identified and leveraged for downstream predictive tasks. Furthermore, it underscores the importance of community efforts to generate and curate QM property datasets -and to include as many informative descriptors as possible in them. At the end of this contribution, we briefly reflect on the transferability of the approach to other reaction types.
At the same time, the literature suggests that even a subset of the descriptors mentioned above could be sufficient to build an informative representation: for small subsets of the chemical space we are interested in, various groups have previously constructed effective multivariate linear models, as well as some more advanced architectures (random forests, neural networks, etc. ) with a subset of the descriptors listed above. 46, 51, 52, 70  As already indicated above, encouraged by these findings, we selected the BDE-db dataset as a suitable source of descriptors. This dataset contains (relaxed) BDFEs -at room temperature in the gasphase -and BDEs, (Mulliken) atomic charges, and (Mulliken) spin densities for over 200,000 organic radical compounds. 29  Additionally, since accurate geometries are included in this dataset, we were able to expand this descriptor list with buried volume values at a marginal computational cost.
Finally, we also used these geometries to rapidly compute vertical, i. e. , frozen or unrelaxed, BDE values at extended tight-binding (xTB) level of theory (these final low-level descriptors were found to be remarkably predictive -even at this low theory level -for activation energies of HAT reactions between P450 enzymes and organic compounds in a recent study by Gingrich et al. 46  -see the Methodology section for more details). With this dataset-derived representation selected, we set out to train a surrogate descriptor model, in combination with a toplevel reactivity model, to probe HAT reactivity on the fly.
Extracting and curating the descriptor data from the BDE-db dataset As indicated in the introduction, the training data for the surrogate neural network (vide infra) was extracted from a dataset recently released by Paton and co-workers, consisting of QM properties for radicals and associated closed shell molecules of less than ten heavy atoms, without formal charges on any of the atoms, and containing exclusively C, H, N, and O atoms. We complemented the registered descriptors, i. e. , BDFEs, spin densities, and partial charges, with V bur values around the radical site and frozen BDE values computed at GFN2-xTB level of theory. V bur values were computed with the help of the Morfeus package  71  with a radius of 3. 5 Å. 72  Frozen BDEs were computed by re-optimizing the extracted geometries at GFN2-xTB level of theory and then performing single-point calculations on the bond fragments, i.
e. , the organic radical and a hydrogen radical respectively. In total, the complete set of descriptors could be obtained for 186,240 dissociation reactions in this manner. The entire dataset in its curated form can be accessed and downloaded from https://figshare. com/articles/dataset/ dataset_surrogate_model_cs/24754341. It should be noted that during the preparation of this manuscript, an expanded version of the dataset, covering the elements S, Cl, F, P, Br and I as well was released. 73  As such, the approach described here could be expanded straightforwardly to a much broader scope of HAT reactions in the near future.
Generation of the HAT dataset To enable comparisons between various types of machine learning model architectures, a dataset of HAT reaction profiles was generated by combining multiple bond dissociation reactions included in the BDE-db dataset. A fully automated reaction profile computation workflow based on autodE  74  and Gaussian16  75  was set up to generate this dataset, starting from an input list of reaction SMILES. This workflow is analogous to the workflow recently developed for cycloaddition reactions by Stuyver et al. ;  76  an in-depth discussion can be found in Section S1 of the Supporting Information. It should be noted that around 40 billion HAT reaction SMILES can in principle be constructed by making combinations between bond dissociation reactions in the BDE-db dataset. Most of these reactions are very similar, involving HAT among two carbon atoms.
Based on a hyperparameter grid search, the depth was set to 5 for the D-MPNN and 4 for the FFNN. For the hidden size, 1200 neurons were selected for the D-MPNN and 900 for the FFNN (cf. Section S2. 1 in the Supporting Information). Once a reasonable set of hyperparameters was selected, the model was trained on a new data split, where all the radical species and molecules present in the constructed HAT reactivity dataset (vide supra) constitute the test set, and the remaining points are split between training (80%) and validation (20%) set.
Datasets In total, 3 application domains have been considered to showcase the ability of our model architectures to efficiently learn barrier heights in various settings. In the first application domain, we focused on HAT predictions related to chemical synthesis. We started by analyzing a dataset consisting of 238 computed reaction profiles for alkoxy radicals abstracting hydrogens from hydrocarbons and heterosubstituted compounds in an acetonitrile solution. 70  Geometry optimizations and frequency calculations were performed at ωB97X-D/6-31G(d) level of theory  [84] [85] [86] [87]  , single-point energy refinements at UωB97X-D/6-311++G(d,p) level of theory, and solvation was taken into account through the SMD model. 88  Since two of the radicals involved in this dataset contain Cl and F atoms -ele-ments that are not present in the training data of the surrogate model -we computed the descriptors for these species explicitly.
Subsequently, we aimed to apply our trained models to an experimental dataset of rate constants/activation energies for 45 HAT reactions from C(sp 3 )-H bonds by cumyloxyl radical (CumO • ), compiled by Bietti et al. 54  Finally, we also considered a small dataset of experimentally reported selectivities for 6 hydrocarbons by CH 3 O • determined by Zuo et al. 50  More specifically, we aimed to reproduce the observed selectivities by inferring the relative barrier heights associated with individual carbon attack positions (15 in total). In a second application, we focused on the prediction of metabolites of medicinally relevant compounds. As such, we considered a second dataset consisting of 24 activation energies for hydrogen atom transfer by the cytochrome P450 enzyme from organic compounds,  46  computed at B3LYP/6-311++G(2d,2p)//B3LYP/6-31G(d) (Fe=SVP) level of theory.
[89] [90] [91]  5 of the HAT reactions involve compounds with elements outside of the domain of the training data of our surrogate model and hence were discarded in our analysis. The models trained on the resulting extremely small dataset were subsequently applied to predict activation energies for 6 additional computed HAT reaction profiles for unseen substrates. As a third application, we focus on HAT reactions encountered in atmospheric chemistry. To this end, around 300 HAT reactions were extracted from RMechDB. RMechDB is a database consisting of elementary radical reactions recently assembled by Tavakoli et al. 38  In its current form, the dataset is dominated by gas-phase reactions of isoprene and other organic molecules. No labels, i. e. , computed or predicted activation energies, are available for the reactions in this database, and hence we computed reaction profiles ourselves with the same high-throughput workflow used to generate the previously discussed in-house HAT dataset.
Performance of the surrogate model In Figure  3 , the performance of the trained surrogate model on the held-out test set, i. e. , the radicals and closed-shell molecules present in the BDE-db dataset, is presented. For (relaxed) BDFEs, atomic charges, spin densities, and buried volumes, excellent correlations between predicted and computed values are obtained (R 2 ≥ 0. 97); for frozen BDEs, the correlation is slightly worse (R 2 = 0. 89), but the quantitative trends are still largely retained.
Predicting activation energies for our in-house generated dataset With a performant surrogate model at our disposal, we turned to the in-house generated HAT dataset. In Figure  4 , the distribution of the computed activation and reaction energies, as well as the correlation between both quantities, is presented. Remarkably, activation and reaction energies only correlate moderately (R 2 = 0. 71), indicating that the Bell-Evans-Polanyi principle  92, 93  is not sufficient to fully explain the trends in the activation energies, and that building a predictive model including multiple descriptors is warranted for this dataset. Various baseline models, not making use of the informative representation predicted by our surrogate GNN, were considered as baselines. Due to the relatively small size and inherent diversity present in the constructed HAT dataset, all of these model architectures perform remarkably poorly.
RF, KNN and XGBoost models taking structural fingerprints, i. e. , differential reaction fingerprints (DRFP), as input, reach a mean absolute error (MAE) of 5. 2-5. 3 kcal/mol (corresponding to R 2 around 0) in 10-fold crossvalidation. This is barely any better than the standard deviation (7 kcal/mol), i. e. , the hypothetical accuracy that would be obtained for a model that always predicts the mean value for the dataset. The GNN architectures tested -a Weisfeiler-Lehmann (WL) one and Chemprop -perform somewhat better, but the MAE still amounts to ∼2. 9-3. 5 kcal/mol (R 2 ∼ 0. 65). The geometrybased kernel ridge regression model, based on the recently introduced Bond-Based Reaction Representation (B 2 R 2 l ) reaches a similar performance (MAE = 3. 55 kcal/mol). 94  The results quoted above are in great contrast to the results obtained when the predicted informative representation is used as the input for the reactivity model.
Reducing the training set size, the multivariate linear model takes over as the most accurate model when fewer than 400 training points are considered. In Fig. 6a , a schematic overview of the impact of the individual descriptors of the (predicted) informative representation on the accuracy of the multivariate linear model is provided. Our analysis indicates that each descriptor has at least some predictive power, i. e. , they all correlate to some extent with the activation energies, so that the prediction accuracy for the corresponding univariate linear regression models exceeds the standard deviation across the dataset by a non-negligble margin. As could have been expected from the Bell-Evans-Polanyi analysis in Fig. 4 , the (relaxed) BDFE values correlate extremely well; a linear regression model based exclusively on this (predicted) descriptor reaches an MAE of 2.
Applications In this final section, we will showcase the usefulness of our approach by considering a set of applications. Particularly, we will exploit the data efficiency of models based on our VB-inspired representation to learn activation energies across some extremely small HAT datasets found in the literature (vide supra). As indicated above, the first application area focuses on HAT predictions related to chemical synthesis. More specifically, we started by considering a HAT dataset involving alkoxy radicals and hydrocarbons/heterosubstituted compounds. 70  In previous work, an empirical model based on five computed parameters, respectively describing the electronegativities and delocalization of the reactants and products, as well as the thermodynamic driving force, was proposed. 51  Applying this empirical model to the full dataset, an MAE 0.
85 kcal/mol (R 2 =0. 84) is obtained on the training set, and 1. 24 kcal/mol (R 2 =0. 72) on the test set. With the help of our on-the-fly predicted VB-inspired representation, MAEs of 0. 99 kcal/mol (R 2 =0. 76) and 1. 14 kcal/mol (R 2 =0. 72) are reached, respectively on the training and test set, with a simple multivariate linear model. While not significantly better than the results obtained through the tailored, empirical expression by Liu et al. , it is important to note that our predictions can be performed on a millisecond scale, whereas the computed descriptor inputs needed in the empirical model require geometry generation and a succession of tedious DFT calculations. We also considered the performance of non-linear model ar- Another remarkable observation is that the predictive power of the respective descriptors in our VB-inspired representation has changed significantly upon switching from the HAT dataset discussed in the previous section to the current one (Figure  6 ; cf.
Section S5. 1 in the Supporting Information). As mentioned before, in the diverse dataset constructed as part of this study, the inclusion of the predicted BDFEs reduced the MAE of the linear model the most, and the inclusion of V bur had no observable effect on the model accuracy. In the small alkoxy radical dataset, however, BDFE values are not nearly as predictive. Atomic charges improve the model accuracy the most now, and the inclusion of V bur leads to significant further improvements. These differences can be connected to the more pronounced polar effects in alkoxy radicals, due to the involvement of a more electronegative radical center. 57, [95] [96] [97]  In more general terms, these findings about descriptor importance underscore that local correlations in small subsections of chemical space are not necessarily transferable and that caution is needed when aiming to draw universal conclusions about reactivity trends based purely on empirical trends.
Next, we turn to the experimental dataset of 45 HAT activation energies, assembled by Bietti and co-workers. Our models, trained on the alkoxy radical dataset, predict the (rescaled) activation energies in this experimental set with acceptable accuracy (∼ 1 kcal/mol; cf. Section S5. 2 of the Supporting Information), correctly identifying all of the reaction barriers to be low. However, the coefficients of determination, i. e. , the extent to which fine-grained trends across these 45 reactions are recovered, are remarkably poor (R 2 ∼ 0. 35). The main reason for this failure to recover the reactivity trends is the distinct data distribution, i. e. , the activation energies in this specific experimental dataset are "outliers" with respect to the reactions present in Ma et al. 's dataset, and -to a lesser extent -the fairly low spread in the targets (the standard deviation amounts to only 1.
Also quantitatively, our models reach good accuracy as compared to the computed DFT barriers in this application. The most accurate model is again the FFNN with 4 ensembles (MAE = 0. 84 kcal/mol, RMSE = 1. 07 kcal/mol, R 2 = 0. 74), with only one prediction resulting in an error over 2 kcal/mol with respect to the DFT calculations. The RF model, despite being the least successful at qualitatively predicting the most likely HAT position, reaches only a slightly worse quantitative accuracy (MAE = 0. 94 kcal/mol, RMSE = 1. 37 kcal/mol; R 2 = 0. 64). The second application area focuses on P450 metabolism of medicinally relevant compounds. As discussed in the Methodology Section, predictive models were trained on a tiny dataset of 19 HAT activation energies, and subsequently, these models were applied to predict 6 barriers for unseen substrates, in line with the work by Gingrich et al.
In total, we were able to extract 268 reactions, falling within the scope of our surrogate model, from this database; HAT reactions between organic compounds and HO • and HOO • radicals, i. e. , the onset of atmospheric oxidation. With our autodEbased workflow for reaction profile computation, we only managed to successfully compute profiles for 73 of these reactions (failure sources are detailed in Section S7). At first, we tried to directly reproduce the computed activation energies with the predictive models trained on our in-house assembled HAT dataset, but even our best model (ensembled FFNN) reached an inadequate MAE of 4. 58 kcal/mol (RMSE = 5. 11 kcal/mol, R 2 =0. 05). A reasonable explanation for this poor performance is (once again) the distribution of the data: our in-house generated dataset contained no examples involving HO • or HOO • radicals, and the activation energies for the reactions extracted from RMechDB (mean ∆G ‡ corr = 9.