dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset & Scoring Scalar couplings are critically dependent on the 3D structure of the molecule for which they are being measured; however at the time we carried out this work, we were unaware of accurate experimental databases linking pair-wise mutiple-bond NMR scalar couplings to well-defined 3D molecular structures. Therefore, we decided to run the CKC utilizing molecular structures included in the QM9 dataset, a publicly available benchmark for developing ML models of 3D structure-property relationships. (37) QM9 includes ~134k molecules comprised of carbon, fluorine, nitrogen, oxygen and hydrogen. The molecules included within QM9 have no more than 9 heavy atoms (non-hydrogen), with a maximum of 29 total atoms. To obtain a corresponding set of scalar couplings, we extended the QM9 computational methodology, using the B3LYP functional  (38)  and the 6-31g(2df,p) basis set  (39) (40) (41) (42)  to compute NMR parameters on the optimized QM9 structures.
The computed QM9 scalar coupling constants are available under Creative Commons CC-NC-BY 4. 0, enabling others to build on this work. To remove the possibility of CKC participants overfitting their models to the entire set of computed QM9 scalar couplings, 65% of molecules in the dataset were randomly partitioned into a training set and the other 35% to a testing set. The test set was further split, with 29% of the data in a 'public' test set, and 71% of the data in a 'private' test set (competitors were unaware of the specifics of the private/public split). Both the training and test sets included the molecular geometries and indices of the coupling atoms. Unlike the test set, the training set included a range of other data, including the calculated scalar coupling values, their breakdown into Fermi contact (FC), spin-dipole (SD), paramagnetic spin-orbit (PSO) and diamagnetic spin-orbit (DSO) components, and a range of auxiliary information obtained from the QM computations (e.
g. , potential energy, dipole moment vectors, magnetic shielding tensors and Mulliken charges). As the CKC progressed, participating teams continually iterated and improved their models. A regularly updated and publicly visible leaderboard enabled each team to see where their model ranked in predicting the public test set data compared to the model predictions made by all of the other teams. The leaderboard scores were determined using a function which accounted for the 8 different types of coupling constants included in the training and testing datasets: 1 JHC, 1 JHN, 2 JHH, 2 JHC, 2 JHN, 3 JHH, 3 JHC and 3 JHN (where the superscript indicates the number of covalent bonds separating the atom pairs indicated by the subscript). Since the number of couplings of each type differed (e. g. , the molecular composition of the QM9 test set included 811,999 3 JHC couplings compared to 24,195 1 JHN couplings) and spanned different value ranges, the scoring function used the average of the logarithm of the mean absolute error for each type of coupling constant: score = 1 ( ) log , 1 -! )|/ " -/ 1 " | # ! " 2 Eq (1) $ ! where t is an index that runs over the T = 8 different scalar coupling types, i is an index that spans 1.
S1 Overview Section S2 and S3 contains details on how the dataset was generated, and what considerations went into the choices made. Section S4 describes the data set available to the partitipants, as well as details on how submissions were scored. Section S5 contains various plots and analysis strategies that didn't make it into the main paper, as well as details on how the ensembles were fitted and the correlations were calculated. Section S7, S8, S9, S10, S11 and S12 contain detailed write-ups explaining the model architecture of the five winning teams as well as the 12th placed team. Finally, section S13 outlines the code and data available for download, including all the winning solutions.
S2 Dataset considerations We opted to use structures from the QM9 dataset  [1]  to construct our own dataset as well as the same computational methodology. QM9 consists of around 130,000 molecules comprised of carbon, fluorine, nitrogen, oxygen and hydrogen. Each molecule has up to 9 heavy atoms (non-hydrogen) and up to 29 total number of atoms. Each structure was optimized by the authors using Density functional theory (DFT), with the B3LYP functional  [2]  and the 6-31g(2df,p) basis set  [3, 4, 5, 6]. There were several advantages of basing our dataset on QM9, as well as several possible disadvantages that needed to be considered. QM9 has historically had an important role as a benchmark dataset in the development of machine learning models for 3D structure-property relationships, and we deemed it beneficial to augment this with magnetic properties.
This is particularly true for pairwise properties like scalar coupling constants, as there to our knowledge currently does not exist any dataset that includes pair-wise properties. Using an existing set of structures also eased the workload of generating a dataset considerably. We initially had two primary concerns about using 1 QM9 structures specifically and one primary concern about using computational methods in general. • The QM9 molecules are quite small compared to molecules commonly used as e. g. medicines, and the solutions to the competition might not work (due to memory use etc. ) on larger molecules. However, since the strength of atomic interactions decay with distance (with the exception of large aromatic systems), we believe that we could ultimately modify the solutions to scale better with system size by introducing an interaction distance cuto↵.
At this time, we have not researched the current size limit of the winning solutions. • All of the QM9 molecules have been optimized to form a local minimum on the potential energy surface (equilibrium). In the future we might be interested in studying non-equilibrium structures, and the solutions might perform less well on these (such as methods ignoring 3D relationships, and that relies entirely on connectivity). All of the top solutions incorporated 3D relationships, and at this time we have no reason to believe that any of them will perform poorly on a non-equilibrium dataset. • By relying on quantum chemistry calculations that directly relate a molecular 3D structure to the target property, participants could guess the specific methodology and compute the target properties of the test set, thus knowing the correct answer.
S3 Dataset generation Of the 130,831 molecules in QM9 that passed the geometry consistency check, we removed an additional 42 molecules as these had no hydrogen atoms. The remaining structures were modified to a valid xyz format and input files for Gaussian NMR computations were constructed. The scalar coupling constants (including contributions from separate terms), dipole moment vectors, nuclear shielding tensors, mulliken charges and potential energy were parsed from the Gaussian output files and the dataset was written in an easily accessible csv format. The molecule structures were made available in xyz formatted files as well as a single csv file. While the target observable was the scalar coupling constants, the auxiliary data was included in case additional learning from these could be achieved.
S4 Competition details The goal of the competition was to predict the scalar coupling constants from the interaction between a hydrogen and a hydrogen, carbon or nitrogen atom 1,2 or 3 bonds apart. This was to be done without the competitors knowing any information other than the coordinates and type of element of the atoms in the molecules. The dataset was randomly split into a 65%/35% training/test split. The scalar coupling distributions seemed to be very similar with random splits so stratified splits did not seem necessary. The competitors were provided the following files: • structures. csv, which contains the coordinates and element type of each of the 2,358,657 atoms of the 130,775 molecules that constitutes the combined test and training set. • train. csv, which contains the values in Hz of the 4,658,147 scalar coupling constants in the training set as well as which atom pairs in which molecule the coupling is between.
S5. 2 Correlation We investigated how similar the submissions from the top teams were by computing the correlation between a scaled subset of their submissions on the test data. 20,000 data points for each coupling type were drawn randomly from the test set. For each coupling type the submissions were the scaled by their individual root mean square error (RMSE). These two pre-processing steps were done to make each coupling type have an equal impact on the correlation. The same analysis was done on the top 50 team submissions where the submissions were clustered with hierarchical clustering using the 'complete' linkage in the Scipy module  [9].
From these, we use the open source package RDKit  [14]  to generate additional features for each atom, specifically bond order and partial charge. Below we list the total set of features of each molecular object that we use in the network. • Atoms: element, number of neighbors, order of neighboring bonds, partial charge, angle with nearest neighbors. All atoms are included. • Bonds: each atom's element, coupling type (if applicable), bond order (if applicable), distance. These are included regardless of whether there is a chemical bond or not. • Triplets: each atom's element, bond angle. These are included only if there is a chemical bond (one central atom bonded to a pair of other atoms). The methods described can be generalized to include quadruplets (and dihedral angles) as well, although the addition of quadruplets was found to not be helpful in this problem.
S8 Solution 2 -Quantum Uncertainty S8. 1 Data Augmentation The key to the success of this model lies in the data augmentation, and comes from the insight that pair-wise properties can be modelled as atom-wise properties, given that relative position to the paired atom is encoded in the input. This is achieved by making duplicate 'sibling' molecules that are translated to have a di↵erent atom as origin. For a given molecule of N atoms, where M of these are of atom type H, C or N, M siblings are constructed, each centered on a di↵erent H, C or N atom. Furthermore the sibling molecules are padded with dummy atoms to make all molecules have the same number of atoms (29 for this competition), which are later masked in the model. Each atom in the sibling molecules are then matched with the corresponding coupling constant label, where dummy values are used for coupling types that are not present in the dataset, and these are similarly masked later in the model.
S9. 4 Features An atomic pair feature is composed of two atoms' properties and pair properties. (See Figure  S11  ) For atomic properties, the position of atom, atomic number and atomic charge, available in QM9 dataset are used and distance between them and type of coupling are for the pair properties. Those values are embedded and concatenated to build feature vector. Unlike the sum operation, concatenation is not a commutative operation so our feature vectors are dependent on the order of atoms in a pair. To make the feature vector independent of the order of atoms, test time augmentation or data augmentation can be applied, but in the data set, the order of atoms in a pair are sorted. Therefore, we use none of them. For clarity, it should be noticed that this order-dependency does not necessarily lead to permutation dependency of atomic pairs belonging to molecules.
S9. 4. 1 Data Augmentation As described in the previous section, positions of atoms are used in input sequences, therefore rotational and translational invariances are not conserved in our model. Although both invariances are not mathematically preserved, by training augmented data, we make parameters of our model guided to have pseudo-invariance in a certain range of translational and rotational changes. Fortunately, positions of atoms belonging to the QM9 set do not have extreme values, which means this kind of less rigorous strategy can cover all the test cases in the competition. If we need to cover general molecular geometry, this kind of strategy might not work. Generation of noise can be done in every epoch with low computational cost, the size of the original training database being enlarged 10 times with randomly selected noise.
S9. 5 Training and Ensembling In order to achieve a high score, we employed an ensemble technique and pseudolabeling. An ensemble technique is to merge the results of various models and pseudo-labeling is to use new dataset which contains unlabeled data whose label is assumed as the results of previously trained model. We originally planned to use a seed ensemble which uses a number of identical models with di↵erent random seed numbers but because of inequalities of teammates' computing resources. Therefore, models with di↵erent N values (in Figure  S9 ) and epochs are used. An overall training process is illustrated in Figure  S12. At first stage, 15 models are trained using only training dataset with dropout. At second stage, parameters of models were fine-tuned by turning o↵ the dropouts.
By using models trained up to the second phase, labels for test sets (so-called pseudolabel) are predicted. To minimize error of pseudo-label, results of 4-8 trained models up to the second phase are employed. In the third phase, newly constructed training sets which including both original training dataset and pseudolabeled test set are used. Because of due date of competition, we only take 8 models and further trained with the expanded datasets. At phase 3 and 4, 20 epochs are progressed. To mitigate the unpredicted bias from pseudo-labeling, at the last phase, the models are trained with only the original training dataset. Adam optimizer, typical choice, is employed with gradient clipping. Linear warmup and linear decay are used for the stable convergence of training. All weights are initialized with uniform distribution (mean=0.
S10. 2 Features In addition to the molecular graph structure (atoms and bonds as inferred by OpenBabel  [34] ) the model uses a number of atom and bond features. For atoms we use: electronegativity, first ionization energy, and electron a nity for each atom type, as well as atom Mulliken charge taken from the QM9 dataset  [35, 36]. For edges we use features: bond length, bond angle (for artificial 1-3 edges), and the dihedral angle (for artificial 1-4 edges). All features were standardized based on the training set statistics. Labels were standardised for all models, except those used for predicting 1JHC, where we only subtract the mean (which we empirically determined to produce better results).
We also used the Mulliken charges as initial features for pretraining. Our training proceeded in three stages. 1. Three nets were trained on the 1J, 2J, and 3J couplings with the Mulliken Charges on an internal split on the training data. 2. Eight nets were trained on the 1JHC, 1JHN, 2JHH, 2JHC, etc. couplings with the Mulliken Charges on an internal split on the training data, with weights initialized to be the values in stage 1. 3. Each of the nets in stage 2 was trained on the full dataset, without access to the Mulliken charges. The internal split was constructed using an 80/10/10 train/validation/test split on the training dataset provided in the Kaggle competition. The model was optimized using the AMSGrad optimizer for 200 epochs. The learning rate was varied using cosine annealing.
S13 Code and data Code and data is available on http://osf. io/kcaht and http://github. com/ larsbratholm/champs_kaggle. These contain the following • List of molecules removed as they did not contain hydrogens. • List of molecules removed due to one or more couplings being outliers. • List of training and test molecules. • Script to convert QM9 extended XYZ-files into both regularly formatted XYZ-files as well as input files for Gaussian NMR computations. • Script to create the Kaggle dataset from the output files of Gaussian NMR computations. • Archive of all extended XYZ-files of QM9 that passed the consistency check of the original paper. • Archive of all regular formatted XYZ files parsed from the extended XYZ file format of the QM9 dataset. • Archive of all Gaussian input files. • Archive of all Gaussian output files.
Acknowledgments. Part of this work was carried out using the computational facilities of the  Advanced Computing Research Centre, University of Bristol. WG is partially supported by the  EPSRC National Productivity Investment Fund (NPIF) for Doctoral Studentship funding. LAB thanks the  Alan Turing Institute  under the  EPSRC grant   EP/N510129/1. DRG is supported by the  Leverhulme Trust  ( Philip Leverhulme Prize ) and  Royal Society  ( URF/R/180033 ). LAB and DRG acknowledge support of this work through the  "CHAMPS" EPSRC programme grant   EP/P021123/1. SC was supported by  National Research Foundation of Korea  ( 2018R1D1A1B07049981 ,  2019M3E5D4065968 ) funded by the  Ministry of Science and ICT. We furthermore thank  Prof. Jan H. Jensen  for access to additional computing resources and  Dr.
opted to use a custom score function to evaluate the submissions. The dataset included 8 di↵erent types of coupling: 1JHC (coupling between a hy- drogen and a carbon separated by 1 covalent bond), 1JHN, 2JHH, 2JHC, 2JHN, 3JHH, 3JHC and 3JHN. Since the number of couplings of each type di↵ered dra- matically (811,999 3JHC couplings in the test set, but only 24,195 1JHN) and spanned di↵erent ranges, a bad choice of scoring metric could easily be domi- nated by the performance on the 3JHC coupling constants. Our loss function is the logarithm of the geometric mean of the mean absolute error for each type: score.
, 2018; Pecl et al. , 2017). Plant communities are no exception to this biodiversity crisis  (Cardinale et al. , 2011; Lenoir et al. , 2008; Staude et al. , 2020). This is particularly worrying since terrestrial vegetation accounts for 80% (450 Gt C) of the living biomass on Earth  (Bar-On et al. , 2018). Given the central role of vegetation in ecosystem productivity, structure, stability and functioning  (Cardinale et al. , 2011) , assessing biodiversity status and trends in plant communities is paramount for other kingdoms of life and human societies alike. Monitoring trends in plant biodiversity requires adequate data across a range of spatio-temporal scales  (Kühl et al. , 2020; Pimm, 2021). Large independent collections of plant occurrence ferred, information seldom provided by existing global plant datasets.
Although many vegetation plots have been recorded, most are not available to the global research community. A recent initiative, called 'sPlot', compiled the first global vegetation plot database, and continues to grow and curate it. The sPlot database, however, is extremely unbalanced spatially and environmentally, and is not open-access. Here, we address both these issues by (a) resampling the vegetation plots using several environmental variables as sampling strata and (b) securing permission from data holders of 105 local-to-regional datasets to openly release data. We thus present sPlotOpen, the largest open-access dataset of vegetation plots ever released. sPlotOpen can be used to explore global diversity at the plant community level, as ground truth data in remote sensing applications, or as a baseline for biodiversity monitoring.
Main types of variable contained: Vegetation plots (n = 95,104) recording cover or abundance of naturally co-occurring vascular plant species within delimited areas. sPlotOpen contains three partially overlapping resampled datasets (c. 50,000 plots each), to be used as replicates in global analyses. Besides geographical location, date, plot size, biome, elevation, slope, aspect, vegetation type, naturalness, coverage of various vegetation layers, and source dataset, plot-level data also include communityweighted means and variances of 18 plant functional traits from the TRY Plant Trait Database. Spatial location and grain: Global, 0. 01-40,000 m². Time period and grain: 1888-2015, recording dates. Major taxa and level of measurement: 42,677 vascular plant taxa, plot-level records. Software format: Three main matrices (.
K E Y W O R D S big data, biodiversity, biogeography, database, functional traits, macroecology, vascular plants, vegetation plots data do exist at the global or continental extent via the Botanical Information and Ecology Network (BIEN;  Enquist et al. , 2016) , the Global Inventory of Floras and Traits (GIFT;  Weigelt et al. , 2020)  or the Global Biodiversity Information Facility (GBIF; https://www. gbif. org/). However, these databases suffer from one or several of the following limitations: (a) imbalance towards tree species only; (b) lack of data on how individual plant species co-occur and interact locally to form plant communities; and (c) coarse spatial resolutions (e. g. , one-degree grid cells), which preclude intersection with high resolution remote sensing data and the assessment of biodiversity trends at the plant community level  (Boakes et al.
, 2018). It is not surprising, therefore, that these data are rarely used in global-scale research on the biodiversity of plant communities  (Aubin et al. , 2020; Franklin et al. , 2017; Wiser, 2016). The sPlot initiative tries to close this data gap. It consolidates numerous local to regional vegetation-plot datasets to create a harmonized and comprehensive global database of georeferenced terrestrial plant species assemblages  (Bruelheide et al. , 2019). Established in 2013, sPlot v3. 0 currently contains more than 1. 9 million vegetation plots, and is fully integrated with the TRY database  (Kattge et al. , 2020) , from which it derives information on plant functional traits. The sPlot database is increasingly being used to study continental-to-global scale vegetation patterns  (Cai et al.
, 2021; Testolin, Attorre, et al. , 2021; Testolin, Carmona, et al. , 2021) , such as the relative contribution of regional versus local factors to the global patterns of fern richness  (Weigand et al. , 2020) , the mechanisms underlying the spread and abundance of native versus invasive tree species  (van der Sande et al. , 2020) , and worldwide trait-environment relationships in plant communities  (Bruelheide et al. , 2018). Yet, most of these data are not open-access. Here, we secured permission from data holders in the sPlot database to openly release a dataset composed of 95,104 vegetation plots. We selected the plots to be released using a replicated environmental stratification, in order to represent the entire environmental space covered by the sPlot database. This maximizes the benefits of releasing these data for a wide range of potential uses.
The selected vegetation plots stem from 105 databases and span 114 countries (Figure  1 ). This resampled dataset (sPlotOpen -hereafter) is composed of: (a) plotlevel information, including metadata and basic vegetation structure descriptors; (b) the vascular plant species composition of each vegetation plot, including species cover or abundance information when available; and (c) community-level functional information obtained by intersection with the TRY database  (Kattge et al. , 2020). sPlotOpen is specifically designed for global macroecological studies, for example, the exploration of functional diversity patterns of communities with continental-to-global extent. We expect, however, that sPlotOpen might likewise prove useful to answer a range of different questions, related for instance to species co-occurrence patterns, the definition of species pools, the link between regional versus local determinants of species diversity, or the niche overlap between co-occurring species.