dataesr/scientific-paragraphs-categorization huggingface dataset:
Data and Software Availability The METASPACE-ML code for inference is available at https://github. com/metaspace2020/metaspace and covers scores calculation, METASPACE-ML and FDR calculation based on the trained model. The METASPACE-ML model is available at https://github. com/metaspace2020/metaspace/tree/master/metaspace/scoring-models. The functionality is available through METASPACE in the annotations settings, see the model "v2. 20230517 (METASPACE-ML)" in "Analysis version". The public training and testing datasets are available for download on METASPACE and more information on each dataset can be accessed from Tables  S2  and S3 , respectively.
We found a significant difference in time spent in a grazing position and distance traveled in the first 5 min of observation, relative to subsequent five-minute bins, suggesting the presence of an observer effect; therefore, we removed the first 5 min of every follow. As fish observations had different durations (30-60 min), distance traveled was standardized per minute (divided over total follow duration), and analysis of KUD was done by capping all follows at 30 min (total of 25 min excluding initial 5 min), as total follow time may affect total space use. The proportion of time spent in grazing position, proportion of territorial invasions out of all grazing events, and associations by heterospecific fish (predatory and nonpredatory) were calculated across all follow durations. We computed distance traveled using the adehabitatLT package in R and 95% utilization kernel using a biased   (Calenge, 2006; R Core Team, 2020; RStudio Team, 2020).
| Analysis All computations were conducted in R (version 4. 0. 3) using R studio and the tidyverse package  (RStudio Team, 2020; Wickham et al. , 2019). We used linear mixed-effects models to explain variations in distance traveled, 95% KUD, proportion of grazing events that were territorial invasions, and associations with predatory and nonpredatory fish for A. triostegus on Palmyra Atoll and Mo'orea. We specified full models with the nlme package  (Pinheiro et al. , 2022) , using distance traveled (per min), 25-min 95% KUD, proportion of grazing events that were territorial invasions, associations with predatory fish, and associations with nonpredatory fish as response variables; social status (shoaling or solitary) as a fixed effect; and site, island, and time of day as random effects, since time of day can affect surgeonfish behavior (Table  A2 ;  Montgomery et al.
, 1989 , Zemke-White et al. , 2002). As our behavioral observations were done on shoals of different sizes, we used linear mixed-effects models fit by maximum likelihood (ML) to explain variations in the response variables mentioned above for shoaling A. triostegus only. We specified full models the same as above using the nlme package and added shoal size as a fixed effect instead of social status (shoaling or solitary). Best-fit models were selected according to small-samples corrected AIC (AICc) using the package MuMIn  (Barton, 2020). For 95% KUD, the data distribution was non-normal; thus, we transformed the data with a log normal transformation as suggested by  Zuur et al. (2009). As time spent in grazing position data was collected differently between shoaling and solitary fish (shoal-scale vs.
| Fish sampling To directly test whether any differences in foraging and movement behavior that were detected between shoaling and solitary A. triostegus affected their diet and trophic ecology, we collected 100 individuals (25 shoaling and 25 solitary from two different sites in Mo'orea only; Figure  A1 , sites P and H, which are ~3 km apart and separated by a channel) to compare muscle tissue stable isotope values and assess the nutritional quality of stomach contents and fecal matter. Fish were collected using hand spears. Shoals of 50 individuals or larger were defined as 'shoaling fish' for this study. All fish were collected between 1000-1600 h, to ensure the fish had been feeding for sufficient time to have contents in their stomach (i. e. , based on gut throughput time data from congeners in  Polunin et al.
3 mg were loaded into tin capsules, which were sent to the University of California, Davis Stable Isotope Facility for analysis. Samples were analyzed for δ 13 C and δ 15 N isotopes using a PDZ Europa ANCA-GSL elemental analyzer interfaced with a PDZ Europa 20-20 isotope ratio mass spectrometer (Sercon Ltd. ). We used linear models to explain variations in nitrogen and carbon stable isotope values for shoaling and solitary A. triostegus on Mo'orea. We specified full models using the nlme package  (Pinheiro et al. , 2022) , with δ 15 N and δ 13 C as response variables; and social status (shoaling or solitary), site, and fish size (standard length) as predictors (Table  A2 ). Best-fit models were selected according to small-samples corrected AIC (AICc) using the package MuMIn  (Barton, 2020) , which compares all possible iterations of combined and individual predictors from the full model.
Additionally, we generated Bayesian standard ellipses (40% confidence level) for each social behavior (shoaling or solitary) and backreef site using the SIBER package to estimate isotopic niche space  (Jackson et al. , 2011). We compared the size of the ellipses by fitting Bayesian models adjusted for small sample sizes (SEAc) and calculated overlap in ellipse area between the two sites and social behaviors, which can be used to determine overlap in diets and niche space  (Eurich et al. , 2019). Shared overlap of >60% was considered a significant shared niche space  (Eurich et al. , 2019; Schoener, 1968).
To measure ash content, we precombusted aluminum weigh boats at 450°C for 6 h and preheated the samples in an oven at 100°C overnight to ensure full water loss. We then combusted preweighed samples in a muffle furnace for 6 h at 450°C and reweighed samples to measure ash content. Finally, we estimated total carbohydrates using a method commonly used for estimating carbohydrate content in food, as carbohydrates = 100-proteins-lipids-ash, where variables are in % dry weight  (Opstvedt et al. , 2003; Rempel et al. , 2022; Southgate, 1969). We used linear models to explain variations in macronutrients for A. triostegus on Mo'orea. We specified full models using the nlme package  (Pinheiro et al. , 2022) , using percent dry matter of protein, carbohydrates, and lipids in stomach contents and feces as response variables; social status (shoaling or solitary), site, the interaction of social behavior and site, and fish size (standard length) as predictors (Table  A2 ).
This benthic data was collected asynchronously with fish survey data and not at the same time at two sites limiting strength of comparisons. However, while these differences in temporal sampling might be expected to exacerbate any underlying differences in benthic habitat analysis showed no statistically significant differences within any type of cover across islands although there were trends for more live coral and CCA at Palmyra Atoll as compared to Mo'orea. This tendency could, however, be an artifact of temporal differences in sampling, noting that there were significant coral die off events in both regions, but particularly French Polynesia, in 2019.
The stripe along the posterior end appears similar, although the caudal fin has been damaged and it is not possible to assess the markings along its dorsal side. (C) A solitary A. triostegus sighted in 2019 and potential resight in 2022 with same melanistic patterns. The fish shows an interruption along the dorsal portion of the 5 th , an incomplete bar in at the caudal peduncle, and otherwise complete bars with no mottling. Downloaded from https://onlinelibrary. wiley. com/doi/10. 1002/ece3. 9686 by Cochrane France, Wiley Online Library on [27/02/2023]. See the Terms and Conditions (https://onlinelibrary. wiley. com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License To understand how variation in foraging behavior may have been influenced by variation in benthic habitats across and within islets we utilized existing benthic data (Palmyra Atoll collected in 2006) supplemented with new benthic data collected with identical methods by the same observers at Mo'orea Island(2022).
In 2021 the facial markings appear to have grown to almost fully occupy the anterior side of the second bar. The stripe along the posterior end appears similar, although the caudal fin has been damaged and it is not possible to assess the markings along its dorsal side. (C) A solitary A. triostegus sighted in 2019 and potential resight in 2022 with same melanistic patterns. The fish shows an interruption along the dorsal portion of the 5 th , an incomplete bar in at the caudal peduncle, and otherwise complete bars with no mottling. Downloaded from https://onlinelibrary. wiley. com/doi/10. 1002/ece3. 9686 by Cochrane France, Wiley Online Library on [27/02/2023]. See the Terms and Conditions (https://onlinelibrary. wiley. com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License To understand how variation in foraging behavior may have been influenced by variation in benthic habitats across and within islets we utilized existing benthic data (Palmyra Atoll collected in 2006) supplemented with new benthic data collected with identical methods by the same observers at Mo'orea Island(2022).
We show that this architecture allows for highly transferable models that are able to accurately generalize on large molecular systems, as well as in the condensed phase aer being trained on small monomers and dimers only. This paper is organized as follows. Section 2 describes the model architecture, from the Allegro 31 equivariant embedding to the output and physics modules. It also describes the particular FF terms that we used for the pretrained model, named FENNIX-OP1, that we provide with this work. Section III focuses on the FENNIX-OP1 model and provides details on its construction, its target properties, the datasets used for training and the different training stages that were required. In Section 4, we validate the model via several applications. First, we show that the model predicts accurate dissociation energy curves of some simple molecules.
Model architecture The FENNIX (Force-eld-Enhanced Neural Network Interactions) model is based on a local multi-output equivariant model that processes atomic neighborhoods and predicts multiple atomic or pairwise properties. As an example for this work, we will present a model that outputs local pairwise energy contributions, charges and atomic volumes. The output is subsequently enriched by a "physical" module that computes force eld terms such as electrostatic and dispersion energy terms. The core of our model is a slightly modied version of the Allegro local equivariant model presented in ref. 31. We use the Allegro model as a general embedding of atomic pairs which is then fed into independent neural networks that predict the target properties. The choice of the Allegro model is mostly motivated by two of its properties: (i) it is strictly local and thus allows for favourable scaling with system size and efficient parallelization; (ii) it is equivariant and thus allows the prediction of tensorial properties such as atomic multipoles (which are not used in this work but will be the focus of future improvements of the model).
A compact ow diagram of the model is shown in Fig. 1. In this section, we will briey describe the Allegro architecture and our modications to the model. The theoretical analysis of this architecture was thoroughly done in the original paper  31  so that we will only review here the main points necessary to the understanding of the model and our improvements to the architecture. We will then present the output module that processes the Allegro embedding. Finally, we will describe the physical module and the particular functional form for the FENNIX-OP1 potential energy surface that we used in this work. with Rik = Rk -Ri the vector going from the position of atom i to atom k. The rst part of the descriptor x ij is built so that it is invariant under the action of certain geometric symmetries (i.
60  In this work, we propose to use a positional encoding that encodes coordinates in the periodic table using sine and cosine functions of different frequencies. It is inspired from ref. 61  where it is used to encode the position of words in sentences in the context of natural language processing. The column index c is encoded as a vector e c of dimension d col as: ck˛0;. ; d col ; ðe c Þ k ¼ ( sin À c g col 2i=d col Á if k ¼ 2i cos À c g col 2i=d col Á if k ¼ 2i þ 1 (8) and similarly for the row index with dimension d row and frequency parameter g row. For this work, we x the dimensions and frequency parameters to d col = 10, d row = 5, g col = 1000 and g row = 100 which provide a good compromise between the compactness and richness of the representation. These could also be treated as hyperparameters or even learned during training (for the frequency parameters).
This allows the output MLP to easily access simple pairwise information and should let the Allegro embedding specialize in ner many-body correlations. This compositional approach allows for a great exibility in the model's output, which is especially useful when experimenting with a ML parametrization of physical models. The output module for the FENNIX-OP1 model is composed of three targets: a local energy contribution E NN i ¼ P j˛N ðiÞ E NN ij (which is a simple scalar output), an atomic partial charge q NN i ¼ P j˛N ðiÞ Dq ij À Dq ji (through antisymmetrized charge exchange) and an atomic volume v NN i (constrained to be positive).
Physics module and energy functional form Finally, the physics module uses the results from the output module and feed them into physically-motivated models to enrich the output. In the case of FENNIX-OP1, the force eld module is composed of an electrostatic energy term E C ij and a pairwise dispersion term E D ij. The functional form of FENNIX-OP1 is then given by: E OP1 ¼ X i E NN i þ X i;j\i E C ij þ X i;j\i E D ij (11) The rst term E NN i is the neural network contribution that accounts for short-range interactions that we introduced in Section 2. 2. We model the electrostatic interaction E C ij via a Coulomb potential with uctuating charges and the Piquemal charge penetration model: 62 E C ij ¼ 1 R ij 2 6 6 6 6 6 4 N i N j þ N j ðq i À N i Þf a À R ij r vdw i Á þN i À q j À N j Á f a R ij.
We furthermore mention that the physics module is not limited in principle to compute energy terms. The aim of this module is to be a general physical interface between the ML embedding and the nal target properties. For example, we use it in the FENNIX-OP1 model to correct for ML-predicted charge exchanges that would deplete the valence shell of an atom. The implementation also provides charge exchange via a simple bond-capacity model 64 that leverages ML-predicted atom-inmolecule electronegativities and which capabilities will be explored in future iterations of the FENNIX model. Each physical model is implemented as a simple Pytorch Module that takes as input a dictionary which contains previously computed properties. It then adds its contribution and outputs the enriched dictionary. These physical submodules can then easily be chained, and it facilitates the implementation of additional models.
Datasets and training procedure In this section, we start by providing some details on the construction of FENNIX-OP1 model. We then review the datasets that we used for training the model and its target properties. Finally, we will focus on the non-trivial task of training a multi-output FENNIX model. The output module is composed of three independent identical MLPs for the short-range energy contribution, the charge exchange and the atomic volume. The input of these MLPs is the concatenation of the two-body scalar embedding and the nal scalar embedding. They have 5 hidden layers with 256, 128, 64, 32 and 16 neurons. In total, the model has approximately 1. 2 million parameters. The output module also has a "constant" submodule that simply provides the atomic reference energies. Importantly, we used the CCSD(T) isolated atom energies as a reference instead of the average atomic energy over the dataset that is typically advised when training a ML model.
Datasets and target properties For the FENNIX-OP1 model, we chose to reproduce high-level coupled-cluster energies and we thus selected three of the few freely-available and generalist datasets that provide such data: the ANI-1ccx  16  dataset, the DES370K 66 dataset and the q-AQUA dimers dataset. 20  The ANI-1ccx dataset provides approximately 500 000 CCSD(T)/CBS total energies of various neutral monomers and dimers (composed of the elements H,C,N,O) in equilibrium and out-of-equilibrium congurations. It also provides atomic forces at DFT level that were crucial to speed-up the beginning of the training procedure. Importantly, it provides partial charges and atomic volumes obtained from a Minimal Basis Iterative Stockholder 67 (MBIS) partitioning of the DFT electronic density that were used to parameterize the force eld terms.
The DES370K dataset is composed of approximately 370 000 CCSD(T)/CBS interaction energies of diverse dimers in various congurations. It comprises both neutral and charged molecules. The latter were discarded for the training of the FENNIX-OP1 model as out-of-scope for this study. It also provides SAPT decompositions of the interaction energies that we used to regularize the Coulomb interactions. The q-AQUA dimers dataset is composed of more than 70 000 water dimer interaction energies at CCSD(T)/CBS level. We used this dataset in the end of the training in order to ne-tune the model for handling water. The FENNIX-OP1 model has then three target properties that use the available data: the CCSD(T) total energy of the system as modelled by V OP1 described in Section 2. 3, the MBIS partial charges q NN i and the ratio of MBIS volumes to free atom volumes v NN i /v free i.
Throughout, we used the AdamW optimizer 68 implemented in Pytorch, as its algorithm for weight decay provides one of the best compromise between training speed and accuracy. We used a fairly strong weight decay parameter of 0. 5 for all the parameters in the output module and no weight decay for the Allegro parameters. In all stages, we used the same random 10% of the dataset as a validation set and optimized the model using mini-batches of 256 congurations from the ANI-1ccx and 64 congurations from DES370K and q-AQUA when they are used. The training procedure for the FENNIX-OP1 model required four stages. In the rst stage, the model was trained to reproduce DFT total energies and forces using the short-range contribution V NN only. We also trained at the same time the charges and atomic volumes to reproduce the MBIS targets.
At this stage, only the ANI-1ccx dataset was used. The loss function for this stage is given by: L ð1Þ ¼ l E À E DFT À E NN Á 2 þ l F X 3 j¼1 X Nat i¼1 F DFT ij À F NN ij 2 þl q X Nat i¼1 À q MBIS i À q NN i Á 2 þ l v X Nat i¼1 v MBIS i v free i À v NN i v free i ! 2 (18) with l E = 0. 001, l F = 1 and l q = l v = 1000 and F NN is the model's predicted force obtained by automatic differentiation (Pytorch's autograd) of the total energy E NN. As suggested in previous studies,  31  we strongly favour learning forces over energies in the beginning to accelerate the training procedure. We note that the provided loss is for a single conguration and that, in practice, it is averaged over the congurations in the mini-batch. We further added a regularization in order to minimize the offdiagonal elements of the covariance matrix of the scalar embedding's features over a batch.
This promotes learning statistically independent features in the embedding which should be favourable for a multi-output network and we found that it led to models with better generalization capabilities. In this stage, we train all the parameters in the embedding and output modules with a starting learning rate of 10 -3. Furthermore, we used a learning rate scheduler that reduces the learning rate when the error on the training set stops diminishing for a few steps (we set the patience of the scheduler to 10 epochs and the learning rate scaling factor to 0. 8). Aer about 100 epochs, progress of both training and validation steps slowed down and we modied the energy and force parameters to l E = 0. 01 and l F = 0. 1 in order to obtain a more balanced training. We stopped the rst stage when the learning rate reached 10 -4.
In the third stage, the embedding and charge and volumes MLPs are frozen again and the energy MLP is nally retrained to reproduce CCSD(T) energies from both ANI-1ccx total energies and DES370K interaction energies. We used the same type of mean-square loss functions with l E = 0. 1 and l DES = 5. Again, we train the energy MLP until the error is close to the previous stage, unfreeze the whole model and optimize again all the parameters. For this stage, we also optimize the parameters from the physical module in order to reach the lowest error possible. We stop the training when the learning rate reaches 10 -5. In the last training stage, we rene the model for water by including the q-AQUA water dimers interaction energies in the loss function. We also generated batches of randomly deformed water monomers (with very large deformations) and trained the model to reproduce forces from the highly accurate Partridge-Schwenke potential energy surface (that was itself tted on high-accuracy coupled-cluster data).
This was particularly helpful to reproduce the molecule's bending energy surface away from equilibrium where fewer data are available in the ANI-1ccx dataset. At the end of the training procedure, the model reached a root mean square error (RMSE) of less than 4 kcal mol -1 for CCSD(T) total energies on both validation and training sets of the ANI-1ccx dataset. While it was possible to reach much lower errors with less regularized models (less than 1 kcal mol -1 ), we found that they were unstable when performing MD and concluded that they were overtting the data. We thus favoured a more strongly regularized, perhaps slightly undertted model that allowed for better generalization in the condensed phase. The model also reached a RMSE of about 0. 35 kcal mol -1 on both DES370K and q-AQUA datasets.
Finally, it reached a RMSE of about 0. 017e for charges and about 0. 017 for volume ratios. As for total energies, less regularized models allowed for much lower errors (less than 0. 008e for charges for example) but with visible overtting, leading to irregular coulomb interactions and unstable dynamics. As a validation for the water interactions, we used the standard energy benchmarks when training force elds for water. The model gives a RMSE of 0. 13 kcal mol -1 on the Smith dimer set  69  which are representative of the most stable water dimer congurations. This low error is not surprising as the Smith dimers are very close to congurations present in the q-AQUA dataset on which the model was trained. For a more challenging test, we used a set of typical water clusters up to hexamers.