dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements The authors gratefully acknowledge funding for this work; The  European Space Agency  supported the two phases of the  Climate Change Initiative (CCI)  for  Sea Level  and the  CCI+ phase for Coastal Sea Level , which has provided the majority of the support leading to the outcomes herein described, in addition to the specific grant reference 4000126561/19/I-NB for Yvan Gouzenes. We thank two anonymous reviewers for their comments that helped us to improve the manuscript.
A Quantitative Structure-Activity Relationship (QSAR) and Quantitative Structure-Property Relationship (QSPR) are among the most used tools in the nanotoxicology prediction. Villaverde et al. [8]  analyzed QSAR/QSPR tools for risk assessment, modeling methods, and validation procedures with regard to their potential for meeting requirements within the European legislative framework for authorization of nano-formulations. The authors argued that the standardization of protocols is needed, even for high-quality and well-described datasets. Quik et al. [9]  analyzed available models and their parametrization related to NM properties for risk assessment. The authors showed an opportunity for development of new predictive in silico methods when full mechanistic functioning of the NM-biological surfaces system is accounted for.
The Nanoinformatics Roadmap 2030  [5]  is a compilation of state-of-the-art commentaries from multiple scientific fields dealing with issues involving NM risk assessment and governance. The authors addressed three recognized challenges that nanoinformatics face in general such as limited data sets, limited data access, and regulatory requirements for validating and accepting computational models. The authors warned for the need of interconnecting harmonized databases in a framework that entails early use of data for regulatory purposes, e. g. , read-across method of filling data gaps, to prevent unstructured progress in generating data. Schemes for clustering NMs have been proposed and reviewed elsewhere  [10] [11] [12]. Lamon et al. [11]  addressed categorization schemes, grouping for read-across approaches and computational applications for ranking NMs.
The authors stated that the few studies dealing with NM similarities used non user-friendly tools on limited datasets. The authors suggested that toxicity datasets and nano-specific properties should both be investigated to identify groups of NMs. Giusti et al. [12]  noted how in silico methods contribute at different stages of NM grouping such as in developing vs. supporting initial grouping hypotheses. The methods to be used vary, from read-across, unsupervised, and supervised machine learning (ML) methods to several QSAR approaches. The Organization for Economic Cooperation and Development (OECD) have published a set of validation principles of QSAR models  [13]. These principles detect that models should have a well-defined endpoint, unambiguous algorithm, defined domain of applicability, appropriate measure of goodness-of-fit, robustness and predictivity, and mechanistic interpretation.
The templates include an adaptation of the QMRF, a reporting template for Physiologically-Based PharmacoKinetic (PBPK) and environmental exposure models, applicable to NMs. The authors demonstrated the value of these templates on reporting different models and overviewing the landscape of available models for NMs. ToxRTool (Toxicological data Reliability assessment Tool), which is a compilation of reliability assessment questions, can also be employed to asses meta-analyzed studies for human health hazard assessments  [17]. Based on the above reviews, it is evident that a lot of effort and research is needed so that in silico tools are both accepted by regulators and implemented in a harmonized way to maximize their utility. The applicability domain was discussed through the existing reviews as well as the limitations of the dataset (e.
This study shows that tree algorithms (i. e. , random forest) are the most common ML used due to insensitivity of data defects, resistance to overfitting, and robustness in small datasets. Regression models traditionally used in classic QSARs are still common but trending shifts are toward nonlinear algorithms. Artificial Neural Networks have a great deal of potentiality but data paucity limits their use for the time being. This review is preceded by another analysis of the literature identified herein, focusing on data collection, curation, and utilization  [18]  as a precursor to data pre-processing and model implementation.
Analysis Each of the 86 identified articles were reviewed in detail and information related to the feature selection process, data processing techniques, model implementation (model category and algorithm), model validation, and applicability domain was extracted. There were no definite guidelines in choosing pre-processing techniques, model implementation, and validation metrics in order to assess the performance and applicability domain of computational models. Figure  1  shows a summary diagram of the process steps applied to the identified, which follows a generalized roadmap from data extraction to model validation and applicability domain. We focused on ML models predicting ecotoxicological (e. g. , effects on terrestrial organisms, aquatic toxicity, etc. ) and human health toxicological endpoints.
Analysis Each of the 86 identified articles were reviewed in detail and information related to the feature selection process, data processing techniques, model implementation (model category and algorithm), model validation, and applicability domain was extracted. There were no definite guidelines in choosing pre-processing techniques, model implementation, and validation metrics in order to assess the performance and applicability domain of computational models. Figure  1  shows a summary diagram of the process steps applied to the identified, which follows a generalized roadmap from data extraction to model validation and applicability domain. This roadmap comprises five main sequential parts and our focus herein are the four subparts consisting of data pre-processing, model implementation, validation, and applicability domain.
Dataset Formation The first part in processes of ML model implementation, Dataset formation (Figure  1 ), contains four subparts. First, data collection is carried out, either from existing literature and databases or from new data experimentally created. A combination of the previously mentioned sources can also be used. Second, the information on NPs is extracted, including nano-specific descriptors (size, coating, zeta potential, etc. ) and the type of NPs (metal, metal oxide, carbon-based, etc. ), derived either from the data sources or elsewhere, e. g. , the manufacturer data sheets. Besides nano-specific descriptors as inputs and theoretical descriptors can be generated using available software and used as input data. Third, inputs including study design information is attained, such as the testing system (in vitro, in vivo), species (human, bacteria, etc.
Sequential forward selection (SFS) and sequential forward floating selection (SFFS) At each step of the selection process, the descriptor that led to the highest model performance is retained until a specified number of descriptors are selected. As an extension to SFS, after each forward selection step, SFFS conducts backward elimination to evaluate descriptors that can be removed. [  [49] [50] [51] [52]  Multiple linear regression (MLR) feature selections (1) In MLR, a set of models is examined for stability and validity. (2) One of the most commonly used methods is the GA-MLR. GA deals with optimizing the nonlinear parameters, while the linear ones are calculated by MLR. (3) MLR with expectation maximization (MLREM) is an iterative method that increases the dataset sparsity varying the values of control hyperparameters.
Normalization and Discretization In Danauskas and Jurs  [22] , the base-10 logarithm was applied to limit the range of data while others used normalization of inputs and outputs for increasing accuracy  [68]. Another method for homogeneous normalization was taken by References  [38, 69] , where the descriptor pool was pre-processed prior to modeling by autoscaling. This approach is necessary when the data consists of variables with different scaling. A robust z-score is used to normalize the data in order to minimize the influence of outliers  [70]. Choi et al. [60]  examined several normalization techniques (z-score, min-max, log10) for each attribute in order to reduce the skewness of the data and choose the most appropriate showing that each dataset (and each variable) may require different normalization techniques.
However, there are cases of models where dependent variables are encoded to indicators that only express presence or absence in each dataset instance. One hot encoding is a procedure of converting categorical variables into numeric data to be applied to ML algorithms. These variables take values of 0 or 1 depending on whether a particular nano-feature or experimental endpoint is absent or present  [56, 71]. Studies used one-hot encoding in models where categories cannot be used, such as in linear regression  [60, 72]. Within the reviewed articles, one hot encoding was applied in 7% of the studies. Attribute transformation, such as discretization of numerical attributes and functional transformation, are also commonly performed  [73]. Discretization of input was performed in two of the reviewed studies  [74, 75]  based on expert judgment or equal frequency distributions.
Class Balancing An issue encountered in both the training and evaluation phases is that hazard classes (i. e. , the toxicity classes) are often unbalanced, which means that the number of samples corresponding to one value of the class (e. g. , non-toxic) is much higher than the number of samples corresponding to the other values of the class (e. g. , toxic)  [14]. This imbalance in a dataset, which is an issue particularly prevalent in nanotoxicology, has a negative effect on the algorithm performance. Furthermore, 8% of the studies mention that their dataset had equal outcome classes, while, on the other hand, 4% of studies tackled the imbalance issue by resampling the training dataset. Resampling can be done by applying the Synthetic Minority Oversampling Technique (SMOTE), which is a supervised instance algorithm that oversamples the minority instances using the k-nearest-neighbor (kNN)  [60, 67, 77].
Missing Values Handling missing values enhances the reliability of the dataset and expands data interoperability, which offers the nano-safety community complete datasets to be used in novel modelling. There are three types of supervised data filling approaches, such as QSAR methods  [78] , trend analysis, and read-across (interpolation or extrapolation). They are based on different assumptions and, as such, require a different minimum number of data points  [5]. Gajewicz  [79]  mention that existing methods of read-across methodologies are expert knowledge-dependent making the prediction prone to bias. To tackle this issue, they propose a novel quantitative read-across approach based on a simple transparent algorithm for filling data gaps. Several computational tools have been developed for supporting grouping and read-across.
Giusti et al. [12]  provide an update of existing approaches in NMs grouping while suggesting future recommendations. Other approaches for filling data have been proposed that are dataset-specific. For example,  Ban et al. [80]  used curve-fitting to calculate missing ages based on the age-weight relationships of different species. While assessing data quality and completeness, nano-specific filling in of missing values using manufacturer's specifications and/or estimations  [60, 64]  was suggested within the Safe and Sustainable Nanotechnology (S2NANO) (http://portal. s2nano. org/ (Webpage accessed autumn 2019)) database. Furxhi et al. [72]  investigated the robustness of several ML tools on generated versions of the dataset by removing values artificially. Recently, an integration of two data gap filling techniques to predict neurotoxicity for non-NMs was implemented, which demonstrated the capacity of integrating methodologies  [81].
Molecular Structures' Codification An additional issue in the pre-processing of data is the description of molecular structures. Among the most common methods to codify chemical structures are (i) the chemical graph, which represents structures by connection tables, (ii) the linear notations as Simplified Molecular Input-Line Entry System (SMILES), and iii) the de-facto standard chemical formats. SMILES can be obtained by common software like ChemSketch (https://www. acdlabs. com/resources/freeware/chemsketch/ (Webpage accessed autumn 2019)). From the cases gathered, 23 of them use line notations such as SMILES (5 cases), based optimal quasi-SMILES (14 cases), and Improved SMILES (4 cases). Experimental in vitro characteristics and exposure conditions are important variables in the representation of a potential toxicity since the same type of NPs may exhibit diverse effects in different biological conditions.
Data Splitting The final component of data pre-processing/transformation is the splitting of the dataset prior to model implementation. Surprisingly, only 41% of studies mention the technique used and even fewer mention the presence of outliers and how the removal improved model performance. Often such information is omitted as unimportant. Yet such details ensure the reproducibility of a method. Datasets are split into different sub-sets with different roles of (i) a set for training a statistically significant and reliable model, (ii) the test set to measure robustness, and (iii) the validation set to assess predictability of the trained model. Training is done to adjust the model parameters while preventing overfitting. Good predictivity may be achieved for substances significantly similar to those in the training set.
A model will perform inadequately for test set substances that differ from the training set. Thus, instances should be selected in a way that ensures that test set substances lie within the properties space defined by the training set  [46, 87]. In some cases, for further evaluation, unseen datasets are used in order to test the model on data, which are absent in the training and validation step  [60, 88, 89]. Distribution of variables into the training and validation set has an influence on model performance  [89]. Several techniques are mentioned in the reviewed studies, including balanced splitting based on one specific variable  [90]. Keeping extreme responses (i. e. , the highest and lowest range of a variable in the training set  [46, 84, 91] ) avoids the risk of extrapolating out of the response range.
Mikolajczyk et al. [94]  sorted NPs along increasing values of zeta potential, and then included every third NP in the validation set, using the remaining NPs to form the training set  [35]. The same methodology has also been followed elsewhere  [48, 84]. The methodology used by Puzyn et al. [35]  added to the validation set some cases, which do not fall in the range of the training set (validation and reliability testing at the same time). The complete dataset should be provided to potential dataset users, including nanomaterial, endpoints, and descriptor information, together with the clearly defined training and test sets  [4].
Model Implementation The third component of the roadmap is the model implementation of linear or nonlinear models (Figure  1 ). In this section, the second OECD validation principle-an unambiguous algorithm-requires full model structure and accurate values of all the model parameters to be specified. Of the 86 studies reviewed, 48 performed linear modelling, 51 performed non-linear analysis, and 13 performed both modelling techniques. For each of the studies examined, the combination of model implementation, validation metrics, and the applicability domain were recorded separately, which causes many extractions per study. If, for example, another model was created with the above specifications unchanged, this would be introduced as new case within the analysis. However, if a different dataset is used with the same model, this also leads to a new insertion.
PLS is a fusion of MLR and Principal Component Regression (PCR) and it is one of the most popular approaches in QSARs. Through a linear combination of the original variables, PLS produces a set of components to best represent the output in the descriptor space  [40]. Two rules models were found to be employed in the studies reviewed as two versions of Decision Table algorithms (DT) (Figure  2 ). Rules, as classifiers, include algorithms that dissect the dataset by rules. DT classifiers carry all links between input and output data using the majority of values or the nearest neighbors in the case of unknown data. DT/naive Bayes (DTNB) hybrid classifier splits the attributes into two sub-assemblies: one for DT and the other for naive Bayes  [96]. Such Rules models have been used in the studies reviewed to predict only cellular viability.
In the Decision Tree Forest (DTF), independent trees are developed in parallel without interacting. Learning sets are then drawn randomly with replacement from the training dataset, which produces different models to predict the entire dataset. The models are then aggregated. The DTF uses data rows left out to validate the model without the requirement of a separate data set. Kovalishyn et al. [102]  built an ensemble of backpropagation neural networks while applying the kNN method to determine the local correction of the Associative Neural Networks (ASNN). Their ASNN ensemble included 100 networks. While Bayes models offer visual representation of the variables' connection and perform well with missing values, only nine cases were found to be applied in the reviewed literature including seven of which were Bayesian Networks (BN) and two were Naïve Bayes (Figure  2 ).
Reference NMs Category Output  [77]  Metal, Metal oxide, polymeric Disrupted cellular processes  [59]  Quantum Dots IC50, viability  [75]  Metal, Metal Oxide Aggregated  [127]  Carbon-based, Metal, Metal Oxide  [128]  Metal, Metal Oxide  [72]  Metal Oxide Viability  [73]  Metal  [96]  Dendrimers BN can be fed with varying datasets that may lack data through their ability to iteratively refine prediction as novel knowledge becomes accessible  [128]. The structure of the model is optimized using data for every node and the conditional probability tables to determine the ideal configuration of the nodes' interactions  [127]. Naive Bayes uses posterior probability to predict the target attribute's value. The classifier tries to find the value that maximizes the conditional probability of the target attribute by using a given input  [96].
To avoid the possibility of overestimation by using only leave-one-out cross validation, a bootstrap procedure (Q 2 Boot ) is suggested  [23]  and is mainly suitable for a limited number of training cases  [50]. These approaches systematically take out data points from the training set, reconstructing the model, and then predict the left-out data points. The leave-many-out approach remove a different number of values from the data set (10%, 20%, 25%, or 50%), depending on the size of the dataset even though there is no rule-of-thumb as to the percentages one should apply for cross validation or data split. Besides Q 2  LOO , the root-mean square error of cross-validation (R 2 CV ) can be calculated  [38, 94]. The minimum criteria for a successful QSAR model is R 2 ≥ 0. 6 and Q 2  LMO of ≥ 0.
Ranking of Classifiers Roy et al. [132]  proposed a composite score of predictions using a reliability indicator. This is a tool based on absolute prediction errors to rank the quality of predictions. The tool ranks the models into good, moderate, and bad, using three criteria. However, the tool is presently valid only for MLR models. Furxhi et al. [72]  proposed a composite score based on a Copeland index to rank classifiers according to their performance on diverse datasets, validation stages, and performance metrics. Tamvakis et al. [133]  proposed a dissimilarity performance index based on their voting performance to recommend the optimal ensemble combination. A variety of different datasets were used in this scenario to evaluate the relationship between voting results and dissimilarity measurements.
Several studies have developed quasi-QSARs using line notation methods, such as SMILES, to represent the structure of a molecule in a character string. This codification enables using the SMILES-specific models to classify non-SMILES descriptors. It should be noted that the application of a mixture of SMILES generated by different software packages is improper  [93]. Optimal SMILES-based models outperform models based on optimal descriptors, since combining global attributes and SMILES components provides more information on the molecular structure than traditional descriptors  [130]. Class imbalance reflects an unequal distribution of class values within a dataset and poses a challenging problem because classifiers exhibit biases of the results. This has been rarely accounted for properly during training  [60, 67, 72, 74].
The most common technique used was SMOTE. SMOTE looks at the feature space for the minority class data points and generates new points considering its k nearest neighbors. The class imbalance not only affects model performance, but it also affects features correlation. Once a balanced dataset is attained, feature correlation becomes more accurate. Regarding data normalization, it is advisable to select a different normalization technique (z-score, min-max, log10) for each variable, according to the skewness of feature data  [60]. Single random splitting was common across the studies. However, our research shows clearly that multiple random training and validation distributions should be examined to investigate the influence that the split may have on attribute distribution and to ensure randomness.
Even though there is a no general rule-of-thumb for setting a splitting point, the 80/20 was the most commonly used ratio, often referred to as the Pareto principle  [145]. If the dataset is not balanced, the data should be stratified before splitting. However, such information is often not reported. Since most of the datasets used in nanotoxicology are quite small and splitting may hinder a satisfactory variance in the estimates, k-cross validation should be performed. Moreover, after splitting, correlation between the data in the training or test set should be minimal and the test data should be contained within the chemical space identified by the training data. The latter can be covered by using the PCA score plot or diversity analysis  [46, 55]  and investigation of multiple splitting can be performed following the methodology from Puzyn et al.
[35] , which ensures that validation data are evenly distributed within the range of toxicity of the training dataset. The complete dataset of substances, endpoints, and descriptor values should be annexed in each analysis, along with clearly defined learning and test sets  [4]. In order to evaluate model performances, it is essential to provide proper metrics and statistics and 78% of the reviewed studies presented evidence of internal validation. Almost half of the studies investigated robustness performance and 60% of the studies performed external validation. k-fold validation provides a superior estimate of the generalization error since it is less affected by overfitting  [73]. R 2 can be artificially increased by adding parameters while Q 2 cv, decreases when a system is over-parameterized, which makes Q 2 cv a more accurate measure of models' predictability  [54].