dataesr/scientific-paragraphs-categorization huggingface dataset:
Separate models were built for attained LAZ and WHZ and growth velocity outcomes  13. Growth velocity was defined as LAZ/WHZ units of change per day between the specified study visit and the subsequent study visit. We assessed microbiota composition and functional pathways separately. XGBoost models were fit using the H20. ai engine and h2o R package interface with the XGBoost package. XGBoost model performance was evaluated using pseudo-R 2 and mean absolute error (MAE). Pseudo-R 2 values <0 indicated that the prediction of the model was worse than the mean response. Scaled relative importance for each model feature was used to identify the twenty most informative variables for further interpretation, where the most important variable is ranked first, and the importance of subsequent variables are relative to the first variable.
The marginal relationships between the twenty most important features and each growth outcome were visualized for interpretation  40  using accumulated local effects plots (ALE). ALE plots can be interpreted as showing a marginal effect, adjusted for all covariates retained in the final model, showing the expected change in the outcome variable per increment in a model feature. The resulting effect sizes are plotted cumulatively and centered about the average effect size  41. ALEs were generated using the ALEplot package, modified to compute confidence intervals, and were plotted using ggplot2. Standard deviations were calculated per increment and were used to calculate and plot increment-wise 95% confidence intervals. The code for the implementation of XGBoost and the ALE plots is available at https://github.
Acknowledgements We thank all the mothers, babies, and their families who participated in the SHINE trial and all members of the SHINE trial team (all members listed here: https://doi. org/10. 1093/cid/civ844). We particularly thank the leadership and staff of the  Ministry of Health and Child Care in Chirumanzu  and  Shurugwi districts and Midlands Province  (especially environmental health, nursing, and nutrition) for their roles in operationalization of the study procedures, the  Ministry of Local Government  officials in each district who supported and facilitated field operations,  Phillipa Rambanepasi  and her team for proficient management of all the finances,  Virginia Sauramba  for management of compliance issues, and the programme officers at the  Gates Foundation  and the  Department for International Development , who enthusiastically worked with us over a long period to make SHINE happen.
Data availability The raw metagenome sequencing data generated in this study have been deposited in the European Bioinformatics Database under accession code PRJEB51728. The UniRef90 database used to annotate metabolic pathways is publicly available. (https://www. uniprot. org/ help/uniref). Epidemiologic data files and final processed and annotated metagenome sequencing data files (taxa and pathways) are available at https://doi. org/10. 5281/zenodo. 7471082.
Indeed, this workflow is intended to be (1) generic, as it is suitable regardless of the data volume (small/big) and regardless of the nature of the variables (continuous/qualitative/mixed), (  2 ) easy to implement, as it is based on few easy-to-use software packages, and (3) robust, through the stability evaluation of the final clusters and through recognized algorithms and implementations. This workflow can be easily automated and/or routinely applied on a wide range of clustering projects. A synthesis of the literature on data clustering as well as the scientific rationale supporting the proposed workflow is also provided. Finally, a detailed application of the workflow on a concrete use case is provided, along with a practical discussion for data scientists. An implementation on the Dataiku platform is available upon request to the authors.
In this paper, we propose in section 2 a synthesis of the different methods related to unsupervised clustering in the literature and in relation to the implementations available in R or Python. In section 3, we propose Qluster, a generic workflow for clustering tabular data of any nature and size, while considering (1) the literature guidelines on how to perform robust clustering, and (2) the availability and ease of use of R or Python implementations for data scientist users. Then, in section 4, we detail this workflow through a step-by-step application on the open access Cardiovascular Disease 1  practice dataset to help data scientists to reapply this workflow on their own project with concrete recommendations. We then provide a practical discussion in section 5 and conclude in section 6.
Moreover, hierarchical-based methods generally have higher time and space complexities than partitioning-based ones, and rely on more input parameters which leaves more room for subjectivity regarding the choice of settings, with a direct impact on the generated clusters  (Fahad et al. 2014). Some studies have also shown that hierarchical-based algorithms lead to worse clustering results than partitioning algorithms, suggesting that the latter are well-suited for clustering large datasets due to not only their relatively low computational requirements, but also comparable or even better clustering performance  (Kaushik et al. 2014; Zhao et al. 2002). The third type of method does not either explicitly require a number of clusters nor does it mainly rely on a distance threshold from a "center" (like partitioning-based methods do).
On the contrary, density-based methods rely on the estimated density of observations to perform the partitioning. Such a method is in this sense more local and allows to represent clusters whose topology are less induced by the sole distance used (like hyper-spheres when using the Euclidean distance in partitioning-based methods). This strategy may however be associated with a greater propensity to overfit data and greater difficulties to set up the hyper-parameters. Beyond these three main types of methods there are many other alternative clustering approaches including grid-based methods (e. g. Wave-Cluster  (Sheikholeslami et al. 1998) , STING  (Wang et al. 1997),. ). They perform the clustering on grids rather than on the whole dataset. There are also model-based methods that optimize the fit between the data and predefined models, assuming that the data is generated by a mixture of underlying probability distribution (e.
g. mixture density model (EM  (Do et al. 2008 )), conceptual clustering (COBWEB  (Fisher 1987) ), neural networks model (SOMs  (Ciampi et al. 2000) )). More recently, a wide range of new approaches to clustering based on deep learning have emerged  (Aljalbout et al. 2018) ; they are mainly used to cluster unstructured data. Deep neural networks (DNN) can be effective ways to map a high-dimensional data space to a lower-dimensional feature space, and thus improving clustering results. Nevertheless, DNN often requires large datasets and a procedure for post-hoc interpretability of the clusters (the representation learned by DNN architectures is not easily understandable).
Choosing an appropriate clustering approach The choice of the appropriate approach to be used rely on many aspects (e. g. Fahad et al. 2014; Ahmad et al. 2019 ) such as: • its ability to handle the desired type of data (binary / nominal / ordinal / numerical), • the dimensionality of the data (see e. g. Mittal et al. (2019) ), • the size of the data (small to large data), • the availability of reliable implementations in software (e. g. R and Python -the 2 most used statistical software by data scientists). Partitioning-based methods are known to be composed of many variants to directly handle continuous (K-means, PAM, CLARA, FCM, etc. ), categorical (K-modes, K-medoid, etc. ) and mixed variables (K-prototype, KAMILA  (Foss et al. 2018), etc. ). In addition, the ability of some of these algorithms to directly handle input dissimilarity matrices facilitates the pre-transformation of the original data into data of the desired type prior to clustering, using suited distance measure (e.
g. McCane et al. 2008 ). This method is particularly used to convert categorical or mixed data into numerical data, as there is more literature and algorithms implemented in software for continuous data. (e. g. scikit-learn 3  in Python, or cluster 4  and fpc 5  R packages. The two latter packages both provide a large number of clustering and clusters stability assessments methods, as well as functions to compute dissimilarity matrices and describe the results). Another known alternative consists of one-hot-encoding categorical data into binary variables and treating the latter as continuous (e. g. in  Li et al. 2017). It is however necessary to downweigh the obtained variables so that no more weight is given to the original variables with more modalities. Finally, dimensionality reduction methods, such as factor analysis (Principal Component Analysis (PCA) for continuous data, Multiple Correspondence Analysis (MCA) for qualitative data, Factor Analysis of Mixed Data (FAMD) for mixed data (Pagès and Husson 2017)), can be used before the clustering as a first step to transform the data into numerical components (i.
In addition to the ease in which the two-step sequential approach can be implemented, there can be substantive reasons for adopting this approach as well  (Green et al. 1995). Alternative methods consist of using simultaneously both MCA and a clustering approach in a single framework, so that the low-dimensional data can be chosen to facilitate the identification of clusters  (Bock 1987; DeSarbo et al. 1991; De Soete et al. 1994 ). However, these methods lack implementations (both in R and Python), hindering their use within a clustering workflow. Finally, the selection of the number of components to be kept is then the key step. This can be based on several methods including permutation tests (Hwang and Takane 2010), cross-validation-based methods  (Bro et al. 2008 , Josse, Chavent, et al.
2012) , or methods based on the amount of information carried by each of the dimensions generated by the analysis, either compared to an average value equivalent to the Kaiser's rule in PCA (Lorenzo-Seva 2011) and/or represented by a scree plot  (Clausen 1998; Drennan 2010 ). The latter method has been found to perform fairly well and is the most widely used to select the optimal number of dimensions  (Zwick et al. 1986; Bandalos et al. 2009). It consists of looking at the bend in the falling curve (so called "elbow") indicating an optimal dimensionality (if there is no obvious elbow, one can choose the number of components just before a flat appears). It has been adapted from PCA  (Cattell 1966)  and used in the context of correspondence analysis  (Costa et al. 2013). All factor analysis methods can notably be found in R in the well-known FactoMineR 6  package, and in Python in the prince 7  GitHub library (although issues are still open for the latter).
Methods for estimating the number of dimensions to be kept can be found in R in many packages such as the FactoMineR and missMDA 8  packages, using cross-validation methods, or in the factoextra 9  package (e. g. scree plot). In Python and to the best of authors knowledge, one would need to code these methods to apply them as no specific functions were found. Whether a factor analysis is performed as a first step or not, the need to choose a distance measure is critical, as some of them are only appropriate according to the type of data, or are preferred in some cases. Indeed, continuous data require appropriate distance measure to obtain the dissimilarity matrix (e. g. Euclidean and Manhattan distances), while categorical data are widely handled with simple matching methods (e. g. Hamming distance for symmetric measures, which is equivalent to Manhattan distance on binary variables, and Jaccard distance for asymmetric measures to favor positive co-occurrences over the negative ones).
Methods to handle mixed data can consist of combining above-mentioned methods, such as the Gower distance (i. e. simple matching methods and Manhattan distance). Finally, some aspects need careful attention when dealing with large data. This includes the size of the dataset, as the candidate algorithm must handle either or both high dimensionality and a massive number of observations (including outliers/noisy data), which makes difficult, and sometimes impossible, dissimilarity matrices to be computed. Equally, fast running time is essential with large data as the clustering needs to be performed several times, notably to assess cluster stability and optimize the clustering hyperparameters (e. g. in the use case in section 4, clustering was replicated 550 times). Few strategies exist to deal with massive data, such as relying on algorithms of lower complexity (e.
g. K-modes and FCM that are O(n),  (Fahad et al. 2014) ). The latter however is quickly limited as computing time increases linearly with the size of data. Alternative methods consist of working on subsets of the whole dataset to cluster smaller datasets before generalizing them (e. g. Mini Batch K-means (Sculley 2010), CLARA  (Kaufman et al. 2009) , CLARANS  (Ng et al. 2002) ). The CLARA algorithm is an extension to K-medoids methods (e. g. PAM), that is known to be more robust than K-means-based algorithms as they minimize a sum of dissimilarities instead of a sum of squared Euclidean distances  (Jin et al. 2010). CLARA allows to deal with data containing a large number of observations (more than several thousand) using a sampling approach, in order to reduce computing time and RAM storage problems.
Instead of finding medoids for the entire dataset, CLARA considers a small sample of the data and applies the PAM algorithm to generate an optimal set of medoids. CLARA repeats the sampling and clustering processes several times in order to minimize the sampling bias. In practice, its strength lies in the possibility to adjust the number of samples and the sample sizes, in order to both make calculation time acceptable and storage in RAM possible. This is indeed essential to enable the stability of clustering to be assessed and the best partitioning to be found by repeating the clustering process many times. Compared to CLARA, CLARANS presents a trade-off between the cost and the effectiveness of using samples to obtain clustering. Mini-batch K-means, CLARA and CLARANS can all be found both in R (e.
e. the inter-cluster distances) over the compactness (intra-cluster distances). Although previous works have shown that there is no single internal cluster validation index that outperforms the other indices, Arbelaitz et al. (  2013 ) compare a set of internal cluster validation indices in many distinct scenarios, indicating that the Silhouette coefficient yielded the best results in most cases. Alternatives exist for estimating the number of clusters in a dataset regardless of the clustering methods, such as the "gap statistic" that compares the change in within-cluster dispersion with that expected under an appropriate reference null distribution (R. Tibshirani et al. 2001). Finally, several methods for assessing the stability of clustering are proposed in the literature, by repeating the clustering process several times under conditions that are different from those of origin.
These include procedures used in bioinformatics that remove one column at a time  (Datta et al. 2006; Handl et al. 2005). Several metrics can then be computed between the set of clusters  (Brock et al. 2008) , such as the average proportion of non-overlap (APN), the average distance (AD) and the figure of merit (FOM). These methods are notably proposed in the clValid 14 R package and its main function clValid(). The latter includes many clustering algorithms (hierarchical, K-means, diana, fanny, som, model, sota, pam, clara, and agnes) and allows for a direct assessment of clusters' stability through the "validation" argument. Other approaches consist in perturbing the original data, either using bootstrapping  (Efron 1979 ; Efron and R. J. Tibshirani 1994), noising and/or sampling methods  (Hennig 2008 ).
The Jaccard similarity statistic is then often used as a metric for assessing stability, by computing the similarities of the original clusters to the most similar clusters in the resampled data. Such methods are implemented in the FPC R package, notably in the clusterboot() function. The latter is an all-inclusive package that also allows for clustering using a wide variety of algorithms. (e. g. K-means, hierarchical clustering, normal mixture models, PAM, CLARA, DBSCAN, spectral clustering, etc. ), making it easy for a data scientist to generate, compare and assess stability of clusters. As for Python, while there are packages to evaluate the internal validity of clusters (Silhouette coefficient, Rand Index, Calinski-Harabasz Index  (Caliński et al. 1974) , see in particular the sklearn.
cluster library), no Python library was found to evaluate the stability of generated clusters. This reinforces the fact that Python does not cover as easily as R the whole clustering process, as there is no Python package that includes all the steps of interest (clusters generation, internal validity evaluation, clustering optimization, clusters stability evaluation and clusters description), as in the FPC R package. Finally, methods for testing cluster stability on a hold-out dataset are barely mentioned in the literature. This would consist of pre-allocating the observations in the test set into the clusters obtained from the learning set, and by clustering the test dataset to check for good allocation. Though no implementation has been found either in R or Python. We can see a simplified application of clustering of 2 independent datasets in Saint  Pierre et al.
As mentioned in section 2. 2, the latter also allows for dealing with collinearity, high-dimensionality and noise which makes the work for a clustering algorithm easier, as there are both fewer variables to deal with and greater clarity in information to cluster (factor analysis methods are themselves meant to uncover profiles into components of richer information). • Adapted to datasets of any volume, be it small or large data. Indeed, the same partitioning-based algorithm is used (PAM), either applied entirely on a dataset of reason-able size, or on samples of a large 15  dataset (CLARA algorithm), using the same pamk() function from the FPC R package. In addition to this practical aspect, PAM was chosen over the (widely used) K-means algorithm based on its ability to deal with Manhattan distance which is less sensitive to outliers than the Euclidean distance  (Jin et al.
2010 ). Moreover, PAM is known for its simplicity of use (fewer parameters e. g. than with DBSCAN, Hierarchical Agglomerative Clustering (HAC) or DENCLUE  (Fahad et al. 2014) ) and is also implemented in an easy-to-use all-inclusive package (not the case for e. g. CLARANS, KAMILA, Mini batch K-means, DENCLUE and STING that are suited for large datasets but where assessing clusters stability would require extensive code development by data scientists). More details on the choice of the clustering algorithm can be found in section 5. In addition, the Qluster workflow relies solely on four state-of-the-art R packages, allowing data scientists to quickly manage data of different nature and volume and perform robust clustering: • Both the tasks of clustering and clusters stability assessment are handled using the FPC R package (functions pamk() and clusterboot() respectively).
R has been chosen over Python because the former offers all the clustering methods desired, and no package including all the steps of interest to clustering was found for the latter (one would have to code some steps by him/herself, see section 2 for more details). The clusterboot() function offers many ways to assess clusters' stability, but one selects the two followings for routine practice and for their complementarity as mentioned in  Hennig (2008) : bootstrapping and noising. • The factor analysis part is handled using the FactoMineR R package (function PCA(), MCA(), FAMD(), for continuous, categorical, and mixed data respectively -the latter function generalizing the others). This step is optional in the case where only continuous variables are in input 16  (although recommended). To select the optimal number of components to keep, one recommends for small data to use the deterministic cross-validation technique implemented in the missMDA package (function estim ncpPCA(), estim ncpMCA(), estim ncpFAMD()).
As this method requires high computing time, the standard "elbow" method in a scree plot is recommended for large data, using the factoextra R package (function fviz eig()). Finally, the Qluster workflow is operationalizable and implementable from end to end (see in Appendix in section A a picture of implementation in the Dataiku 17  platform. Available upon request: contact@quinten-france. com). This generic workflow, usable in most situations, can be described through the following pseudocode (Algorithm 1): Algorithm 1: The Qluster pseudo-code input : X: The input data packages: FactoMineR, factoextra, FPC, missMDA output : Q: A clustering of X and associated measures if X is continuous only then The Cardiovascular Disease 18  dataset includes 70,000 patients with or without a cardiovascular disease, and 12 variables (5 of them are continuous).
Step-by-step application of the proposed workflow The following section details the application of the Qluster workflow to the cardiovascular dataset to help scientists use it for their own project. Additional elements to the ones presented in section 2 supporting the present methodology are also provided when relevant. We first present the preprocessing of the dataset, in which notably the few continuous variables are converted into qualitative data, before applying a MCA, that is a data-reduction technique for exploring the associations among multiple categorical variables  (Greenacre and Blasius 2006; Greenacre 1984; Warwick et al. 1989; Murtagh 2005; Nishisato 2019 ). Then, given the large size of the database, the CLARA algorithm is applied and optimized. Finally, clusters' stability is assessed and a brief interpretation of clusters is provided.
Data preparation Features derivation and selection First, outliers were detected by defining for each quantitative variable thresholds above or below which values are more likely to be inaccurate. Acceptable values should be in the following ranges: 0 <Age <120, 0 <BMI <100, SBP ≤ 400, DBP ≤ 200 (Ortega  et al. 2016, Mayo Clinic 19 , French HTA (HAS) reccomandations 20  ). For simplicity, patients with at least one outlier were removed from the analysis (sensitivity analyses could be performed). Quantitative variables were then discretized in order to both create variables with clinical sense and enable the use of the MCA algorithm (see Table  1 ). Nota bene: the database has no missing values. Finally, the variables selected to discriminate the population must be chosen according to their medical relevance to the context of the study.
To this end, the user must always consider the results he would obtain if a variable is included or not. In particular, the user has to ask himself whether active discrimination of the clusters by a variable is sought: considering the example of the two common variables age and race, if they are are actively included in the clustering step, it will tend to create groups of young versus old, Caucasian versus non-Caucasian patients. If not, such variables can be kept for a passive analysis of the generated clusters and assess a posteriori a possible heterogeneity on these variables. In this use case, we removed the height, weight, systolic and diastolic blood pressure features, as they are used to create the derived features listed above and are not useful alone for clustering. At the end of these treatments, we obtained a database of 34,134 patients described with 11 variables.