dataesr/scientific-paragraphs-categorization huggingface dataset:
Similarly to Mixmod, an EM algorithm is used to determine the optimal partition. When the selection of relevant variables is enabled, a penalization on the Bayesian Information Criterion (BIC) or the Maximum Integrated Complete-data Likelihood (MICL) is applied at the maximization step. The selection of the number of clusters is achieved by running the algorithm for each number of clusters in a specified range, and selecting the one which yields the best value of the selected criterion. These selection features were not used in the present study due to the lengthy computing time, although the user may find convenient having an all-in-one tool via this function. Latent class analysis (LCA). This clustering technique is derived from the Latent Class Regression 14 and implemented in the poLCA R package  15.
Simulation framework. General process. To assess the performance of each previous clustering method, 1000 datasets were generated with a particular design (e. g. set of parameters), yielding continuous and categorical variables which were representative of a true known partition. Each method then yielded its predicted partition that was compared to the original partition. The agreement scores over the 1000 repetitions were then summarized and graphically represented, allowing a visual comparison of the clustering performance for each method. Investigated scenarios. The datasets were simulated under various scenarios in order to approach the diversity of real-life data. These scenarios were defined by controlling the following parameters: • The size of the population (300, 600, 1200); • The number of clusters (2, 6, 10); • The ratio between the number of continuous and categorical variables; • The proportion of relevant (non-noisy) variables (20%, 50%, 90%); • The degree of relevance of the variables (low, mild, high) Ideally, all combinations of the different values of the parameters should be tested to avoid hidden interactions.
However, due to the overwhelming computational time, it was decided to investigate one parameter at a time. By default, the size of the population was 300, the number of clusters was 6, the continuous and categorical variables were equally represented (4 each), mildly relevant (see below for the definitions) and without any fully irrelevant variables. The seven scenarios and their investigated parameters are summarized in Table  2. For example, in scenario n°1, three series of 1000 datasets were generated, corresponding to respective population sizes of 300, 600 and 1200. This enabled the visualization of the impact of sample size on clustering performance.
Simulated datasets. Each population was created according to the following process: Continuous variables were generated by Qiu and Joe's method  17  , an improvement of Milligan's method  18. This algorithm finds a multivariate normal distribution for each cluster such that a degree of separation of each cluster with its closest neighbor is verified. The N × q covariates values are then sampled and returned with the true partition. In practice, the continuous covariates were generated by using the genRandomClust function from the clustergeneration package. This function uses the separation index as a parameter, included in the ] -2, +2[ interval. The larger the index, the more the clusters are separated. Therefore, this index can be deemed as a degree of relevance for continuous variables.
In the scenarios presented in Table  2 , the separation index values "-0. 3", "0" and "0. 3" represent a "low", "mild" and "high" degree of relevance, respectively. In practice, when using the Euclidean distance such as in K-prototypes, continuous data should be scaled (i. e. subtracted by the mean and divided by the standard deviation). Indeed, variables having the highest values tend to play an over-represented role when computing the distance to the closest center of the class. Although all simulated continuous variables had roughly the same scale, the datasets were scaled for precautionary reasons. For the purpose of equity, all methods utilized the same scaled dataset as input, since we were more interested in relative performance than in absolute performance. LogLik(x|θ) = N � i=1 log e   G � g=1 τ g h � x|α g �   Categorical variables were then added to the population based on the existing partition.
1 ). Similarly to continuous variables, the introduced proportion of noise can be considered as a degree of relevance for categorical variables. In the scenarios presented in Table  2 , the "low", "mild" and "high" degrees of relevance represent noise proportion of 95%, 85% and 75%, respectively. Noisy variables, i. e. which do not carry information regarding the latent partition, can also be added to the dataset in order to modify the proportion of relevant variables in the dataset, as described for scenarios 6 and 7 in Table  2 , leading to proportions of 1/5, 1/2 and 9/10 for a total of 10 variables for the varying type. For continuous variables, the genRandomClust function automatically produces the latter, while for categorical variables, a full random sampling of the levels of the new variable is sufficient.
Parameters of the clustering methods. The various clustering methods presented in "Clustering algorithms" section feature parameters which directly influence the resulting partition. Since thousands of datasets were generated for each of the seven scenarios, it was not possible to fine-tune the parameters individually. Thus, default parameters were chosen following the guidelines of the documentation, and the same parameters were implemented for every dataset. Particular attention was given to the number of initializations (e. g. for UET, K-Proto) which represents a trade-off between the robustness of the partition and computation time. Typically, the true number of clusters is unknown and is determined a posteriori, either according to an objective criterion or according to practitioner knowledge.
Application on a real-life dataset: the EPHESUS study. The EPHESUS study 4 is a randomized multicenter double-blind placebo controlled clinical trial, conducted on 6632 patients having a recent acute Myocardial Infarction (MI) and a Left Ventricular Ejection Fraction (LVEF) lower than 40%, with heart failure or diabetes. The treated group, which received eplerenone, had significantly lower mortality and hospitalization rates. This study has been used previously by our group as a first approach of clustering for clinical trial data, using the latent class analysis (LCA) method  16. The clustering techniques presented above were used to partition the patients of the EPHESUS study on 16 clinical features recorded at baseline. The six continuous variables consisted of age (AGE), body mass index (BMI), left ventricular ejection fraction (LVEF), estimated glomerular filtration rate (GFR), potassium (K), and sodium (NA).
Results Simulated datasets. Impact of population properties. Figure  1  shows the impact of population size (left panel) and the number of clusters (right panel) on the mean ARI calculated with each clustering method (scenarios 1 and 2 respectively in Table  2 ). The first five sets of boxplots (in green) are from the distance-based methods, while the four remaining sets of boxplots (in blue) are from the model-based techniques. An increase in the number of subjects (Fig. 1 , left panel) appeared to only positively impact LCA and Mixmod, two model-based methods. Meanwhile, Kamila, Kproto and LCM exhibited a constant good performance irrespective of population size, surpassing the ARIs achieved with other methods. When increasing the number of clusters up to 10 (Fig. 1 , right panel), the mean ARIs dropped for all methods.
All of these numerous comparisons (96 tests) retrieved p values < 0. 0001 (Supplementary Table  1 ). Importantly, runtimes differed across methods as reported in Supplementary Table  2  for scenarios 1 and 2. Yet none of them exceeded 20 s. In summary, distance-based methods (except the K-prototypes) displayed low ARIs in this simulation framework. The lowest ARIs were observed for Gower's distance either with PAM or hierarchical clustering. In contrast,   2. This figure title was generated using R (R: A Language and Environment for Statistical Computing, R Core Team, R Foundation for Statistical Computing, Vienna, Austria, 2020, https ://www. R-proje ct. org).
Application to real-life data: the EPHESUS clinical trial. The results obtained with distance-based and model-based clustering methods on the EPHESUS dataset are respectively presented in Figs. 3  and 4. The clusters are numbered from 1 to 4 according to the ascendant overall risk as estimated by the Kaplan-Meier curve. Thus, regardless of the method used, group 1 (green) had the lowest risk of hospitalization or death related to heart failure, while group 4 (red) had the highest risk.
Discussion The present findings highlight substantial differences in clustering performance (as measured by ARI on simulated data) between the various methods tested; these methods cannot consequently be considered as interchangeable in the setting of heterogeneous data. Overall, our simulations demonstrate the dominance of K-prototypes, Kamila and LCM over all other methods. In addition, the classical methods using dissimilarity matrices such as Partitioning Around Medoids and Hierarchical Clustering generally performed poorly in comparison with model-based methods. Furthermore, when applying the clustering methods to a real-life clinical dataset, LCM yielded the most promising results, in that it (1) featured the most striking differences in clinical profiles across clusters, (2) exhibited the best prognostic performance of all clustering methods (highest C-index) and (3) identified a subgroup of patients with substantial treatment effect (HR < 0.
6). Large differences in performance of clustering methods in the benchmark study. Complex situations in cluster analysis have already been emphasized (e. g. in the scikit-learn documentation, https ://sciki t-learn. org/stabl e/modul es/clust ering. html). Yet, in our analysis, as shown in Supplementary Fig. 2 , the clusters are relatively well defined, and have "classical" shapes as illustrated by the t-SNE analysis. This suggests that the complexity/difficulty of the clustering is emerging from the heterogeneity of the data rather than the "shape" of the clusters. The satisfactory performance of LCM and Kamila was partly expected since the simulated data matched their assumptions (better grasp of data heterogeneity). However, Kamila appeared to better tackle the high imbalance between continuous and categorical data than any other method.
Although conceptually similar to PAM, its weighted combination of distances may be the key to its success. However, its performance may be greatly diminished on unfavorable datasets with continuous variables of low relevance, (as illustrated in Fig. 2 ). The under-performance of LCA may be due to the choice of the cut-points to discretize the continuous variables  21. Indeed, the cut-point was determined automatically such that three balanced classes arose from each continuous variable, which is not likely to reflect the underlying partition. Nevertheless, this approach represents the routine use of LCA, which is a popular clustering approach. Choosing the most appropriate clustering methods. With the present results in mind, the following points could help researchers in choosing adequate clustering methods in the setting of heterogeneous data.
• If the dataset is assumed to stem from a normal-multinomial distribution, model-based methods should logically be preferred. If supplementary assumptions are made relative to cluster size, orientation, volume and levels, a sound option could be to use the corresponding model offered by the Mixmod method. However, using LCM (VarSelLCM package) will probably be sufficient, including additional extra features such as a selection of covariates, handling of missing data, and selection of the number of clusters. • When the normal-multinomial assumption does not hold, the use of Kamila seemingly appears as a safer choice as opposed to LCM or Mixmod. The K-prototypes method could also be a good alternative, as shown here for datasets with scaled continuous variables (or when continuous variables do not need scaling because they have the same magnitude), and for datasets in which each level of categorical variables is equally represented.
• From a time-efficiency standpoint, the Kamila method offers the best performance when dealing with large datasets (thousands of observations, dozens of variables). Indeed, the method has been specifically implemented to work in a Big-Data setting by taking advantage of the scalability of its algorithm. For other modelbased methods such as LCM and Mixmod, the computation time depends on the complexity of the selected model, the number of iterations and the additional features. Perspectives. The present analysis aimed to provide some guidance in the choice of a clustering method for heterogeneous data. Given the growing access to multiple data sources, it has become crucial to be able to manage all types of variables through recent advances in statistics and machine-learning. While a few clustering tools for mixed data have been investigated in the present analysis, many more have been developed for this purpose.
Nonetheless, this abundance of techniques may confuse end users, since they generally ignore the performance of these techniques relative to each other. As noted by Ahmad and Khan 1 , this mainly stems from the fact that the methods are often evaluated on a restricted number of datasets which cannot be extrapolated to every usage. For example, the popular "Heart-Cleveland" dataset has an arbitrary target variable, while the "Australian Credit" dataset has only a binary target. The generated simulated datasets herein were hence based on a combination of the Qiu and Joe's 17 method and a simple home-made stratified sampling. The benchmark itself, although conducted over a limited selection of methods, revealed a huge performance gap observed between "popular" algorithms (such as Hierarchical Clustering and PAM) and more sophisticated methods (such as Kamila or LCM) in all tested scenarios.
However, this does not imply that the former will never be useful, or that the latter will always perform better. For instance, in a setting where groups are nested circles, Kamila will fail to identify the clusters, whereas a hierarchical algorithm with a single linkage aggregation method will easily complete the task. Even if a clustering technique yields excellent results, its potential is nonetheless ineffective if such technique cannot be implemented in current popular software packages. Indeed, efficiently translating an algorithm into a program is an arduous task for most users of such tools. Of even greater importance is that the quality of the implementation directly determines the computation time. Despite the increased performance of current computers, the clustering time can rapidly become overwhelming, especially when multiple runs are required to determine the number of clusters, or in selecting variables such as in the LCM algorithm.
This aspect should not be underestimated as the datasets can quickly reach huge proportions in a big-data context. As already stated above, the Kamila algorithm is particularly well suited for a big-data setting. According to Hennig  22  , multiple relevant partitions can be found in a population, just as a deck of cards can be clustered according to colors, shapes, values or faces. Consequently, cluster analysis can be considered as successful only if the partition makes sense for the practitioner. Therefore, the involvement of field specialists in the clustering process is essential to determine the question of interest as well as a suitable knowledge of the relevant variables or the number of clusters. Despite the immense progress enabled by artificial intelligence in recent years, human experience and intuition remain the best judge in cluster analysis.
That being said, it is very likely that useful clinical information will arise only from clustering algorithms displaying good intrinsic performance. In keeping with the latter, the results of the present benchmark analysis could strengthen the collaboration of dataanalysts, clinical researchers and physicians by using the most appropriate machine-learning tools. Importantly, our results provide guidance on how to use all available (heterogeneous) clinical, biological and imaging data for clustering analysis in clinical research cohorts. UET as a stochastic-based method for computing pairwise distances while avoiding the burden of data preparation is particularly relevant both for large heterogeneous datasets (with thousands of variables) and structures  6. Conversely, its underperformance could be related to the simulated datasets using normal distributions for continuous variables, which are a better fit with model-based methods.
Nonetheless, a number of continuous variables often follow a normal distribution in clinical research. Limitations. Although the genRandomClust function allows unbalanced cluster sizes and outliers, these features were not used herein due to lack of time. The impact of these characteristics would be worth investigating in subsequent studies. In addition, scenarios with non-normal continuous variables may yield different results and should be tested in subsequent studies. However, our simulation study already covers a number of scenarios which could already be useful in a range of clinical studies. An important limitation of our work is the use of the Hamming distance for the categorical data in the distance-based algorithm implemented in R software (as shown in Table  1 ). The use of other approaches could have resulted in largely different clustering performance.
Conclusion The results from the present simulation study focused on "ready-to-use" tools from R and suggest that modelbased tools (p. e. the Kamila and LCM packages implemented in R) usually perform better than distance-based tools (except K-prototypes packages implemented in R) in the setting of heterogeneous data such as clinical research datasets including both numeric and categorical variables. The present results suggest that model-based tools that are currently readily available to biology and clinical researchers can be useful practical solutions for performing clustering in situations involving heterogeneous data. Future work should continue this effort for benchmarking "ready-to-use" clustering tools on mixed data, possibly using our simulated datasets (available on http://mbi. loria.
Acknowledgements This work and the publication of this article were funded by the  Agence Nationale de la Recherche  (grant number  ANR-15-RHUS-0004: RHU FIGHT-HF ) and by the  CPER IT2MP  ( Contrat Plan État Région, Innovations Technologiques, Modélisation & Médecine Personnalisée ) and  FEDER  ( Fonds Européen de Développement Régional). Kevin Dalleau  was recipient of a  RHU-Region Lorraine doctoral fellowship.
With an increasing need for secured water supply, a better understanding of the water consumption behavior is beneficial. This can be achieved through end-use classification, i. e. , identifying end-uses such as toilets, showers or dishwashers from water consumption data. Previously, both supervised and unsupervised machine learning (ML) techniques are employed, demonstrating accurate classification results on particular datasets. However, a comprehensive comparison of ML techniques on a common dataset is still missing. Hence, in this study, we are aiming at a quantitative evaluation of various ML techniques on a common dataset. For this purpose, a stochastic water consumption simulation tool with high capability to model the real-world water consumption pattern is applied to generate residential data.
On a basic level, ML techniques can be subdivided into supervised and unsupervised approaches. The former directly perform a categorization/classification of data with given categories. Consequently, training data with labels Water 2021, 13, 236 2 of 16 annotating the samples' categories are necessary. The latter search for groups of similar data points in the dataset in the absence of labels. Both unsupervised and supervised approaches are widely applied in studies on enduse classifications of water use events. For example, Pastor-Jabaloyes et al. [1]  proposed an unsupervised approach formulating a Partition Around Medoids clustering method using Gower distance as a similarity measure in the feature space, which is spanned by the volume and the average flow rate of the water use events. A clustering algorithm based on Dynamic Time Warping (DTW) as the similarity measure between events is described by Nguyen et al.
[19]  identified residential water end-uses in data measured by precision water meters equipped with pulse emitters using SVMs. Both supervised and unsupervised approaches proposed in the literature demonstrate high performance in terms of accuracy with the particular dataset utilized in corresponding experiments. However, these datasets are often acquired in different countries or regions. Consequently, the underlying consumer behavior represented by the consumption data varies. Moreover, each dataset has a specific time resolution. Thus, the characteristic of the datasets utilized in the state-of-the-art literature differs largely. As the methods proposed in the aforementioned studies are mainly customized to the particular dataset, the generalizability of each method is not proven. Moreover, the particular datasets applied in the state-of-the-arts are commonly not publicly available.
Hence, the corresponding implementation (e. g. , setting of model hyperparameters) of the proposed ML models is not traceable. This results in a decreased reproducibility of the methods. Finally, to the best of our knowledge a comprehensive comparison of both supervised and unsupervised approaches on a common database has not been performed yet. In this paper, we focus on a comparative evaluation of supervised and unsupervised ML techniques on a common dataset for the application of residential water end-use classification and aim at deriving a decision support for the selection of the appropriate approach, revealing possible pitfalls of water end-use classification. For this purpose, we implemented a large variety of ML-based approaches proposed in the state-of-the-art literature, i. e. , the clustering algorithm based on DTW established by Nguyen et al.
[2] , k-means, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Ordering Points To Identify the Clustering Structure (OPTICS), Clustering in Arbitrary Subspaces based on the Hough transform (CASH), as well as two classifiers based on Support Vector Machines (SVM). Subsequently, the methods are evaluated on a common database generated using a stochastic water consumption simulation framework. The validation of different algorithms with a common dataset guarantees the comparability of the evaluation results. Finally, evaluation results are analyzed and discussed in detail, with the intention to draw implications and recommendations of experiment set-ups for data-driven water end-use classification in general.
Methods In this section, the simulated common water consumption database as well as the ML techniques employed and evaluated in this work are described in detail. An overview of supervised and unsupervised ML methods is presented in Figure  1. For the supervised methods as well as the clustering algorithms k-means, DBSCAN and OPTICS, the open-source library scikit-learn  [20]  is used. The clustering algorithm based on DTW is im-plemented after Nguyen et al. [2] , and the open-source data mining framework ELKI  [21]  provides the clustering algorithm CASH. Both labels and water consumption data are generated with a stochastic simulation proposed in  [16]. Water 2021, 13, x FOR PEER REVIEW 3 of 16 methods as well as the clustering algorithms k-means, DBSCAN and OPTICS, the opensource library scikit-learn  [20]  is used.