Package: neureval
NeuRevalA stability-based relative clustering validation method to determine the best number of clusters based on neuroimaging data.Table of contentsProject OverviewInstallation and RequirementsHow to use NeuRevalInput structureGrid-search cross-validation for parameters' tuningRun NeuReval with opitmized clustering/classifier/preprocessing algorithmsCompute internal measuresExampleReferences1. Project overviewNeuRevalimplements a stability-based relative clustering approach within a cross-validation framework to identify the clustering solution that best replicates on unseen data. Compared to commonly used internal measures that rely on the inherent characteristics of the data, this approach has the advantage to identify clusters that are robust and reproducible in other samples of the same population.NeuRevalis based onrevalPython package (https://github.com/IIT-LAND/reval_clustering) and extends its application to neuroimaging data. For more details about the theoretical background ofreval, please see Landi et al. (2021).This package allows to:Select any classification algorithm fromsklearnlibrary;Select a clustering algorithm withn_clustersparameter (i.e., KMeans, AgglomerativeClustering, and SpectralClustering), Gaussian Mixture Models withn_componentsparameter, and HDBSCAN density-based algorithm;Perform (repeated) k-fold cross-validation to determine the best number of clusters;Test the final model on an held-out dataset.The following changes were made torevalto be performed on neuroimaging data:Standardization and covariates adjustement within cross-validation;Combine different kind of neuroimaging data and apply different set of covariates to each neuroimaging modality;Implementation of data reduction techniques (e.g., PCA, UMAP) and optimization of their parameters within cross-validation.2. Installation and RequirementsWork in progress3. How to use NeuRevali. Input structureNeuRevalrequires that input features and covariates are organized as file excel in the following way:for database withinput features (database.xlsx):First column: subject IDSecond column: diagnosis (e.g., patients=1, healthy controls=0). In caseNeuRevalis run on a single diagnostic group, provide a costant value for all subjects.From the third column: featuresNotes: in case you want to combine with difffusion tensor imaging (DTI) extracted tract-based features, please add them after all the other neuroimaging features. The first DTI feature should be "ACR".Example of database structure for input features:Subject_IDDiagnosisFeature_01Feature_02sub_000100.266492212.13888054sub_000210.326675900.67116539sub_000300.354067572.35572978for database withcovariates (covariates.xlsx):First column: subject IDSecond column: diagnosis (e.g., patients=1, healhty controls=0). In caseNeuRevalis run on a single diagnostic group, provide a costant value for all subjects.From the third column: covariatesNotes: if you want to correct neuroimaging features also for total intracranial volume (TIV), please add it as the last column of the database.Example of database structure for covariates:Subject_IDDiagnosisAgeSexTIVsub_000105401213.76sub_000213711372.93sub_000304301285.88Templates for both datasets are provided in the folderNeuReval/example_data.ii. Grid-search cross-validation for parameters' tuningFirst, parameters for fixed classifier/clustering/preprocessing algorithms can be optimized through a grid-search cross-validation. This can be done with theParamSelectionConfoundsclass:ParamSelectionConfounds(params,cv,s,c,preprocessing,nrand=10,n_jobs=-1,iter_cv=1,strat=None,clust_range=None,combined_data=False)Parameters to be specified:params: dictionary of dictionaries of the form {‘s’: {classifier parameter grid}, ‘c’: {clustering parameter grid}} including the lists of classifiers and clustering methods to fit to the data. In case you want to optimize also preprocessing parameters (e.g., PCA or UMAP components), specify {'preprocessing':{preprocessing parameter grid}} within the dictionary.cv: cross-validation foldss: classifier objectc: clustering objectpreprocessing: data reduction algorithm objectnrand: number of random labelling iterations, default 10n_jobs: number of jobs to run in parallel, default (number of cpus - 1)iter_cv: number of repeated cross-validation, default 1clust_range: list with number of clusters, default Nonestrat: stratification vector for cross-validation splits, defaultNonecombined_data: define whether multimodal data are used as input features. IfTrue, different sets of covariates will be applied for each modality (e.g. correction for TIV only for grey matter features). DefaultFalseOnce theParamSelectionConfoundsclass is initialized, thefit(data_tr, cov_tr, nclass=None)class method can be used to run grid-search cross-validation.
It returns the optimal number of clusters (i.e., minimum normalized stability), the corresponding normalized stability, and the selected classifier/clustering/preprocessing parameters.iii. Run NeuReval with opitmized clustering/classifier/preprocessing algorithmsAfter the selection of the best clustering/classifier/preprocessing parameters through grid-search cross-vallidation, we can initalize theFindBestClustCVConfoundsclass to assess the normalized stability associated to the best clustering solution and the corresponding clusters' labelsFindBestClustCVConfounds(s,c,preprocessing=None,nrand=10,nfold=2,n_jobs=-1,nclust_range=None)Parameters to be specified:s: classifier object (with opitmized parameters)c: clustering object (with optimized parameters)preprocessing: data reduction algorithm object (with optimized parameters), default Nonenrand: number of random labelling iterations, default 10nfold: number of cross-validation folds, default 2n_jobs: number of jobs to run in parallel, default (number of cpus - 1)clust_range: list with number of clusters, default NoneOnce the class has been initialized, thebest_nclust_confounds(data, covariates, iter_cv=10, strat_vect=None, combined_data=False)method can be used to obtain the normalized stability, the number of clusters associated to the optimal clustering solution, and clusters' labels. It returns:metrics: normalized stabilitybestncl: best number of clusterstr_lab: clusters' labelsiv. Compute internal measuresTogether with normalized stability,NeuRevalalso allows to compute internal measures for comparisons between the stability-based relative validation and internal validation approaches. This can be done with theneureval.internal_baselines_confoundsmethod and the functionselect_bestto select the best number of clusters that maximize/minimize the selected internal measure:neureval.internal_baselines_confounds.select_best(data,covariates,c,int_measure,preprocessing=None,select='max',nclust_range=None,combined_data=False)Parameters to be specified:data: features datasetcovariates: covariates datasetc: clustering algorithm class (with optimized parameters)int_measure: internal measure function (e.g., silhouette score, Davies-Bouldin score)preprocessing:  data reduction algorithm object (with optimized parameters), defaultNoneselect: it can be ‘min’, if the internal measure is to be minimized or ‘max’ if the internal measure should be maximizednclust_range: range of clusters to consider, defaultNonecombined_data: define whether multimodal data are used as input features. IfTrue, different sets of covariates will be applied for each modality (e.g. correction for TIV only for grey matter features). DefaultFalseNotes: in case Gaussian Mixture Model was implemented as clustering algorithm, theselect_best_bic_aicfunction can be used to compute Akaike and Bayesian Information Criterion (AIC, BIC) and used them for model's selection.4. ExampleAn example of how to performNeuRevalcan be found in the folderNeuReval/scripts. These codes show the application ofNeuRevalusing Gaussian Mixture Model as clustering algorithm, Support Vector Machine as classifier, and UMAP as dimensionality reduction algorithm:01_grid_search: code to perform grid-search cross-validation for clustering/classifier/preprocessing parameters tuning02_run_findbestclustcv: code to performNeuRevalwith the optimized clustering/classifier/preprocessing algorithms. This script also provides codes to compute different kind of internal measures03_visualization: code to create a plot for clusters' representation5. ReferencesLandi, I., Mandelli, V., & Lombardo, M. V. (2021). reval: A Python package to determine best clustering solutions with stability-based relative clustering validation.Patterns, 2(4), 100228.