Package: psis
Projection Separability IndicesThis python package is based on the MATLAB project namedprojection-separability-indices.DescriptionThe projection separability indices (PSIs) are projection-based statistical measures specifically designed to assess and quantify the group separability of data samples in a geometrical space. For instance, PSIs can be used to evaluate the quality of the dimensionality reduction analyses produced by embedding algorithms. Currently, this package implements four different PSIs for evaluating group separability and a statistical test termedtrustworthiness, which is based on a null model to assess the statistical significance of each PSI by ap-value.For more details seeMeasuring group separability in geometrical space for evaluation of pattern recognition and dimension reduction algorithms.PSI measurespsi-p: Based on the Mann-Whitney U-test p-value [1]psi-roc: Based on the Area Under the ROC-Curve [2]psi-pr: Based on the Area Under the Precision-Recall Curve [3]psi-mcc: Based on the Matthews Correlation Coefficient [4][1] H. B. Mann and D. R. Whitney, “On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other,” Ann. Math. Stat., vol. 18, no. 1, pp. 50–60, 1947, doi: 10.1214/aoms/1177730491.[2] J. S. Hanley and B. J. McNeil, “The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve,” Radiology, vol. 143, no. 1, pp. 29–36, 1982.[3] V. Raghavan, P. Bollmann, and G. S. Jung, “A critical investigation of recall and precision as measures of retrieval system performance,” ACM Trans. Inf. Syst., vol. 7, no. 3, pp. 205–229, 1989, doi: 10.1145/65943.65945.[4] B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme,” BBA - Protein Struct., vol. 405, no. 2, pp. 442–451, 1975, doi: 10.1016/0005-2795(75)90109-9.InstallationRun the following to install:pipinstallpsisUsageCompute PSIsimportnumpyasnpfrompsisimportindices"""Simulated embedding obtained by a dimension reduction method.In this example, only two dimensions are used. However, an arbitrarynumber of dimensions can be evaluated.Note: It is expected to receive samples as rows and the features/variables as columns."""embedding=np.array([[1,2],[3,4],[5,6],[7,8],[10,11],[12,13],[14,15],[16,17]])"""List of sample labels (groups/classes).In this example, only two different groups are used. However, an arbitrarynumber of classes can be evaluated."""labels=np.array(['group1','group1','group1','group1','group2','group2','group2','group2'])"""List of positive samples.Depending on the study, positive classes are usually ranked asthe labels for which a particular prediction is desired.For instance:- sick patients (positive class) versus controls (negative class)- burnout (positive class), depression (positive class), versus control (negative class)If you are not sure which are your positive classes, then omit this input and thealgorithm will take the groups with the lower number of samples as positive"""positives=np.array(['group1'])"""Base approach for projecting the points.Available options are:- centroid [default]- lda"""projection_type='centroid'"""Base approach for defining the groups' centroids.NOTE: Only applicable if projection_type is centroid, ignored otherwise.Available options are:- mean- median [default]- mode"""center_formula='median'# Group separability evaluationpsi_p,psi_roc,psi_pr,psi_mcc=indices.compute_psis(embedding,labels,positives,projection_type,center_formula)print(psi_p)print(psi_roc)print(psi_pr)print(psi_mcc)Compute trustworthiness of PSIsfromsklearn.datasetsimportload_irisfrompsisimportindices# Sample data. Details at https://scikit-learn.org/stable/datasets/toy_dataset.htmldata=load_iris()# Number of iterations for the Null modeliterations=50# Random seed (for reproducibility)seed=10# Linear Discriminant Analysis (LDA) based projectionprojection='lda'# Group separability evaluation.# In this example, the evaluation of group separability is directly# assessed in the High-Dimensional (HD) space, and the trustworthiness of# each PSI (together with details about their null model evaluation) is returnedresults=indices.compute_trustworthiness(data.data,data.target,iterations=iterations,projection_type=projection,seed=seed)# Accessing the results# In this example only 'psi_roc' is evaluated. The other indices' results can be# accessed in the same wayprint(results['psi_roc']['value'])# Initial index valueprint(results['psi_roc']['min'])# Minimum permuted valueprint(results['psi_roc']['max'])# Maximum permuted valueprint(results['psi_roc']['p_value'])# Separability significance (p-value)IssuesPlease, report any issue atpsis/issues