Package: zadu
ZADUA-to-Zpython library for evalUatingDimensionality reductionZADU is a Python library that provides distortion measures for evaluating and analyzing dimensionality reduction (DR) embeddings. The library supports a diverse set of local, cluster-level, and global distortion measures, allowing users to assess DR techniques from various structural perspectives. By offering an optimized execution and pointwise local distortions, ZADU enables efficient and in-depth analysis of DR embeddings.InstallationYou can install ZADU viapip:pipinstallzaduSupported Distortion MeasuresZADU currently supports a total of 18 distortion measures, including:7 local measures5 cluster-level measures6 global measuresFor a complete list of supported measures, refer tomeasures. The library initially provided 17 measures when it was first introduced by our academic paper, and we added one more measure (label trustworthiness & continuity) to the library.How To Use ZADUZADU provides two different interfaces for executing distortion measures.
You can either use the main class that wraps the measures, or directly access and invoke the functions that define each distortion measure.Using the Main ClassUse the main class of ZADU to compute distortion measures.
This approach benefits from the optimization, providing faster performance when executing multiple measures.fromzaduimportzaduhd,ld=load_datasets()spec=[{"id":"tnc","params":{"k":20},},{"id":"snc","params":{"k":30,"clustering_strategy":"dbscan"}}]scores=zadu.ZADU(spec,hd).measure(ld)print("T&C:",scores[0])print("S&C:",scores[1])hdrepresents high-dimensional data,ldrepresents low-dimensional dataZADU ClassThe ZADU class provides the main interface for the Zadu library, allowing users to evaluate and analyze dimensionality reduction (DR) embeddings effectively and reliably.Class ConstructorThe ZADU class constructor has the following signature:classZADU(spec:List[Dict[str,Union[str,dict]]],hd:np.ndarray,return_local:bool=False)Parameters:specA list of dictionaries that define the distortion measures to execute and their hyperparameters.
Each dictionary must contain the following keys:"id": The identifier of the distortion measure, such as"tnc"or"snc"."params": A dictionary containing hyperparameters specific to the chosen distortion measure.List of ID/Parameters for Each FunctionWarning: While usingdsc,ivm,c_evm,nh, andca_tnc, please be aware that these measures assume that class labels arewell-separatedin the original high-dimensional space. If the class labels are not well-separated, the measures may produce unreliable results. Use the measure only if you are confident that the class labels are well-separated. Please refer to the relatedacademic paperfor more detail.Local MeasuresMeasureIDParametersRangeOptimumTrustworthiness & Continuitytnck=20[0.5, 1]1Mean Relative Rank Errorsmrrek=20[0, 1]1Local Continuity Meta-Criterialcmck=20[0, 1]1Neighborhood hitnhk=20[0, 1]1Neighbor Dissimilarityndk=20R+0Class-Aware Trustworthiness & Continuityca_tnck=20[0.5, 1]1Procrustes Measureprock=20R+0Cluster-level MeasuresMeasureIDParametersRangeOptimumSteadiness & Cohesivenesssnciteration=150, walk_num_ratio=0.3, alpha=0.1, k=50, clustering_strategy="dbscan"[0, 1]1Distance Consistencydsc[0.5, 1]0.5Internal Validation Measuresivmmeasure="silhouette"Depends on IVMDepends on IVMClustering + External Clustering Validation Measuresc_evmmeasure="arand", clustering="kmeans", clustering_args=NoneDepends on EVMDepends on EVMLabel Trustworthiness & Continuityl_tnccvm="dsc"[0, 1]1Global MeasuresMeasureIDParametersRangeOptimumStressstressR+0Kullback-Leibler Divergencekl_divsigma=0.1R+0Distance-to-Measuredtmsigma=0.1R+0Topographic Producttopok=20R0Pearson’s correlation coefficientpr[-1, 1]1Spearman’s rank correlation coefficientsrho[-1, 1]1hdA high-dimensional dataset (numpy array) to register and reuse during the evaluation process.return_localA boolean flag that, when set toTrue, enables the computation of local pointwise distortions for each data point. The default value isFalse.Directly Accessing FunctionsYou can also directly access and invoke the functions defining each distortion measure for greater flexibility.fromzadu.measuresimport*mrre=mean_relative_rank_error.measure(hd,ld,k=20)pr=pearson_r.measure(hd,ld)nh=neighborhood_hit.measure(ld,label,k=20)Advanced FeaturesOptimizing the ExecutionZADU automatically optimizes the execution of multiple distortion measures. It minimizes the computational overhead associated with preprocessing stages such as pairwise distance calculation, pointwise distance ranking determination, and k-nearest neighbor identification.Computing Pointwise Local DistortionsUsers can obtain local pointwise distortions by setting the return_local flag. If a specified distortion measure produces local pointwise distortion as intermediate results, it returns a list of pointwise distortions when the flag is raised.fromzaduimportzaduspec=[{"id":"dtm","params":{}},{"id":"mrre","params":{"k":30}}]zadu_obj=zadu.ZADU(spec,hd,return_local=True)global_,local_=zadu_obj.measure(ld)print("MRRE local distortions:",local_[1])Visualizing Local DistortionsWith the pointwise local distortions obtained from ZADU, users can visualize the distortions using various distortion visualizations. We provide ZADUVis, a python library that enables the rendering of two disotortion visualizations:CheckVizand theReliability Map.fromzaduimportzadufromzaduvisimportzaduvisimportmatplotlib.pyplotaspltfromsklearn.manifoldimportTSNEfromsklearn.datasetsimportfetch_openmlhd=fetch_openml('mnist_784',version=1,cache=True).data.to_numpy()[::7]ld=TSNE().fit_transform(hd)## Computing local pointwise distortionsspec=[{"id":"tnc","params":{"k":25}},{"id":"snc","params":{"k":50}}]zadu_obj=zadu.ZADU(spec,hd,return_local=True)scores,local_list=zadu_obj.measure(ld)tnc_local=local_list[0]snc_local=local_list[1]local_trustworthiness=tnc_local["local_trustworthiness"]local_continuity=tnc_local["local_continuity"]local_steadiness=snc_local["local_steadiness"]local_cohesiveness=snc_local["local_cohesiveness"]fig,ax=plt.subplots(1,4,figsize=(50,12.5))zaduvis.checkviz(ld,local_trustworthiness,local_continuity,ax=ax[0])zaduvis.reliability_map(ld,local_trustworthiness,local_continuity,k=10,ax=ax[1])zaduvis.checkviz(ld,local_steadiness,local_cohesiveness,ax=ax[2])zaduvis.reliability_map(ld,local_steadiness,local_cohesiveness,k=10,ax=ax[3])The above code snippet demonstrates how to visualize local pointwise distortions using CheckViz and Reliability Map plots.DocumentationFor more information about the available distortion measures, their use cases, and examples, please refer to our paper (IEEE VIS 2023 Short).CitationHyeon Jeon, Aeri Cho, Jinhwa Jang, Soohyun Lee, Jake Hyun, Hyung-Kwon Ko, Jaemin Jo, and Jinwook Seo. Zadu: A python library for evaluating the reliability of dimensionality reduction embeddings. In 2023 IEEE Visualization and Visual Analytics (VIS), 2023. to appear.@inproceedings{jeon23vis,author={Jeon, Hyeon and Cho, Aeri and Jang, Jinhwa and Lee, Soohyun and Hyun, Jake and Ko, Hyung-Kwon and Jo, Jaemin and Seo, Jinwook},booktitle={2023 IEEE Visualization and Visual Analytics (VIS)},title={ZADU: A Python Library for Evaluating the Reliability of Dimensionality Reduction Embeddings},year={2023},volume={},number={},pages={},doi={},note={to appear}}