Package: simplexai
SimplEx - Explaining Latent Representations with a Corpus of ExamplesCode Author: Jonathan Crabbé (jc2133@cam.ac.uk)This repository contains the implementation of SimplEx, a method to explain the latent
representations of black-box models with the help of a corpus of examples.
For more details, please read ourNeurIPS 2021 paper: 'Explaining Latent Representations with a Corpus of Examples'.:rocket: InstallationThe library can be installed from PyPI using$pipinstallsimplexaior from source, using$pipinstall.Toy exampleBellow, you can find a toy demonstration where we make a corpus decomposition of test examples representations.
All the relevant code can be found in the filesimplex.fromsimplexai.explainers.simpleximportSimplexfromsimplexai.models.image_recognitionimportMnistClassifierfromsimplexai.experiments.mnistimportload_mnist# Get a modelmodel=MnistClassifier()# Model should have the BlackBox interface# Load corpus and test inputscorpus_loader=load_mnist(subset_size=100,train=True,batch_size=100)# MNIST train loadertest_loader=load_mnist(subset_size=10,train=True,batch_size=10)# MNIST test loadercorpus_inputs,_=next(iter(corpus_loader))# A tensor of corpus inputstest_inputs,_=next(iter(test_loader))# A set of inputs to explain# Compute the corpus and test latent representationscorpus_latents=model.latent_representation(corpus_inputs).detach()test_latents=model.latent_representation(test_inputs).detach()# Initialize SimplEX, fit it on test examplessimplex=Simplex(corpus_examples=corpus_inputs,corpus_latent_reps=corpus_latents)simplex.fit(test_examples=test_inputs,test_latent_reps=test_latents,reg_factor=0)# Get the weights of each corpus decompositionweights=simplex.weightsWe get a tensor weights that can be interpreted as follows:weights[i,c] = weight of corpus example c in the decomposition of example i.We can get the importance of each corpus feature for the decomposition
of a given exampleiin the following way:importtorch# Compute the Integrated Jacobian for a particular examplei=4input_baseline=torch.zeros(corpus_inputs.shape)# Baseline tensor of the same shape as corpus_inputssimplex.jacobian_projection(test_id=i,model=model,input_baseline=input_baseline)result=simplex.decompose(i)We get a listresultwhere each element of the list corresponds to a corpus example.
This list is sorted by decreasing order of importance in the corpus decomposition.
Each element of the list is a tuple structured as follows:w_c,x_c,proj_jacobian_c=result[c]Wherew_ccorresponds to the weightweights[i,c],x_ccorresponds tocorpus_inputs[c]andproj_jacobianis a tensor such thatproj_jacobian_c[k]is the Projected Jacobian
of featurekfrom corpus examplec.Reproducing the paper resultsReproducing MNIST Approximation Quality ExperimentRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 9)python-msimplexai.experiments.mnist-experiment"approximation_quality"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.mnist.quality.plot_results-cv_listCV1CV2CV3...The resulting plots and data are savedhere.Reproducing Prostate Cancer Approximation Quality ExperimentThis experiment requires the access to the private datasets CUTRACT
and SEER decribed in the paper.Copy the filescutract_internal_all.csvandseer_external_imputed_new.csvare in the folderdata/Prostate CancerRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 9)python-msimplexai.experiments.prostate_cancer-experiment"approximation_quality"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.prostate.quality.plot_results-cv_listCV1CV2CV3...The resulting plots are savedhere.Reproducing Prostate Cancer Outlier ExperimentThis experiment requires the access to the private datasets CUTRACT
and SEER decribed in the paper.Make sure that the filescutract_internal_all.csvandseer_external_imputed_new.csvare in the folderdata/Prostate CancerRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 9)python-msimplexai.experiments.prostate_cancer-experiment"outlier_detection"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.prostate.outlier.plot_results-cv_listCV1CV2CV3...The resulting plots are savedhere.Reproducing MNIST Jacobian Projection Significance ExperimentRun the following scriptpython-msimplexai.experiments.mnist-experiment"jacobian_corruption"2.The resulting plots and data are savedhere.Reproducing MNIST Outlier Detection ExperimentRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 9)python-msimplexai.experiments.mnist-experiment"outlier_detection"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.mnist.outlier.plot_results-cv_listCV1CV2CV3...The resulting plots and data are savedhere.Reproducing MNIST Influence Function ExperimentRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 4)python-msimplexai.experiments.mnist-experiment"influence"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.mnist.influence.plot_results-cv_listCV1CV2CV3...The resulting plots and data are savedhere.Note: some problems can appear with the packagePytorch Influence Functions.
If this is the case, please changecalc_influence_function.pyin the following way:343:influences.append(tmp_influence)==>influences.append(tmp_influence.cpu())438:influences_meta['test_sample_index_list']=sample_list==>#influences_meta['test_sample_index_list'] = sample_listReproducing AR Approximation Quality ExperimentRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 4)python-msimplexai.experiments.time_series-experiment"approximation_quality"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.ar.quality.plot_results-cv_listCV1CV2CV3...The resulting plots and data are savedhere.Reproducing AR Outlier Detection ExperimentRun the following script for different values of CV (the results from the paper
were obtained by taking all integer CV between 0 and 4)python-msimplexai.experiments.time_series-experiment"outlier_detection"-cvCVRun the following script by adding all the values of CV from the previous steppython-msimplexai.experiments.results.ar.outlier.plot_results-cv_listCV1CV2CV3...The resulting plots and data are savedhere.:hammer: TestsInstall the testing dependencies usingpipinstall.[testing]The tests can be executed usingpytest-vsxCitingIf you use this code, please cite the associated paper:@inproceedings{Crabbe2021Simplex,
 author = {Crabbe, Jonathan and Qian, Zhaozhi and Imrie, Fergus and van der Schaar, Mihaela},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},
 pages = {12154--12166},
 publisher = {Curran Associates, Inc.},
 title = {Explaining Latent Representations with a Corpus of Examples},
 url = {https://proceedings.neurips.cc/paper/2021/file/65658fde58ab3c2b6e5132a39fae7cb9-Paper.pdf},
 volume = {34},
 year = {2021}
}