Package: cfbench
Universal Counterfactual Benchmark FrameworkFastest way to test your tabular counterfactuals, evaluating 22 different datasets/models. All models are Keras/TensorFlow NN.RankingYou can see the ranking of the best counterfactual explanation generation algorithms in this repository:https://github.com/rmazzine/Ranking-Tabular-CFInstallationpipinstallcfbenchUsageThis code will just run the counterfacutal generatormy_cf_generatoron all factual instances and models.
Not creating any data, analysis or submitting to the benchmark.
If you want to do that, see the examples in Further Examples.importnumpyasnpfromcfbench.cfbenchimportBenchmarkCF# A simple CF generator, when the factual class is 1# return full 0 array, otherwise return full 1 arraydefmy_cf_generator(factual_array,model):ifmodel.predict(np.array([factual_array]))[0][0]>0.5:return[0]*len(factual_array)else:return[1]*len(factual_array)# Create Benchmark Generatorbenchmark_generator=BenchmarkCF().create_generator()# The Benchmark loopforbenchmark_datainbenchmark_generator:# Get factual arrayfactual_array=benchmark_data['factual_oh']# Get Keras TensorFlow modelmodel=benchmark_data['model']# Create CFcf=my_cf_generator(factual_array,model)# Get Evaluatorevaluator=benchmark_data['cf_evaluator']# Evaluate CFevaluator(cf,verbose=True,algorithm_name="my_cf_generator")Further informationWe understand that different counterfactual generators need different data, so our generator provide multiple data described in the following table:Click here for detailed infoTheBenchmarkCF().create_generator()method returns a generator that provides the following data:keyTypeDescriptionfactual_ohlistFactual, one hot encoded (if categorical features), datamodeltf.Keras.ModelModel to be explainedfactuallistFactual data (WITHOUT one hot encoding)num_featslistIndexes of the numerical continuous featurescat_featslistIndexes of the categorical featurescf_evaluatorBenchmarkGenerator.cf_evaluatorEvaluates if the CF is indeed a CF. Returns [True, cf_array] if a CF and [False, nan_array] otherwiseoh_convertercfbench.cfg.OHConverter.ConverterConverts to one hot.convert_to_ohor from one hot.convertdf_trainpandas.DataFrameDataframe of model's training data (WITHOUT one hot encoding)df_oh_trainpandas.DataFrameDataframe of model's training data (WITH one hot encoding)df_testpandas.DataFrameDataframe of model's test data (WITHOUT one hot encoding)df_oh_testpandas.DataFrameDataframe of model's test data (WITH one hot encoding)df_factualpandas.DataFrameDataframe of factual data (WITHOUT one hot encoding)tf_sessiontf.SessionTensorFlow sessionfactual_idxintIndex of the factual data in the factual datasetfactual_classintModel's prediction (0 or 1) of the factual datadsnamestrName of the datasetI want to get general metrics of my counterfactual generatorIf you want to get general metrics (coverage, sparsity, l2, mean absolute deviation, Mahalanobis distance, and
generation time), you can use the sample code belowClick here to see the codeTo generate a global analysis, you must create experiment data with the evaluator (benchmark_data['cf_evaluator'])
and assigning thecf_generation_time(the time it took to generate the CF) andsave_resultsas True, to
create the data to be analyzed (that will be in the folder./cfbench_results.)Then, theanalyze_resultsmethod makes the global analysis and returns a dataframe with all
results processed in the folder./cfbench_results_processed/. And it also prints the global
metrics in the console.importtimeimportnumpyasnpfromcfbench.cfbenchimportBenchmarkCF,analyze_results# A simple CF generator, when the factual class is 1# return full 0 array, otherwise return full 1 arraydefmy_cf_generator(factual_array,model):ifmodel.predict(np.array([factual_array]))[0][0]>0.5:return[0]*len(factual_array)else:return[1]*len(factual_array)# Create Benchmark Generatorbenchmark_generator=BenchmarkCF().create_generator()# The Benchmark loopforbenchmark_datainbenchmark_generator:# Get factual arrayfactual_array=benchmark_data['factual_oh']# Get Keras TensorFlow modelmodel=benchmark_data['model']# Create CF measuring how long it takes to generate the CFstart_time=time.time()cf=my_cf_generator(factual_array,model)cf_generation_time=time.time()-start_time# Get Evaluatorevaluator=benchmark_data['cf_evaluator']# Evaluate CFevaluator(cf_out=cf,algorithm_name='my_cf_generator',cf_generation_time=cf_generation_time,save_results=True)analyze_results('my_cf_generator')I want to rank my algorithmIf you want to compare your algorithm with others, you can use the code below.Click here to see the codeTo correctly send the results, you must create experiment data with the evaluator (benchmark_data['cf_evaluator'])
and assigning thecf_generation_time(the time it took to generate the CF) andsave_resultsas True, to
create the data to be sent (that will be in the folder./cfbench_results.)After the experiment loop, you must call thesend_resultsmethod of the evaluator,
to send the results to the server.This function will also create in the folder./cfbench_results_processed/a file with the
processed results of your algorithm.importtimeimportnumpyasnpfromcfbench.cfbenchimportBenchmarkCF,send_results# A simple CF generator, when the factual class is 1# return full 0 array, otherwise return full 1 arraydefmy_cf_generator(factual_array,model):ifmodel.predict(np.array([factual_array]))[0][0]>0.5:return[0]*len(factual_array)else:return[1]*len(factual_array)# Create Benchmark Generatorbenchmark_generator=BenchmarkCF().create_generator()# The Benchmark loopforbenchmark_datainbenchmark_generator:# Get factual arrayfactual_array=benchmark_data['factual_oh']# Get Keras TensorFlow modelmodel=benchmark_data['model']# Create CF measuring how long it takes to generate the CFstart_time=time.time()cf=my_cf_generator(factual_array,model)cf_generation_time=time.time()-start_time# Get Evaluatorevaluator=benchmark_data['cf_evaluator']# Evaluate CFevaluator(cf_out=cf,algorithm_name='my_cf_generator',cf_generation_time=cf_generation_time,save_results=True)send_results('my_cf_generator')After making the experiments and creating the analysis, you must forkthis repository.Then, you must provide the SSH path to your forked repo and, then, finally make a pull request to the main repository.All these details are included in the algorithm, in a step-by-step process.TensorFlow Version compatibilityThis framework is supposed to be compatible with TensorFlow 1 and 2, however, problems can arise. Therefore,
if you encounter any problem, please open an issue.ReferenceIf you used this package on your experiments, here's the reference paper:@Article{app11167274,AUTHOR={de Oliveira, Raphael Mazzine Barbosa and Martens, David},TITLE={A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data},JOURNAL={Applied Sciences},VOLUME={11},YEAR={2021},NUMBER={16},ARTICLE-NUMBER={7274},URL={https://www.mdpi.com/2076-3417/11/16/7274},ISSN={2076-3417},DOI={10.3390/app11167274}}0.0.9 / 2022-09-09Allow disable/enable(default) TF session request0.0.7 / 2022-09-03Allow to enable/disable TensorFlow 2 behavior0.0.7 / 2022-09-02[BUGFIX] Remove binary features from one-hot encoding evaluation[BUGFIX] Fix validity calculation0.0.6 / 2022-09-01Allow to add initial and final indexes0.0.5 / 2022-09-01Fix metric reportingAdd experiment progress information0.0.4 / 2022-08-31Implement method to send results to ranking repositoryImplement global metrics methodUpdate README.md0.0.3 / 2022-08-29==================Fix data files0.0.2 / 2022-08-29==================Simplified interface to run the benchmark with module cfbenchUpdated README.md