Package: aia-fairness
AIA_fairnessCode repository of the article.
The repositries segments the pipeline as to isolate key componants of the experimental analysis.Installation and dependenciesBy using pypi :pip install aia_fairnessOr clone the repository then runpip install --editable .in the directory containing pyproject.tomlAll dependencies are specified in pyproject.toml and installed automatically by pip.ConfigurationDefault configuration is loaded automatically and can be found atsrc/aia_fairness/config/defaulf.py.
To set custom configuration, first runpython-maia_fairness.configIt will create a fileconfig.pyin your current directory containing the default configuration.
You can then edit this at you liking.
If the fileconfig.pyexist in your current directory it is loaded.
If not, the default configuration is loaded.How to useDataset automatic download and processingPart of the dataset uses the kaggle API to download the data.
Hence you need to include your API key in~/.kaggle/kaggle.json.aia_fairness provides automatic download, formatic and saving of the dataset used in the article.
To use this feature and thus saving data in thedata_formatdirectory simply run oncepython-maia_fairness.dataset_processing.fetchThen you can load any dataset easily form anywhere in your code withimportaia_fairness.dataset_processingasdpdata=dp.load_format(dset,attrib)Dataset evaluationEach dataset can be evaluated along different axes:Sizesfromaia_fairness.dataset_processingimportmetricmetric.counting(<dataset>)Fairnessfromaia_fairness.dataset_processingimportmetricmetric.dp_lvl(<dataset>,<attribute>)To run all the evalution simply executepython-maia_fairness.dataset_processing.evaluationYou can refer to the full implementation intest/target_training.py.Running all the expriments form the paper** Heavy computing power required **One you have downloaded all the data you can run all the expirement of the article by runningpython-maia_fairness.experimental_stackor importaia_fairness.experimental_stackin the python interpreter.Plotting all the experiments of the paperRun the same shell command as for running the experiment adding theplotargument:python-maia_fairness.experimental_stackplotTraining a target modelaia_fairness.models.targetcontains various target model type.
The target models available are :RandomForestRandomForest_EGDfairlearn is used to impose EGDNeuralNetworkNeuralNetwork_EGDNeuralNetwork_FairgradOriginal implementation of the fairgrad paperNeuralNetwork_AdversarialDebiasingUses the fairlearn implementation of Adversarial DebisaingFor instance to train a random forest (based on sklearn) you canimportaia_fairness.models.targetastargetsT=dp.split(data,0)target.fit(T["train"]["x"],T["train"]["y"])yhat=target.predict(T["test"]["x"])Evaluation of a target modelaia_fairness.evaluationprovides the metrics used in the article.importaia_fairness.evaluationasevaluationsutility=evaluations.utility()fairness=evaluations.fairness()utility.add_result(T["test"]["y"],yhat,T["test"]["z"])fairness.add_result(T["test"]["y"],yhat,T["test"]["z"])utility.save(type(target).__name__,dset,attrib)fairness.save(type(target).__name__,dset,attrib)The save method, as called in the example, creates a directory structure of the form :#|result/target/<target type>
#|  |<name of the metric class>
#|  |   |<dset>
#|  |   |   |<attrib>
#|  |   |   |   <Name of metric 1>.pickle
#|  |   |   |   <Name of metric 2>.pickle
#|  |   |   |   <Name of metric ..>.pickle
#|  |<name of another metric class>
#|  |   |<dset>
#|  |   |   |<attrib>
#|  |   |   |   <Name of another metric 1>.pickle
#|  |   |   |   <Name of another metric 2>.pickle
#|  |   |   |   <Name of another metric ..>.pickleTraining an attackaia_fairness.modelsprovides the two types of AIA attack described in the paper:Classificationfor hard labelsRegressionfor soft labelsfromaia_fairness.modelsimportattackasattacksaux={"y":T["test"]["y"],"z":T["test"]["z"],"yhat":yhat}aux_split=dp.split(aux,0)classif=attacks.Classification()classif.fit(aux_split["train"]["yhat"],aux_split["train"]["z"])Evaluation of an attackSimilarly with evaluating the target modelaia_fairness.evaluation.attackcan be used to save the accuracy and the balanced accuracy of the attack.Plots and graphsTODOTests scriptsVarious tests are provided in thetestdirectory.download_data.pyFetches all the data form the diferent sources (don't forget to set your kaggle API key)target_training.pyLoads a dataset, split it with 5-folding (cross validation), train a target model on the data, and computes the metrics for this target modelDirectories structuredata_processing contains code that downloads, preprocess and saves the dataset in a uniformed pickle format exploitable by the rest of the pipeline usingload_format(dataset,attribute)function of the utils.py file.