Package: faim
FAIMFAIM (FAir Interpolation Method), described inBeyond Incompatibility: Interpolation between Mutually
Exclusive Fairness Criteria in Classification Problems,
is a post-processing algorithm for achieving a combination of group-fairness criteria
(equalized false positive rates, equalized false negative rates, group calibration).This README.md is under construction!InstallationEnvironmentEnsure you have a environment with Python>=3.7 and pip>=2.21, preferably by creating a virtual environment.One way to do this is usingminiconda.  Install miniconda following
the instructions onthis pageand create a python 3.10 environment:condacreate-nfaimpython=3.10Activate the environmentcondaactivatefaimCheck that versions of python are >=3.7 and >=2.21, respectively:python--version
pip--versionPython PackageIf you intend to develop the package and/or contribute, follow the install instructions in theDevelopment Environmentsection below instead.  Otherwise, follow these instructions.The package and experiment CLI can be installed with pip:pipinstall"faim[experiment]"Note the[experiment]notation is required for now since, for the moment, the algorithm can only be run in experiment
mode for recreating experimental results in thepaper.
In the future,faimwill be made available for post-processing classifier scores
(given ground truth and group information), going beyond reproducing paper experiments.RemovalFrom the environment where you installed the package, runpipuninstallfaimUsageInstalling faim also (currently) installs one command line interface (CLI) tool,faim-experimentwhich can be
used to reproduce the work in the paper.[A general API will be added soon]ExperimentsThis section contains information for reproducing experiments in ourpaper.Ensure the package has been installed with[experiment]extra requirements before continuing
(seeInstallation | Python Package)!Prepare DataThe CLI can be used to prepare any of the three datasets used in thepaper:faim-experiment--prepare-dataDATASETwhereDATASETis one of:synthetic-from-papercompaszalando[waiting for permission to release, contact us for more information]The dataset will be downloaded, and prepared to a folder calledprepared-data.The following sections include info about each dataset:Synthetic dataThe raw dataset in the GitHub repo corresponds to synthetic prediction and ground truth scores for two groups,
for each group sampling from a corresponding binormal distribution.COMPAS dataThe raw data was obtained fromProPublica's COMPAS Analysis repository.Zalando dataUnder construction, more information to follow!Run ExperimentHaving prepared data following the instruction above, you are ready to run a FAIM experiment:faim-experiment--runPREPARED-DATASETLOW_SCORE_VAL,HIGH_SCORE_VALTHETASPREPARED_DATA_FILEPATHPREPARED-DATASETis now one of the following options (depending on what has been prepared):syntheticTwoGroups(prepared using--prepare-data synthetic)compasGender(prepared using--prepare-data compas)compasRace(prepared using--prepare-data compas)compasAge(prepared using--prepare-data compas)zalando(prepared using--prepare-data zalando) [waiting for permission to release, contact us for more information]LOW_SCORE_VAL,HIGH_SCORE_VALare two numbers that define the score range.THETAScorrespond to the fairness compromise you want. There are three thetas per group corresponding to the
desired amount of the three fairness criteria that the system should achieve:group calibrationequalized false negative ratesequalized false positive ratesNote, as discussed in the paper, thetas = 1,1,1 does not indicate that the system will simultaneously achieve all
three (mutually incompatible) fairness criteria, but rather the result will be a compromise between all three.See thepaperfor more details.Finally,PREPARED_DATA_FILEPATHcorresponds to the filepath of the prepared data.ExamplesRun all of the following from the same folder wherefaim-experiment --prepare-datawas run.In each example, a FAIM post-processor is trained and evaluated with results saved under theresultsfolder:Train FAIM model on synthetic dataset with callibration as fairness correctionfaim-experiment--runsyntheticTwoGroups0.11,0,0,1,0,0prepared-data/synthetic/2groups/2022-01-12/dataset.csvTrain FAIM model on synthetic dataset to achieve a combination of all three fairness criteria.faim-experiment--runsyntheticTwoGroups0.11,1,1,1,1,1prepared-data/synthetic/2groups/2022-01-12/dataset.csvVisualize ResultsNeeds documentation!Development EnvironmentTo develop and/or contribute, clone the repositorygitclone<thisrepoURL>From the root directory of the git repository, install the package with pip in editable mode (-e)
with extra requirements for experiments (experiment) and development (dev):pipinstall-e".[experiment,dev]"Don't confuse the[]to mean optional.  The".[experiment, dev]"notation tells pip to install extra
"experiment" and "dev" requirements including things likepytestandpre-commit.When contributing, be sure to install (and use) ourpre-commithooks:pre-commitinstall-tpre-commit-tpre-push