Package: parity-fairness
ParityThis repository contains codes that demonstrate the use of fairness metrics, bias mitigations and explainability tool.InstallationIn order for the explainability modules to work, first you have to install shap throughcondalike so:foo@bar:~$condainstall-cconda-forgeshapInstall using:foo@bar:~$pipinstallparity-fairnessBias Measurement UsageSetup the data such that the target column is a binary string target. Then find out which features are theprivileged categoriesand which values areprivileged values. Afterwards, feed them into the function calledshow_biaslike:from parity.fairness_metrics import show_bias

priv_category = 'Race-White'
priv_value = 'True'
target_label = 'high pay'
unencoded_target_label = 'True'
cols_to_drop = ''

show_bias(data, priv_category, priv_value, target_label, unencoded_target_label, cols_to_drop)Bias and FairnessA common problem with most machine learning models is bias from data. This notebook shows how to measure those biases and perform bias mitigation. A python package calledaif360can give us metrics and algorithms for bias measurement and mitigation.MetricsStatistical Parity DifferenceEqual Opportunity DifferenceAverage Absolute Odds DifferenceDisparate ImpactTheil IndexSome metrics need predictions while others just the original dataset. This is why we will use 2 classes of the aif360 package :ClassificationMetricandBinaryLabelDatasetMetric.For metrics that require predictions:Equal Opportunity Difference:equal_opportunity_difference()Average Absolute Odds Difference:average_abs_odds_difference()Theil Index :theil_index()For metrics that don't require predictions:Statistical Parity Difference:statistical_parity_difference()Disparate Impact:disparate_impact()People + AI ResearchA project by Google, this module helps visualize inference results, visualize feature attributions, arrange datapoints by similarity, edit a datapoint and see how it performs, compare counterfactuals to datapoints, and test algorithmic fairness constraints. More details can be foundhere.To use, simply add:from parity import pairSample code to show the GUI:from parity.pair import *

### Insert more code here

# Load up the test dataset
test_csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test'
test_df = pd.read_csv(test_csv_path, names=csv_columns, skipinitialspace=True,
  skiprows=1)

num_datapoints = 2000
tool_height_in_px = 1000
make_label_column_numeric(test_df, label_column, lambda val: val == '>50K.')
test_examples = df_to_examples(test_df[0:num_datapoints])
labels = ['Under 50K', 'Over 50K']

# Setup the tool with the test examples and the trained classifier
config_builder = config(test_examples, classifier, labels, feature_spec)

## Visualization only appears through jupyter notebook (not jupyter lab)
WitWidget(config_builder, height=tool_height_in_px)Bias MitigationThere are three types of bias mitigation:Pre-processing algorithms : they are used before training the modelIn-processing algorithms : they are fair classifiers so it's during the trainingPost-processing algorithms : they are used after training the modelBias Mitigation UsageBefore using the different bias mitigation functions, necessary objects must be prepared using the functionprepare_data. Once the dataset is processed which in this case is stored indata_orig, split this into testing and training datasets. Along with other different objects:encoders,numerical_featuresandcategorical_featuresthese will be fed appropriately when using any mitigation tools.from parity.fairness_metrics import prepare_data

priv_category = 'Race-White'
priv_value = 'True'
target_label = 'high pay'
unencoded_target_label = 'True'
cols_to_drop = ''

data_orig, encoders, numerical_features, categorical_features = prepare_data(data, priv_category, priv_value, target_label, 
                             priv_target_value=unencoded_target_label, ignore_cols=cols_to_drop)
                             
data_orig_train, data_orig_test = data_orig.split([0.7], shuffle=True)Preprocess MitigationsUsing the prepared data and identifying whichpriveleged categoery, bias is removed using any of the preprocessing mitigation function tools. Afterwards, this must be decoded into pandas dataframe for readability by feeding the transformed data and other objects into thedecode_datasetfunction.Disparate Impact Remover:disparate_impact_remover()from parity.fair import disparate_impact_remover, decode_dataset

data_transf_df = disparate_impact_remover(data_orig_train)

decoded_df = decode_dataset(data_transf_df, encoders, numerical_features, categorical_features)Learning Fair Representation:learning_fair_representation()from parity.fair import learning_fair_representation, decode_dataset

data_transf_df = learning_fair_representation(data_orig_train, priv_category)

decoded_df = decode_dataset(data_transf_df, encoders, numerical_features, categorical_features)Reweighing:reweight()from parity.fair import reweight, decode_dataset

data_transf_df = reweight(data_orig_train, priv_category)

decoded_df = decode_dataset(data_transf_df, encoders, numerical_features, categorical_features)Inprocess MitigationsFor inprocessing, thestructured_data_train_test_splitis used to split the prepared original dataset. These are then fed on a chosen mitigation tool and must be decoded into pandas dataframe for readability usingdecode_datasetfunction.data_orig_train, data_orig_test = structured_data_train_test_split(data_orig)Adversarial Debiasing:adversarial_debias()from parity.fair import adversarial_debias, decode_dataset

data_transf_train_df, data_transf_test_df = adversarial_debias(data_orig_train, data_orig_test, priv_category)

decoded_train_df = decode_dataset(data_transf_train_df, encoders, numerical_features, categorical_features)

decoded_test_df = decode_dataset(data_transf_test_df, encoders, numerical_features, categorical_features)Prejudice Remover Regularizer:prejudice_remover()from parity.fair import prejudice_remover, decode_dataset

data_transf_train_df, data_transf_test_df = prejudice_remover(data_orig_train, data_orig_test, priv_category, eta=25)

decoded_train_df = decode_dataset(data_transf_train_df, encoders, numerical_features, categorical_features)

decoded_test_df = decode_dataset(data_transf_test_df, encoders, numerical_features, categorical_features)Grid Search:gridSearch()This preprocessing tool has a different approch of debiasing. It works by generating a sequence of relabellings and reweighings, and trains a predictor for each. These values are then fed in the function calledshow_comparisonto create a dashboard that shows the comparison of before and after the mitigation.from parity.reduction import gridSearch

values = gridSearch(model = unmitigated_model, X_train = X_train, Y_train = Y_train, A_train = A_train, grid_size = 71)

show_comparison(model = unmitigated_model, X_test = X_test, Y_test = Y_test,
                          A_test = A_test, protected_features =['sex'], non_dominated = values)Postprocess MitigationsThis type of mitigation needs theprediction scoresfrom using the original model. Along with this, prepared testing and training datasets are fed into the mitigation function tool. Afterwards, the newly mitigated predictions must be decoded into a pandas dataframe for readability usingdecode_datasetfunction.Calibrated Equality of Oddscalibrate_equality_of_odds()data_trans_pred = calibrate_equality_of_odds(data_orig_train, data_orig_test, data_orig_test_pred, priv_category)

decoded_pred_df = decode_dataset(data_trans_pred, encoders, numerical_features, categorical_features)Reject Option Classificationreject_optiondata_trans_pred = reject_option(data_orig_train, data_orig_test, data_orig_test_pred, priv_category)

decoded_pred_df = decode_dataset(data_trans_pred, encoders, numerical_features, categorical_features)Fairness DashboardWhen creating a FairlearnDashboard one must provide theinput datato be analyzed for fairness metrics,sensitive attributes, andlist of ground truths and prediction. Once these are fed to the funciton calledfairness_dashboard, a dashboard should displays the sensitive attributes followed by options of model performance metrics. After selecting, performance and prediction disparity must be shown.from parity.fair import fairness_dashboard

sensitive_feature_names = ['Age', 'Sex']
A_test = data.iloc[X_test.index][sensitive_feature_names]

fairness_dashboard(A_test=A_test, protected_features=sensitive_feature_names, y_true=y_test, y_pred=y_pred)ExplainabilityAnother common problem in machine learning is determining which features significantly explain the prediction of a model. TheSHAP or SHapley Additive exPlanationslibrary can be use to address this. To simply get scores of how features are useful at predicting a target variable, usefeature_importancefunction. These are computed by providing themodeland theinput train data.from parity.explainer import shap_feature_explainer, plot_prediction_causes, dependence_plots, feature_importances

features = feature_importances(model, X_train)To compute SHAP values for each feature, feed themodelandinput train datain theshap_feature_explainerfunction.shap_values = shap_feature_explainer(model, X_train)To plot and determine the prediction causes for each feature from model given a specific row orindexin the data, use theplot_prediction_causesfunction.plot_prediction_causes(model, X_train, shap_values, index=1)On the otherhand, to plot the interaction effects of SHAP values of features with each other with respect to the target variable, usedependence_plotsfunction.dependence_plots(X_train, shap_values)