Dive into Bias Metrics and Model Explainability with Amazon SageMaker Clarify
By: Emily Webber
Tags: Bias, Machine Learning, Data Science, Sagemaker, AWS

Pre-Training Metrics

Pre-training metrics are a great first-pass on a dataset. We can run a job with SageMaker Clarify to identify key differences in our dataset across various attributes of interest, or variables for which we’d like to learn about any imbalances or samples that are present in the dataset which may be likely to lead to a biased model. SageMaker Clarify will produce a downloadable bias report with key statistics, and in Studio we can view all of these generated metrics with a rich visualization. To make it even easier, we also have metric description cards integrated within Studio for customers to surface the intuition behind the metric, read through examples and understand suggested ranges. Many of these cards will even point to the paper proposing that statistic on arxiv.org! If you’re interested, you can jump straight to one commonly referenced paper right here.



It’s helpful to point out that, in addition to looking at differences across attributes of interest, many of these metrics are looking at this difference in terms of the label. That is to say, to date these are primarily solving for supervised learning.

Class Imbalance

Equation for Class Imbalance

Our first metric to pick apart is called class imbalance. Simply put, this metric is telling us the difference of data samples we have for the advantaged group, n_a, relative to the disadvantaged group, n_d. Your class imbalance statistic falls in a range between [-1, +1], with a perfectly balanced dataset giving us a class imbalance of 0. One way to improve this statistic is actually downsampling on your advantaged group, or even augmenting your set of disadvantaged samples to improve their representation. Masked transformers might be a nice way of doing that!

Difference in Positive Proportions of True Labels

Equation for DPPL

This statistic is simply looking at the difference in the ratio of positive cases to total sample amounts between advantaged and disadvantaged groups. That is to say, we’ll look at the number of positive cases for the advantaged group, n_a(1), and compare that to the total number of positive samples we have, n_a. Next, we’ll do the same thing and compare the number of positive cases in the disadvantaged group, n_d(1), and compare that to the total number of disadvantaged samples we have, n_d. This is a valuable metric because it tells us the relatively likelihood of members from both the advantaged and the disadvantaged group to fall into the positive class.

If this statistic is close to 0, then we say that demographic parity has been achieved.

Kullback-Liebler & Jensen-Shannon Divergences

Equation for KL Divergence

These two metrics are a nice pair. They both describe the difference between two label distributions. The first term, P_a, refers to the distribution of the advantaged group, while P_d refers to the distribution for the disadvantaged group. They’re actually nice equations to get comfortable with, because you’re very likely to see them come up under other circumstances, such as generative deep learning. The Kullback-Liebler divergence is the relative distance between the distribution of advantaged cases versus the disadvantaged cases.

Equation for JS Divergence

The Jensen-Shannon divergence builds on this to bring symmetry into the equation, literally speaking, because we can reverse the order of operations and still get the same result.

Conditional Demographic Disparity in Labels

Equation for CDDL

The conditional demographic disparity in labels helps us understand whether the disadvantaged group has a larger proportion of rejected outcomes than accepted outcomes.

This statistic uses the demographic disparity statistic, or DD, which is the difference between the proportion of rejected outcomes and the proportion of accepted outcomes for a given group. The conditional demographic disparity in labels statistic looks at the weighted average of DD across different groups, each weighted by its size.

There are a variety of additional pre-training metrics to study, but in the interest of time we’ll move onto the post-training metrics.

Post-Training Metrics

Disparate (Adverse) Impact and Difference in Positive Proportions in Predicted Labels

Equation for DPPL

These two related statistics are helpful in understanding the ratio of cases from the disadvantaged group that are accepted, q_d, from that of the advantaged group, q_a. This first statistic is simply looking at the difference between these ratios.

Equation for Disparate Impact

In the disparate (adverse) impact statistic, we simply divide the q_d term by the q_a term to assess the magnitude of this difference. There is some legal precedent for considering a result on this statistics in the range of [4/5, 5/4] as being fair.

Difference in conditional outcome (acceptance and rejection)

Equation for Difference in Conditional Outcome

This statistic looks at the relationship between the actual labels and the predicted labels, and helps us see if they are the same across classes.

In the case of DCA, we’re looking at this in the case of accepted samples. In the case of DCR, we’re doing the same thing for rejected samples. This helps us pick on the type of bias when a lender may be giving more loans to men than prescribed by a model, but fewer loans to women than prescribed by the model.

Recall, precision, and accuracy differences

Equation for Recall Difference

This term helps us understand how well the model is performing in terms of recall within both the advantaged and disadvantaged groups. It may be the case that recall is high for a model only within the advantaged group, but low within the disadvantaged group, meaning it’s better at finding positive samples there.

The terms mentioned here come from a confusion matrix — a common table used to identify true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), for a given confidence level within a set of predictions. This is well defined elsewhere, so we won’t define them here.

It’s common to look at similar statistics for precision and accuracy.

Treatment Equality

Equation for Treatment Equality

This statistic looks at the difference in the ratio of false positives to false negatives between the advantaged and disadvantaged groups. You may have cases where both groups have the same levels of accuracy, but different ratios of false accepts to rejects. For example, let’s say 100 men and 8 women apply for a loan. Out of those, 8 men were wrongly denied, while 6 were wrongly approved. For women, 5 were wrongly denied, but only 2 were wrongly approved. Using a treatment equality statistic brings this to the forefront.

Counterfactual Flip Test

Equation for Counterfactual Flip Test

This metric looks at pairs of similar candidates across advantaged and disadvantaged groups, and helps us understand whether or not the model treats them the same. Typically these pairs are chosen using k-nearest neighbors — that is to say, the F+ and F- actually refer to similar pairs of data objects selected via KNN based on alternate samples.

Believe it or not but this list is really just the beginning of all of the metrics available with SageMaker Clarify! For an exhaustive list with an even deeper treatment, take a look at our whitepaper.

Explainability

Now, let’s take a quick look at another way of analyzing our data, which is to produce global and local feature importances. This uses a method called SHAP, which is a model-agnostic method inspired by game theory. SHAP actually produces a variety of candidate datasets, each without a given feature, and sends these against the trained model endpoint. Based on the model response, and the baseline statistics and samples that we provide when we configure the job, SHAP infers the relative importance of that feature.