DPA: A one-stop metric to measure bias amplification in classification datasets
Authors: Bhanu Tokas
Rahul Nair
Hannah Kerner
Venue: NeurIPS 2025 poster ()
Keywords: algorithmic biases
fairness
accountability
AI ethics
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We measure directional bias amplification in balanced and unbalanced classification datasets

Abstract: Most ML datasets today contain biases. When we train models on these datasets, they often not only learn these biases but can worsen them --- a phenomenon known as bias amplification. Several co-occurrence-based metrics have been proposed to measure bias amplification in classification datasets. They measure bias amplification between a protected attribute (e.g., gender) and a task (e.g., cooking). These metrics also support fine-grained bias analysis by identifying the direction in which a model amplifies biases. However, co-occurrence-based metrics have limitations --- some fail to measure bias amplification in balanced datasets, while others fail to measure negative bias amplification. To solve these issues, recent work proposed a predictability-based metric called leakage amplification (LA). However, LA cannot identify the direction in which a model amplifies biases. We propose Directional Predictability Amplification (DPA), a predictability-based metric that is  (1) directional, (2) works with balanced and unbalanced datasets, and (3) correctly identifies positive and negative bias amplification. DPA eliminates the need to evaluate models on multiple metrics to verify these three aspects. DPA also improves over prior predictability-based metrics like LA: it is less sensitive to the choice of attacker function (a hyperparameter in predictability-based metrics), reports scores within a bounded range, and accounts for dataset bias by measuring relative changes in predictability. Our experiments on well-known datasets like COMPAS (a tabular dataset), COCO, and ImSitu (image datasets) show that DPA is the most reliable metric to measure bias amplification in classification problems.

DPA: A One-stop Metric to Measure Bias Amplification in Classification Datasets
Bhanu Tokas" Arizona State University bhanu. tokaseasu.cm edu
Rahul Nair* Arizona State University rnair210asu.er edu
Hannah Kerner Arizona State University hkernerGasu, edu
Abstract
Most ML datasets today contain biases. When train models these datasets, they often not only learn these biases but can worsen them - a phenomenon known bias amplification. Several co-occurrence-bait -basd metrics have been proposed to measure bias amplification in classification datasets. They measure bias amplification between protected attribute (e.g., gender) and task (e.g., cooking). These metrics also support fine-grained bias analysis by identifying the direction in which model amplifies biases. However, co-occurrence-basee metrics have limitations - some fail to measure bias amplification in balanced datasets, while others fail to measure negative bias amplification. To solve these issues, recent work proposed a predictability- (LA). However, LA cannot identify the direction in which model amplifies biases. We propose Directional Predictability Amplification (DPA), a predictability-base metric that (1) directional, (2) works with balanced and unbalanced datasets, (3) correctly identifies positive and negative bias amplification. DPA eliminates the need to evaluate models on multiple metrics to verify these three aspects. DPA also improves over prior predictability-basse sensitive to the choice of attacker function (a hyperparameter predictability- based metrics), reports scores within a bounded range, and accounts for dataset bias by measuring relative changes in predictability. Our experiments on well- known datasets like COMPAS tabular dataset), COCO, and ImSitu (image datasets) show that DPA the most reliable metric to measure bias amplification in classification problems. To compare DPA with existing bias amplification metrics, we released one-stop library of major bias amplification metrics at attps://github. com
metric called leakage amplification
metrics like LA: it is less
www.eeoooeeeeeeeeeee kerner- Bacab/Bias-aae Aalaaa-mmplffcatiion..
Introduction
Machine learning models should perform fairly across demographics, genders, and other groups. However, ensuring fairness is challenging when training datasets are biased, as is the case with many datasets. For instance, in the ImSitu dataset []]. 67% of the images labeled "cooking" feature females, indicating a gender bias that women are more likely to be associated with cooking than men [2]. Given biased training set, it is not surprising for a model to learn these dataset biases. Surprisingly, models not only learn dataset biases but can also amplify them 2 B 6]. In the example from ImSitu, where females and cooking co-occurred 67% of the time, bias amplification occurs when 67% of the images predicted as cooking feature females.
Several co-occurrence-bart based metrics (BA,, Multi B BAMALS) have been proposed to measure bias amplification in classification datasets. They measure the bias amplification between a protected
attribute (e.g., gender), denoted as A, and a task (e.g., cooking), denoted as T 2 3
4 If A and
'These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

T co-occur more than random in the training dataset, these metrics measure how much more they co-oceur in the model's predictions. For instance, if the co-oceurrence between females (A) and cooking (T) is 67% in the training dataset and 90% at test time, the bias amplification value is 23%.
snxccurrence-baset metrics help with a fine-grained bias analysis as they can identify the direction in which model amplified biases. In the cooking example from ImSitu, metrics like B.A, and M Mutii.. can measure how much model amplified the bias towards predicting women as cooking (A + T) and towards predicting cooks as women (T' + A). Such disentanglement of bias amplification is important, as it helps practitioners come up with targeted bias intervention efforts.
However, co-occurrence--aee based metrics have a limitation: they cannot measure bias amplification if A is balanced with T. Metries like B.A,, and B. B.AALS assume that if attributes and tasks are balanced in the training dataset, there are no dataset biases to amplify. Previous works have shown that simply balancing attributes and tasks does not ensure an unbiased dataset. Biases may emerge from parts of the dataset that are not annotated 77]
Suppose we balance imSitu such that 50% of the images labeled "cooking" feature females. Now, assume that cooking objects in ImSitu, like hairnets, are not annotated. If most of the cooking images with females have hairnets, while most of the cooking images with males do not, the model may learn a spurious correlation between hairnets, cooking, and females. Hence, the model may more often predict the presence of female when cooking images have hairnets in the test set, leading to bias amplification between females and cooking. However, since gender appears balanced with respect to the cooking labels, BA_, and B AM ALS would report 0 bias amplification. While other co-occurrence metrics like Multi may report non-zero bias amplification for balanced datasets, these values are often misleading, as M Multi cannot capture negative bias amplification scenarios.
To correctly measure bias amplification in balanced datasets (by accounting for biases from unanno- tated elements), Wang et al. 7 proposed a predictability-base dataset leakage (A"), which refers to how well attribute can be predicted from true task labels T, and model leakage (AM), which refers to how well A can be predicted from model's task predictions T. They defined bias amplification as X >P_ A and AM are measured using separate attacker function trained to predict A. In this work, we refer to Wang et al.'s method of calculating bias amplification leakage amplification (LA)). Since LA focuses on predictability, it can measure bias amplification in balanced and unbalanced datasets, However, it cannot identify the direction in which a model amplifies biases (A + T and T + A disentanglement not possible with LA).
metric called leakage. They defined
We do not have one metric that can (1) correctly report positive and negative bias amplification scenarios. (2) accurately measure bias amplification for both balanced and unbalanced datasets, and (3) identify the direction in which a model amplifies biases. Practitioners need to evaluate their models on multiple bias amplification metrics to assess these properties, making the process tedious. We propose Directional Predictability Amplification (DPA) A), a one-stop medictability-basee metric that addresses all these properties of bias amplification. In addition to being one-stop metric, DP A addresses several issues found in earlier predictability-basse as reporting unbounded bias amplification values, measuring absolute change in predictability instead of relative change, and being highly sensitive to attacker function.
metrics like LA, including issues such
Our key contributions: (1) DPA is the only directional metric that can accurately identify positive and negative bias amplification in both balanced and unbalanced classification datasets. (2) DPA A reports bias amplification values within a bounded range of [-1, 1]. enabling easy comparison across different models (3) DPA accounts for the original bias in the dataset as it measures relative change in predictability (instead of absolute change). (4) DP is minimally sensitive to the choice of attacker function. This eliminates the pain of choosing a suitable attacker to measure bias amplification.
2 Related Work
Co-occurrence for Bias Amplification Men Also Like Shopping (BAMALS) 2 proposed the first metric for bias amplification. The proposed metric measured the co-occurrences between protected attributes A and tasks T. For any T - A pairs that showed positive correlation (i.e., the pair occurred more frequently than independent events) in the training dataset, it measured how much the positive correlation increased in model predictions.
2

---

Wang and Russakovsky [3] generalized the BAMALS metric to also measure negative correlation (i.e., the pair occurred less frequently than independent events). Further, Wang and Russakovsky B changed how the positive bias is defined by comparing the independent and joint probability of a pair. But, both B ALS sets (e.g., (Basketball) ) & (Male)). Zhao et al. 4 extended the metric proposed by Wang and Russakovsky [3] to allow T A pairs when T, A are non-singleton sets (e.g.. (Basketball, Sneakers] & (African-American, Male)).
and BA_, (3] could only work for A pairs when 7, were singleton
Lin et g proposed new metric called bias disparity to measure bias amplification in recom- mender systems. Foulds et al. [9] measured bias amplification using the difference in "differential fairness', a measure of the difference in co-occurrences of T A pairs across different values of A. Seshadri et al. [T0] measured bias amplification for text-to-image generation using the increase in percentage bias in generated vs. training samples.
Wang et al. 7 identified that BAMALS [2 failed to measure bias amplification for balanced datasets. They proposed metric that refer to leakage amplification that could measure bias amplification in balanced datasets. While some of the previously discussed metrics (9] TO can measure bias amplification in balanced dataset, these metries do not work for continuous variables, because they use co-occurrences to quantify biases.
Bias Amplification in Balanced Datasets
Leakage amplification quantifies biases in terms of predictabilitt, i.e., how easily model predict the protected attribute from a task T. Attacker functions (f) are trained to predict the attribute (A) from the ground-trut truth observations of the task (T) and model predictions of the task (T). The relative performance of / on T vs. represents the leakage of information from A to T.
As the attacker function be any kind of machine learning model, it process continuous inputs, text, and images. This flexibility gives Ieakage amplification distinct advantage over co-occurrence- based bias amplification metrics. Subsequent work used leakage amplification for quantifying bias amplification in image captioning 11.
Capturing Directionality in Bias Amplification While previous metrics, including leakage ampli- fication 7. could detect the presence of bias, they could not explain its directionality. Wang and Russakovsky (3] was the first to introduce directional bias amplification metric, BA However, the metric only works for unbalanced datasets. Zhao et al. 4 proposed new metric, Multi.,, to measure directional bias amplification for multiple attributes and balanced datasets. Still, the metric cannot distinguish between positive and negative bias amplification, as shown in section A This lack of sign awareness makes Mutii,, unsuitable for many use cases.
In summary, no existing metric can measure the positive and negative directional bias amplification in balanced dataset, as shown in Table 1
Table I: We compare desirable properties of bias amplification metrics. Only DPA has all three.
Method
Balanced Datasets
Directional
Negative Amp.
BAMALS BA-
X
x
M Multi
LA
X
DPA (Ours)
3 Directional Predictability Amplification
In this section, we explain how our predictability metric, DPA, measures directional bias amplification (A - T and T -> A). We also show how DPA A is more robust and easier interpret compared to previous predictability-based metrics like LA. We demonstrate the other two properties of D.A:: (1) its ability to quantify both positive and negative bias amplification, and (2) its efficacy across both balanced and unbalanced classification datasets, through our experiments (Section
3.1 Problem Formulation and Proposed Method
Before introducing our proposed metric, we outline the problem setup. Consider a classification dataset consisting of images 1, where each image is annotated with task labels T and protected attributes A. A model M processes the input images I to predict tasks I and attributes Á.
3

---

Prodictions
True Task Labels (2)
Input Image (7)
Pot, Soap,
Pot. Pan, Toast
Ball, Bat
True Attribute
Label (4)
Attucker function fx
Train fä predict T from A: Dataset bias (9.))
Male
f
Predictions
Model's Task Predictions (T)
Knife, Toast, Pan, Laptop
Pot, Pan, Toast
DPA V
Attacker function f² Train f2 to predict from A: Model bias (V)
$ Predictions
True Attribute Label (4)
Input Image (I)
True Task
Male
Male
Labels (7)
Train f to predict A from Dataset bias (V.))
Pot
Attacker function fz
Pan Toast
f Predictions
Model's Attribute Prediction (Á)
Male
Female
DPA7-A = (W7 ( + w)))
Attacker function fz
Train fit predict Á from T: Model bias (V
Figure A T and T + A bias amplification using our predictability-batt based metric DPA
To measure bias amplification, we quantified how much A influences (A + T bias) and how much T influences A (T - A bias) in both the true labels (dataset bias) and the model M's predictions (model bias). While previous directional metrics, such as B BA,, ,, 3 and Mulli., , [4]. used conditional probabilities quantify dataset and model bias, we used the concept of predictability.
To measure dataset bias in A + T D. we trained an attacker function f that predicts true task labels (7) using the true attribute label (A) as an input. f is any arbitrary model, such as an SVM, decision tree, or neural network. We measured attacker function performance using quality function Q. Accuracy and F-1 scores are examples of some popular quality functions. If f predicts T from input A with high accuracy, it shows that the true attribute label (A) has high predictability for true task labels (T), indicating high dataset bias A TT
To measure model bias A + T (I))), we trained an attacker function fx that predicts model M's task predictions (?) using the attribute label (A) input. If the attacker function I predicts T from input with high accuracy, shows that true attribute label (A) has high predictability for M's task predictions (7), indicating high model bias A 7 (refer Figure
For A + I", we mathematically denote dataset bias (1) and model bias (Vx))
V Q(FT(A))TT)
(1)
V (A),
(2)
Similarly, for > A, we mathematically denote dataset bias ( and model bias (V,41
V7 P Q(A
V M = QUA
(4)
For T + A, / represents an attacker function that takes T as input and predicts A, while / represents an attacker function that takes T as input and predicts Á. We define bias amplification as the difference between the model bias and dataset bias, normalized by the sum of these biases. Using equations and 2 and equations
3 and 4
we define bias amplification in both directions:
V/
M
V
v
DPAA-T
(5)
DPAT-A =
(6)
+
V
+
Note, a small value e + 0+ is added the denominator for numerical stability.
When measuring dataset or model bias in either direction, note that the input to the attacker function the same (true attribute label A for equations [ 2]d and true task label 7 for B This is because we followed Wang et al.'s [3] definition of directionality, where bias amplification

---

(change in biases) should be measured with respect to a fixed prior variable (A or T in this case). We offer a detailed proof of how DPA follows Wang al.'s definition of directionality in Section
Note that the model M's predictions are not 1 100% accurate and may contain errors. For example, assume we are measuring bias amplification in A - T, and model M achieves 70% accuracy on task predictions T'. Since the true task labels T are 100% accurate (as they represent ground-truth), this 30% gap in accuracy could influence bias amplification values. To prevent conflating prediction errors with bias, we adopt the procedure of Wang al. 7 to equalize the dataset and model accuracy. In this case, we randomly flip 30% of the true task labels T so that their accuracy aligns with the model M's s 70%. As the bias I (or A for T + A) can vary significantly between two iterations of random flips, we measured bias amplification using confidence intervals across multiple iterations.
3.2 Benefits over existing predictability-basee metrics
In addition directionality, DPA has these benefits over other predictability-bast metrics like LA: Reports bias amplification in a fixed range For any quality function Q (such that its range [0, x)) or [0, RtD. D) the range of DPA is restricted to [-1,1]. Since DPA reports bounded values, it is easy to compare bias amplification across different models.
Note: While selecting Q, users must ensure that 0 represents the worst possible performance by the attacker (i.e., low predictability or no bias), and the upper bound for Q represents the best possible performance by the attacker (i.e., high predictability or significant bias). This is true for most typical choices for quality functions, such as accuracy or F1 score, but not for certain losses like cross-entropy. For cross-entropy, lower value would indicate good performance by the attacker (i.e., high predictability or significant bias). To use cross-entropy (or similar loss functions) as our Q, we should modify it to (1 (1/cross-entropy) or (1-cross-entropyy
Measures Relative Amplification DPA A accounts for dataset bias by measuring the relative change in bias (or predictability). rather than the absolute change as done by LA. Assume we have an unbiased dataset D1 with a dataset bias of O. and a highly biased dataset D2 with a dataset bias of 0.9. We train identical models M1 and M2 on D2 and D2, respectively. Assume the resulting model biases are 0.05 for Mj and 0.95 for M2. If we measure absolute change like LA,, both D1 and D2 show the same increase in bias (0.05). making the two scenarios appear equally problematic. In practical settings, a model introducing bias an originally unbiased dataset (D1) is more concerning than one adding the same amount of bias to already biased dataset ( (D2). If we measure relative change like DPA, we can distinguish between the two scenarios - the relative increase bias (or amplification) for D2 is 1, while for D2 it is only about 0.06. To further show how relative change in DPA accounts for dataset bias, refer to the analysis we conducted in Scction C
Robust to the choice of attacker function Predictability-bant bbsed metrics are highly sensitive to the hyperparameters of the attacker function used, as shown in [2. Since DPA measures relative change in predictability, it is more robust to different hyperparameters of the attacker function. In section D we performed experiments on the ImSitu dataset and a controlled synthetic dataset to show how measuring relative change in predictability improves DP D'A's robustness to attacker function.
4 Experiments and Results
To show that DPA is the most reliable metric to measure bias amplification in classification datasets, we conducted experiments one tabular dataset (COMPAS [3) and two image datasets (COCO 14] and ImSitu (T). For COCO and ImSitu, we used gender as the protected attribute. In this section, we describe the experimental details and results for these datasets. For the COMPAS dataset, we used race as the protected attribute. The experiment setup and results for COMPAS are in Section
B
4.1 COCO Experiment Setup
COCO [4] is an image dataset where each image has one or more people (male or female) along with various objects (e.g., ball, book). We performed an experiment similar to Wang et al. [.] to test whether different bias amplification metrics can correctly identify the direction in which model amplifies bias. We gradually increased the T - bias in COCO through image masking and evaluated which metries were able to detect this increasing bias.
5

---

We used the gender-annn annotated version of the COCO dataset released by Wang et al. 77]. Each image is labeled with gender (A) and object (I) annotations. A holds single value: {Female 0, Male 1]. If the image has multiple people, A corresponds to the gender of the person in the foreground. T is one-dimensional vector of 78 binary values: [40, (1, ...: [T7] (e.g., teddy bear, skateboard). 4, = if at least one instance of that 7th object is present in the image, else t, O. We sampled balanced and unbalanced sub-datasets from COCO. The balanced dataset is subject to the constraint in Equation 7
Vy #mm y) = # (f.
Where #(m. y) represents the number of images where male performs task y and #ff.2 y) represents the number of images where a female performs task y. As these constraints are hard to satisfy. only a subset of 12 objects (or tasks) was used in the balanced sub-dataset. This resulted in 6156 images in the sub-dataset (3078 male and 3078 female images). We used the same 12 objects for the unbalanced sub-dataset, but relaxed the constraint from Equation 7 as shown in Equation g sub-dataset of 15743 images (8885 male and 6588 female images)
This resulted in a
Vy: 1 #(m 3) #(.v))
<3
(8)
To gradually increase T + A bias, we created four versions of each sub-dataset: (1) the original sub-dataset, and three other versions where the person in the image is progressively masked by (2) partially masking the segment, (3) completely masking the segment, and (4) completely the masking bounding box. As we progressively mask more of the person, the model loses access to visual gender cues and is forced to rely heavily on surrounding objects (tasks) to predict gender. This causes the T + A bias in the model to gradually increase. We evaluated whether bias amplification metrics report steady increase in T - A scores the - A model bias increases across the four versions. Since all models are trained on COCO, the dataset bias remains constant. We can directly compare changes in amplification scores with changes in model bias.
We used the image masking annotations for COCO from [3]. We used two models: ViT b 16 [S]
and VGG16 [16] (both pretrained on ImageNet- 1 [7].. We replaced the last fully connected layer of these pre-trained models with a layer of size [2']. We fine-tuned these models for 12 epochs on dataset versions (4 versions of the balanced and 4 versions of the unbalanced sub-dataset). We used only two models as fine-tuning a model on 8 datasets is computationally expensive.
We measured bias amplification using co-occurrence-based metrics like BA.,, BAMALS, Mulli.,, metries like LA and DPA (ours). For LA and DPA, we used an MLP with two hidden layers as our attacker function. Additional experiment details are in Section ]
and predictability-bant
4.2 COCO Results
To quantify the T + A model bias in the VIT (or VGG), we computed feature attribution score using Gradient Shap [18. The attribution score measures the percentage contribution of non-person pixels in the ViT's prediction of the person's 's gender. A higher attribution score means the VIT is relying more on the background objects rather than the person to predict gender. In Table 2, we show the average attribution score for the VIT model, along with bias amplification values from different metrics. Row 2 reports results on various masked versions of the unbalanced COCO sub-dataset, while row 3 reports results for the balanced sub-datased
In the unbalanced sub-dataset, the attribution score increased with each masked version. This confirms that as the person was progressively masked, the ViT increasingly used background objects to predict gender, indicating a rise in T + A model bias. Only DPA and BA reported increasing T + A scores, correctly capturing the growing model bias. Since metries like BAMALS and LA lack directionality, they failed to capture the increasing T + A model bias. Although Multi,, is directional, it too failed to capture the increasing T + A bias. This is because Mutti,, cannot capture negative bias amplification scenarios (we have clearly demonstrated this using the COMPAS experiment in Section E)).
In the balanced COCO sub-dataset, labeled background objects (i.e., task objects) are balanced with gender, so they should not provide predictive signals for attribution score with each masked version. This indicates that the model used background cues that were not part of the labeled object (or task) set.
gender. However, we observed rising

---

Table 2: T - A bias amplification as the person is progressively masked in the unbalanced and balanced COCO sub-datasets, Attribution scores indicate the T + A model bias in the ViT _T_b_16 16 model. For the unbalanced sub-dataset, DPA and BA. captured the increasing T + A model bias. For the balanced sub-dataset, only DPA captured the increasing T - A model bias. Subscript shows confidence intervals.
Dataset Split
Metric
Original
Partial Masked
Segment Masked
Bounding-Boo Masked
Image
Attribution Map
Attribution
0. 0.38276.000
0. 0.432To.oom 0o03
U. 0.4461p 0020
U.32447g 0002
DPA (ours)
0..0152 0017
0. 0.1365e 0013
0.6015p co0s
0.8085g 0003
Unbalanced
BA+
0.0227g.oo .000s
0.0097g.o
0.0188p
0. 0.06001..mg
Malti-.
0.150 0002
0.1179p 0004
0.3606p con3
0.56077 0010
LA
0. 0.0368o.oo
0.071 5gg
0.09422
0.02655.
BAMALS
0.000 lo. 0.00011..oo ooo0
0. 0.0004g.com .oooo
0.00166..00 acon
0.001 50000000
Attrbuttio
0. 0.4568o 0026
0.46844 .0026
0. 0.4834p 0026
0 0.4933% .022
DP (ours)
0. 2.02300. 0008
0. 0.0280g .oon
0.0380p 003o
0. 1.390...ooo noo6
Balanced
BA
0. 0.00000.com
0. 0.0000g .oono
0. 0.0000g. cono
0. 0.00000.oomm
Malti-
1.00088..om ocoe
0.0010p coso
0.0038p cono
0.00433, po00
LA
0. 0,00006..rae 00002
0.0028g oo02
0.0020p oons
0,0019g
BAMALS
0.00000.000 0000
0.0000g 000o
0.0000g 0000
0.0000g 0000
To investigate this further, we visualized attribution maps for an image from the balanced COCO sub-dataset, as shown in Table As the person was gradually masked, the ViT model increasingly focused on unlabeled objects, such as skis and ski poles, to predict gender. This shows that even if datasets are balanced, models can amplify bias from unlabeled objects at test time. This behavior is captured by DPA as it reports increasing T A scores. While Multi showed the same trend as DPA, it is not reliable as it does not capture negative bias amplification (as mentioned carlier). BA.,, and BA ALS incorrectly assume that a balanced dataset contains no bias. They failed to capture the increasing T + A model bias and consistently reported zero bias amplification.
While the attribution maps in Table show the ViT's reliance on unlabeled objects, they may not fully convince readers that balanced datasets can amplify biases from unlabeled objects. To make this point clear, we designed controlled experiment (Section FF) where we introduced hidden A T bias into the balanced COCO sub-dataset using one-pixel shortcuts (OPS) [19]. This setup mimics the presence of bias from unlabeled objects. Only DPA A was able to detect the bias amplification caused by this hidden bias. To further convince readers, we conducted another controlled experiment on hidden biases using the CMNIST dataset in Section
The VGG model showed similar trends to the ViT model on both the unbalanced and balanced COCO sub-datasets. We showed VGG results in Section H
4.3 ImSitu Experiment Setup
ImSitu [ is an image dataset where a person (male or female) performs an activity (e.g., playing, cating). We examined which metrics can correctly measure directional bias amplification in ImSitu. ImSitu did not provide image mask annotations, so we were unable to perform progressive person masking to verify directionality. Instead, we examined which metrics could detect A + T and T A bias amplification in the original (unmasked) dataset.
We used the cender-annotated version of the ImSitu dataset released by Wang et al. [7]. Each image has gender (A) and activity (T) annotations. A holds single value: (Female 0, Male 1). the image has multiple people. A corresponds to the gender of the person performing the activity. T is one-dimensional vector 211 activities: (lo, 41,..., 4210] (e.g., repairing, curling). Each image has only one activity annotation that activity is assigned 1 and the rest are 0.
7

---

We sampled unbalanced and balanced dub-datasets from ImSitu. To sample the balanced sub-dataset, we used the constraint in Equation 7 This resulted in a sub-dataset of 14600 images (7300 male and 7300 female images). For the unbalanced sub-dataset, we used modified constrained (Fquation ( This resulted in a sub-dataset of 24301 images (14199 male and 10102 female images). We selected subset of 205 activities (tasks) for both sub-datasets.
#(m, v) #(f.u)
Vy:
<3
The ImSitu experiment was much less expensive than the COCO masking experiment, as we had to train each model only sub-datasets. Along with ViT 16 [5] and VGG16 [6] (used in the
COCO setup), we evaluated seven more models - Max [20]. 15 ViT_b_3t b_32 [5], SqueezeNet [2]]. Wide ResNet50 [22]. Wide ResNet101 [22). MobileNet V2 [23] and Swin Tiny [24]. For each
of the sub-datasets, we measured bias amplification caused by these models in two directions: bias amplification caused by gender (AA) on activities (T): A T,, and the bias amplification caused by
activities (T) on gender (A)): T + A.
As mentioned in Section 4.1 since all models are trained the same dataset (in this case, ImSitu), bias amplification only depends on model bias. We examined which bias amplification metrics correctly captured A T and T + A model biases. We used co-occurrence-base metries like BA B MAMAL ALS, Multi.,, and predictabilityybasee metrics like LA and DPA (ours). Similar to COCO, for LA and DP A, we used MI. with two hidden layers our attacker model. Additional experiment details are in Section
4.4 ImSitu Results
To quantify A + T and T model bias in ImSitu, we could feature attribution scores as ImSitu lacks person mask annotations. Instead, we used "conceptual sensitivity" scores based on Concept Activation Vectors (CAVs) (25). Sensitivity score (Sen) quantifies how much model's prediction is influenced by specific concept.
We define SCRA-T the influence of the gender concept (male female) in the overall task predictions. We define Seny-A as the contribution of task concepts (activities like playing, eating) in the overall gender predictions. If Sen is high, it means that the concept "male" (or female) highly influences the model's 's task predictions, indicating a strong - T model bias. If Seny is high, it means that the task concepts (e.g., eating) highly influence the model's attribute predictions, indicating a strong T + A model bias.
In Table 3 showed the sensitivity and bias amplification scores in the A + T direction for the unbalanced sub-dataset. In Table 4 we showed corresponding results for the balanced sub-dataset. For both tables, ranked all models (from 1 to 9) by their SEn scores, from least biased to most biased. We checked which bias amplification metrics produced rankings closest to this model which metrics best captured changes in model bias.
bias ranking. This allowed us
In the unbalanced case (Table 3} DPA's ranking was very close to the model bias ranking differed by only one position (rank 5 vs. 6). All other metrics showed significant mismatches. In balanced case (Table 4. DPA fully matched the Sen, A- rankings. Other metrics, like LA and Multi,,, differed significantly. Metrics like BA,, and B BAMAL ALS reported zero bias amplification because they assume that a balanced dataset has no bias. We observed similar trends for the unbalanced and balanced sub-datasets in T A. The T + A results shown in Section [
Our experiments on COMPAS, COCO, and ImSitu show that only DPA reliably captures the three properties of a bias amplification metric: directionality, accurately measuring both positive and negative bias amplification, and working with both unbalanced and balanced datasets. No other metrics satisfy all three properties. . Multi,, cannot measure negative bias amplification. BA,,, and B A M ALS always report zero amplification on balanced datasets. LA cannot capture the direction in which model amplifies biases. We also ran controlled experiment on the directional metrics (DPA, BA Multi-,) Section K to show how BA,, fails to measure bias amplification in balanced datasets, and how Mult Multi_ fails to capture negative bias amplification scenarios.
8

---

Table 3: A - T results for the ImSitu unbalanced sub-dataset. Models are shown in increasing order of model bias, indicated by their S-R scores. For each metric, the number in brackets shows the rank it assigned to that model. Rank is the lowest (most negative) bias amplification, and rank 9 is the highest (most positive). Red numbers show where the metric's s rank differs from the Sen rank. DPA is the most reliable metric, as its model rankings best match the -enA-T rankings. All values are scaled by 100.
Models
SeRA-T
DPAA-TT
LA
BA-
Multi-,
BAMALA ALS
MaxViT
20
0.017 (1)
-0.147 (1)
0.130 (4)
-0.157 (3)
0.553 (6)
0.211 (6)
ViT b_32
[
0.018 (2)
-0.050 (2)
0.209 (6)
-0.146 (G)
0.606 (8)
0.248 (8)
1iT_b_16
[5]
0.025 (3)
-0.036 (3)
(5)
-0.145 (7)
0.568 (7)
0.228 (7)
SqueezeNet 21
0.082 (4)
-0.022 (4)
0.009
-0.172 (1)
48.40 (9)
25.58 (9)
Wide ResNet101
22
0.144 (5)
0.064 (6)
0.281 (9)
-0.146 (5)
0.348 (4)
0.109 (4)
MobileNet V2
23
0.156 (6)
-0.011 (5)
0.087
-0.158 (2)
0.505 (5)
0.193
Wide ResNet50
22
0.190 (7)
0.118 (7)
0.253 (8)
- -0.141 (8)
0.260 (3)
0.061 (3)
Swin Tiny
0.225 (8)
0.318 (8)
0.064 (2)
00.15 (4)
0.159 (2)
-0.074 (1)
VGG16
IC
0.272 (9)
0.329 (9)
0,210 (7)
-0.051 (9)
0.097 (1)
-0.045 (2)
Table 4: A + T results for the ImSitu halanced sub-dataset. This table follows the same setup Table 3 Red
numbers show where the metric's rank differs from the SEnA-T rank. DPA A is the most reliable metric, its model rankings exactly match the Sen a rankings.
Models
S
DP D
LA
BA_
Multi_
BAMALS
Squeezzeeet
21
0.000 (1)
0.003 (1)
0.003 (8)
0.000 (1)
48.47 (9)
0.000 (1)
Wide ResNet50
[2]
0.089
0.085
0.002 (7)
0.000 (1)
0.705 (3)
0.000 (1)
Wide ResNer10]
20
0.098 (3)
0.094 (3)
0.000 (3)
0.000 (1)
0. 730(4)
0.000 (1)
MobileNet V2
[3]
0.112 (4)
0,250 (4)
0.007 (9)
0.000 (1)
1.379 (5)
0.000 (1)
Swin Tiay 24
0.211 (5)
0.282 (5)
0,001 (5)
0.000 (1)
0.038 (1)
0,000 (1)
VTT b_32
15
0.283 (6)
0.285 (6)
0.001 (5)
0.000 (1)
1.510 (6)
0.000 (1)
VIT b 16
0.302 (7)
0.302
0.000 (2)
0.000 (1)
1.599 (7)
0.000 (1)
MaxViT 20
0.478 (8)
0.326 (8)
-0.002 (1)
0.000 (1)
1.762 (8)
0.000 (1)
VGG16
0.497 (9)
0.341 (9)
0.000 (3)
0.000 (1)
0.073 (2)
0.000 (1)
5 Discussion
5.1 Why and when should we use DPA?
Table 5: Example dataset and model prediction tables to demonstrate when we should use BA, and DPA. We used the same tables for the hiring scenario and the indoor-outdoor scenario.
(a) Original Dataset
(b) Model Predictions
T= 0
T=
f
0
f
150
A= 0 A=1
15
185
A=
22
78
91
In real-world machine learning applications, common to train multiple models on the same dataset to determine which model performs best. Often, these models achieve similar accuracies, making model selection difficult - phenomenon known as the Rashomon effect (26 27 In such cases, metrics like DPA can offer valuable insights into model selection by highlighting which models preserve, reduce, or worsen existing biases.
Based on our experiments, we found DPA to be the most reliable metric to measure bias amplification. However, there are cases where other directional metrics like BA, could be more appropriate. Let us take two scenarios where we measure A - T bias amplification in DPA and BA.,, Consider a hiring dataset where 200 men (A 0) and 100 women (A = 1) apply for a job. Out of these, 50 men and 22 women are hired (T 0), while the rest are rejected (T = 1) refer to Table 5a Assume we train model (M) on this dataset, which gives predictions as shown in Table 5bb
According to DPA, the A + T bias amplification is large. This is because for both men and women, Í much more unbalanced than T. For men: 7 ratio = 3: (150/50) Í ratio = 12.3: 1 (185/15) 115). For women: T ratio- = 3.5: 1 1188/117221 /2)), T ratio = 10.1: : (91199999)))) (9). According to BAA,, bias amplification small. This is because the ratio of hired men and women similar in the dataset and in the model's predictions. Hired men and women in the dataset: 25% (50/200) and 22%
respectively. Hired men and women in the model's predictions: 7.5% (15/200) and 9% (9/ (9/100), respectively. Our goal is to ensure men and women have a similar acceptance ratio. Since the acceptance ratio between

---

men and women is similar in the dataset and in the model's predictions, the small bias amplification reported by BA_, is appropriate. In scenarios where we desire equal opportunity (e.g., hiring datasets where men and women should have equal acceptance ratios), using B B.A,, is better.
In another scenario, consider a dataset of men (A 0) and women (A 1), where each person is either indoors (T' = 0) or outdoors (T' = 1). Assume the same tables as before (Tables 5a and (5d Here, our goal is to ensure that we have a balanced count of men and women, indoors and outdoors. We should report high bias amplification as 7 is highly unbalanced with respect T (for both men and women). In this case, the large bias amplification reported by DPA is appropriate. BA,,, and DPA capture different notions of bias, and the right choice of metric depends on the type of bias we seek to address.
5.2 Limitations and Future Work
Predictability-basee metrics like DPA (and LA) rely training attacker models to estimate bias amplification. These attackers can be sensitive to training instabilities - such as getting stuck in local minima. This may lead inconsistent results. To ensure stable results, must run the attacker model multiple times, which increases computational cost. In contrast, co-occurrence-basse metrics like B BA,, Multi are cheaper to compute. We have compared the runtime for all bias amplification metrics in Section L
DPA A cannot measure bias amplification at the level of individual A-T pairs - the bias amplification caused by a specific A (e.g., women) on a specific T (e.g.. cooking), or vice versa. Instead, it measures overall A T and T - AA bias amplification, aggregated across all A and T. Suppose model amplifies bias on the cooking task when women are present, but reduces bias on the gardening task when men are present. DPA would report only the net effect across both A's and both T's, masking these opposing trends.
As part of future work, we aim to disaggregate bias amplification into individual A T pairs. We can use techniques like feature attribution to find out the influence of a specific attribute = A, (e.g.. male), in predicting a specific task T = Tj (e.g., cooking). This attribution value can be used as weak proxy to measure dataset or model bias between A,, Ty. Causal attacker models or interpretable attackers are other potential approaches to disaggregate bias amplification into individual A T pairs. This finer-grained view can support targeted bias intervention strategies.
5.3 Conclusion
In this work, we introduced DPA, one-stop predictability-base metric to measure bias amplifica- tion classification problems. DPA the only metric that can (1) accurately identify positive and negative bias amplification scenarios, offering valuable insights into model selection by highlighting which models preserve, worsen, or reduce data bias, (2) identify the source of bias amplification as the metric is directional (3) measure bias amplification for both unbalanced and balanced datasets. DPA A eliminates the need evaluate models on multiple bias amplification metrics to verify each of the three aspects of bias amplification. DPA is also more robust and easier to interpret than current predictability-baee metrics like LA. While DPA is generally the most reliable metric to measure bias amplification, there are bias scenarios (e.g., hiring datasets where rejections are almost always more than acceptances) where other metries like B BA_, may be more suitable. Before using PAA, A, must have accurate understanding of the type of bias we want to address.
6 Acknowledgments
We acknowledge the Research Computing at Arizona State University for providing HPC re- sources (28] that have contributed to the results reported in this paper. We also thank Mirali Purohit for her helpful suggestions throughout the development of this work.
References
[1] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 5534-5542, 2016.
10

---

[2] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints, arXiv preprint urXiv 2017.
[3] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference on Machine Learning, pages
PMLR, 2021.
[4] Dora Zhao, Jerone Andrews, and Alice Xiang. Men also do laundry: Multi-attribute bias amplification. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 42000- Jul 2023. URL https: https://prroeeedingg mlr. www.pprsssee. /202s//222/// zhao23a 42017. PMLR, 23-29 29
html
[5] Arjun Subramonian, Samuel J. Bell, Levent Sagun, and Elvis Dohmatob, An effective theory of bias amplification, 2024. URL
17263
[6] Ze Wang. Zekun Wu, Jeremy Zhang. Navya Jain, Xin Guan, and Adriano Koshiyama. Bias amplification: Language models as increasingly biased media, 2024. URL https://arxiv org/ vrg/abs/244e 200bbb/211..11334
[7] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5310-5319,
2019.
[8] Kun Lin, Nasim Sonboli, Bamshad Mobasher, and Robin Burke. Crank up the volume: preference bias amplification in collaborative recommendation, 2019. URL https://axxiv org//bs//99n 09362
[9] James R. Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. An intersectional definition of fairness. In 2020 /EEE 36th International Conference on Data Engineering (ICDE), pages 1918-1922 2020. doi: 10.1 109/1/D//444307 202.000000.0000000...
[10] Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to- image generation. arXiv preprint arXiv:2308. 00755, 2023.
[11] Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Quantifying societal bias amplification in image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[12] Eslam AbdeIrahman, Pengzhan Sun, Li Erran Li, and Mohamed Elhoseiny. Imagecaptioner2: image captioner for image captioning bias amplification assessment. Proceedings of the Thirty- Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAT2A4AAATT4/EAAAA AAAI Press, 2024. ISBN 978- 1-57735- 887-9. doi: ww..00/aaai..v8889.... 30000000 URL httph:ttt///ot org 10. 1609/ aaai v38119 30080
[13] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pages 254-264 Auerbach Publications, 2022.
[14] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
(15) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[16] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL https: htttps://anae //tt:////iiv.///aa/iaa http//axxvv.rrgab////////sss 1409 1556
11

---

[17] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision doi: 10. (007///1263-----0811
(IJCV), 115(3)211-252,, 2015.
[18] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017. URL https: https:: neurips. ccpppsccc//// wttpppsscccppppppppeeeep paapperriieessppppp///////11//11 622088221766222ffc444226666776
pdf
[19] Shutong Wu, Sizhe Chen, Cihang Xie, and Xiaolin Huang. One-pixel shortcut: On the learning preference of deep neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL /tttps://openreyy//
[20] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459-479, Springer, 2022.
[21] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07336 2016.
[22] Sergey Zagoruyko and Nikos Komodakis. 1605v:1605.07144 2016.
Wide residual networks.
arXiv preprint
[23] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Mobilenetv2: Inverted residuals and linear bottlnnekks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 4520, 2018.
Liang-Chich Chich Chen.
[24] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[25] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 1668-2677.221 PMLR, 2018.
[26] Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner, Amazing things come from having many good models. arXiv preprint arXiv:2407. 00040604466 2024.
[27] Sebastian Müller, Vanessa Toborek, Katharina Beckh, Matthias Jakobs, Christian Bauckhage, and Pascal Welke. An empirical evaluation of the rashomon effect in explainable machine learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 462-478. Springer, 2023.
[28] Douglas M. Jennewein, Johnathan Lee, Chris Kurtz, William Dizon, Ian Shaeffer, Alan Chap- man, Alejandro Chiquete, Josh Burks, Amber Carlson, Natalie Mason, Arhat Kobawala, Thirug- nanam Jagadeesan, Praful Bhargav Basani, Torey Battelle, Rebecca Belshe, Deb McCaffrey, Marisa Brazil, Chaitanya Inumella, Kirby Kuznia, Jade Buzinski, Dhruvil Deepakbhai Shah, Sean M. Dudley, Gil Speyer, and Jason Yalim. The sol supercomputer at arizona state uni- versity. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC '23, page 296-301, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. 3557573
doi: 111144/5599911359777 URL
https:
[29] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916):1411142, 2012.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims are aligned with the contributions and overall scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitation of our metric (inability to disaggregate biases) is discussed in Section
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: We provided proofs for theorems whenever we introduced a new concept. We also referenced existing proofs wherever necessary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All information to reproduce our results are available in the Experiment Setup and Supplementary Sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide open access to our code repositories, with sufficient instructions to reproduce the main experimental results. We do not introduce new dataset. However, links to the datasets we used can be found in the paper and our code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
sooololiccii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are mentioned in the Supplementary. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported error bars for all bias amplification metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: As our work introduces a metric, information about compute resources does not add much value to work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and followed the Code of Ethics set by NeurIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since we introduce metric, there is no societal impact of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not relevant for work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited all original owners of the code and assets we used by referencing their work in our paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Not relevant for our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not relevant for our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for developing our core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp::///euupss.... http//rrppsscccooofferreeeee//// Cc/Connferences/2225LILI. for what should or should be described.
19

---

A Multi,, explanation
To understand why Multi,, cannot differentiate between positive bias amplification and negative bias amplification (i.e., bias reduction), let us take a look at its formulation.
Multi X, V aaar( VarrA gm)
x=
Σ Σ
Ugm Amml + (1 1gm)
ym
(10)
9
M
DEG MEM
where,
Vgm 1[P(m 1. P(g = Pm 1)]
ann,
P(g 1 m =) P(g 1 1|m 1)
if measuring M + G
(11)
gm
P(m = 1g =) P(m = 1/g 1) if measuring G + M
Following 4. M represents the attribute groups, and G represents the task groups.
From Equation 10
get
X
Σ MEM
3gm
Agml
+
Ugm Aggll
G M
DEG
X
Σ
Agmm
(12)
G M
9E9 MEM
Hence, we see from Equations 11 and 12 that Multi simply measures the average absolute differences for the conditional probabilities. Due to the absolute term, Multi cannot capture negative bias amplification.
B Explaining Directionality
To prove that DPA follows Wang et al.'s definition of directionality B} let us see how DPA measures bias amplification in A T. As per equation [ to calculate dataset bias RV) the attacker function S tries to model the relationship of P(T]A). For Q function of accuracy, we can rewrite the equation as:
V = TT Σ P(A.))(Tmm)A)) AEA
(13)
where Tmi is such that p(Tmil|) (A) P(Ta|A,)Vk
Similarly, we can rewrite equation 2 as:
v M = QUA(A), = Σ P(A))((.))))
(14)
A:EA
Then DPA can be defined as:
DPA Σ ((A))(((fmmAAA)) P(TmalA.))
(15)
A.EA
Similarly, for > A direction we can say:
DPA--A Σ P(T))(((Am - P(Amil (T))
(16)
T.ET

---

Let us compare this to Wang et al. 's definition of directionality B in the following manner:
They defined their metric BA
BA
Σ
Yat + (1- (--aa)--Aaaa)
(17)
A||T
aEA,EET
where,
Mat I[P(A, 1, = P(A, 1)P(T) 1)]
(18)
P(T: 1|Ag = 1) P(T: 1/A, =)
if measuring -
Aat
(19)
P(Á. = 1[T; 1) P(A. = -1 = 1) if measuring T - A
For T + A, BA measures the change in P(A[Z) with respect to P(A[T), i.., change in the conditional probability of vs. A with respect to fixed prior T. Similarly, for A + T. BA measures change in the conditional probability of Tim vs. Tii with respect to a fixed prior A,.
Thus, we see both BA,,, and DPA A are proportional to the similar changes in conditional probabilities with fixed priors. Therefore, DPA follows the same concept of directionality as BA...
C Relative vs. Absolute Change in Predictability
To further show how relative change DPA accounts for dataset bias, we plotted the relationship between LA and model bias (A*) ) in Figure22. and between DPA A and model bias (V.b) in Figure 2b, across three values dataset bias O.T, 1, and (dataset bias denoted with DD for DPA and X for LA), ). We observed that the slope of the LA vs. AM curve remains constant regardless of the dataset bias (AD). In contrast, the DPA vs. model bias ( curve shows steeper slopes for smaller dataset biases (vr and flatter slopes for higher dataset biases. This means that for nearly unbiased datasets, DAA reports high amplification even for small increases in model bias, whereas for highly biased datasets, reports lower amplification for similar increase in bias. This shows that, unlike absolute change, relative change (used in DAA) accounts for dataset bias.
Leakage Amp.
10
A
1.00
0.50
A"
0.00
-0.25
6 0.1
-0.50
6
-0.75
6 =
8 10
A
UMM
(a) LA vs. A
(b) DPA A us. M
Figure 2: The graphs show trends between (a) LA vs model bias (Aa),, and between (b) DPA vs model bias (d ), at different values dataset bias (A for LA and V for DP For the same model bias, DP reported much higher bias amplification values (compared to LA) when the dataset bias small.
D Attacker Robustness
To show how measuring relative change in predictability improves DPA A's robustness to attacker function, we conduct an experiment using a controlled synthetic dataset and real-world dataset like ImSitu II.
D.1 Synthetic Experiment
We define A : :3)32)). We define T and T in the following manner:
T = poly(A + (a1 p).
(20)

---

f = po (az c).
(21)
Here poly(æ, p) represents any tt degree polynomial of x and c :: (00 1). demonstrate positive bias amplification, we want to better predictor of A, compared to Hence, we set 02 < 01-
As the attacker is generally a simple polynomial function, we used simple Fully Connected Network as the attacker. We parameterized the depths (d) and width (w) of the attacker functioo. This parameterization helped us create a large variety of attacker functions (each with different depth and width). The parameters of the experiment are shown in Table For each attacker, we used a combination of TanH and ReLU activations.
We created two versions of DPA- (1) DPA with absolute change in bias (or predictability), to mimic the behavior of current metrics like LA, (2) DPA A with relative change in bias, which is our formulation. For both versions, used the inverse of RMSE loss the quality function (Q).
10
Bias Amplification
Non-Normalized
Normalized
4
2
0
20000 40000 60000 80000 100000 Number of parameters
Figure 3: For different attacker functions, we show the bias amplification reported by DPA with absolute change (blue line) and DPA with relative change (red line) on the synthetic dataset.
For different attacker functions, we compared the values reported DPA A with absolute change (blue line) and by DPA A with relative change (red line) in Figure 3 DPA with absolute change shows unstable bias amplification scores with high variance across attackers. DPA with relative change shows stable bias amplification score with minimal variance across attackers. This shows that DPA A with relative change is more robust to attacker function hyperparameters.
Table 6: Experiment Parameters
Parameter
p
Or
Q2
d
Value
2
[20, 500]
[2, 6]
D.2 ImSitu Experiment
We show that measuring relative change in predictability (as done in DPA) is robust to attacker hyperparameters on real-world dataset like ImSitu. We used the same setup the ImSitu experiment in section 4.3 We trained ViT b 16 model on the unbalanced ImSitu sub-dataset. We measured bias amplification using attackers with different hyperparameters.
We used a Fully Connected Network as our attacker function. We parameterized the width w (in the range of [5, 40] neurons) of the attacker function. This parameterization helped create large variety of attacker functions (each with a different width). For each attacker, we used ReLU activations. We used the Adam Optimizer with learning rate of 5 x 1
3

---

We created two versions of DPA - (1) DPA with absolute change in bias (or predictability), to mimic the behavior of current metrics like LA, (2) DPA with relative change bias, which is our formulation. For both versions, we used the inverse of categorical cross- cross-entropy loss as the quality function (Q).
For different attacker functions, we compared the values reported DPA with absolute change (blue line) and by DPA with relative change (red line) Figure DPA with absolute change shows unstable bias amplification scores with high variance across attackers. DPA with relative change shows stable bias amplification score with minimal variance across attackers. This shows that even in real-world datasets like ImSitu, DPA with relative change is more robust to attacker function hyperparameters.
0
Bias Amplification
-2
44
Non-Normalized
Normalized
6
-8
-10
5
10
15 20
25 Width
30 35 40
For different attacker functions, show the bias amplification reported by DPA with absolute change (blue line) and DPA with relative change (red line) on the ImSitu dataset.
Figure 4:
E COMPAS experiment
E.1 Experiment Setup
COMPAS is a tabular dataset containing information about individuals who have been previously arrested. Each entry is associated with 52 features. We used five features: age, juv_fel_coun juv_misd_count, jttpjjjtr___oottteeeeeeeeee _ount, priors_count.
Prrican-AAmricann) which we used the protected attribute (A). The task (T) was recidivism (i.e., if the person was arrested again for a crime in the next years). Hence, A [Caucasian 0, affrican-American 1} and T (No Recidivism 0, Reciiiiiisn 1).
We limited the dataset 2 races (Caucasian
We created balanced and unbalanced versions of the COMPAS dataset. For the unbalanced dataset, we sampled all available COMPAS instances (attributes, race labels, and recidivism labels) for cach of the four A and I' pairs. For the balanced dataset, sampled equal number of instances across the four and T' pairs. The counts for the and T pairs the unbalanced dataset are shown in the top-left quadrant of Table 7al and for the balanced dataset, in the top-left quadrant of Table 7b
We trained decision tree model on the unbalanced and the balanced COMPAS datasets. Each model predicts person's 's race (A) and recidivism (T') based on the 5 selected features. We measured the bias amplification caused by each model two directions: bias amplification caused by race (A) on recidivism (7), referred to as A + T, and the bias amplification caused by recidivism (T) race (A), referred to as - A. In this experiment we only evaluated directional metrics DPA (ours), BAA,, and M ulti...

---

The problem of mapping a binary variable (A) to another binary variable (T) for the A + T direction and vice-versa is trivial and can be done using contingency matrix as well. Thus, we use two variations of D.A. For DPA (ANN), we used a 3-layer dense neural network (with hidden layer of size and sigmoid activations) as the attacker function. For DPA (MAT), we calculate the probabilities based on a contingency matrix instead of an attacker function. Note that quality equalization is still applied as a preprocessing step.
E.2 Results
While interpreting COMPAS results, note that co-occurreecee-ased metric like BA and predictability-basee metric like DPA capture different notions of bias.
B.A, classifies each A - 'I' pair in the dataset as a majority or minority pair using equation 18 It only measures if the counts of the majority pair increased (positive bias amplification) or decreased (negative bias amplification) in the model predictions or vice versa.
minority A T pair. It measures the change in the task distribution given the attribute (and vice versa). For instance, if A and T are binary, DPA measures if the absolute difference in counts between T 0 and T increased (positive bias amplification) or decreased (negative bias amplification) in the model predictions. Both BA and DPA offer different yet valuable insights into bias amplification.
DPA does not select majority
Table 7: COMPAS Dataset:
Counts of the protected attribute (race) and task (recidivism) in the dataset (represented as A and T) and in the model predictions (represented as A and 7') for the balanced and unbal- anced COMPAS set. Here: A {Caucasian Recidivism 1)}.
1} and T {No Recidivism
À
T 0
1229
1402
1056
1575
T =
874
874
1083
665
874
1773
1115
1532
T
874
874
896
852
1165
1546
T
1145
948
==1
938
1629
1=
603
800
(a) Unbsllanced COMPAS Set
(b) Balanced COMPAS Set
Table 8: COMPAS Results:
The first two columns depict the bias amplification values for the unbalanced COMPAS set (Table 7ap. while the last two columns depict the bias amplification values for the balanced COMPAS (eable Pb). Subscript shows confidence intervals.
Method
Unbalanced
Balanced
T + A
T
T A
A T
BA Multi
0.078g.051
-0.038e.00
0.000g .000
0.000g.000
0.078p.026
0.038o.
0. 0.066g.0o .60g
0.099p.one
DPA (ANN)
0.063p.om5
-0.010g.om
0.061g bos
0.100p.o04
DPA (MAT)
0.0222.om
-0.036p.oo
0. 0.066g
0.098p ond
E.2.1 Unbalanced COMPAS dataset
The bias amplification scores for the unbalanced case are shown in the first two columns of Table g
T + A: For BA when T 0, the count of the majority class A O decreased from 1229 in the dataset to 1056 in the model predictions. Similarly, when T = 1, the count of the majority class A = 1 decreased from 1773 in the dataset to 1532 in the model predictions. Since the count of the majority classes decreased in the model predictions, BA_, reported a negative bias amplification in T 1--A A.
For DPA, when T 0. the difference in counts between A and A 1 increased from 173 (1402- - 1229 = 173) in the dataset to 519 (1575 1056 = 519) in the model predictions. decreased from 899 (1773 - 874 = 899) in the dataset to 417 (1532 - 1115 417) in the model predictions. Since the
However, when T = 1, the difference in counts between A = 0 and A
decrease in bias when = 1 larger than the increase bias when T = 0 (899- 417 > 519 173), might naively assume negative bias amplification in T A.
5

---

This naive assumption overlooks the conflation of model errors and model biases (discussed in Section 3.1 Here, the decision tree model has low accuracy when predicting A (approx. 69%); hence, 31% of instances in are randomly flipped to match the model's accuracy. As a result, the biases in the perturbed are less than A, indicating a positive bias amplification. The positive score reported by DPA A is not incorrect result. It is the low model accuracy that misleadingly suggests negative bias amplification. Multi,, reports positive bias amplification as it cannot capture negative bias amplification scenarios (discussed in Section A
A + T: For BA_,, when A O, the count of the majority class 7 decreased from 1229 in the dataset to 1165 in the model predictions. Similarly, when A = 1, the count of the majority class = 1 decreased from 1773 in the dataset to 1546 in the model predictions. Since the count of the majority classes decreased in the model predictions, BA reported negative bias amplification in 7.
For DPA, when A = 0. the difference in counts between T = and T = 1 decreased from 355 (1229 874 355) the dataset to 227 (1165 938 227) in the model predictions. Similarly, when A = 1. the difference in counts between T = 0 and T = 1 decreased from 371 (1773 1402 371) in the dataset to 83 (1629- 1546 83) in the model predictions. Since the overall count difference decreased in the model predictions, DP reported negative bias amplification in A T.
Multi reported positive bias amplification as it cannot capture negative bias amplification. It only measures the magnitude of bias amplification but not its sign.
E.2.2 Balanced COMPAS Dataset
The bias amplification scores for the balanced case are shown in the last two columns of Table g
T + A: Since BA,, assumes a balanced dataset is unbiased. BA,,, reported zero bias amplification increased from
in T + A. For DP A, when T = 0, the count difference between = 0 and
0 (874- - 874 = 0) in the dataset to 418 (1083 665 418) in the model predictions. Similarly, increased from (874 874 = 0) in the dataset to 44 (896 852- 44) in the model predictions. Since the overall difference increased in the model predictions, D P reported positive bias amplification in T + A.
when == 1, the difference in counts between A and
A + T: Since the dataset is balanced, BA, reported zero bias amplification in A - 4 T. For DPA, when A= O, the difference in counts between T and = 1 increased from (874 874 = 0) in the dataset to 542 (1145- - 603 542) in the model predictions. Similarly, when A = 1, the difference in counts between T and 1 increased from 0 (874 874 in the dataset to 148 (948- - 800 = 148) in the model predictions. Since the overall count difference increased in the model predictions, DPA reported positive bias amplification in + T.
Multi reported positive bias amplification as it only looks at the magnitude of amplification scores.
F
Simulating Biases in a Balanced COCO Dataset
Table 9: Bias Amplification scores for various metrics at different values of in the COCO dataset
Test Accuracy (wlo shoriculs)
Test Accuracy (shortcuts)
I.
BA
Mutti
DPA
BAMALS
L.A
(ours)
55. 35.3%%
71 71.20%
0.05
0. 0.00000.mo o00n
0. 0.2569e.cm co0o
0. 0.08296.onor
0. 0.2586o 8000
0.0000g.om 0000
55.34%
72.38%
0.00000,.000
0.2707a.coma
0.0032o 0.09930.0014
8000
0.0000c.o0od
66%
74.31%
0.15
0.0000,,
0.2879p. co00
0.10988.onor
0. 0.139
0.0000
54.50%
75.01%
0.20
0.0000g.06o0
7o.00o
0810
0.2448g none
0.0000g.0000
In this controlled experiment, we show that even if attributes and tasks are balanced in a dataset, models can pick up biases from unlabeled elements and amplify them at test time. We chose out of the 12 task objects from the COCO balanced dataset used in Section 4.1 A = {Female O, Male 1} and = (Sink: O, Not Sink: 1}. Here, "Not Sink refers to COCO images that do not have sink.
Our balanced dataset has 288 images for each A and T pair.

---

We created biases in our balanced dataset using one-pixel shortcuts (OPS) 19. In OPS, fixed pixel with position (2,, 9i) is assigned the same value in all images of a class. This makes it easier for model to predict class, as it only needs to learn that all images in that class share the same pixel. By applying OPS, we simulated how an unlabeled element (here, the shorteut pixel) could create biases in balanced dataset.
We used two parameters, B1 and 82, where 0 S 11 100 and 31 > B2, control how many shortcuts were added. For "sink" images, we applied OPS to 3 % images with males and 3, % images with females. For "not sink" images, we swapped the parameters, applying OPS to 32% images with males and 81 % images with females.
Since 31 > 82. "sink" images with males had more shortcuts than those with females, while "not sink" images with females had more shortcuts than those with males. Although the dataset is balanced in terms of attributes and tasks, model trained "sink" images and females for "not sink" images at test time. In essence, we introduced an A - T bias simulate unlabeled bias elements in the balanced COCO dataset.
this dataset is more likely to predict males for
Similar to our experiment in Section 4.1 we trained a VGGI6 model on this balanced COCO dataset. We trained the model for 15 epochs with batch size of 32. We used the SGD optimizer, with learning rate of 0.01 and momentum of 0.5. We used the binary cross-entropy loss for training.
Similar to the training dataset, we used 288 instances for each A and T pair in our test set. To verify OPS introduced an A - T bias the trained VGG16 model, we created two versions of our test set. In version 1, we did not introduce any shortcuts. In version 2, we applied shortcuts using parameters 31 and A3 as described above.
If the accuracy of VGG16 in version 2 of the test set (where shortcuts are applied) is greater than version (where no shortcuts exist), we can confirm that the model has learned an A - T bias. This is because if the VGG16 model has learned the shortcuts (or biases) we introduced during training, it will give a higher accuracy on the test set where these shorteuts are available.
We show our results on four different configurations of 31 and 3g values in Table 9 For simplicity, we introduce new term L 31 - 82, where is L is always greater than O. We see that for all values of I., the accuracy of the test set with shorteuts is greater than the test set without shortcuts, which confirms the presence of A T bias.
As L increases, the A + T model bias increases (as the number of shortcuts increases with L). Only DPA reports increasing A + T scores as L increases. All other metrics fail to report increasing bias amplification scores. This shows that only DPA correctly captures the amplification of hidden biases (similar to bias amplification from unlabeled objects) in balanced datasets.
G Simulating Biases in a Balanced CMNIST Dataset
In this controlled experiment, we use the CMNIST dataset to show that even if attributes (color) and tasks (digit) are balanced in dataset, models can pick up biases from unlabeled elements and amplify them at test time.
In the CMNIST dataset, we used images of handwritten digits from the original MNIST 29 dataset and added color to them by replacing white pixels with colored pixels. Each image is assigned a single color. The images are the features X, digit labels are the task T, and color is the protected
attribute A. Each label (i.c., digit from 0 9) is correlated with a particular color (i.e., 0- - red, 1- blue. green and so on). The magnitude of this correlation controlled using a. If = 0.7. 70%
image examples of ( are assigned red, and the remaining 30% images are randomly assigned any of the remaining 9 colors. a allows us to control the bias in the dataset.
Table 10: CMNIST controlled experiment.
0.10
= 0.20
0.30
0.40
DPAA ir
0.063 + 0.006
0.085
0.006
0.098
0.008
0.119 +: 0.006
BAA-T
0.000 1 0.000
0.000 0.000
0.000 1: 0.000
0.000 1: 0.000
LA
0.043 1: 0.009
0.060
0.014
0.075 4 0.015
0.071 4: 0.017
7

---

We trained a simple CNN model (with depth 2) on this dataset. We used the trained model to get predictions, and post-processed the predictions using parameter predicted labels that are overwritten based on the color in the input image. If /3 = 0.1, we randomly select 10% of the predicted labels. These labels are replaced based on the color in the image. This allows us to increase the A + T bias in the model predictions. For a fixed value of a, 3 allows us to control the model bias. We used accuracy as our Q function.
defines the percentage of
We started with a balanced CMNIST dataset (a 0.1). We then varied the 3 parameter from 0.1 to 0.4 to gradually increase the model bias. We showed the bias amplification captured by different metrics in [O]
As
in Table 10, BAA, shows constant value for varying values of B. Meanwhile, LA is unable to show a clear trend due to the large confidence intervals. Only DPA A shows a monotonic increase in reported bias amplification 3 value increases. This indicates that only D) is able to accurately capture the bias amplification our controlled experiment.
H COCO Masking experiment on VGG-16
Table 11: - A bias amplification as the person progressively masked in the unbalanced and balanced COCO sub-datasets. Attribution scores indicate the I + A model bias in the VGG16 model. For the unbalanced sub-dataset, DPA and BAA,, correctly capture the increasing T + A model bias. For the balances sub-datasct, only A correctly captures the increasing T + A model bias. Subscript shows confidence intervals.
Dutaset Split
Metric
Original
Partial Masked
Segment Masked
Bounding Masked
Imuge
Attribation Map
Autribttion
0. 0.6202p.
0.6777a
0.7321
0.7973,
DPA (ours)
0. 0.0034g. oooo
0.0195g.cm 0007
0.021 4p 0010
0.02442g.oo 0004
Unbalanced
BA.,
0.0029, 0002
0.0072g coos
0.0108g 0007
0.0140 0007
Malti
0. 0.0057 ooos
0.0091 o cocos
0.0109p oons
0.02 19, 0011
LA
0. 0.0005a.o oo0o
0.0068p .0099
0.0032p. 0084
0.001 10.0010
Attribution
0.6292 0027
0.6992a.com .024
0.7367 0019
0.8065g.oz 0158
DP (ours)
0.0019a.com
0.0024g.com
0.0068p
0.0100g.000 00111
Balanced
BA
0. 1.0000g.com cono
0. 0.0000g .o0o
0. 0.00000 0o0o
0. 0.0000g, 0000
alltii..
0.0035 0002
0.0056g 0004
0. 0.0060g c003
0. 0.0099g poro
LA
0.0012g, 00002
0.0055a 0016
0.0026g cons
0. 0.0065g 0011
We recreated our COCO masking experiment from section are shown Table II
with with VGG16 [6 model. The results
For the unbalanced COCO sub-dataset (row 2 of Table IT]. DPA A and BA., report increasing T + A scores across versions, correctly capturing the growing model bias (indicated by the growing model attribution scores). While Multi,, shows the same trend as these metrics, it is not a reliable metric as it cannot capture negative bias amplification (discussed in Section
For the balanced sub-dataset, we observe growing model attribution scores indicating a rising T + A model bias. DPA correctly captures this trend by reporting increasing T - A scores. While Multi shows the same trend as DPA, as discussed earlier, Multi is not a reliable metric. LA fails to capture the growing model bias as it is not directional. BAA,, assumes that a balanced dataset unbiased. Hence, fails to capture the growing model bias and constantly reports zero bias amplification.
We also show additional examples for attribution maps for the VGG16 model on images from our balanced COCO sub-dataset in Figure 5 We observe in these images that we increase masking
8

---

the person, i.e., attribution examples from left to right, the attribution shifts from the person to either the background or unlabeled task objects. This shows that in balanced datasets, as masking increases, the model relies more on unlabeled background objects to determine the gender of the person.
Original Image
No Masking
Partial Masking
Segment Masking
Box Masking
Figure 5: VGG-16 model attributions for images from the balanced COCO sub-dataset. For all cases, as the person in the image is progressively masked, the VGG-16 often relies on unlabeled background objects to predict gender.
I ImSitu Experiment: T - A results
In Table showed the sensitivity and bias amplification scores in the T' + A direction for the unbalanced sub-dataset. Table [3] we showed corresponding results for the balanced sub-dataset. For both tables, ranked all models (from 1 to 9) by their Seny-+ scores, from least biased to most biased. We checked which bias amplification metrics produced rankings closest to this model which metrics best captured changes in model bias.
bias ranking. This allowed us
In the unbalanced case (Table DPA's ranking is closest to the model bias ranking - differs by only three positions, All other metries show significant mismatches.

---

In the balanced case (Table 13,, DPA's ranking is closest to the model bias ranking - it again differs by only three positions. Other metrics, like LA and Multi differ significantly. Metrics like BA and BAMALS report zero bias amplification because they assume that a balanced dataset has no bias.
Table 12: T -> A results for the ImSitu unhalanced sub-dataset. Models are shown in increasing order of model bias, based on their SeRz- scores, For each metric, the number in parentheses shows the rank it assigned to that model where rank 1 means the lowest (most negative) bias amplification and rank 9 means the highest (most positive). Red numbers highlight where the metric's rank differs from model bias rank assigned by Senr-A- D is the most reliable metric here, as model rankings closely match Senz +A rankings. All values are scaled by 100.
Models
Seny
DP APAT-1
LA
BAr
Multir
BA BAMALS
SqueezeNet
21
0.000 [1)
-0.013 (3)
0.009 (1)
16..26 (1)
18.90 (9)
25.59 (9)
VIT h 32
15
0.033 (2)
-0.025 (1)
0.209 (6)
-6.891 (6)
7.252 (5)
0.248 (8)
Wide ResNet50
0.036
-0.020 (2)
0.253
-4.494
5.289 (3)
0.061 (4)
Wide ResNet10]
[2]
0.098 (4)
-0.012 (4)
0.281 (9)
-3.612 (8)
4.790 (2)
0.109 (3)
(4)
-8.143
ViT_b_I6 [1
0.190 (6)
0.001 (6)
0.195 (5)
-7.214 (G)
7.719 (4)
0.229 (7)
MobileNet 23
0.566 (7)
0,009 (7)
0.087 (3)
-9.085 (3)
9.761 (7)
0.193 (5)
Swin Tiny 24
2,800 (8)
0.012 (8)
0.064 (2)
-9.782 (2)
10.86 (6)
-0.082 (1)
VGG16 [n
2.938 (9)
0.013 (9)
0.210 (7)
1.620 (9)
3.979 (1)
0.045 (2)
Table 13: T A results for the ImSitu balanced sub-dataset. This table follows the same setup as Table
Red numbers highlight where metric's rank differs from model bias rank assigned by SenT-A DPA the most reliable metric here, as its model rankings exactly match the model bias ranking given by S All values are scaled by 100.
Models
SenT
DPAr
LA
BAT
Multir
BAMALS
SqueezeNet 21
0.000 (1)
0.019 (3)
0.003 (8)
0.000 (1)
0.208 (2)
0.000 (1)
Wide ResNer50
22]
0.107 (2)
0.016 (1)
0.002 (7)
0.000 (1)
3.580 (8)
0.000 (1)
Wide RosNet101
0.122
0.018
0.000 (3)
1.815
0,000
MaxVil (20
0.261 (4)
0.021 (4)
-0.002 (1)
0.000 (1)
0.022 (1)
0,000 (1)
23
(5)
(5)
(9)
(1)
(1)
VIT_b_16 15
0.310 (6)
0.021 (6)
0.000 (2)
0.000 (1)
2.174 (5)
0,000 (1)
Swin Tiny 24
0.345 (7)
0.022 (7)
0.001 (5)
0.000 (1)
2.628 (7)
0.000 (1)
VGG16 [6]
0.497 (9)
0.047 (8)
0.000 (3)
0.000 (1)
5.638
0.000
WTT_b_32 [3
0.626 (9)
0.174 (6)
0.001 (5)
0.000 (1)
2.584 (0)
0.000 (1)
Additional Experiment Details
Given below are additional details about the hyperparameters used for DPA and LA metrics in different experiments mentioned in the paper. Since BAMALS. BA and Multi do not train an attacker function, no such hyperparameters are required for them. For all experiments, we used (1/ cross-entropy) as our quality function (Q).
Table 14: COCO Masking additional parameters
Parameter
Optimizer
Attacker Depth Learning Rate
Num. epochs
Batch size
DPA Adam
0.001
100
64
LA
Adam
2
0.001
100
64
Table 1.5: ImSitu additional parameters
Parameter
Optimizer
Attacker Depth
Leamingg Rate
Num. epochs
Batch size
DPA
Adam
2
0.001
100
128
LA
Adam
0.001
100
128
Table 16: COMPAS additional parameters
Parameter
Optimizer
Attacker Depth
Learning Rate
Num. epochs
Batch size
DPA
Adam
0.005
50
512
LA
Adam
2
0.005
50
512
10

---

K Behavior of different directional metrics
BA
DPA
Multi
0.25
06
Om
0.0
0
0.0
-02
-01
-04
-02
0.43
-.22
0.25
9.25
0.25
0.25
dd
ad
ad
(a) BA_,
(b) DPA
(c) ulti...
Figure 6: Bias amplification heatmap for different configurations of the dataset (X-axis) and model predictions (Y-axis). Od creates different configurations of the dataset, while Omm creates different configurations the model predictions. BA and DPA A show similar behavior (except when the dataset is balanced). However, M Multi always reports positive bias amplification.
To analyze the behavior of directional metrics, we simulated dataset with protected attribute A (A = 0, 1) and task (T = 0, 1). Initially, each A and 7 pair had an equal probability of 0.25. To introduce bias in the dataset,
modified the probabilities by adding term ag the group (A == 0J and subtracting Ou from (A = 7 }]. Here, as ranges from -0.25 to 0.25 in steps of 0.005. The dataset is balanced only when O but becomes increasingly unbalanced ax deviates from 0. In the same manner, we use to introduce bias in the model predictions.
With ag and am ranging from -0.25 to 0.25, create 100 different versions of the dataset and model predictions, respectively. For each metric, we plot 100 x 100 heatmap of the reported bias amplification scores. Each pixel in the heatmap represents the bias amplification score for a specific (dataset, model) pair.
Figure shows the heatmaps for all metrics. Figures for BA,, and DPA, respectively, with similar patterns. However, BA., (Figure 6a) shows distinct vertical green line in the center, indicating that when the dataset is balanced (oaa = 0 on the X-axis), bias amplification remains at 0, regardless of changes in model's bias (varying a.m. values on the Y-axis). In contrast, DPA (Figure (6b) accurately detects non-zero bias amplification whenever there a shift in bias in either the dataset or model predictions, making a more reliable metric. M Multi,, (as shown in Figure (c) reports positive bias amplification in all scenarios, making it unreliable.
the bias amplification heatmaps
L Runtime Comparison
In Table 17 WC compared the time taken by different metrics to compute bias amplification on the COCO dataset. This computation was performed over Intel Core i7 165H processor without any GPU acceleration. Note that bias (or bias amplification) analysis is usually conducted offline (and not in real time). While DPA several magnitudes more time-consuming than the other methods, an execution time of two minutes is acceptable for most offline applications.
Table 17: Runtime comparison for different bias amplification metries on the COCO dataset.
BAMALS
BA
Mutli
LA
DPA
Time (ms)
57.57
150.16
453.11
1.90
105
1.20 x
105


Reviews:
[{"review_id": "QnqTAVDtOc", "reviewer": "Reviewer_bRtk", "summary": "The paper proposes DAP metric to quantify the amount by which a predictive model amplifies pre-existing correlations between sensitive attributes $A$ and labels $T$. It is demonstrated experimentally that models can amplify such correlations even when trained on curated datasets where $A$ is statistically independent from $Y$. The authors argue that this is because the model can leverage spurious correlations to infer $A$ from the image. Since alternatives to DAP do not highlight such shortcuts, DAP is a worthy addition to the literature.", "questions": "## Clarifications Regarding the DAP\nFrom what I understand, the bias measures $\\Psi_A^D$ consists of reporting the accuracy of an attacker $f_A^T$ trained to predict $T$ from $A$. However, in Appendix B equation (13), it is stated that $\\Psi_A^D \\propto P(T|A)$. I think this equation is incorrect, which undermines the connection the authors were trying to make with the \n$BA_{\\rightarrow}$ method . Indeed, assuming $A$ and $T$ are binary, the accuracy of the optimal attacker is $f_A^T$ is $\\Phi_A^D = P(A=0) \\text{max}\\\\{P(T=1|A=0), P(T=0|A=0)\\\\} + P(A=1) \\text{max}\\\\{ P(T=1|A=1), P(T=0|A=1)\\\\}$. This is not proportional to a conditional probability. Accordingly, Section B should be clarified or removed.\n\n## Clarifications regarding Section 5.1\nFirst of all, it is stated that DAP could be used for model selection. However, I think this is only true for the $DAP_{A\\rightarrow T}$ which only requires that the model outputs labels $\\hat{T}$. The metric $DAP_{T \\rightarrow A}$ requires that the model predicts the sentitive attribute $\\hat{A}$, and I cannot think of any practical application where we would want to use such predictive model. I would argue that $DAP_{T \\rightarrow A}$ is more useful for identifying spurious correlations in datasets and potentially curating them.\n\nAlso, I am confused by the toy example provided in lines 333-342. In this example, we have $A=0$ for men, $A=1$ for women, $T=0$ for hired and $T=1$ for rejected. It is claimed that this dataset is biased according to DAP. But isn't DAP an amplification metric that requires a predictive model $M$ as well? What is you model in this toy example. Also DAP is directional, so which direction is considered ($A\\rightarrow T$ or $T\\rightarrow A$) in this toy example? Since this toy example is brought in the conclusion, it would be important to clarify why DAP claims it is biased.\n\n## Missing Experimental Details\nThere are not enough details regarding the model training on Coco. Indeed, a single image can contain multiple objects of different/same classes (Figure 1 shows an image with three different objects for instance). Is the model required to localize these objects? Is it required to differentiate two different objects of the same class (e.g. two different pans)? Or simply predict there is *at least* one pan in the image? \nI suspect that you treat Coco as a *multi-label classification* problem where image $I$ is labeled as $y=(y^{(1)}, y^{(2)}, \\\\ldots, y^{|T|})$ with $y^{(t)}=1$ if at least one object of class $t$ is present and $y^{(t)}=0$ otherwise. This would be would be important to clarify in the manuscript along with descriptions of how the architectures pretrained on Imagenet multiclass classification were adapted for Coco.\n\nSection I (line 1012) of the appendix does not contain enough details on the Attacker network that is used to predict $A$ from $T$. For Coco, the ground-truth label $T$ fed to the attacker is actually a **set** (see Figure 1 bottom) and not a single numerical value. MLPs are not suited to take sets as inputs so I suspect that the labels were One-Hot-Encoded to $\\\\{0, 1\\\\}^{|T|}$ before being fed to the attacker. It would be important to clarify if this is the case or not. If not, how were MLPs able to process sets of objects as their inputs?\n\n## Fitting a MLP on COMPAS\nIn COMPAS, the labels and sensitive attributes are binary. Thus, I do not see any benefits to fitting a MLP to predict $A$ from $T$ (or $T$ from $A$). In this case, computing conditional probabilities from the 2x2 contingency table should be sufficient. Can the authors argue why a MLP was used as an attacker on COMPAS?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The paper is well written and easy to follow.\n- The experiments consisting of progressively masking individuals and trying to infer their gender $A$ based on the ground-truth $T$ (or predicted $\\hat{T}$) objects is a sound way to quantify spurious correlations in the dataset/model.\n- Relevant classifications models are studied (ViT newer and VGG16 older).\n- The DAP metric is a significant contribution to the community because it highlights a scenario that other metrics fail to identify. Indeed, DAP can detect when a model infers gender based on spurious correlations although the dataset is curated to render $A$ and $T$ statistically independent.\n- The work is original and cites revelant prior work.\n\n# Weaknesses\n- The clarity of the technical sections could be improved (more on this below in the **Questions** section)\n- The experimental details are not sufficient. Notably there are missing details on the trained classifiers $M$ and the Attackers (more on this below in the **Questions** section).\n\n\n## Minor Comments\nThe current work investigates statistical measures of fairness so there is no notion of causality involved. However, at line 112, the word *causality* is used as a synonym for directionality. This is not the correct terminology. $A\\rightarrow T$ essentially means $P(T|A)$ and\n$T\\rightarrow A$ refers to $P(A|T)$, but using the wording *causality* along side arrow-based notation $\\rightarrow$ suggests that you are doing causal inference. I would avoid writting *causality*.", "was_revised": true, "final_justification": "Increased my score due to the rebuttal.", "rebuttal": "Thank you for your insightful and detailed review.\n\nWe have answered to your concerns in the responses below:\n\n$\\newline$\n\n**Minor Comments**:\n\nThank you for pointing this out. We will be removing the term “causality” from our manuscript to avoid any confusion.\n\n$\\newline$\n\n**DPA clarification**:\n\nYou are indeed correct, we had simplified the expression to $P(T|A)$. We will modify the manuscript to reflect the expression you provided. Note that this new expression still ensures that the metric is directional. We prove this here:\n\nAs a generalization of your expression, $\\Psi_{A}^{D}$ can also be written as: $\\Psi_{A}^{D} = \\sum_{A_{i} \\in A} P(A_i) P(T_{mi}|A_{i})$. \n\nWhere, $T_{mi}$ is such that $P(T_{mi}|A{i}) \\geq P(T_{k}|A{i}) \\forall k$. \n\n\nSimilarly, $\\Psi_{A}^{M} = \\sum_{A_{i} \\in A} P(A_i) P(\\hat{T_{mi}}|A_{i})$.\n\nHence, your final expression can be written as: $\\sum_{A_{i} \\in A} P(A_i) P(\\hat{T_{mi}}|A_{i}) -\\sum_{A_{i} \\in A} P(A_i) P(T_{mi}|A_{i})$. We see that bias amplification is still measured with respect to a fixed a prior variable (in this case $A_{i}$), which is how a directional metric should work.\n\n$\\newline$\n\n**Section 5.1**:\n\nDiscussion Section Part 1: You have raised very relevant points. We will add the following clarifications to paragraph 1 of Section 5.1: $DPA_{ A \\rightarrow T}$ can be used for model selection, and $DPA_{ T \\rightarrow A}$ can be used for identifying spurious correlations in datasets.\n\nDiscussion Section Part 2: While addressing your comments for the second part of Section 5.1, we found an error in how we wrote this portion. We mentioned in line 340 — “Compared to $BA_{\\rightarrow}$, DPA may under-report bias amplification...” This is incorrect as $BA_{\\rightarrow}$ would give a “0” bias amplification if the dataset bias is 0. To convey our point correctly, we updated the second part of Section 5.1. Please read updates here:\n\n**Table 1**\n\n|       | $T=0$       | $T=1$       |\n|-------|-------------|-------------|\n| $A=0$ | $150$      | $50$      |\n| $A=1$ | $78$      | $22$      |\n\n**Table 2**\n\n|       | $\\hat{T}=0$       | $\\hat{T}=1$       |\n|-------|-------------|-------------|\n| $A=0$ | $185$      | $15$      |\n| $A=1$ | $91$      | $9$      |\n\n\n\"Based on our experiments, we found $DPA$ to be the most reliable metric to measure bias amplification. However, there are cases where other directional metrics like $BA_{\\rightarrow}$ could be more appropriate. Let us take two scenarios where we measure $A\\rightarrow T$ bias amplification in $BA_{\\rightarrow}$ and $DPA$.\n\nConsider a hiring dataset where $200$ men ($A = 0$) and $100$ women ($A = 1$) apply for a job. Out of these, $50$ men and $22$ women are hired ($T = 0$), while the rest are rejected ($T = 1$) — refer to Table 1. Now, assume we train a model (M) on this dataset, which gives predictions as shown in Table 2.\n\nAccording to DPA, the $A\\rightarrow T$ bias amplification is large. This is because the counts of $T$ and $\\hat{T}$ have grown farther apart for both men {initial difference = 100 (150 - 50), new difference = 170 (185 - 15)} and women {initial difference = 56 (78 - 22), new difference = 82 (91 - 9)}. According to $BA_{\\rightarrow}$, $A\\rightarrow T$ bias amplification is small. This is because the ratio of hired men and hired women is similar in $T$ (25% vs 22%) and $\\hat{T}$ (7.5% vs 9%). Here, our goal is to ensure that men and women have a similar acceptance ratio. Since the acceptance ratio between men and women is similar in the dataset and in the model’s predictions, the small bias amplification reported by $BA_{A\\rightarrow T}$ is appropriate. In scenarios where we desire equal opportunity (e.g., hiring datasets where both men and women should have equal acceptance ratios), using $BA_{\\rightarrow}$ is better as it measures bias amplification by comparing task ratios between men and women.\n\nIn another scenario, consider a dataset of men ($A = 0$) and women ($A = 1$), where each person is either indoors ($T = 0$) or outdoors ($T = 1$). Let us consider the same tables for the dataset (Table 1) and the model’s predictions (Table 2). Here, our goal is to ensure that we have a similar count of men and women, indoors and outdoors. Keeping our goal in mind, we should report a high $A\\rightarrow T$ bias amplification as the count of $\\hat{T}$ has changed significantly with respect to $T$ (for both men and women). Here, the large bias amplification reported by $DPA_{A \\rightarrow T}$ is appropriate. In scenarios where we desire equal counts across $A–T$ pairs, comparing task ratios (as done by $BA_{\\rightarrow}$) is not meaningful. Instead, the count-based comparison used by DPA better reflects bias amplification.  $BA_{\\rightarrow}$ and $DPA$ capture different notions of bias, and the right choice of metric depends on the type of bias we seek to address.\"\n\n$\\newline$\n\n**Experiment Details**:\n\nYou are right. We used annotations from the MS-COCO multi-label benchmark. The label $y = (y_1, y_2, …. y_{|T|})$ is a binary variable. $y_{i} = 1$ indicates that at least one instance of the $i^{th}$ object is present in the image. To adapt the ImageNet pretrained models to COCO, we removed the last MLP layer from these models and replaced it with an MLP layer of size |T|. The models were then fine-tuned on the COCO dataset. We will make sure to add these points along with more details on the model training. For the attacker models, the labels are one-hot encoded. This was not shown in the figure for the sake of simplicity. We will clarify this in the text to avoid any confusion.\n\n$\\newline$\n\n**MLP on COMPAS**:\n\nYes, the 2x2 contingency table should be sufficient in the COMPAS example. For the sake of uniformity, we chose to use the MLP as the attacker function. We can compute DPA using the 2x2 contingency matrix, if you suggest that it adds value to the study."}, {"review_id": "CVfd4UjGFF", "reviewer": "Reviewer_n6n6", "summary": "### Problem:\n_a.k.a. What problem/question does this work purport to solve/answer?_\n\nA particular aspect of bias within ML/AI that is particularly concerning is bias amplification -- the phenomena in which a ML model does not (as naive theory might suggest) automatically learn to ameliorate data biases but actually amplify them in downstream use. In order to effectively reduce bias amplification, we need to be able to measure it. This paper proposes a new method to do so that solves several key challenges: \n  1. Directionality\n  2. Working across balanced and unbalanced datasets\n  3. Works with both positive and negative bias\n\n### Impact:\n_a.k.a. Is the problem/question impactful?_\n\nBias amplification is a critical property that intersects both with the high-impact nature of fairness challenges and the possibility of meaningful technical solution through new innovations, so I feel this is a key direction for impact.\n\n### Proposed Solution:\n_a.k.a. How does this work claim it will solve/answer the problem/question?_\n\nThey define their metric to be the relative change in predictability of a protected attribute from a real vs. predicted label (or the reverse).\n\n### Novelty/Why hasn't this been done before:\n_a.k.a. How does this work differ from the existing literature?_\n\nPrior methods have examined an absolute change in predictability, rather than a relative change. The authors claim this is a significant problem, because it conflates a small relative increase in bias in settings with high dataset bias and a large increase in relative bias for settings with low dataset bias.  As far as I can tell, this difference is the largest source of novelty in the work.\n\n### Evidence that this work solves the problem:\n_a.k.a. How does this work prove that their solution/answer is valid?_\n\nThe authors employ a very clever experiment (inspired in part by some prior work, I believe) in which they artificially introduce a setting in which the model must amplify bias in order to make effective predictions at all, then sees to what extent their metric catches that bias amplification versus competing approaches.", "questions": "1. I don't understand why you flip some of the predicted labels T based on M's accuracy. Can you further explain this? What is the error case you're concerned with here?\n\n2. How would a technique like multicalibration impact the success of your approach here? I think that warrants testing.", "limitations": "No comment", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n### Key Strengths\n_a.k.a. reasons I would advocate this paper be accepted._\n\n  1. This problem is very high impact and new metrics to detect bias amplification are important.\n  2. I like your controlled masking experiments as a vehicle to artificially inflate the expected bias amplification.\n\n### Minor Strengths\n_a.k.a. things I like but would not be sufficient to motivate acceptance in isolation._\n\n  1. I like the use of proxy signals (like attribution maps and CAVs) to reinforce what the expected behavior is (though I would not trust them absolutely). \n\n## Weaknesses\n### Key Weaknesses\n_a.k.a. reasons I would advocate this paper be rejected._\n\n#### Flaw in your results\nI do not believe your results can be correct as presented -- in particular the relative performance of LA vs. DPA. Let's let $Q$ be $\\lambda$ -- both are (as you describe them) measures of how well the attribute can be predicted from the task in the dataset vs. the predicted task by the model, unless I've misunderstood. $Q_D$ in your experiment should be fixed across all masking levels -- in the data, the task predicts the attribute in a manner that is unaffected by masking. $Q_M$ changes as you adjust the masking strategy and rate -- and in particular _should_ if your experiment has internal validity be increasing as we move to the right (in Table 2). This means that LA -- defined as $Q_M$ - $Q_D$ -- _should also be increasing monotonically as we move to the right_. But, instead, you have the last value in the Bounding-Box Masked column being lower than any other. Why is this? What has gone wrong in the use of the LA metric or my understanding of it to deviate from what should be the result here? \n\nWithout understanding or explaining this, this result undercuts your entire argument, because it calls the internal validity of your experiment or the validity of the baselines into question, so this is essential to correct.\n\n#### Insufficient Novelty\nYour method is functionally just a prior method (LA) divided by the sum of the two quality terms. The fact that DPA can disentangle the two directions is merely because you repeat the analysis of LA but with a swapped internal definition of A and T. This is very much stretching the novelty bar for a venue like NeurIPS. This weakness may be overcome by your results, which are very promising, but it is a major challenge nonetheless. Fundamentally, this weakness compounds some of the skepticism about results, as -- even if your reported improvements over LA are valid -- it is such a tiny change to improve LA in a manner that re-captures your proposed approach.\n\n### Minor Weaknesses\n_a.k.a. things I dislike but would not be sufficient to motivate rejection in isolation._\n\n  1. Your evaluation metrics need to be summarized in some way so that we can read off whether DPA is superior to other metrics in one pass, rather than needing to read all the numbers in Table 2. E.g., report the correlation between the % masked and the metric, or something like that. \n  2. I'd love for your metric to be more theoretically principled, rather than just a subjective justification on a change from an absolute difference to a relative one. E.g., is there a probabilistic perspective you can leverage to suggest a relative metric is superior under a given biasing method?\n  3. You should generate some purely synthetic experiments that unambiguously show the improvement of your method, so you don't need to rely on heuristics or proxy variables to establish what the \"right\" bias direction should be.\n\n## Final Recommendation:\n_My overall recommendation for this paper, and its justification._\n\nReject\n\nI recommend reject because, without further clarification of the results, I cannot trust the conclusions presented (see key weakness 1). If this is clarified and my misunderstandings addressed, I could see my score rising as high as a Marginal Accept even with the concern regarding novelty.\n\n## What do I want to see changed:\n### Changes to Increase my Score:\n_What things, were they provided to my satisfaction, would motivate me improve my score?_\n\nClarify (and convince me) why the LA results do not undermine your argument and experimental coherence and more generally re-present your results or present additional results that establish the superiority of DPA in a more convincing, internally consistent manner. In addition, some further justification (perhaps merely promoted from the appendix) that demonstrates more nuanced aspects of superiority vs. LA independent from the specific choices of quality functions and attacker functions used in the original LA formulation (unless those are somehow thought to be essential to the approach in a manner I'm missing).\n\n### Changes I'd recommend for Future Submissions:\n_What changes do I recommend for future submissions, but would not necessarily change my score now?_\n\nSignificantly rework results presentation to better highlight in a single pass why DPA is better.", "was_revised": true, "final_justification": "See my rebuttal response and original review for justification.", "rebuttal": "We thank you for your inquisitive and candid review. Please find our responses below.\n\n$\\newline$\n\n**Key Weaknesses** - Flaw in your results:\n\nThe result from our masking experiment is correct. The non-monotonic behavior shown in LA is a result of its conflation of biases from the $A\\rightarrow T$ and $T\\rightarrow A$ directions.\n\nIn the aforementioned experiment, we progressively mask the person to monotonically decrease the $A\\rightarrow T$ bias. But, this masking also leads to a monotonic increase in the $T\\rightarrow A$ biases because, as the model has less information on the gender ($A$), it relies more on task ($T$) to make its predictions. \n\n\nAs $DPA_{T\\rightarrow A}$ measures biases only in the $T\\rightarrow A$ direction, it shows a monotonic increasing behavior. On the other hand, LA confounds the biases from both directions. I.e., An increase in $T\\rightarrow A$ direction may be offset by decreases in $A\\rightarrow T$ direction, resulting in non-monotonic behavior. Please read the next section -  [**Key Weaknesses** - Insufficient novelty over LA] to understand why LA conflates biases. \n\nThus, the experiment shows us the limitations of LA and how it can give misleading results. Meanwhile, DPA, being a directional metric, can give users a clearer understanding of the bias amplification.\n \n$\\newline$\n\n**Key Weaknesses** - Insufficient novelty over LA:\n\nA major improvement over LA (our novelty) is this — LA measures bias amplification (or change in bias) w.r.t varying priors. DPA (our metric) measures bias amplification with respect to a fixed prior.\n\nWhat does LA do? To measure dataset bias, LA uses an attacker model to predict the ground-truth attribute $A$ using the ground-truth task $T$ as input. In probabilistic terms, this is proportional to $P(A|T)$. To measure model bias, LA uses an attacker model to predict the ground-truth attribute $A$ using the task predictions $\\hat{T}$ as input. This is proportional to $P(A/\\hat{T})$. Bias amplification in LA (model bias - dataset bias) is proportional to $P(A|\\hat{T}) - P(A|T)$. Here, the prior variable changes from $\\hat{T}$ to $T$. The change in bias (or bias amplification) is not grounded w.r.t a fixed variable.\n\nWhat does $DPA_{T\\rightarrow A}$ do? To measure dataset bias, DPA is the same as LA: dataset bias in DPA is proportional to $P(A|T)$. To measure model bias, DPA uses an attacker model to predict the ground-truth predictions $\\hat{A}$ using the ground-truth task $T$ as input. In probabilistic terms, this is proportional to $P(\\hat{A}|T)$. Bias amplification in DPA (model bias - dataset bias) is proportional to $P(\\hat{A}|T) - P(A|T)$. In DPA, the change in bias (or bias amplification) is grounded with respect to a fixed variable (in this case, $T$). \n\n“Fundamentally, a change in bias should be measured with respect to a fixed quantity” — this idea was conveyed by Wang et al. [1] in their paper, and we use the same idea for our work. This is our main improvement over LA (LA does not follow this idea --- thus, it conflates biases in $A \\rightarrow T$ and  $T \\rightarrow A$). We attempted to explain this aspect of our novelty in the Appendix, Section B. If you feel this aspect of our novelty should be highlighted better in the main paper, we would be happy to do that.\n\nNote: $P(\\hat{A}|T) - P(A|T)$ is a simplified expression for DPA. The precise expression for DPA is $\\sum P(T_{i})P(\\hat{A_{mi}}|T_{i}) - \\sum P(T_{i}) P(A_{mi}|T_{i})$. This expression also ensures a fixed prior variable. For more details, please read the section: **DPA clarification**  of our response to Reviewer bRtk)\n\n$\\newline$\n\n**Minor Weakness 1**\n\nThank you for the suggestion. We will add correlation-based scores in our tables to summarize the results in a single row. This will allow readers to quickly parse and compare different metrics.\n\n**Minor Weakness 2**\n\nPresently, we do not have a \"theoretical\" justification for why a relative metric is superior. We have shown the practical advantages of using relative metrics over absolute ones, like robustness to different attacker models. \n\n**Minor Weakness 3**:\n\nWe have a controlled experiment on the COCO dataset as described in Appendix F. Herein, we use shortcuts created using the One Pixel Shortcut[2] technique to control the hidden bias. In addition to this experiment, we have added another experiment on the CMNIST dataset (see Controlled experiment in response to Reviewer dTPG).\n\n$\\newline$\n\n**Question 1**:\nWe flipped $\\hat{T}$ to ensure $\\hat{T}$ and  $T$ have the same accuracy value. If $\\hat{T}$ and $T$ do not have the same accuracy, bias amplification would be $P(\\hat{T}|A) - P(T|A) + f(\\text{difference in accuracy})$. Here, $f$ is some arbitrary function. The bias amplification score should only reflect the gap between the model and the dataset bias. This score should not incorporate the gap in accuracy between the model and the dataset. If the accuracy of $\\hat{T}$ is $70$%, we flipped $30$% of the labels in $T$ so that $T$ comes down to the same accuracy as $\\hat{T}$. This ensures that bias amplification is $P(\\hat{T}|A) - P(T|A)$.\n\n$\\newline$\n\n**Question 2**:\nWhile we are not fully familiar with multicalibration, we understand that it is a bias mitigation technique. We feel that metrics like DPA can be used to identify models that require or could benefit from multi-calibration. \n\n$\\newline$\n\n**Formatting Concern**:\nThank you for pointing this out. For practical use, a small value epsilon ~1e-10 is added to the denominator to ensure numerical stability. We will update the manuscript to reflect this.\n\n$\\newline$\n\nReferences\n\n[1] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference on Machine Learning, pages 10882–10893. PMLR, 2021.\n\n[2] Shutong Wu, Sizhe Chen, Cihang Xie, Xiaolin Huang. \"One-Pixel Shortcut: On the Learning Preference of Deep Neural Networks.\" The Eleventh International Conference on Learning Representations. 2023."}, {"review_id": "c6wVq5eLwQ", "reviewer": "Reviewer_SYXG", "summary": "This paper introduces Directional Predictability Amplification (DPA), a novel metric for measuring bias amplification in classification datasets. The authors address limitations in existing co-occurrence-based metrics (BA→, Multi→, BAMALS) which fail to detect bias amplification in balanced datasets or cannot identify negative bias amplification. DPA uses predictability-based measurements with attacker functions to quantify how well protected attributes (e.g., gender) can predict task labels, providing directional analysis whilst working across both balanced and unbalanced datasets. The metric reports bounded scores in [-1, 1] and measures relative rather than absolute changes in predictability. Experiments on COMPAS, COCO, and ImSitu datasets demonstrate DPA's superiority over existing metrics.", "questions": "1. Attacker Function Sensitivity: Line 79 claims DPA is \"minimally sensitive\" to attacker function choice. Could you provide more rigorous theoretical or empirical analysis of this claim? Under what conditions might this sensitivity become problematic?\n2. Generalisation Beyond Gender: Your experiments focus predominantly on gender bias. How does DPA perform with other protected attributes like race, age, or socioeconomic status? Could you demonstrate the metric's effectiveness on non-visual domains (text, audio)?\n3. Computational Complexity: What are the computational requirements of DPA compared to co-occurrence-based metrics? How does training time for attacker functions scale with dataset size and complexity?\n4. Failure Modes: Under what theoretical or practical conditions might DPA produce misleading results? How can practitioners identify when the metric may be unreliable?\n5. Individual A-T Pair Analysis: You acknowledge in limitations that DPA cannot measure bias amplification for specific A-T pairs. Could you elaborate on potential approaches to address this limitation, perhaps through ensemble methods or hierarchical analysis?", "limitations": "YEs", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Quality: The paper presents a technically sound solution to genuine limitations in existing bias amplification metrics. The mathematical formulation is rigorous, with proper normalisation ensuring bounded outputs and relative change measurement accounting for dataset bias. The experimental methodology is comprehensive, using multiple datasets (tabular and image) with proper controls including progressive masking experiments and synthetic data validation.\n- Clarity: The paper is well-structured with clear mathematical notation and effective use of tables and figures. The motivation is well-established through concrete examples (e.g., ImSitu cooking bias), and the comparison with existing metrics is systematic and thorough.\n- Significance: This work addresses an important problem in fairness research. The ability to detect bias amplification in balanced datasets is particularly valuable, as practitioners often assume balanced datasets are unbiased. The directional analysis capability provides actionable insights for targeted bias mitigation.\n- Originality: The combination of predictability-based measurement with directionality is novel. Whilst building on Wang et al.'s leakage amplification concept, the addition of directional analysis and the relative change formulation represents a meaningful advancement.\n\nWeaknesses\n- Limited Scope: The evaluation focuses heavily on gender bias across image datasets. Testing on other protected attributes (race, age) and domains (text, audio) would strengthen generalisability claims. The choice of attacker function sensitivity claims require more rigorous validation.\n- Theoretical Gaps: The paper lacks theoretical analysis of when DPA might fail or produce misleading results. The relationship between attacker function choice and metric reliability needs deeper investigation beyond the limited robustness experiments.\n- Practical Considerations: Limited discussion of computational requirements compared to simpler co-occurrence metrics. The need to train attacker functions adds complexity that may hinder adoption.", "was_revised": true, "final_justification": "The rebuttal addressed several practical concerns effectively. The authors provided useful timing data showing DPA requires ~2 minutes for COCO dataset analysis, which is acceptable for offline bias evaluation. They clarified their scope includes racial bias evaluation (COMPAS dataset) beyond gender-focused experiments and acknowledged key failure modes like underfitting/overfitting with proposed monitoring approaches. These responses demonstrate good engagement and practical awareness.\nHowever, critical limitations remain unresolved that prevent a higher rating. Most significantly, the core claim that DPA is \"minimally sensitive\" to attacker function choice lacks sufficient empirical validation - the evidence in Appendix D is limited, and authors acknowledge time constraints prevented deeper analysis. The evaluation scope remains narrow (predominantly visual domains, gender bias) despite claims of broad applicability for a \"one-stop\" metric. Additionally, the work lacks theoretical analysis of when DPA might fail beyond basic scenarios. I assign highest weight to the technical soundness and genuine contribution of addressing bias amplification in balanced datasets, but the unresolved robustness validation significantly undermines confidence in the metric's reliability across diverse applications. The work merits publication as it addresses important gaps in fairness research, but requires stronger validation and broader evaluation to achieve its ambitious goals as a comprehensive bias amplification solution.", "rebuttal": "We thank the reviewer for their insightful feedback. We address their questions below.\n\n$\\newline$\n\n**Question 1**:\n\nGiven the time constraints, we were not able to provide any further analysis on the attacker function apart from the one currently present in Appendix D. If you recommend, we could add an experiment showing the reduced sensitivity by varying other model parameters (e.g., architecture, depth, and activation functions) by the camera-ready deadline.\n\nHigh attacker sensitivity can be especially problematic as it can cause misleading results. High attacker sensitivity can lead to different bias amplification scores for the same model or change the relative magnitude of biases across different models. I.e., for bias amplification in image captioning, Bakr et. al[1] show that the LIC[2] metric is inconsistent in its ranking of bias amplification in different models due to attacker model sensitivity.\n\n\n$\\newline$\n\n**Question 2**:\n\nTo stay consistent with previous works on bias amplification, we measured bias amplification in visual and synthetic datasets on attributes like gender and race. Our experiment on racial biases using the COMPAS dataset (COMPAS is a synthetic dataset) is available in the Appendix – Section E. \n\nText or NLP-focused datasets (e.g., image captioning datasets) have semantic nuances. For such datasets, there are specialized metrics (like LIC) that incorporate text-based semantics when computing bias amplification. The metrics discussed in this work (and related previous works) are largely focused on classification problems in the visual and synthetic domains. We did not investigate audio datasets as we were not aware of publicly available datasets with protected attribute information. It would be helpful if you could suggest audio datasets that could be studied for bias amplification.\n\n$\\newline$\n\n**Question 3**:\n\nIn the table below, we compared the time taken by different metrics to compute bias amplification on the COCO dataset. This computation was performed over an Intel Core Ultra 7 165H processor without any GPU acceleration.\n\n|           | $BA_{MALS}$    | $BA_{\\rightarrow}$        | $Multi_{\\rightarrow}$       | $LA$               | $DPA$                 |\n|-----------|-------------|------------|------------|-----------------------|----------------------|\n| Time (ms) | 57.57 ms    | 150.16 ms  | 453.11 ms  | $1.90 \\times 10^{5}$ ms | $1.2 \\times 10^{5}$ ms |\n\nNote that bias amplification is often measured in offline analysis. Hence, execution time is not the utmost priority. Moreover, DPA takes ~ 2 minutes on the COCO dataset, which is sufficient for an offline analysis in most applications. \n\nWith regards to training time scaling with dataset size, we expect a linear trend. The training process itself is similar to conventional model training and hence, would follow the same linear trend with respect to increasing dataset size.\n\n$\\newline$\n\n**Question 4**:\n\nPoor choice of attacker models can lead to misleading or unreliable results. If the attacker model grossly underfits or overfits the training data, it can lead to bias amplification being over- or under-reported. Monitoring the training and test loss of the attacker models can help identify such misleading values. We have briefly talked about this in our Limitations section. If you suggest that these details would be beneficial for readers, we can add these details in the Limitations section by the camera-ready version.\n\n$\\newline$\n\n **Question 5**:\n\nThere are several potential approaches to analyze individual {A, T} pairs. One approach can be to use feature attribution. Assume we want to measure dataset and model bias in $ \\ A\\rightarrow T$. If we one-hot encode both $A$ and $T$, we can use feature attribution to find out the influence of a specific attribute $A=A_{i}$, in predicting a specific task $T=T_k$. This attribution value can be used as a weak proxy to measure dataset or model bias for this ${A_i, T_k}$ pair. Causal attacker models or interpretable attackers are other avenues for a similar ${A, T}$ pair analysis. We can add a section discussing such ideas for future work.\n\n$\\newline$\n\nReferences\n\n[1] Eslam Mohamed Bakr, , Pengzhan Sun, Li Erran Li, Mohamed Elhoseiny. \"ImageCaptioner$^2$: Image Captioner for Image Captioning Bias Amplification Assessment.\" (2023).\n\n[2] Hirota, Yusuke, Yuta, Nakashima, Noa, Garcia. \"Quantifying Societal Bias Amplification in Image Captioning.\" 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022."}, {"review_id": "BJiWYjTNOj", "reviewer": "Reviewer_dTPG", "summary": "The authors propose Directional Predictability Amplification (DPA), a new metric for measuring bias amplification in machine learning models. Unlike previous metrics, DPA captures the directionality of bias amplification—even in balanced datasets—and accounts for both its positive and negative amplification. The authors demonstrate DPA's effectiveness on both tabular and image datasets.", "questions": "1. Is there a toy example (e.g., Colored MNIST) where model bias amplification arises due to easily learned shortcut features used as attributes, which can effectively demonstrate the usefulness of the proposed metric?\n\n\n2. How can the benefits of the new metric be utilized in real-world scenarios? Are there tasks where DPA offers improvements that other metrics cannot?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Systematic analysis of the limitations of previous metrics, both conceptually and experimentally.\n2. Thorough explanation of the results for each experiment was good. \n3. Diverse experiments on the architectural selection of attacker models.\n\nWeaknesses:\n1. Requires training additional models for measurement.\n2. The authors use attribution score measures and sensitivity ranking to show the tendency of model bias amplification. However, these are proxies for model bias, making it difficult to compare the absolute effectiveness of metrics (i.e., they only show whether the metric aligns with the tendency or not).", "was_revised": true, "final_justification": "I found the Colored MNIST experiment in the rebuttal particularly compelling.\n\nBy evaluating DPA in a controlled setting with limited confounding factors, the authors effectively demonstrated its advantage over existing metrics.\n\nThe paper identifies key limitations of prior approaches and presents a well-motivated alternative.\n\nWhile the contribution feels more incremental than entirely novel, I believe it is a meaningful step forward.\n\nTherefore, I would lean toward a borderline accept.", "rebuttal": "We thank the reviewer for their constructive feedback. We address each point below.\n\n$\\newline$\n\n**Question 1**:\n\nWe have shown a controlled experiment on COCO using OPS[1] based shortcuts in Appendix F. But based on your recommendation, we have also added another experiment on the CMNIST dataset to get more empirical proof.\n\nExperiment Setup: \n\nThe experiment is controlled by two control variables $\\alpha$ and $\\beta$ to control the dataset and model bias, respectively.\n\nIn the CMNIST dataset, we modify the traditional MNIST dataset by replacing the black and white images with their colored versions.  The images are the features ($X$), digit labels are the task ($T$), and color is the protected attribute ($A$). Each label (i.e., digit from 0-9) is correlated with a particular color (i.e., 0- red, 1- blue, 2- green, …). The magnitude of this correlation is controlled using alpha. I.e., if $\\alpha =0.7$, 70% of examples of 0 are red, and the remaining 30% are randomly assigned any of the remaining 9 colors. Thus, $\\alpha$ allows us to control the bias in the dataset.\n\nA simple CNN model (with depth = 2) is trained on this dataset. The trained model is used to get predictions. The predictions go through post-processing based on $\\beta$. $\\beta$ defines the percentage of predicted labels that are overwritten based on the color in the input image. I.e., if $\\beta=0.1$, 10% of the predicted labels will be randomly selected. These labels will then be replaced based on the color in the image. This allows us to increase the $A\\rightarrow T$ bias in the model predictions. Thus, for a fixed value of $\\alpha$, $\\beta$ allows us to control the model bias. For $Q$ function, accuracy is used.\n\nResults:\n\n$\\newline$\n\n$\\beta = 0.0$\n  |                   | $\\alpha = 0.60$     | $\\alpha = 0.65$     | $\\alpha = 0.70$     | $\\alpha = 0.75$     |\n|-------------------|---------------------|---------------------|---------------------|---------------------|\n| $\\Psi_{A}^{D}$  | $0.5816 \\pm 0.010$   |    $0.6536 \\pm 0.008$ | $0.6944 \\pm 0.003$ | $0.7507 \\pm 0.012$ |\n\nAs we see from the table above, $\\Psi_{A}^{D}$ shows a clear increasing trend as the controlled variable $\\alpha$ increases. I.e., the dataset bias detected by $DPA$ increases directly as the actual dataset bias (controlled by $\\alpha$) increases. This controlled experiment allows us to verify that $DPA$ is capable of measuring the dataset bias. \n\n$\\newline$\n\n$\\alpha = 0.75$\n|                         | $\\beta = 0.00$       | $\\beta = 0.05$       | $\\beta = 0.10$       | $\\beta = 0.15$       |\n|-------------------------|----------------------|----------------------|----------------------|----------------------|\n| $\\text{DPA}_{A \\rightarrow T}$   | $0.021 \\pm 0.008$   |    $ 0.057 \\pm 0.007$ | $0.094 \\pm 0.017$ | $0.132 \\pm 0.016$ |\n\nAs we see from the table above, $\\text{DPA}_{A\\rightarrow T}$ shows a clear increasing trend as the controlled variable $\\beta$ increases. I.e., the bias amplification detected by $DPA$ increases directly as the actual model bias (controlled by $\\beta$) increases. This controlled experiment allows us to verify that $DPA$ is capable of accurately measuring bias amplification. \n\n\nWe shall be happy to add this experiment in a more detailed fashion in the camera-ready version.\n\n$\\newline$\n\n**Question 2**:\n\nDirectional bias amplification metrics (like $BA_{\\rightarrow}$ and DPA) can be used to verify if bias mitigation strategies work correctly. Let us revisit an example shown by Wang et al. [2] where the Equalizer model, a model that mitigates gender bias amplification in image captions, accidentally increases $A\\rightarrow T$ bias amplification.\n\nConsider an image of a woman holding a book, and the following gender-biased caption: “A man holding a book”. The Equalizer model corrects the gender of this caption: “A woman holding a kite”. However, it incorrectly changes the object in the caption. While the Equalizer model mitigates $T\\rightarrow A$, it accidentally increases $A\\rightarrow T$ bias amplification. \n\nFor this caption, $DPA$ becomes useful if $A$ and $T$ are balanced in the training dataset (ML practitioners often balance datasets, assuming that balancing gives them an unbiased dataset). If $A$ and $T$ are balanced, even though the Equalizer model predicted the object incorrectly (kite instead of book) while correcting gender, the $A\\rightarrow T$ bias amplification reported by $BA_{\\rightarrow}$ would be 0. This is because $BA_{\\rightarrow}$ incorrectly assumes that a model cannot amplify biases if it is trained on a balanced dataset. However, DPA would report the intended positive bias amplification in $A\\rightarrow T$.\n\n$\\newline$\n\nReferences\n\n[1] Shutong Wu, , Sizhe Chen, Cihang Xie, Xiaolin Huang. \"One-Pixel Shortcut: On the Learning Preference of Deep Neural Networks.\" The Eleventh International Conference on Learning Representations . 2023.\n\n[2] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference on Machine Learning, pages 10882–10893. PMLR, 2021."}]

Author rebuttal:

Meta-review: The authors tackle the problem of quantifying bias amplification in classification datasets. They proposes Directional Predictability Amplification (DPA), a predictability-based metric for directional bias amplification, which differs from prior work by working on balanced datasets and being able to detect negative bias amplification. The authors empirically demonstrates the superiority of DPA over existing metrics on three datasets.

**Strengths**

The problem is impactful and timely. The paper is well-written and easy to follow, with the formulation of the metric being clear and sound. Reviewers were generally happy with the experimental evaluations.

**Initial Weaknesses**

1. SYXG and bRtk asked for stronger evidence that DPA is minimally sensitive to attacker choice and for broader attributes/domains. SYXG also requested computational cost details.
2. n6n6 raised some concerns about novelty over LA.
3. n6n6 felt that the metric could be more theoretically principled, and SYXG asked for better theoretical characterization of potential failure modes.
4. bRtk and n6n6 raised issues about clarity in various parts of the paper.


**Rebuttal Period**

The authors provided clarifications on novelty over LA and LA non-monotonicity which satisfied Reviewer n6n6. They added a CMNIST experiment in response to Reviewer dTPG, and provided a runtime analysis for Reviewer SYXG. They also promised various clarity improvements in the revision.



**Overall Evaluation**

The paper fills a gap in the literature by providing a metric that is directional, sign-aware, and valid on balanced and unbalanced datasets. The empirical evidence is convincing. Reviewers raised valid concerns about lack of theoretical justification for some aspects (e.g. failure modes, minimal sensitivity to attacker function choice, more formal justification for dominance over LA) which have not been addressed, but I do not think this is significant enough for rejection. All reviewers eventually converged to acceptance. As such, I recommend acceptance. The authors should make the promised changes, as well as run the 2x2 contingency table experiment on COMPAS suggested by Reviewer bRtk, for the camera ready.
Decision: Accept (poster)