Title: DPA: A one-stop metric to measure bias amplification in classification datasets
Venue: NeurIPS 2025
Accepted: yes
Keywords: algorithmic biases

fairness

accountability

AI ethics
Abstract: Most ML datasets today contain biases. When we train models on these datasets, they often not only learn these biases but can worsen them --- a phenomenon known as bias amplification. Several co-occurrence-based metrics have been proposed to measure bias amplification in classification datasets. They measure bias amplification between a protected attribute (e.g., gender) and a task (e.g., cooking). These metrics also support fine-grained bias analysis by identifying the direction in which a model amplifies biases. However, co-occurrence-based metrics have limitations --- some fail to measure bias amplification in balanced datasets, while others fail to measure negative bias amplification. To solve these issues, recent work proposed a predictability-based metric called leakage amplification (LA). However, LA cannot identify the direction in which a model amplifies biases. We propose Directional Predictability Amplification (DPA), a predictability-based metric that is  (1) directional, (2) works with balanced and unbalanced datasets, and (3) correctly identifies positive and negative bias amplification. DPA eliminates the need to evaluate models on multiple metrics to verify these three aspects. DPA also improves over prior predictability-based metrics like LA: it is less sensitive to the choice of attacker function (a hyperparameter in predictability-based metrics), reports scores within a bounded range, and accounts for dataset bias by measuring relative changes in predictability. Our experiments on well-known datasets like COMPAS (a tabular dataset), COCO, and ImSitu (image datasets) show that DPA is the most reliable metric to measure bias amplification in classification problems.


Peer reviews:
comments: SUMMARY:
The authors propose Directional Predictability Amplification (DPA), a new metric for measuring bias amplification in machine learning models. Unlike previous metrics, DPA captures the directionality of bias amplification—even in balanced datasets—and accounts for both its positive and negative amplification. The authors demonstrate DPA's effectiveness on both tabular and image datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Systematic analysis of the limitations of previous metrics, both conceptually and experimentally.
2. Thorough explanation of the results for each experiment was good. 
3. Diverse experiments on the architectural selection of attacker models.

Weaknesses:
1. Requires training additional models for measurement.
2. The authors use attribution score measures and sensitivity ranking to show the tendency of model bias amplification. However, these are proxies for model bias, making it difficult to compare the absolute effectiveness of metrics (i.e., they only show whether the metric aligns with the tendency or not).

QUESTIONS:
1. Is there a toy example (e.g., Colored MNIST) where model bias amplification arises due to easily learned shortcut features used as attributes, which can effectively demonstrate the usefulness of the proposed metric?


2. How can the benefits of the new metric be utilized in real-world scenarios? Are there tasks where DPA offers improvements that other metrics cannot?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Directional Predictability Amplification (DPA), a novel metric for measuring bias amplification in classification datasets. The authors address limitations in existing co-occurrence-based metrics (BA→, Multi→, BAMALS) which fail to detect bias amplification in balanced datasets or cannot identify negative bias amplification. DPA uses predictability-based measurements with attacker functions to quantify how well protected attributes (e.g., gender) can predict task labels, providing directional analysis whilst working across both balanced and unbalanced datasets. The metric reports bounded scores in [-1, 1] and measures relative rather than absolute changes in predictability. Experiments on COMPAS, COCO, and ImSitu datasets demonstrate DPA's superiority over existing metrics.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Quality: The paper presents a technically sound solution to genuine limitations in existing bias amplification metrics. The mathematical formulation is rigorous, with proper normalisation ensuring bounded outputs and relative change measurement accounting for dataset bias. The experimental methodology is comprehensive, using multiple datasets (tabular and image) with proper controls including progressive masking experiments and synthetic data validation.\n- Clarity: The paper is well-structured with clear mathematical notation and effective use of tables and figures. The motivation is well-established through concrete examples (e.g., ImSitu cooking bias), and the comparison with existing metrics is systematic and thorough.\n- Significance: This work addresses an important problem in fairness research. The ability to detect bias amplification in balanced datasets is particularly valuable, as practitioners often assume balanced datasets are unbiased. The directional analysis capability provides actionable insights for targeted bias mitigation.\n- Originality: The combination of predictability-based measurement with directionality is novel. Whilst building on Wang et al.'s leakage amplification concept, the addition of directional analysis and the relative change formulation represents a meaningful advancement.\n\nWeaknesses\n- Limited Scope: The evaluation focuses heavily on gender bias across image datasets. Testing on other protected attributes (race, age) and domains (text, audio) would strengthen generalisability claims. The choice of attacker function sensitivity claims require more rigorous validation.\n- Theoretical Gaps: The paper lacks theoretical analysis of when DPA might fail or produce misleading results. The relationship between attacker function choice and metric reliability needs deeper investigation beyond the limited robustness experiments.\n- Practical Considerations: Limited discussion of computational requirements compared to simpler co-occurrence metrics. The need to train attacker functions adds complexity that may hinder adoption.\n\nQUESTIONS:\n1. Attacker Function Sensitivity: Line 79 claims DPA is \"minimally sensitive\" to attacker function choice. Could you provide more rigorous theoretical or empirical analysis of this claim? Under what conditions might this sensitivity become problematic?\n2. Generalisation Beyond Gender: Your experiments focus predominantly on gender bias. How does DPA perform with other protected attributes like race, age, or socioeconomic status? Could you demonstrate the metric's effectiveness on non-visual domains (text, audio)?\n3. Computational Complexity: What are the computational requirements of DPA compared to co-occurrence-based metrics? How does training time for attacker functions scale with dataset size and complexity?\n4. Failure Modes: Under what theoretical or practical conditions might DPA produce misleading results? How can practitioners identify when the metric may be unreliable?\n5. Individual A-T Pair Analysis: You acknowledge in limitations that DPA cannot measure bias amplification for specific A-T pairs. Could you elaborate on potential approaches to address this limitation, perhaps through ensemble methods or hierarchical analysis?\n\nLIMITATIONS:\nYEs", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Problem:\n_a.k.a. What problem/question does this work purport to solve/answer?_\n\nA particular aspect of bias within ML/AI that is particularly concerning is bias amplification -- the phenomena in which a ML model does not (as naive theory might suggest) automatically learn to ameliorate data biases but actually amplify them in downstream use. In order to effectively reduce bias amplification, we need to be able to measure it. This paper proposes a new method to do so that solves several key challenges: \n  1. Directionality\n  2. Working across balanced and unbalanced datasets\n  3. Works with both positive and negative bias\n\n### Impact:\n_a.k.a. Is the problem/question impactful?_\n\nBias amplification is a critical property that intersects both with the high-impact nature of fairness challenges and the possibility of meaningful technical solution through new innovations, so I feel this is a key direction for impact.\n\n### Proposed Solution:\n_a.k.a. How does this work claim it will solve/answer the problem/question?_\n\nThey define their metric to be the relative change in predictability of a protected attribute from a real vs. predicted label (or the reverse).\n\n### Novelty/Why hasn't this been done before:\n_a.k.a. How does this work differ from the existing literature?_\n\nPrior methods have examined an absolute change in predictability, rather than a relative change. The authors claim this is a significant problem, because it conflates a small relative increase in bias in settings with high dataset bias and a large increase in relative bias for settings with low dataset bias.  As far as I can tell, this difference is the largest source of novelty in the work.\n\n### Evidence that this work solves the problem:\n_a.k.a. How does this work prove that their solution/answer is valid?_\n\nThe authors employ a very clever experiment (inspired in part by some prior work, I believe) in which they artificially introduce a setting in which the model must amplify bias in order to make effective predictions at all, then sees to what extent their metric catches that bias amplification versus competing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n### Key Strengths\n_a.k.a. reasons I would advocate this paper be accepted._\n\n  1. This problem is very high impact and new metrics to detect bias amplification are important.\n  2. I like your controlled masking experiments as a vehicle to artificially inflate the expected bias amplification.\n\n### Minor Strengths\n_a.k.a. things I like but would not be sufficient to motivate acceptance in isolation._\n\n  1. I like the use of proxy signals (like attribution maps and CAVs) to reinforce what the expected behavior is (though I would not trust them absolutely). \n\n## Weaknesses\n### Key Weaknesses\n_a.k.a. reasons I would advocate this paper be rejected._\n\n#### Flaw in your results\nI do not believe your results can be correct as presented -- in particular the relative performance of LA vs. DPA. Let's let $Q$ be $\\lambda$ -- both are (as you describe them) measures of how well the attribute can be predicted from the task in the dataset vs. the predicted task by the model, unless I've misunderstood. $Q_D$ in your experiment should be fixed across all masking levels -- in the data, the task predicts the attribute in a manner that is unaffected by masking. $Q_M$ changes as you adjust the masking strategy and rate -- and in particular _should_ if your experiment has internal validity be increasing as we move to the right (in Table 2). This means that LA -- defined as $Q_M$ - $Q_D$ -- _should also be increasing monotonically as we move to the right_. But, instead, you have the last value in the Bounding-Box Masked column being lower than any other. Why is this? What has gone wrong in the use of the LA metric or my understanding of it to deviate from what should be the result here? \n\nWithout understanding or explaining this, this result undercuts your entire argument, because it calls the internal validity of your experiment or the validity of the baselines into question, so this is essential to correct.\n\n#### Insufficient Novelty\nYour method is functionally just a prior method (LA) divided by the sum of the two quality terms. The fact that DPA can disentangle the two directions is merely because you repeat the analysis of LA but with a swapped internal definition of A and T. This is very much stretching the novelty bar for a venue like NeurIPS. This weakness may be overcome by your results, which are very promising, but it is a major challenge nonetheless. Fundamentally, this weakness compounds some of the skepticism about results, as -- even if your reported improvements over LA are valid -- it is such a tiny change to improve LA in a manner that re-captures your proposed approach.\n\n### Minor Weaknesses\n_a.k.a. things I dislike but would not be sufficient to motivate rejection in isolation._\n\n  1. Your evaluation metrics need to be summarized in some way so that we can read off whether DPA is superior to other metrics in one pass, rather than needing to read all the numbers in Table 2. E.g., report the correlation between the % masked and the metric, or something like that. \n  2. I'd love for your metric to be more theoretically principled, rather than just a subjective justification on a change from an absolute difference to a relative one. E.g., is there a probabilistic perspective you can leverage to suggest a relative metric is superior under a given biasing method?\n  3. You should generate some purely synthetic experiments that unambiguously show the improvement of your method, so you don't need to rely on heuristics or proxy variables to establish what the \"right\" bias direction should be.\n\n## Final Recommendation:\n_My overall recommendation for this paper, and its justification._\n\nReject\n\nI recommend reject because, without further clarification of the results, I cannot trust the conclusions presented (see key weakness 1). If this is clarified and my misunderstandings addressed, I could see my score rising as high as a Marginal Accept even with the concern regarding novelty.\n\n## What do I want to see changed:\n### Changes to Increase my Score:\n_What things, were they provided to my satisfaction, would motivate me improve my score?_\n\nClarify (and convince me) why the LA results do not undermine your argument and experimental coherence and more generally re-present your results or present additional results that establish the superiority of DPA in a more convincing, internally consistent manner. In addition, some further justification (perhaps merely promoted from the appendix) that demonstrates more nuanced aspects of superiority vs. LA independent from the specific choices of quality functions and attacker functions used in the original LA formulation (unless those are somehow thought to be essential to the approach in a manner I'm missing).\n\n### Changes I'd recommend for Future Submissions:\n_What changes do I recommend for future submissions, but would not necessarily change my score now?_\n\nSignificantly rework results presentation to better highlight in a single pass why DPA is better.\n\nQUESTIONS:\n1. I don't understand why you flip some of the predicted labels T based on M's accuracy. Can you further explain this? What is the error case you're concerned with here?\n\n2. How would a technique like multicalibration impact the success of your approach here? I think that warrants testing.\n\nLIMITATIONS:\nNo comment", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DAP metric to quantify the amount by which a predictive model amplifies pre-existing correlations between sensitive attributes $A$ and labels $T$. It is demonstrated experimentally that models can amplify such correlations even when trained on curated datasets where $A$ is statistically independent from $Y$. The authors argue that this is because the model can leverage spurious correlations to infer $A$ from the image. Since alternatives to DAP do not highlight such shortcuts, DAP is a worthy addition to the literature.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The paper is well written and easy to follow.\n- The experiments consisting of progressively masking individuals and trying to infer their gender $A$ based on the ground-truth $T$ (or predicted $\\hat{T}$) objects is a sound way to quantify spurious correlations in the dataset/model.\n- Relevant classifications models are studied (ViT newer and VGG16 older).\n- The DAP metric is a significant contribution to the community because it highlights a scenario that other metrics fail to identify. Indeed, DAP can detect when a model infers gender based on spurious correlations although the dataset is curated to render $A$ and $T$ statistically independent.\n- The work is original and cites revelant prior work.\n\n# Weaknesses\n- The clarity of the technical sections could be improved (more on this below in the **Questions** section)\n- The experimental details are not sufficient. Notably there are missing details on the trained classifiers $M$ and the Attackers (more on this below in the **Questions** section).\n\n\n## Minor Comments\nThe current work investigates statistical measures of fairness so there is no notion of causality involved. However, at line 112, the word *causality* is used as a synonym for directionality. This is not the correct terminology. $A\\rightarrow T$ essentially means $P(T|A)$ and\n$T\\rightarrow A$ refers to $P(A|T)$, but using the wording *causality* along side arrow-based notation $\\rightarrow$ suggests that you are doing causal inference. I would avoid writting *causality*.\n\nQUESTIONS:\n## Clarifications Regarding the DAP\nFrom what I understand, the bias measures $\\Psi_A^D$ consists of reporting the accuracy of an attacker $f_A^T$ trained to predict $T$ from $A$. However, in Appendix B equation (13), it is stated that $\\Psi_A^D \\propto P(T|A)$. I think this equation is incorrect, which undermines the connection the authors were trying to make with the \n$BA_{\\rightarrow}$ method . Indeed, assuming $A$ and $T$ are binary, the accuracy of the optimal attacker is $f_A^T$ is $\\Phi_A^D = P(A=0) \\text{max}\\\\{P(T=1|A=0), P(T=0|A=0)\\\\} + P(A=1) \\text{max}\\\\{ P(T=1|A=1), P(T=0|A=1)\\\\}$. This is not proportional to a conditional probability. Accordingly, Section B should be clarified or removed.\n\n## Clarifications regarding Section 5.1\nFirst of all, it is stated that DAP could be used for model selection. However, I think this is only true for the $DAP_{A\\rightarrow T}$ which only requires that the model outputs labels $\\hat{T}$. The metric $DAP_{T \\rightarrow A}$ requires that the model predicts the sentitive attribute $\\hat{A}$, and I cannot think of any practical application where we would want to use such predictive model. I would argue that $DAP_{T \\rightarrow A}$ is more useful for identifying spurious correlations in datasets and potentially curating them.\n\nAlso, I am confused by the toy example provided in lines 333-342. In this example, we have $A=0$ for men, $A=1$ for women, $T=0$ for hired and $T=1$ for rejected. It is claimed that this dataset is biased according to DAP. But isn't DAP an amplification metric that requires a predictive model $M$ as well? What is you model in this toy example. Also DAP is directional, so which direction is considered ($A\\rightarrow T$ or $T\\rightarrow A$) in this toy example? Since this toy example is brought in the conclusion, it would be important to clarify why DAP claims it is biased.\n\n## Missing Experimental Details\nThere are not enough details regarding the model training on Coco. Indeed, a single image can contain multiple objects of different/same classes (Figure 1 shows an image with three different objects for instance). Is the model required to localize these objects? Is it required to differentiate two different objects of the same class (e.g. two different pans)? Or simply predict there is *at least* one pan in the image? \nI suspect that you treat Coco as a *multi-label classification* problem where image $I$ is labeled as $y=(y^{(1)}, y^{(2)}, \\\\ldots, y^{|T|})$ with $y^{(t)}=1$ if at least one object of class $t$ is present and $y^{(t)}=0$ otherwise. This would be would be important to clarify in the manuscript along with descriptions of how the architectures pretrained on Imagenet multiclass classification were adapted for Coco.\n\nSection I (line 1012) of the appendix does not contain enough details on the Attacker network that is used to predict $A$ from $T$. For Coco, the ground-truth label $T$ fed to the attacker is actually a **set** (see Figure 1 bottom) and not a single numerical value. MLPs are not suited to take sets as inputs so I suspect that the labels were One-Hot-Encoded to $\\\\{0, 1\\\\}^{|T|}$ before being fed to the attacker. It would be important to clarify if this is the case or not. If not, how were MLPs able to process sets of objects as their inputs?\n\n## Fitting a MLP on COMPAS\nIn COMPAS, the labels and sensitive attributes are binary. Thus, I do not see any benefits to fitting a MLP to predict $A$ from $T$ (or $T$ from $A$). In this case, computing conditional probabilities from the 2x2 contingency table should be sufficient. Can the authors argue why a MLP was used as an attacker on COMPAS?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}