Preserving AUC Fairness in Learning with Noisy Protected Groups
Authors: Mingyang Wu
Li Lin
Wenbin Zhang
Xin Wang
Zhenhuan Yang
Shu Hu
Venue: ICML 2025 poster (2025)
Keywords: Robust Machine Learning
AUC Optimization
Fairness
Primary area: general_machine_learning->supervised_learning
TL;DR: We propose the first robust AUC fairness framework under noisy group labels with theoretical guarantees.

Abstract: The Area Under the ROC Curve (AUC) is a key metric for classification, especially under class imbalance, with growing research focus on optimizing AUC over accuracy in applications like medical image analysis and deepfake detection. This leads to fairness in AUC optimization becoming crucial as biases can impact protected groups. While various fairness mitigation techniques exist, fairness considerations in AUC optimization remain in their early stages, with most research focusing on improving AUC fairness under the
assumption of clean protected groups. However, these studies often overlook the impact of noisy protected groups, leading to fairness violations in practice. To address this, we propose the first robust AUC fairness approach under noisy protected groups with fairness theoretical guarantees using distributionally robust optimization. Extensive experiments on tabular and image datasets show that our method outperforms state-of-the-art approaches in preserving AUC fairness. The code is in https://github.com/Purdue-M2/AUC_Fairness_with_Noisy_Groups.

Preserving AUC Fairness in Learning with Noisy Protected Groups
Mingyang Wu"! Lin' Wenbin Zhang²
Xin Wang? Zhenhuan Yang* Shu Hu
Abstract
domains like information retrieval (Cortes & Mohri, 2003), medical image analysis (Yuan et al., 2021), and deepfake detection (Pu et al., 2022). In particular, in deepfake de- tection, misclassifying fake content as real can lead to the widespread dissemination of misinformation, potentially un- dermining public trust, manipulating discourse, or enabling fraud. AUC is thus preferred over -hee-thre metrics, as it captures model performance comprehensively across varying operational settings.
The Area Under the ROC Curve (AUC) is key metrie for classification, especially under class imbalance, with growing research focus opti- mizing AUC over accuracy in applications like medical image analysis and deepfake detection. This leads to fairness in AUC optimization becom- ing crucial as biases can impact protected groups. While various fairness mitigation techniques ex- ist, fairness considerations in AUC optimization remain in their early stages, with most research focusing on improving AUC fairness under the assumption of clean protected groups. However, these studies often overlook the impact of noisy protected groups, leading to fairness violations in practice. To address this, we propose the first robust AUC fairness approach under noisy pro- tected groups with fairness theoretical guarantees using distributionally robust optimization. Exten- sive experiments on tabular and image datasets show that our method outperforms mate-offtheean approaches in preserving AUC faireess. The code is https: /tt:://gitttbb com/ Purdue-M2/ AAU__Fairnese with w Noisy Groppp.
This has motivated numerous studies (Yang. 2021; Kumagai et al.; Guo et al., 2022; Zhang et al.. 2023) focusing on training Al models to maximize AUC rather than relying on traditional loss functions (e.g., cross-entropy loss), as this approach directly optimizes a metric better aligned with the desired application outcomes. By doing so, models achieve improved performance in scenarios where distinguishing between classes with high sensitivity and specificity is es- sential.
The optimization of AUC in machine learning models ne- cessitates a focus on fairness, particularly in light of grow- ing concerns that algorithmic decisions often exacerbate inequities faced by vulnerable groups defined by sensitive attributes such gender and race, also known protected groups. Recent studies (Caton & Haas, 2024; Kenfack et al., 2024) have highlighted how machine learning models, if left unchecked, can perpetuate or worsen biases in allocation decisions, leading to unfavorable outcomes for these groups. To address this, a range of bias mitigation techniques (Hu & Chen, 2024; Lin et al., 2024; Tian al., 2025; Kollias et al., 2024: Ju et al., 2024) has been developed, including methods that focus on statistical faimess metrics derived from confusion matrices. However, faimesss considerations in AUC optimization (Yang et al., 2023; Yao et al., 2023) remain an early stage of exploration despite its significance in many applications and most of them focus on the group- level AUC fairness.
1. Introduction
The Area Under the ROC Curve (AUC) (Hanley & McNeil, 1982) is one of the most widely used performance metrics in classification tasks, particularly when addressing challenges such as class imbalance or uncertain relative costs of false positives and false negatives. It provides a measure of classifier's ability distinguish between classes across all possible decision thresholds, making it especially relevant in
"Equal contribution 'Department of Computer and Infor- mation Technology, Purduc University, West Lafayette, USA "nnight Foundation School of Computing and Information Sci- ences, Florida International University, Miami, USA Depaatment of Epidemiology and Biostatisties. University at Albany, State University of New York, New York, USA AAmazon, New York, USA. Correspondence to: Shu Hu <hu968 purdue.edm edu>.
The group-level fairness for AUC leads to three categories of metrics, each addressing different aspects of disparate impacts. First, intra- group AUC (Beutel et al., 2019; Yao et al., 2023) focuses constraining both positive and negative examples to the same group. Second, inter-group AUC (Beutel et al., 2019; Kallus & Zhou, 2019; Yao et al., 2023) considers ranking fairness between groups, evaluating the
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
proup Clean
Intergroup, Nols
Intre-group.
Intra group. Notsy
1.0
0.8
0.0
0.8
0.8
Group
Group
Group
Group
AUD Gape
AUC Gape
AUC Gape
AUC Gap
0.4
Our:: 0.028
0.0
Oure:
0.830
0.1
Dura: 0.038
0.4
Oura: 0,635
8.064
NirinasFaAAUC 0.044
NininasFarAUG: 0.042
0.2
0.2
800%dup - 1,37
3,7
-1.7
R0076 - 1,7
- -+)
0.0
0.0)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
6.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
3.0
0.0
0.2
044
0.6
0u 1.0
FPR
Figure 1. Illustrative inter-/intra-gron AUC dicrepancy examples of existing MinimaxFairAUC method (Yang et al., 2023) (dashed curves) and our method (solid curves) on befault (Yeh & Lien, 2009) dataset with noisy levels 0 and 0.3, respectively. Notations are defined Section 4. In general, our method is better than MininaxFairAUC in preserving AUC fairness, demonstrating robustness to noisy groups.
metric by comparing positive examples from one group to negative examples from another. Lastly, few works (Yang et al., 2023; Yao et al., 2023) have sought to considering both intra-group and inter-group AUC fairness during the learning process.
pose an empirical estimation method leveraging pre-trained foundation models. Finally, we design effi- cient stochastic gradient descent-ascent (SGDA) algorithm to optimize the proposed learning objective, enhancing both AUC fairness and the model's generalization capabilities in deployment scenarios. Our key contributions are:
Nevertheless, existing AUC fairness studies often overlook the reliability of protected group information. This raises crucial question: Can AUC fairness notions be accurately measured or effectively enforced when the protected group data is noisy or unreliable? Noisy protected group labels are prevalent in many scenarios. For instance, survey par- ticipants may intentionally obfuscate their responses due to concerns about privacy, fear of disclosure, or potential discrimination, leading to response biases (Krumpal, 2013). Similarly, in deepfake datasets, demographic annotations often inferred using deep learning models (Lin et al., 2025). However, the accuracy of these annotations is inher- ently limited, as the true demographic information cannot be verified or tracked when the faces are Al-generated. Our practical evaluation (see Fig. 1) reveals that training with traditional AUC fairness under noisy protected group labels can result in significant group AUC gap in model deploy- ment. This highlights the critical need designing a robust approach to ensure reliuble AUC fairness.
1. We present the first experimental analysis of the impact of noisy protected group labels on the existing AUC fairness learning method.
2. We introduce novel AUC faimesss metric and propose the first approach to preserve AUC fairness under noisy protected groups with theoretical guarantees.
3. Extensive experiments on tabular and image datasets show that our method surpasses state-of-the-arr ap- proaches across applications like socioeconomic anal- ysis and deepfake detection.
2. Related Work
AUC-based Fairness. A pioneering effort by Dixon et al. (2018) introduced the Pinned AUC metric for text classifica- tion tasks, which involves resampling the data so that each of the two groups constitutes half of the dataset, followed by calculating the AUC difference on the resampled data, Building on this foundation, Beutel et al. (2019) proposed intra-group and inter-group AUC metrics for recommender systems, assessing whether clicked items are ranked above unclicked items both within and across protected groups Their method also incorporated a regularization term to reduce ranking unfairness. To address disparities across groups, Kallus Zhou (2019) developed the cross-AUC (xAUC) metric, which identifies systematic biases where positive instances from one group may be ranked below negative instances from another.
In this work, we propose the first robust AUC fairness ap- proach for learning under noisy protected groups, providing theoretical fairness guarantees. We begin by conducting ex- periments to illustrate the adverse effects of noisy protected group labels on existing AUC fairness methods. Next, we introduce novel and general AUC fairness metric that ac- counts for both intra-group and inter-group AUC. Based on this metric, we formulate a new learning objective for AUC fairness using a distributionally robust optimization (DRO) framework (Duchi& & Namkoong, 2021), which bounds the Total Variation (TV) distance between clean and noisy group distributions. We also provide a theoretical analysis demon- strating that our approach ensures faimess even under noisy protected group labels. To estimate the TV distance bound, we reformulate it in terms of noisy label ratios and pro-
In the context of general pairwise ranking, Narasimhan et al. (2020) proposed maximizing AUC under faimess con- straints, further advancing the exploration of cross-group AUC fairness. Other works, such as Vogel et al. (2021),
2

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
focused on fairness defined directly in terms of the ROC curve, employing regularization balance overall AUC performance with group-level fairesss requirements. More recently, Yang et al. (2023) introduced a minimax faimess framework that simultaneously addresses intra-group and inter-group AUC disparities using a Rawlsian approach, sup- ported by an efficient optimization algorithm with proven convergence guarantees. Similarly, Yao et al. (2023) pro- posed scalable and efficient stochastic optimization frame- work for AUC-based fairesss constraints, demonstrating its ability to balance classification performance and fairness in both online and offline learning scenarios. However, all of the aforementioned works assume clean and accurately labeled protected groups, disregurding the impuct of noisy group labels that are common in real-world datasets. Ad- dressing this limitation is the central focus of our paper.
0000
Ner ArrtaaTarAAC
AUC Faimess Violation
AUC Faimess Violation
60037
s01
(a)
15)
0034
Noise Level
Noise Level
Figure 2. Impact of noisy protected group labels on AUC fairness violation (lower values indicate better AUC fairness) in sce- narios: {a) Sacioeconomic Analysis and (b) Deepfake Detection. Mean value shown in black line. The standard deviation shown in blue background. where three random runs for each noise level.
3. Motivation
Fairness with Noisy Protected Groups. Group fairness methods typically assume accurate knowledge of protected group labels. but in practice, these labels are often noisy or unreliable. Enforcing fairesss constraints based on such noisy labels fails to guarantee fairness with respect to the clean labels (Gupta et al., 2018). To address this issue, Lahoti al. (2020) proposed an adversarial reweighting approach that leverages correlations between non non- propprtee features, task labels, and potentially unobserved group mem- bership, demonstrating improved fairness under label uncer- tainty in tabular datasets. However, extending this approach to image data poses significant challenges.
To demonstrate the impact of noisy protected group levels on AUC fairness, we conduct experiments on the tabular Adult dataset (for socioeconomic analysis) (Asuncion et al., 2007) and the image-based FF++ dataset (for deepfake de- tection) (Rossler et al., 2019; Lin et al., 2024). Noise is introduced by flipping a portion (ranging from O to 50%) of the protected group labels (e.g., gender) in the training sets. For the Adult dataset, we evaluate the performance of the latest AUC fairness method, MinimaxFairAUC (Yang et al., 2023), while for the FF++ dataset, assess the state-of- the-art fairness approach, DAW-FDD (Ju et al., 2024). All experiments follow the original settings specified in these methods. We use AUC faimess violation as the evaluation metric, which quantifies the maximum gap between group- level (intra-group or inter-group) AUC and the overall AUC. Then, we present the mean and standard deviation scores on the test sets over three random runs for each noise setting.
Under the more conservative assumption of no protected group information, Hashimoto et al. (2018) applied distribu- tionally robust optimization (DRO) to enforce what Lahoti et al. (2020) termed Rawlsian Max-Min fairesss. Although DRO-based methods can achieve reasonable fairness results without explicit protected group labels, they are often less effective than approaches that incorporate such information. Building on Hashimoto et al. (2018), Wang et al. (2020) introduced maximum total variation distance bound in the DRO procedure, offering the first fairness framework with guarantees under noisy protected-group labels. Further advancements include the work of Celis et al. (2021), who proposed an optimization framework with provable guar- antees on both accuracy and fairness for classifiers trained with noisy protected attributes. Similarly, Mchrotra & Vish- noi (2022) introduced novel method for improving fair rankings in the presence of noisy group labels. Additionally, Ghazimatin et al. (2022) and Ghosh et al. (2023) conducted empirical evaluations of fairness approaches under noisy sensitive information. However, the aforementioned ap- proaches are either not generalizable or are unsuitable for direct application to pairwise ranking optimization prob- lems. These limitations leave AUC fairness largely unex- plored in these contexts.
As shown in Fig. 2, AUC fairness violation increases sig- nificantly as the accuracy of protected group annotations declines across all three scenarios. Specifically, Fig. 2(a) demonstrates that conventional AUC fairness enhancement method is highly sensitive to noisy protected group labels in the training data. Similarly, while DAW-FDD designed to improve general faimess, Fig. 2(b) reveals it struggles in maintaining AUC faimess under noisy group conditions. These findings underscore the critical need for robust AUC fairesss approaches capable of tolerating noisy groups.
4. Methodology
In this section, we introduce a novel robust AUC fairness approach to address the challenges discussed in the previous section. Let X E X' C Rd represent the random variable for input features, Y E y = { denote the binary label, and Z E Z = {1..... m) represent the random protected group. where m is the total number of protected groups. We define a scoring function do : X - R, parameterized by
3

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
0 E O, which maps the input features to a real-valued score.
Beutel al. (2019); Yao et al. (2023). Conversely, when * z'; it aligns with inter-group pairwise fairness, ex- plored in Beutel et al. (2019); Kallus & Zhou (2019); Yao et al. (2023). Furthermore, our AUC faimess function is closely related to the Rawlsian principle of justice (Rawls, 2001), particularly the Rawlsian AUC fairness framework proposed by Yang et al. (2023). Specifically, if we focus solely on maximizing the smallest group-level AUC among all groups, without considering the second term (i.e, the overall AUC) in Eq. (5).
4.1. AUC Fairness
Overall AUC: The overall AUC quantifies the probability that a model correctly ranks positive example (e.g., X)
higher than negative example (e.g., X'). It is formally defined as:
AUC(0) = (x) Y 1,1Y== = -11, (1)
where [[o] is the indicator function, which equals 1 if is true and 0 otherwise. To optimize the AUC score, it is common practice to minimize the complementary AUC risk (Hanley & McNeil, 1982), given by:
However, solely emphasizing and enforcing the faimess con- straint can lead to a trivial solution where 2UC,,,/ (0) = A = 0.5, which reflects no discriminatory power in the model. To prevent this, is necessary to simultane- ously maximize the overall AUC score while ensuring AUC faimesss. It is worth noting that AUC is not only faimess- related metric but also a key performance measure for eval- uating trained models. Consequently, directly optimizing AUC can enhance model performance, as demonstrated in various domains, including medical image analysis (Yuan et al., 2021) and deepfake detection (Pu et al., 2022). In- stead of maximizing the overall AUC, we minimize its cor- responding AUC risk directly. This leads us formulate the following constrained AUC fairness problem:
(0) = 1 - AUC(O)
(2)
= BIllee |Y = 1, Y' = -1].
Group-level AUC. Following (Yang et al., 2023), the group- level AUC is defined as:
AUCz, (0)
= 1,- [[[((x(X)))xxx")) Y 11, 2, Z' z'1
(3)
This metric captures the AUC score for comparisons be- tween positive examples from group and negative exam- ples from group z', reflecting a pairwise dependence with respect to the groups Z, Z' E Z. When 2 = 2', we refer it as intra-group AUC, which measures ranking performance within a single group. Conversely, when A z', it is termed inter-group AUC, assessing the ranking performance across different groups
min Lavc s.t. (0, S O2, E Z.
(6)
4.2. Robust AUC Fairness
While minimizing Eq. (6) can achieve AUC fairness under clean protected groups, it does not guarantee faimess when the protected groups are noisy. To address this, we propose a robust AUC fairness learning objective by leveraging a distributionally robust optimization (DRO) approach (Duchi & Namkoong, 2021), inspired by Wang et al. (2020).
AUC Fairness Metric. Then, we propose the target AUC fairness metric as follows:
(0) = =
Formulation. Specifically, let 2 E 2 be the random vari- able representing the noisy protected group associated with X. The learning objective can then be reformulated as:
(4)
(x I -T
Building on this, we define the group-level AUC fairness functions as:
min (f)) s.t. (0) S 0, 50, 2' E 2,
(7)
= 92.2 = =,, = AUC,,, (0) - AUC(()), V2, E Z.
(5)
where (0) ET((0) =, 2, 2'J. Next, we an- alyze how far a model trained with noisy protected group labels using 1q. (7) deviates from satisfying the fairness con- straints defined for clean protected groups Let p represent the distribution of pairwise data ((X, Y), (X', Y')) ~ p. where = 1 and Y' -1. Define Pa.x' as the distribution of pairwise data conditioned the clean groups 7 and Z' = 2', such that ((), (X', YY)) (Z- = x, Z' 2 ~ ps.z' Similarly, let P..:/ represent the distribution of ((X, (X', Y") conditioned on the noisy groups 2 =
The above formulation quantifies the gap (f)) be- tween any group-level AUC score and the overall AUC score. To achieve AUC fairness, it suffices to enforce the constraint 2 (0) < 0 Vz, z' E Z, ensuring that all group- level AUCs are close to the overall AUC. This, in turn, re- duces disparities among gevup AUCs, fostering more equitable model performance across groups This formula- tion also establishes meaningful connections with several existing fairness measures. For instance, when 2 = z'. it generalizes to intra-group pairwise fairness, as studied in
and Z' = z', such that ((x, Y). (X', YY)) (2 = 2, Z' z²) ~ pr.z- To quantify the difference between P=,=' and
4

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Ps.z' we use the Total Variation (TV) distance (Duchi & Namkoong, 2021), denoted as (V(p..2' P..="). Based this measure, we establish the following theoretical results (the proof is provided in Appendix A.I).
version of Eq. (8) can be rewritten as:
LAUC(0) + +ΣΣ 2 A (0),
min
max
A 20.0 20.20
==1===1
Theorem 4.1. Suppose a model is trained using Eq. (7) with noisy groups and satisfies Iz.x (0) S 0 Vz,z' z' E Z. Let Ya,a" be an upper bound on the TV distance, such that Ya.s" 2 TV (Pe.st (Pe,s',Pe.s'' V2, z' E 2. Then, the fairness measure for the clean groups will be satisfied within a slack of "z,s" for each pairwise group, ensuring 22.2r S Na V2, x' E Z.
s.t. lip - P2, S 27,,'', IIPe,, = 1, V2, a' E 2,
(9)
where Lovc(0) ntn Σ i= Ei= 1==
(0o(x ( (X
is the empirical Σ E5= Pz. T2.2 (06(27))66067))
form
of LAUC(0).
(0)
nnt ntn Di Σj-1 1000 )>on(X) ) is the empirical form of (0,e(()). and || |11,1 is the norm. In practice, the non-differentiable indicator function I can be replaced with (sub) dibf-eiffere and non iocrenccggg surrogate loss function &. For instance, in our experiments, we replace Lagoo with the logistic loss log(1 + exp(-a)).
Relaxation. While Theorem 4.1 provides an upper bound for AUC fairness on the clean groups, our goal is to ensure that (0) S 0, Vz, z' E Z. To achieve this, inspired by Wang et al. (2020), we employ a DRO approach. Specif- ically, any feasible solution to the following constrained optimization problem is guaranteed to satisfy the fairness constraints for the clean groups:
4.3. Noisy Ratio Estimation
min (0),
Theoretical Estimation. The learning objective in Eq. (9) relies on the upper bound 72,s' of the TV distance between Ps.=' and ps..r, as demonstrated in Theorem 4.1. However, in practice, Pa,2' is typically unknown. To address this, we present theoretical result to facilitate the estimation of Tz,,', as detailed below (proof provided in Appendix A.2);
S.L.
max
I (0) 1,, E 2, (8)
: TV (Da.et ) Sf Dere P.P..
where g2.e (0) = (X(X,))x
[h)) and Pe.a
Pz,z' indicates that p=.=' absolutely continuous w.r.t. Pa,a'-
Lemma 4.2. Given a pesitive-negative pair group (z,z'), z'), suppose the prior pairwise clean group probability Pr((2,,)) z²)] unaffected by the noise, i.c., Pr[(Z== = 2, Z, 2, Z' = z')] Then TV (P2.z', S Pr 3') # (? = 2,
Reformulation. To simplify the constrained optimization problem in Eq. (8) and demonstrate its practical applica- tion when the true distributions are unknown, we refor- mulate it as a minimax problem using a Lagrangian for- mulation. Additionally, we replace all expectations with those over the empirical distribution derived from a dataset S : {(X,, Y, )= of n samples. For convenience, denote X; as X,* or X, if it has positive or neg- ative label, respectively. Let n = XXX | E [n]H] and n = H[,, | i E [n]H represent the total num- ber of positive and negative samples, respectively, where [m] = (1.....n), n). Similarly, we denote X, as X or X.- if it belongs to group z with a positive or negative label, respectively. Let n²+ and n² represent the total number of positive and negative samples from group 2, re- spectively. In practice, we let the empirical distribution Po.e' E R²* xn be a matrix where the (i, entry is given by: P.u. = n= n= if the i-th positive sample be- longs to group and the j-th negative sample belongs to group z': otherwise, it is set to O.
=')] = Pr[(Z
According to the above Lemma, the estimation of 72,=' can be reduced to estimating the probability Pr Z') * (2, Ż') (Z 2, Z' = z')]. In robust machine learning, various methods can be employed for this estimation. For example, an auxiliary network can be trained from scratch to estimate this probability (Jiang et al., 2018; Yu et al., 2019), or an auxiliary clean dataset can be utilized for estimation (Kallus et al., 2022; Wang et al., 2020). However, training an auxiliary network introduces additional complexity to the target model, potentially reducing its generalizability across diverse scenarios. Furthermore, many existing approaches rely on single data modality (c.g., images) for estimation, which can limit accuracy and effectiveness. Additionally, obtaining auxiliary datasets that are well-suited to the target problem can be challenging in practice.
The TV distance constraint can then be reformulated to iden- tify an empirical distribution P=.,z' E Bun xx= within ball defined by: B. (P..)) == (pz.2' Σx j==
Empirical Estimation. To address these challenges, we leverage the capabilities of pre- tre-trained multi modal founda- tion models (Li et al., 2024; Gardner et al., 2024) to con- struct noisy label detector, drawing inspiration from recent works (Hu et al., 2023; Wei et al., 2024), without requiring additional training. While we use image data as an illustra- tive example, our method can be easily adapted to other data
Pa
p.d. S 2 En Ej= In+ E []. Denote Ae.s' as the Lagrangian multiplier associated with the pair 2, E 2. Then, the empirical
tij P
1,
p. 0 Vi E
5

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
modalities, such as tabular data, which we leave as future work. Specifically, for each image, utilize its protected group label to design pair of label-specific prompts: pos- itive prompt (P) and a negative prompt (M). For example, P is designed as "a photo of a (group_name))" and N as "a photo without a ggroup.namme]: where (group name) corresponds to the protected group label (Z) of the im- age. These prompts are fed into the text encoder of the CLIP model (Radford et al., 2021) to generate text fea- ture representations T and TW, Simultaneously, the im- age is processed through CLIP's visual encoder to extract its visual feature representation V. We then compute the cosine similarity between the visual features and the text features, resulting in s sim(V, TP) and a mim (W). Fi- nally, we regard the group label of the test image as clean if sim(V, T) > sim(V, J, and noisy otherwise.
Algorithm 1 Robust AUC Fairness
1: Input: A training dataset S of size n, number of itera- tions T, batch size b, learning rates no: 77x 7p. and Ys,"' estimated by Eq. (10)
2: Initialize: 0(1), (1) 2,2 P2,x' for all pairs (z,2')
3: for / to 7 do
4: B Sampler(S.b)
5: Compute c* based on Eq. (11)
6: Compute perturbed parameters: g(D) = B(0) + +c²
7: Update 0: 0((+1) 4- -0(0 - 110 V f g(o)
8:
for each (2, E (11,1),11,2 2),.................. (m, m)) do
9: Update Xo, 2,0" t 2,2 + +1J
(1+1)
x(0)
10: Update P=,z:
22,2' +
mp V
92.
11: Project Pz.e" onto f -orm constraints:
We stress that CLIP is not used for relabeling, classifica- tion, or decision-making, Instead, its strong representational power is utilized to estimate the mismatch rate between protected group labels and corresponding semantic features. This design avoids additional training (e.g., adding MLP heads), thereby maintaining the theoretical guarantees of our fairness framework. Since CLIP predictions are not perfectly reliable, using them for classification would under- mine the provable robustness that our method offers. Based on this estimation, we approximate " follows:
( - -D S 2 (pttt)) 11,1=1
12: end for
13: end for
14: return (((*), where t" is the best iterate satisfying the constraints in Eq. (9) with the lowest objective.
parameters 0, which maximizes the loss. The process is:
E" = arg max (LAUC +
9 (0 + e)
||e IaSu
2=
1/=1
Pr((Z,Z') # (2,2))(2 = 2 z²)]
VOL
arg max E VoL = V | 1ellas
(H(:22) = T {) 1>mim(1) (V," .2. ,,. 2) = .. Siiim() 22j xi}
(11) Here, V controls the perturbation magnitude, and the ap- proximation is derived using a first-order Taylor expansion, assuming is small. The final equation is obtained by solv- ing a dual norm problem, where sign represents the sign function, and voL is the gradient of L with respect to 0. Consequently, the model weights are updated by solving the following optimization problem: ming Z(O + +*). The intu- ition is that perturbing the model parameters in the direction of the gradient norm maximizes the loss value, which in turn encourages the model to explore flatter loss landscape. enhancing its generalizability.
2, = T.. Solm (vp T.t. + ((..5)
(10)
8/ ( T. >>aim() v .,.
+ (((0.0)))===111mm) 2)=='=
2. 2 x: ssimi ! .x. My
/1((1.1) = 2, 23 = 2'H
Here, i E [n+] and j E [n with Vi+ and VjJ as visual feature representations, and T and T as text feature rep- resentations for positive and negative samples, respectively.
To ensure each mini-batch contains both positive and nega- tive samples from all possible groups, we follow Yang et al. (2023) and design a sampling operator (i.e., Sampl er(; )) that randomly selects mini-batch of size by stratifying the training set S based the label and group attribute. The details of the sampling operator are outlined in Appendix B.
4.4. Optimization
Finally, we develop a stochastic gradient descent-ascent (SGDA) method to solve the minimax optimization prob- lem in Eq. (9). To avoid the model becoming stuck in sharp and narrow minima during training, we incorporate the sharpness-award aware minimization (SAM) technique (Foret et al., 2020) to flatten the loss landscape. This flattening is achieved by finding the optimal perturbation e" to the model
The overall optimization procedure is as follows: we first initialize the model parameters 8, Ag.z'. and P2.2'. and use the approach developed in the previous section with Eq. (10) to estimate 7a.2' for all possible positive-negativv group
6

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
pairs (2, z'). Next, we randomly select a mini-batch B using our Sampler operator and perform the following steps for each iteration on B (For more details, refer to Algorithm I);
Implementation Details. All experiments are implemented in PyTorch and trained on an NVIDIA RTX A6000. For training, we set the batch size to 10,000 for socioeconomic analysis and 32 for deepfake detection, with 1,000 and 100 training epochs, respectively. We use the SGD opti- mizer. For socioeconomic analysis, we use 3-layer multi- layer perceptron (MLP) as the model. ? is selected from {0.1, 0.2, 0.3, 0.4, 0.53. For deepfake detection, we use noisy group labels, which are common in datasets like FF++ where demographic attributes are inferred. Evaluating fair- ness under label noise is practical and follows prior work (Celis et al., 2021; Mchrotra & Vishnoi, 2022). We use Xception (Chollet, 2017) and EfficientNet- BB (Tan & Le, 2019) as the detector backbones. y = 0.02 is estimated using Eq. (10). See Appendix F.1 for details.
Compute E" based on Eq. (11).
Update 0 based on the gradient descent: 0 + 0 no VoCloee where T/o is the learning rate.
(m, use gradient ascent to update Az,z' A2,2' +- Xz,2 + 7AA3,.;' (0) + E*) and Ps,z': De,a' 6 Pa,e' Ap V (0...- (0++t)) c"), where 77 and Mp are learn- ing rates. Next, we project Pz,2' onto f1,1-nom con- straints: || - - S 2 1122./ = 1.
For each (2,=') E
The project can be done efficiently with Duchi al. (2008).
5.2. Results
Performance on Tabluar Data. For each of the three tab- ular datasets, we introduce noise into the protected group labels by randomly selecting fraction y of data points and flipping their labels to another group. We evaluate the performance of each method under different noise levels, conducting three random runs per dataset and reporting the mean and standard deviation in Table 1. It is clear that our approach consistently achieves the lowest AUC fair- ness violation and the highest Min/Max AUC score across all datasets and noise levels, demonstrating its effective- ness in preserving faimess under noisy protected group labels. For instance, at noise level of 0.5 on the Adult dataset, our method achieves a faimess violation of 0.0316, 6,, significantly lower than AUCMax (0.0766). InterFairAUC (0.0374), and ManimaxFairAUC (0.0375). While AUCMax attains a higher overall AUC due to the absence of fair- ness constraints, its fairness violation remains substantially higher. Similarly,
5. Experiments
5.1. Settings
Datasets. In experiments, we train models with different methods on both tabular and image datasets. For tabular data, we conduct socioeconomic analysis on three widely used datasets in fair machine learning research (Donini et al.,, 2018): Adult (protected attribute: gender), Bank (protected attribute: age), and Default (protected attribute: gender). Each dataset is randomly split into training, validation, and test sets in a 60%/20%//25s ratio. For image data, we focus on the deepfake detection task using datasets from Lin et al. (2024). Specifically, we train models on the FF++ (Rossler et al., 2019) training set (protected attribute: gender) and evaluate them on the test sets of FF++, DFDC (dee), DFD (Google & Jigsaw, 2019), and Celeb-DF (Li et al., 2020). Further details are provided in Appendix C.
noise level of 0.1 the Default dataset, our method achieves 0.0187 fairness violation, re- ducing it by 5.14% compared to AUCMax. Moreover, as the noise level increases, baseline methods exhibit wors- ening fairness violations, whereas our approach maintains superior fairness performance, underscoring its robustness in handling noisy protected group labels.
Evaluation Metrics. For utility. we use overall AUC as the primary model performance metric. For fairness, we mea- sure AUC fairness violation ('Violation', lower is better), defined as the maximum absolute difference between any group-level AUC score the overall AUC score. Additionally, following Yang al. (2023), we use the Min/Max faimess metric (higher is better), defined as the ratio of the minimum to the maximum group-level AUC score. Their formulations can be found in Appendix D.
Performance on Image Data. Deepfake detection datasets are inherently noisy due to inaccuracies in demographic an- notations, as the protected groups of generated faces cannot be verified in practice. Thus, our proposed noisy estima- tion method is well-suited for estimating the noise ratio, which we determine to be 0.02 in our experiments. We fix this value for subsequent experiments and report the results in Table 2. From the table, we find our method achieves the lowest group AUC fairness violation across all datasets, demonstrating superior fairness preservation under noisy protected groups. For instance, on FF++ (Xcep- tion backbone). our method achieves a faimess violation of
Baselines. For socioeconomic analysis, we compare our method against AUCMax (without fairness constraint) (Yang et al., 2023), InterFairAUC (Vogel al., 2021), and MinimaxFairAUC (Yang et al., 2023). For deepfake detec- tion, we compare our method with the latest fairness meth- ods, including DAG-FDD, DAW-FDD (Ju et al., 2024), and PG-FDD (Lin et al., 2024). The comparison also includes 'Original' (a backbone with cross- cross-cntropy loss). More de- tails can be found in Appendix E.
7

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Nobe: Level
Method
AUCC
Vinlation
Adult
MinMMax
ALC
Violation]
Bank
MinMMax
AUC
Violation
Defullt
MinMax
AUCMas
0.019
007877100018
09115:00012
0.928K
0.0003
011565100016
0.8599000014
0.7762:0.000
00701
0.9076-000011
0.1
MinitaaFFaAAAC
IniecharAACC
09001000000
09029=000000
0.03801000013
000050=0000003
09479_00077
0000
09148=0 0006
0,9068
0005
0.0975
0.9018_000022
0.75%6=00003 08795-00054
0.7399-000031
0000
0.02
0,9619_000009
0.952000.099
Durs
0906)-000002
0.032-00.0077
0.95%--00
0.0005
0007
0JW76--00008
0,9143
0.0029
0.7645-0.001
0.1187-00001
0.9691 -J0019
AUCMax
69112_0.602
00085100011
0,9051
00000
0.9264
01592 1000026
0,85E3
00022
0.0000
0,0565 100011
0,9251_
0.2
Miri
ImeeFarAAUC
09845-000003
09035=000004
00369+000001
00877=000003
0905
0.489-0.0005
0008
009142-00006
0.1021 =00028
01226++0/046
0.8814-00004
0 0.9
0.7526-0.0099
00226
00267+00007
0.9558-00009
0.950=000002
Ours
0.90990000001
0.002810.0006
0,9539 0.0011
0.9163.10 0005
0.10531000026
0,8988
0.0024
0.7624-0.0016
0.0206 10.001
0.9660
AUXM
0.150000.000
008000+0.0005
0.017-000004
0.9203-0.0005
011565+0.0066
0.859-00.0023
0.7768-000006
0.0679+00001
0.9100-0.0015
InterFairAUC
0.8990100.0008
0.007910.0012
0,9182 2.0022.000032
0:0000
0.099:10..000
0,9031
0.7:05_000027
0.02171000009
0,9681100.0019
0.3
MinimasxaaiiAU
0.8977:0.0006
0:0367:0.0013
0 0.990-0.0025
0,9142:00 0006
0.1207:0.0050
0.8853-0.00-0
0.7538::00063
000290:000077
0.9097-000097
Ous
0.036610.0006
0.9513_0.0007
0,9093000000
0.0949 10,009
0,9090-00.0024
0.7687-00001
0.129-10.011
0,9785_000114
AUC 691311000002
007133100007
0,9123 0000
19275_.6.000
016655100025
0.8265_00022
0.7955000000
0.017 100000
0,9035
0.000
0.4
MinimaxFFaAADC
ImsecarAAC
0.9987000008
09056=000002
0008640000022
00077++0.0007
0.93
0.948-000022
09146 0005
0.9000=00000
00944-000000
01246+000066
08795-000055
0:9078=0000099
0.7538-000088
0.7524
0.0256200029
+0.0000
0,059
0.9520000099
Ours
0.0002000002
0.004100.000
0.95M0-0.004
00004
0JW377:00000
0,9086
0.8028
0.7552:000002
A0246:000000
0,9590
0.0066
AUC 691270000003
010/66100000
09141
00005
0.000
011022000017
0.7707 0,053
000155000155
0,017
0.5
FatrAU
09012=000012
0.90122000008
0L0875+-00016
000874+000013
0:9422-00022
0.9482-00005
0.000
0
0:9145:00006
0006
019977000177
0.122900.0022
0.8811-00006
0.2488=0 0.16
003377000037
0.0278
000271+000069
0.9553-0.0095 0.0098
0.9002-0.0027
Ours
0.898-10000066
0.0316.10.0002
0.9554_..0000
0.9094 0004
0.0676.000020
0.9148_0.000
0.7467.000007
0.024310.0001
0,9571_0.000
Table 1. Performance comparison across different noise levels (0.1-0.5). The numbers are reported as 'Mean + Standard Deviation. means higher is better and means lower is better. The best results are shown in Bold.
Backbone
Method
HP4+
DEDC
D2FD
Celeb-IF
AUC1
Violation
MinAMax
AUC-
Violation
Min/Maxi
AUCT
Violation
Min/Max
AUCS
Violation
Min/Max
Original
0:9344
0.0303
0.9652
0.5953
(19492
0,7574
0103376
0.9529
0,6660
0.1477
0.8101
DAG-FDD
0.9628
0.0147
0.9776
0.6058
0.0229
0.9608
0.7770
0.0216
0.9633
0.7059
0.1730
0.7906
Xception
DAW-FDD
0.9650
0.0288
0,9702
0.6037
0.0201
0.9684
0.7815
0,0312
0,9479
0,7092
0.1818
0.7792
PG-FDD
0.9708
0.0111
0,9714
0.6207
0.0184
0.9594
0.8025
0.0113
0.9846
0.7214
0,1412
0.8350
Ours
0.9644
0,0090
0.9857
0.6086
0.0048
0,9930
0.7847
0,0069
0,9881
0,7108
0.0729
0.9117
Original
0.9332
0.0209
0.9684
0.5982
0,0306
0.9320
0.7593
0.0443
0.9322
0.6692
0,2453
0.6962
EfficientNet -B4
DAG-FDD
0.9563
0.0100
0.9869
0.6030
0,0372
0,9210
0.7706
0,0216
0,9626
0,7102
0.2196
0.7415
DAW-FDD
0.9691
0.0169
0,9761
0.59-11
0.0254
0.9119
0.7756
0.0380
0,94110
0.7345
0,2873
0.6792
PG-FDD
0.9721
0.0144
0.9784
0.6043
0.0235
0,9476
0.8033
0,0260
0,9650
0.7366
0.1356
0.8373
Ours
0.9766
0.0061
0.9907
0.6172
0.0136
0.9771
0.8184
0.0135
0.9760
0.7351
0.0928
0.8876
Table 2. Performance comparison on deepfake detection task. f means higher is better and + means lower is better.
0.0090, significantly lower than PG-FDD (0.0111), DAW- FDD (0.0288), and DAG-FDD (0.0147). Similarly, in the DFDC cross-domain scenario, our method attains a violation of 0.0048, reducing fairness violation by 1. 1.36% compared to PG-FDD. This trend persists across DFD and Celeb-DF, where our method consistently maintains the lowest faimess violation. While PG-FDD achieves higher AUC it is the ofate-of-the--at method for fairness generalization in deep- fake detection, its fairness violation remains higher than ours across all datasets. Overall, our results demonstrate superior fairness preservation under noisy groups across different datasets and model backbones, indicating the robustness of our method in real-world image analysis applications.
by its performance across both tabular and image datasets.
Correctness of Noise Ratio Estimation. To evaluate the accuracy of noise ratio estimation in the image-data scenario, we experiment with multiple noise levels (7 e (0.01, 0.02, 0.03, 0.04, 0.05)), as the exact noise ratio is unknown. We apply our method to train deepfake detectors using an EfficientNet-BB B4 backbone and test on the FF++ dataset. As shown in Fig. 3 (Right), our method achieves the lowest faimess violation at 7 = 0.02, which aligns with the estimated y derived from Eq. (10). This consistency validates the correctness of our noise ratio estimation.
5.4. Ablation Study
5.3. Sensitivity Analysis
With vs. Without SAM. We conduct experiments on im- age datasets using the EfficienNNet-B4 backbone without applying SAM during training and report the results in Ta- ble 3. The results show a performance decline (e.g., 0.9656 Min/Max on DFDC) compared to our full method (e.g.. 0.9771), yet it still outperforms the baseline approaches in Table 2. This highlights the importance of SAM in enhanc- ing AUC faimess generalization in our approach.
Inter-/Intra- Group Performance Gap. We examine the inter-group and intra-group AUC gaps across multiple datasets to assess the capability of our method in preserving fairness under noisy groups. As shown in the left of Fig. 3, our method reduces the intra-group AUC gap by 3.40% compared to AAC on the Bank dataset and by 1.70% compared to PG-FDD on the Celeb-DF dataset. These results demonstrate that our method effectively mit- igates both intra-group and inter-group AUC disparities simultaneously in the presence of noisy labels, as evidenced
Robust vs. Non-robust. Under the same experimental setting, we compare our full method with a version that excludes robustness by using Eq. (6). As shown in Table
8

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
1.0
Bank Fieggrup
1.0
Bant, Intragruu
1.
Coleb-DF, group
1.0
intregroup
AUI Rimess Villtion
0.5
0.5
C.6
AUS Gar
AUC Gane
Oea
Grop
0.4
0.4
0.4
Dun: 1,00
Maseadarsi:::
MiseaFartc:: t12
P0-00:
CRTT
PO
-000AT LP
30006
L7
0.
60
02
30006T- 1.7
0.4
0.6
0.3
1.2
0.5
0.2
30016
0.6
0.6
2-
1/2
07
NCATA6T 1,5 (o
01.
05
170
0.2
0.8
0000
0.50
Figure 3. (Left) Comparison of AUC gap on inter-grond group and intra- group across different datasets. For the tabular dataset, we compare our method with FanAM on the Bank dataset under noise level of 0.1. For the image dataset, we compare ours with PG-FDD on Celeb-DE. (Right) AUC fairness violation across different values. 7y has been set manually from {0.01, 0.02, 0.03, 0.04, 0.053.
Buckbone
Method
DFDC
limitation, we aim to extend our approach to other pairwise ranking metries (Yang, 2022) (e.g., partial AUC, average precision) to enhance their group-level fairness.
AUCT
ALC]
Visazonn
MiniMas]
EfficiencNet
B4
0.6122
8.0000
0.008
030133
0.0136
0.971
09K36
0.8055
0.8184
0,0299
0.911]
0,9760
Table Performance comparison between with and without SAM on our method. f means higher is better and means lower is better. The best results are shown in Bold.
Acknowledgements
We thank anonymous reviewers for constructive comments. This work is supported by the U.S. National Science Foun- dation (NSF) under grant IIS-2434967 and the National Artificial Intelligence Research Resource (NAIRR) Pilot and TACC Lonestar6. The views, opinions and/or fndings expressed are those of the author and should not be inter- preted as representing the offcial views or policies of NSF and NAIRR Pilot.
2, the absence of robustness leads to significant perfor- mance drop. For instance, the DFDC dataset, the fairness violation increases by approximately 0.32%, highlighting the effectiveness of our proposed robust approach. More experiments are in Appendix F.2.
6. Conclusion
Impact Statement
Existing methods for enhancing AUC fairess perform well with clean protected groups but fail under noisy groups. To address this, we propose a novel BRO-based approach with theoretical fairness guarantees, achieved by bounding the TV distance between elean and noisy group distributions. We estimate this bound through theoretical analysis and then develop an empirical method leveraging pre-trained multi-modal foundation models. Finally, we design ef- ficient SGDA algorithm to optimize the proposed learning objective, improving both AUC fairness and model general- ization. Experimental results in diverse datasets highlight the superior AUC fairness maintenance capabilities of our method in three application scenarios.
Our work addresses a critical and underexplored challenge in machine learning faimess: how to ensure based fairess when protected group labels are noisy or unreliable, a condition common in real-world datasets such as survey data and demographically annotated synthetic media. By proposing the first theoretically grounded, DRO framework tailored to AUC fairness under noisy protected groups, our research expands the practical reliability and applicability of fairness-aware machine learning.
The potential societal benefits are substantial. In domains like healthcare, finance, and digital media, where both per- formance and fairness are paramount, our method offers a robust solution that can help prevent algorithmic discrimi- nation, especially under imperfect data labeling conditions. For example, it can reduce harm in risk-sensitive applica- tions such as medical image analysis deepfake detection by ensuring more equitable outcomes across groups
Limitation. Although our approach guarantees AUC fair- ness, ensuring both utility and faireess simultaneously when training with noisy groups remains open challenge. Addi- tionally, while we use CLIP as an example of a multi-modal foundation model for noise estimation in image data, we do not imply that CLIP is directly applicable to tabular data. Rather, our statement refers to the potential to leverage domain-specific foundation models, such as those for tabu- lar data, for analogous noise estimation tasks. Since we do not instantiate this component for tabular data in our current experiments, we acknowledge the lack of a concrete noise estimation strategy for tabular modalities as limitation of this work and leave its exploration to future research.
Ethically, our approach prioritizes fairness even when data quality is compromised, an important step toward respon- sible AI. However, the broader implications of deploying fairness-aware systems should still be evaluated in context, especially when group identities are inferred estimated.
We believe our contribution helps move the field toward more trustworthy and fair machine learning models, even under real-world constraints.
Future work. In addition to addressing the aforementioned

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
References
Gardner, J., Perdomo, J. C., and Schmidt, L. Large scale transfer learning for tabular data via language modeling. arXiv preprint arXiv:2406. 12031, 2024.
Deeplake detection challenge. https://www, http://wwww.ggl...eee/ee Ac-
cessed: 2021-04-24.
Ghazimatin, A., Kleindessner, M., Russell, C., Abedjan, Z., and Golebiowski, J. Measuring faimess of rankings under noisy sensitive information. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2263-2279, 2022.
Asuncion, A., Newman, D., et al. Uci machine learning repository, 2007.
Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L.,, Wu, Y., Heldt, L., Zhao, Z., Hong, L., Chi, E. H., al. Fairness in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2212-2220, 2019.
Ghosh, A., Kvitca, P., and Wilson, C. When fair classifi- cation meets noisy protected attributes. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 679-690, 2023.
Caton, S. and Haas, C. Fairness in machine learning: A survey, ACM Computing Surveys, 56(7):1-38, 2024.
Google and Jigsaw, Deepfakes dataset by google & jigsaw. In https: /ttps://ai..o www.geboog.oooooooooooooo attps:/aa..ooooleeboo..oooo com wtt.ccccccc/ccc contri butt totprr/ngg-e---ttteeeeeeeeeeeeee html, 2019.
Celis, L. E., Huang, L.,, Keswani, V., and Vishnoi, N. Fair classification with noisy protected attributes: A frame- work with provable guarantees. In International Con- ference Machine Learning, pp. 1349-1361. PMLR, 2021.
Guo, H., Hu, S., Wang, X., Chang, M.-C,, and Lyu, S. Robust attentive deep neural network for detecting gan- generated faces. IEEE Access, 11.25574-22588 2022.
Chollet, Xception: Deep learning with depthwise separa- ble convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251- 1258, 2017.
Gupta, M., Cotter, A., Fard, M. M., and Wang, S. Proxy fairness. arXiv preprini arXiv: / /212, 2018. URL abwp
Hanley, J. A. and McNeil, B. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
Cortes, C. and Mohri, M. Auc optimization vs. error minimization. Advances in neural information processing systems, 16, 2003.
Dixon, L., Li, J., Sorensen, J., Thain, N., and Vasserman, L. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67-73, 2018.
Hashimoto, T., Srivastava, M., Namkoong, H., and Liang, P. Fairness without demographics in repeated loss minimiza- tion. International Conference on Machine Learning, pp. 1929-1999 PMLR, 2018
Donini, M., Oneto, L., Ben-David, S., Shawe- Tayyor, J. S., and Pontil, M. Empirical risk minimization under faimess constraints. Advances in neural information processing systems, 31, 2018.
Hu, P., Sun, X., Sclaroff, S., and Saenko, K. Dualcoop++: Fast and effective adaptation to multi-label recognition with limited annotations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Duchi, J., Shalev-Shwartz, S., Singer, Y., and Chandra, T. Efficient projections onto the I 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pp. 272-279, 2008.
Hu, S. and Chen, G. H. Fairness in survival analysis with distributionally robust optimization. Journal of Machine Learning Research, 25(246):1-85, 2024.
Jiang, L., Zhou, Z., Leung. T., Li, L.-J,, and Fei-Fei, L. Men- tornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313. PMLR, 2018
Duchi, J. C. and Namkoong, Learning models with uni- form performance via distributionally robust optimization. The Annals of Statistics, 4973):1378-1406 2021.
Edwards, D. A. On the Santorovich-rubinsee theorem. Expositiones Mathematicae, 29(4):387-398, 2011.
Ju, Y., Hu, S., Jia, S., Chen, G. H., and Lyu, S. Improv- ing fairness in deepfake detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 4655 4665, 2024.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- aware minimization for efficiently improving generalization. arXiv preprint 41114 2020.
10

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Kallus, N. and Zhou, A. The faimesss of risk scores be- yond classification: Bipartite ranking and the xauc metric. Advances in neural information processing systems, 32, 2019.
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Mehrotra, A. and Vishnoi, N. Fair ranking with noisy pro- tected attributes. Advances in Neural Information Pro- cessing Systems, 35:31711-31722 2022.
Kallus, N., Mao, X., and Zhou, A. Assessing algorithmic fairness with unobserved protected class using data com- bination. Management Science, 68(3):1959-198) 2022.
Narasimhan, H., Cotter, A., Gupta, M., and Wang, S. Pair- wise faimesss for ranking and regression. In Proceedings of the AAAI Conference on Artificial Intelligence vol- ume 34, pp. 5248-5255, 2020.
Kenfack, P. J., Kahou, S. E., and Aivodji, U. A sur- vey on fairness without demographics. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL
Pu, W. Hu, J., Wang, X., Li, Y., Hu, S., Zhu, B., Song, R., Song, Q.. Wu, X., and Lyu, S. Learning deep dual-level network for robust deepfake detection. Pattern Recognition, 130: 108832, 2022.
id=3HE4VPNIEX,
Kollias, D., Arsenos, A., and Kollias, Domain adaptation explainability & fairness in ai for medical image analysis: Diagnosis of covid-19 19 based on 3-d chest ct-scans. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4907-4914 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Krumpal, I. Determinants of social desirability bias in
sensitive surveys: literature review. Quality & quantity, 47(4):2025-2047 2013.
Rawls, Justice as faimess: A restatement.
Erin
Kelly/Harvard University, 2001.
Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., and Fujiwara, Y. Auc maximization under positive distribu- tion shift. In The eiirty-eighth Annual Conference Neural Information Processing Systems.
Rockafellar, R. T., Uryasev, S., et al. Optimization of condi-
tional value-at-risk. Journal of risk, 2:21-42, 2000.
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thics, J., and Nießner, M. Faceforensics++: Learning to de- tect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1-11, 2019.
Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F., Thain, N., Wang, X., and Chi, E. Fairness without demograph- ies through adversarially reweighted learning. Advances in neural information processing systems, 33:728-740 2020.
Tan, M. and Le, Q. Efficientnet: Rethinking model scal- ing for convolutional neural networks. In International conference on machine learning, pp. 100561144 PMLR, 2019.
Levy, D., Carmon, Y., Duchi, J. C., and Sidford, A. Large- scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33: 8847-8860, 2020.
Tian, Y., Wen, C., Shi, M., Afzal, M. M., Huang, H., Khan, M. O., Luo, Y., Fang, Y., and Wang, M. Fairdomain: Achieving fairness in cross-domain medical image seg- mentation and classification. In European Conference on Computer Vision pp. 251-271. Springer, 2025.
Li, C., Gan, Z. Yang, Z., Yang, J., Li, L.., Wang, L.,, Gao, J., et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-214, 2024.
Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, Celeb-df: A new dataset for deepfake forensies. In CVPR, pp. 6,7, 2020.
Vogel, R., Bellet, A., and Clémençon, S. Learning fair scoring functions: Bipartite ranking under roc-based fair- ness constraints. In International conference artificial intelligence and statistics, pp. 784-792. PMLR, 2021.
Lin, L., He, X., Ju. Y., Wang, X., Ding, F., and Hu, S. Pre- serving fairness generalization in deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 16815-16825, 2024.
Wang. S., Guo, W., Narasimhan, H., Cotter, A., Gupta, M., and Jordan, M. Robust optimization for fairness with noisy protected groups. Advances in neural information processing systems, 33:5190-5200 2020.
Lin, L.,, Santosh, Wu, M., Wang. X., and Hu, S. Ai-face: A million-scaa scalle demographically annotated ai- di-generaeed face dataset and fairness benchmark. Proceedings of the
Wei, T., Li, H.-T., Li, C., Shi, J.-X., Li, Y.FF., and Zhang, M.-L. llnguanl models are strong noisy label

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
detectors. Advances in Neural Information Processing Systems,
2024.
Williamson, R. and Menon, A. Fairness risk measures. International conference on machine learning, pp. 6786- 6797. PMLR, 2019.
Yang. T. Deep auc maximization for medical image clas- sification: Challenges and opportunities. arXiv preprint arXiv:2/11.0224 2021.
Yang. T. Algorithmic foundation of deep x-risk optimization.
arXiv preprint arXiv:2206.00453 2022.
Yang, Z., Ko, Y. L.., Varshney, K. R., and Ying, Y. Minimax auc fairness: Efficient algorithm with provable conver- gence. In Proceedings of the AAAI Conference Artifi- cial Intelligence, volume 37, pp. 11909-11917, 2023.
Yao, Y., Lin, Q., and Yang, T. Stochastic methods for optimization subject auc-based fairness constraints. In International Conference on Artificial Intelligence and Statistics, pp. 10324-10342. PMLR, 2023.
Yeh, I.-C. and Lien, C.-h. The comparisons of data min- ing techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2):2473 2480, 2009.
Yu, X., Han, B., Yao, J., Niu, G., Tsang, I., and Sugiyama, M. How does disagreement help generalization against label corruption? In International conference on machine learning, pp. 7173. PMLR, 2019.
Yuan, Z. Yan, Y., Sonka, M., and Yang, Large-scale robust deep maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 3040-3049, 2021.
Zhang, C., Shi, W., Luo, L., and Gu, B. Doubly robust auc optimization against noisy and adversarial samples. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3195-3205, 2023.
12

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
A. Proofs
A.1. Proof of Theorem 4.1
Our proof is inspired by Wang et al. (2020); however, the key distinction lies in the focus. While Wang et al. (2020) addresses non-ranking- baanking-based fairness measures, we derive results specifically for pairwise ranking- based AUC fairness measures. First, we define TV distance between Ps,s' and Ps,=' follows,
Definition A.1. Let m(x, y) Lanty a metric, and и represent a coupling between the probability distribu- tions Ps,s' and Ps.z'. The TV distance is defined as (P..,. inf, Ex [mnXX, Y)] s.t. [2 Pa (v), at (2, y)dr ())
Then, introduce Lemma as follows.
Lemma A.2. (Edwards, 2011). A function h is called Lipschitz with respect to m if (((z) - h(y)l S m(x y) for all x, y.
and let Hm) denote the space of such functions. If m a metric, the Wasserstein distance can be expressed as:
We(PE.et (2,x))
sup Ex-p... [x(x)] - Ex- [x(X]. NEH(()
Now, let m(x,y) y) = Mayy. this case, the Total Variation (TV) distance becomes:
TV = sup Ex-p... [x(X)] - Ex [h(X)]
1=110
Finally, we prove Theorem 4.1 follows,
Proof. For any pairwise group labels 2, 2/,
(0) - -2.2' (0) (0) + (0) S (0) (0)+ -=...(0))
By Lemma A.2, we have the following result.
(0) (0) (+) Z 7' - 2, = 2'7l s (s.
Given the assumption that 0 satisfies the fairness constraints with respect to the noisy groups, (0) S 0. Therefore, derive the desired result:
(0) < P2,2') S Ta.z'
A.2. Proof of Lemma 4.2
Proof. The Total Variation (TV) distance between the probability measures Pa,s! and pa. is defined as:
TV (V Pe.e)) -upl::s (A) (A)) A is measurable event).
Let be any measurable event under both Pz,z' and Pr,z' By the definition Pz,z', we have (A) Pr[A] (2,2))
13

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Algorithm 2 Samp lerelerrraai (Daattset:: S, batch size: b)
1: for z E 2 and E 3/ do
2: Uniformly sample without replacement B=Y from SaY with size ba [b- (|S**//|S1)
3: end for
4: return B = U2y
(2, 2.) In the context of the pairwise problem between Pz,s' and Pa,st, it follows that:
(A) - (A))
= |Pr[AA|
= ( (x.3)))- (2,z')1
=|Pr|A| (Z.Z') = (2.)) 22) = (2,2)) (Z,Z') = (2,2')
+ P (2,2') =
-Pr(A| (2,2") = (x.1)1222)=xxx)))(((22))))))) (2,2) (Z,Z") = =(2,2')
(6,2)((2, = (2,22']
-Pr(A|(2,2 = (t,
(z,z)) ((,2) = (2,2')l
= |PrjA|(Z,2)) = =2,,)) =
(Pr((2,2) - = (s,,²)] ((:)))(
= (2.)) (z,
- Pr(((2 21) # ((,2) = -x,,2)]
(Pr(A | (2.))) (2,=',
(2,2') - Pr[A| (2,2) =
# (1))
= 10
=
(Pr(A (Z,ZZ)
= (2,2),,,, # (2, Pr|A| (2,2)) = (z,)) * (2))
S 2r( * (2, 2) [2, - -r(() + (2, 2")
= (z,z²))
The second equality follows from the law of total probability. The third and the fourth equalities follow from the assumption
that Pr[ 2') = (z, 22) ) = Pr[ 2') = 22). which implies that Pr[ 2') = (2, 2') | (Z, (') = (z,z/) =
Pr[(Z, Z²) = (2. (2, 2') (2,2²)
since
= (z,z')
Pr(( (2, )' (2. (2,
Pr((Z,7))= =
Pr[(2, 2') = (2,77); = (2,
pr((2,2))= = (x,,(()
= -Pr((2, 21 = = = (2,,)((222,2) =22,2')
This further implies that (2, 2') (7, ( (2,2") = (,, the reason why the last equation holds.
= Pr((2, # (2,2) (
21) = (2, 2')], which is
B. Sampling Method
We denote strata of data as S=Y, where E 2 and Y E y. Then, the sampling algorithm is shown in Algorithm 2.
C. Datasets Details
For tabular datasets, we do socioeconomic analysis on three datasets that have been commonly used in the fair machine learning literature (Donini et al., 2018). In Adult dataset, the sensitive attribute is the gender of the individual, i.e. female (Z- a) or male (Z- = 6). In Bank dataset, the sensitive attribute is the age of the individual: Z = a when the age is less than 25 or over 60 and 7 = b otherwise. In fault dataset (Yeh & Lien, 2009). the sensitive attribute is the gender of the individual, i.c. female (Z- = a) or male (Z- = b).
For image datasets, we do deepfake detection on the most widely used benchmark FaceForressic++(FFF+++ (Rossler et al., 2019). DFDC (dee), DFD (Google & Jigsaw, 2019), and Ce (Li et al., 2020). We only used the test set of the later
14

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Type
Name
# Instances
Group ratio
Class ratio
Adult
48,842
0.48:1
3,03:1
Tabular
Bank
41,188
0.05:1
7.55:1
Default
30,000
1.52:1
3.52:1
FF++
76,139
1.27:1
1:4.89
DFDC
22,857
1.05:1
1.80:1
Image
DFD
9,386
1.22:1
1:2.04
Celeb-DF
28,458
8.79:1
1:3.98
Table Dataset Statistics. The group ratio is given by the protective attribute Z b. The class ratio is given by negative positive class.
three deepfake dataset. Since the original datasets do not have the demographic information of each video or image, we follow Ju et al. (2024) for data processing and data annotation. The sensitive attribute is the gender of the individual, i.e. female (Z = a) or male (Z = b). The summary statistics of the datasets are given in Table 4.
D. Fairness Metrics
For fairness, measure AUC fairness violation ('Violation', lower is better), defined the maximum absolute difference between any group-level AUC score the overall AUC score:
max (2,):()))
2,3 EZ
Additionally, following Yang et al. (2023), we use the Min/Max fairness metric (higher is better), defined as the ratio of the minimum to the maximum group-level AUC score:
min AUC,, max A 2
2,
E. Baselines
The AUCMax algorithm maximizes AUC across the entire dataset without distinguishing between groups. It updates the model parameters using mini-batch SGD.
We select the method by Vogel et al. (2021) as a representative since they considered the same datasets. Their approach, which we refer to as InterFairAUC, ensures fair AUC scores by regularizing the difference between inter-group AUCs.
Mani introduced by Yang al. (2023) is minimax fairness framework that simultaneously addresses intra group and inter-group AUC disparities using Rawlsian approach, sup- ported by efficient optimization algorithm with proven convergence guarantees.
DAG-FDD (Ju et al., 2024), a demographic-awwe aware Fair Deepfake Detection (DAW-FDD) method leverages demographic information and employs an existing fairness risk measure (Williamson Menon, 2019). At high level, DAW-FDD aims ensure that the losses achieved by different user-specified groups of interest (e.g., different races or genders) group another) and, moreover, that the
similar to each other (so that the Al face detector is not accurate losses across all groups are low. Specifically, DAW-FDD uses a CVaR (Levy et al., 2020; Rockafellar et al., 2000) loss function across groups (to address imbalance in demographic groups) and, per group. DAW-FDD uses another CVaR loss function (to address imbalance in real vs A Al-generated training examples).
DAD (Ju et al., 2024), demograppiic-agnosicc Fair Deepfake Detection (DAG-FDD) method, which based on the distributionally robust optimization (DRO) (Hashimoto et al., 2018; Duchi & Namkoong, 2021). To use DAG-FDD, the user does not have to specify which attributes to treat sensitive such as race and gender, only need to specify probability threshold for a minority group without explicitly identifying all possible groups.
PG-FDD (Lin et al., 2024) (Preserving Generalization Fair Deepfake Detection) employs disentanglement learning to extract demographic and domain-agnoste forgery features, promoting fair learning across a flattened loss landscape.
15

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Noise
Methed
AUC1
Adall
Min/Max
Bank
Default
Ours(Non-rubxs)
0.906400.000
00000000:000
0953510.0006
0.909:10:0001
0.000
091022000005
0.75
00012:10.0066
09600000011
0.1
Ours (Robust)
0.0000000002
0,302-0.0007
0.95361000005
0909910:0007
0.0066
0.1433000029
0.764510.0017
00187.000011
0.9611-0.001
0.2
Oas(/on-sohast)
Ours (Robust)
0.909+0.0004
09008203009
0.0328- 0.A00S
036325000000
0.959--0.001
09163+000000
09107=00:000
0.1053+0.0076
0.8868
8888-+000022
0.7634+0.0006
0.7511=00006
0.0200=000000
-0.004
09626200001
Oux(Non-sohrcc)
00007700
0.9562 000500-00015
0:000:+000012
0.093+0000226
000555000025
0.7500=0.0002
01021 100,0010
0.9647
Ours (Robusl)
0.5900000005
1.055-000000
0003
09003:0.0004
0.999:0.0026
0.900010.002
0.8870000001
00029:000011
0
0.4
Oun rubust)
Ours (Robust)
0:000=000000
0.90510.0012
0,
000770000000
0,550
0562210.000
050542000004
0910810:0011
00.1110000100
0.908510.000
0.986-20.0025
0.7552600002
07442900.008
0.066
00265510.000
0.:991000015
0.5
OuzsNNon robust)
Ours (Robust)
0.89990000011
0.8%64=000000
0.316=00.000
0.054
0.555100.000
091081000122
0:904400::004
0.8676=0.0020
09120100000
0.9145+00015
0.747=00.0022
0.70055000001
0.0243-000014
010261 100611000000
0.9571=0.0100
095662000001
Table 5. Performance comparison across different noise levels (0.1-0.5) between robust and non-robust method of ours. The numbers reported as 'Mean + Standard Deviation," means higher is better and &. means lower is better.
Backbone
Method
FF44
DFDC
DRD
Celeh-DP
AUCT
Vinlation
Min/Max]
AUCT
Violation]
Min/Max
ALCT
Violaton
MinMMa
AUCT
Vialatin
Min/Max|
Xception
Chrss(on--cohust
09546
0.0104
03942
0.0014
D019T
0,9575
0.7826
D.009W
0.7108
0.0934
0.8M0T7
Ours
0.9644
0.0090
0.9857
0.6086
0.0048
0.9930
0.7847
0.0069
0.9881
0.7108
0.0729
0.9117
BfficiewNer-B4
(hrssNon-rohuss)
0.7799
0.0100
03942
0.9646
0.8058
DA0EN9
DUNTT
0.7350
0.1046
0.8715
Ours
0.9766
0.0061
0.9907
0.6172
0.0136
0.9771
0.8184
0.0135
0.9760
0.7351
0.0928
0.8876
Table Performance comparison of Ours (Non-robust) and Ours on deepfake detection task. means higher is better and means lower is better.
Its framework combines disentanglement learning, fairness learning, and optimization modules. The disentanglement module introduces a loss to expose demographic and domain-agnostie features that enhance fairness generalization. The fairness learning module combines these features to promote fair learning, guided by generalization principles. The optimization module flattens the loss landscape, helping the model escape suboptimal solutions and strengthen faimess generalization.
F. More Implementation Details and results F1. Additional experimental setup details
In Algorithm 1, we explored the following hyperparameters:
Tabular data:
no {0.001, 0.01,
2. y/ E {0.001, 0.25,
3. Tpp E {0.001, 0.13
4. in Eq. (11) is selected from {0.0005, 0.001,
Image data:
1. n7 e {0.0001, 0.0005, 0.001}
yл E [0.0001, 0.0005, 0.005)
3. Tpp {0.0001, 0.0005, 0.0001
4. v in Eq. (11) is selected from 00.7,
F.2. Additional experimental results
Table 5 and table show the results between robust and non-robust methods of ours. In general, our method with using robust approach shows more robust performance than method without the robust approach.
Label setting. In Table 1, fairness metrics use underlying group labels. In Table 2, we use noisy group labels, which are common in datasets like FF++ where demographic attributes are inferred, Evaluating fairness under label noise is practical and follows prior work ((Celis et al., 2021; Mehrotra & Vishnoi, 2022)). We also test the model on a buman-corrected FF++ test set from the Lin et al. (2025), which contains relatively clean labels, As shown in Table 7, our method still maintains the best performance across all fairness metrics.
16

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
FF++ clean
Backbone
Method
AUCT
Violation.
Min/Max
Ori
0,9384
0.0289
0.9691
DAG-FDD
0.9628
0.0128
0.9808
Xception
DAW-FDD
0.965
0.0256
0.9751
PG-FDD
0.9708
0.0105
0.9840
Ours
0.9644
0.0070
0.9894
Table 7. Performance comparison clean version of FF++ using Xception backbone. means higher is better and means lower better.
Backbone
Method
Training Time per Epoch (minutes)
Original
DAG-FDD
6
Xception
DAW-FDD
9
PG-FDD
28
Ours
15
Table 8. Training time comparison on deepfake detection task on FF++ dataset.
Computational overhead. To evaluate the practicality of our approach at scale, we benchmarked training time on the FaceForensics++ dataset, a widely used, large-scale benchmark for deepfake detection. As shown in table 8, our method introduces moderate overhead compared to some baselines but remains significantly more efficient than others, such as PG- FDD. Specifically, our method requires 15 minutes per epoch, which is faster than PG-FDD (28 min) and reasonably close to other baselines like DAW-FDID and the Original model. This demonstrates that our approach remains computationally feasible and scalable in practice, even for large image datasets and backbone models like Xception.
Experiments with extreme settings. As Table 9 shows, even in extremely high noise levels (60%, 70%, 80%, and 90%), method consistently achieves the lowest AUC fairness violation and the highest Min/Max AUC score. These results strongly align with our claims in the paper, demonstrating that our approach maintains robust faimesss guarantees and balanced group performance in highly noisy settings.
Results on more groups. We add one more experiment about evaluating our method on a dataset with more than two protected groups Specifically, in Table 10, we present results the FaceForensicst* (FF++) dataset, where we consider race as the protected attribute, comprising four groups: White, Black, Asian, and Other. This multi-group setting is more challenging than the binary group setting commonly seen in fairness literature. Nonetheless, our method achieves the lowest fairness violation (0.0161) and the highest Min/Max AUC score (0. (0.9850) among all baselines, demonstrating its effectiveness in ensuring AUC fairness across multiple protected groups. These results confirm that our approach generalizes well to settings involving complex, non-binary group structures, such as race.
CLIP for label prediction. We added experiments with one more baseline to address your concern. Specifically, as shown in Table 11, include baseline called Ours (CIII--lbbeld),, where we directly CLIP predict the protected group labels and then train the model based those labels. In contrast, our full method, Ours (robust), uses CLIP only to estimate the group label noise level, not for prediction relabeling. is clear that Ours (robust) outperforms Ours (CLIP-labeled) in all metrics, achieving higher AUC (0.9766 vs. 0.9725), lower violation (0.0061 vs. 0.0089). and better Min/Max fairness (0.9907 vs. 0.9858). This indicates that directly using CL.PP-reeiicedd labels to mitigate the impact of noise during training does not yield optimal performance. More importantly, this baseline approach does not guarantee fairness under the noisy label setting.
Additional Visualizations. To better communicate the chimess-performann tradeoff across all methods in Table we include two efficiency frontier plots in Fig 4 and Fig 5. We used the results in Table 1, which include performance three method, we compute the average AUC, average
tabular datasets: Adult, Bank, and Default. For each fairness-enhande fairness violation, and average Min/Max AUC ratio across the three datasets for each noise level (0.1-0.3). This gives three points per method, each representing the average values across datasets at a given noise level. Fig 4 visualizes Average AUC vs. Average Fairness Violation, demonstrating the tradeoff of performance and fairness. Our method ("Ours") consistently occupies the top-left region, achieving lower violation than all baselines while maintaining competitive or superior AUC. Fig
17

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Noise Level
Default
Method
AUCT
Violation.
Min/Maxt
AUCMax
0.7771=000005
0.0672-10.000+
0.9108
InterFairAUC
0.7548=10.0222
0.0238=0.0039
0.9586=1.0062
0.6
MinimaxFairAUu
0.7550++00001
0.0249-0.0054
0.9577+0.0078
Ours
0.754010.0024
0.0026=0.0027
0.9676=10.0045
AUCMax
0.7788
0.0696=000000
0.9081#0.0012
InterFairAUC
0.7532++0.002
0.0239=0.0018
0.9594+0.0052
0.7
MinimaxFairAUC
0.7545 1002
0.0265=0.0062
0.9557+0.0079
Ours
0.7041
0.0183=0.0033
0.9687+0.0052
AUCMax
0.0687=0.0014
0.9092+0.0017
InterFairAUC
0.7548+10.0022
0.0251=0.0043
0.9566110.0022
0.8
MinimaxFaarrAUC
0.7532++00021
0.0246=#00039
0.9588+0.0072
Ours
0.0045-10.0025
0.0183
0.9688
AUCMax
0.7095+0000006
0.0675=0.0007
0.9108
InterFairAUC
0.7545=0.0022 0.0252=0.0045
0.9567#0.0062
0.9
MinimaxFairAUe
0.7544+100022
0.0267=0.0061
0.9555+0.0008
Ours
0.7527+0.0022
0.0201=0.0031
0.96611000049
Table 9. Performance comparison across high noise levels (0.6-0.9). The numbers reported as 'Mean + Standard Deviation." meons higher better and & means lower better. best results are shown in Bold.
FF++
Backbone
Method
AUC1
Violation.
Min/Max
Ori
0.9445
0.0335
0,9633
DAG-FDD
0.9647
0.0226
0.9798
Xception
DAW-FDD
0.9670
0.0215
0.9753
PG-FDD
0.9751
0.0172
0.9837
Ours
0.9625
0.0161
0.9850
Table 10. Performance comparison the FF++ dataset with race the protected ultribute. better.
means higher is better and means lower
5 shows Average AUC vs. Average Min/Max AUC Ratio, highlighting group-wise faimess consistency. Our method ranks in the top-right region, attains the best Min/Max ratio with strong AUC, reflecting better fairness stability across groups.
To complement the tabular results in Table 2, we include two efficiency frontier plots based on the average values across all four benchmark datasets (FF++, DFDC, DFD, and Celeb-DF). Fig reveals the trade-off between detection performance and fairness violation. Our method ("Ours") achieves the lowest average violation while maintaining higher AUC than all other methods both Xception and EfficientNet-BA backbones. These results position our method the top-left corner, indicating Pareto efficiency and demonstrating strong performance-faireess trade-offs. Fig 7 captures performance versus group-wise fairness consistency. Our method again ranks the top-right region, with both the highest min/max ratio and competitive superior AUC. This confirms that our method not only performs well but also offers greater fairness stability across demographic groups. We note that while the table provides fine-grained per-dataset results, these plots offer a complementary view by highlighting global efficiency across multiple objectives. The strong position of our method in both plots further supports its overall effectiveness.
18

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
FF++
Backbone
Method
AUCT
Violation
Min/Max
Ori
0.9332
0.0209
0.9684
DAG-FDD
0.9563
0.0100
0.9869
EfficientNet-B4
DAW-FDD
0.9694
0.0169
0.9764
PG-FDD
0.9721
0.0144
0.9784
Ours(CLIP-Iabeled)
0.9725
0.0089
0.9858
Ours(robust)
0.9766
0.0061
0.9907
Table Performance comparison + using EfficientNet-B backbone. f means higher better and J. means lower better.
InterFairAUc
0.3
MinimaxFairAUC
0.2
Ours
0.860
0.858
x
0.1
0.1
Average AUC (1)
0.2
0.856
X
0.3
0.854
0.852
0.850
0.0475
0.0500
0.0525
0.0550
0.0575
0.0600
0.0625
Average Violation (4)
Figure 4. Efficiency frontier showing the rofee-off between Average AUC and Average Fairness Violation across three tabular datasets (Adult, Bank, Default) at varying noise levels (0.1-0.3)
19

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
InterFairAUC
0.3
MinimaxFairAUC
Ours
0.860
0.858
0.1
0.1
X
0.2
Average AUC
0.856
0.3
0.854
0.2
0.852
0.850
0.3
0.9275
0.9300
0.9325
0.9350
0.9375
0.9400
0.9425
0.9450
Average Min/Max (t)
Figure 5, Efficiency frontier showing the trade-off between Average AUC and Average Min/Max AUC across three tabular datasets (Adult, Bank, Default) at varying noise levels (0.1-0.3).
20

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Method
EffB4
Original
DAG-FDD
0.78
Xception
DAW-FDD
PG-FDD
Eff-B4
Ours
Average AUC (1) (1)
0.77
X
Eff-B4
Xception
X
Xception
X
Xception
0.76
Eff-84
0.75
0.74
X
EF-84
Xceptian
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
Average Fairness Violation (4)
Figure 6. Efficiency frontier showing the trade-off between detection performance (AUC) and fairness violation across four benchmark datasets (FF++, DFDC, DFD, Celeb-DF).
21

---

Preserving AUC Fairness in Learning with Noisy Protected Groups
Method
Original
DAG-FDD
Xception
DAW-FDD
0.78
X
PG-FDD
Ours
Average AUC (1)
0.77
Eff-B4
Xception
Xception
Xception
0.76
Eff-B4
0.75
0.74
x
Eff-B4
Xception
0.88
0.90
0.92
0.94
0.96
Average Min/Max (f)
Figure 7. Efficiency frontier showing the trade-off between detection performance (AUC) and Min/Max AUC across four benchmark datasets (FF++, DFDC, DFD, Celeb-DF).
22


Reviews:
[{"review_id": "pPrVaHa1uf", "reviewer": "Reviewer_w8bV", "summary": "The paper addresses the critical problem of preserving AUC fairness in machine learning models when protected group labels are noisy. The authors propose a novel distributionally robust optimization (DRO) approach with theoretical fairness guarantees, which bounds the Total Variation (TV) distance between clean and noisy group distributions. They introduce a new AUC fairness metric that accounts for both intra-group and inter-group AUC disparities and develop an efficient stochastic gradient descent-ascent (SGDA) algorithm to optimize the learning objective. The method is evaluated on multiple tabular and image datasets, demonstrating superior performance in preserving AUC fairness compared to state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for providing valuable input. We are glad to answer them below. \n\n**Other Strengths And Weaknesses**. Thank you for your suggestion. We will add a discussion about the potential extensions of our methods to multi-class classification settings. For example, we can explore Multiclass ROC (https://arxiv.org/pdf/2404.13147) and try to integrate our proposed approach into it.\n\n**Questions For Authors**.\n1) **Computational overhead**. To evaluate the practicality of our approach at scale, we benchmarked training time on the FaceForensics++ dataset, a widely used, large-scale benchmark for deepfake detection. As shown in **Table A.1 (https://imgur.com/a/Zh6RaZ7)**, our method introduces moderate overhead compared to some baselines but remains significantly more efficient than others, such as PG-FDD. Specifically, our method requires **15 minutes per epoch**, which is **faster than PG-FDD (28 min)** and reasonably close to other baselines like DAW-FDD and the Original model. This demonstrates that **our approach remains computationally feasible and scalable** in practice, even for large image datasets and backbone models like Xception.\n2) **Experiments with extreme settings**. To answer this question, we add one more experiment, which implements our method on the Default dataset **with extremely high noise levels (60%, 70%, 80%, and 90%)**. As **Table A.2 (https://imgur.com/a/Zh6RaZ7)** shows, even in extremely high noise levels, our method consistently **achieves the lowest AUC fairness violation and the highest Min/Max AUC score**. These results strongly align with our claims in the paper, demonstrating that our approach maintains robust fairness guarantees and balanced group performance in highly noisy settings.", "claims_and_evidence": "The paper makes several important contributions to the field of fair machine learning. First, it provides a thorough theoretical analysis of the impact of noisy protected group labels on AUC fairness, which has been largely overlooked in previous work. Second, the proposed DRO-based approach with TV distance bounding offers a principled way to preserve fairness under noise, supported by both theoretical guarantees and empirical validation. Third, the extensive experimental evaluation demonstrates the effectiveness of the method across diverse datasets and applications, showing consistent improvements over state-of-the-art baselines.", "methods_and_evaluation_criteria": "The work addresses an important problem in fair machine learning with both theoretical contributions and practical implications. The proposed method shows significant improvements over existing approaches in handling noisy protected group labels, which is a common challenge in real-world applications.", "theoretical_claims": "The authors propose a novel distributionally robust optimization (DRO) approach with theoretical fairness guarantees, which bounds the Total Variation (TV) distance between clean and noisy group distributions.", "experimental_designs_or_analyses": "The extensive experimental evaluation demonstrates the effectiveness of the method across diverse datasets and applications.", "relation_to_broader_scientific_literature": "It is the first robust AUC fairness approach under noisy protected groups with fairness theoretical guarantees using distributionally robust optimization.", "essential_references_not_discussed": "The references seem adequate.", "other_strengths_and_weaknesses": "The paper is well-written and presents a comprehensive solution to preserving AUC fairness under noisy protected groups. Some minor improvements could enhance the paper further, such as discuss potential extensions to multi-class classification settings.", "other_comments_or_suggestions": "The work addresses an important problem in fair machine learning with both theoretical contributions and practical implications. The proposed method shows significant improvements over existing approaches in handling noisy protected group labels, which is a common challenge in real-world applications.", "questions_for_authors": "1. Could you provide more details on the computational overhead of your method compared to baseline approaches, especially for large-scale datasets?\n2. How does your method handle scenarios with extremely high noise levels (e.g., >50%) in protected group labels?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w21vk2nuir", "reviewer": "Reviewer_Gfgv", "summary": "This paper proposes a robust AUC fairness approach under noisy protected group with fairness theoretical guarantees using distributionally robust optimization. Also, experiments have been implemented on tabular and image datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for providing valuable input. We are glad to answer them below. \n\n**Claims And Evidence**. We would like to clarify that our paper **does not aim to provide convergence guarantees** for Algorithm 1. The goal of Section 4.4 is to describe the **practical optimization procedure** (based on stochastic gradient descent-ascent) used to solve our minimax learning objective. **Instead, our theoretical contribution lies in the fairness guarantees**, as stated in Theorem 4.1, which provides a **robust fairness bound under group label noise**. Specifically, Theorem 4.1 shows that, under our robust DRO formulation, the true group fairness deviation is controlled by the specified slack variables $\\gamma_{z,z’}$, even in the presence of noisy group labels.\n\n**Experimental Designs Or Analyses**\n1) **Results on more groups**. We add one more experiment about evaluating our method on a dataset with **more than two protected groups**. Specifically, in **Table A.3 (https://imgur.com/a/zNzRnKP)**, we present results on the **FaceForensics++ (FF++) dataset**, where we consider **race as the protected attribute**, comprising four groups: **White, Black, Asian, and Other**. This multi-group setting is more challenging than the binary group setting commonly seen in fairness literature. Nonetheless, our method achieves **the lowest fairness violation (0.0161)** and the **highest Min/Max AUC score (0.9850)** among all baselines, demonstrating its **effectiveness in ensuring AUC fairness across multiple protected groups**. These results confirm that our approach generalizes well to settings involving **complex, non-binary group structures**, such as race.\n2) **Efficiency frontier visualization**. We appreciate your suggestion. To complement the tabular results in Table 2, we include two efficiency frontier plots based on the **average values across all four benchmark datasets** (FF++, DFDC, DFD, and Celeb-DF). Fig A.1 (https://imgur.com/a/zNzRnKP) reveals the trade-off between detection performance and fairness violation. Our method (\"Ours\") achieves the **lowest average violation** while maintaining a **higher AUC** than all other methods on both Xception and EfficientNet-B4 backbones. These results position our method at the **top-left corner**, indicating Pareto efficiency and demonstrating strong performance–fairness trade-offs. Fig A.2 (https://imgur.com/a/zNzRnKP) captures performance versus group-wise fairness consistency. Our method again ranks at the **top-right region**, with both the **highest min/max ratio** and **competitive or superior AUC**. This confirms that our method not only performs well but also offers **greater fairness stability** across demographic groups. We note that while the table provides fine-grained per-dataset results, these plots offer a complementary view by highlighting **global efficiency across multiple objectives**. The strong position of our method in both plots further supports its overall effectiveness.\n\n**Other Strengths And Weaknesses**. As shown in **Table 1**, our approach **consistently outperforms** baselines on fairness metrics, achieving the **lowest AUC fairness violation** and the **highest Min/Max AUC score** across all datasets and noise levels. These results demonstrate that our method **offers a clear advantage** over existing approaches in terms of fairness. Regarding utility (i.e., overall AUC), while **AUCMax** achieves the highest performance due to the **absence of fairness constraints**, our method achieves **competitive or superior AUC compared to fairness-aware baselines**. For example, on the Default dataset, our method outperforms all baselines except AUCMax in terms of AUC at noise levels ranging from 0.1 to 0.4. In this case, our method **does have advantage over baselines**.\n\n**Questions For Authors**. Our work addresses a fundamentally different setting than that of [Kallus & Zhou, 2019]. Specifically, we focus on a problem in **in-processing approach** where the model **has to be trained** with AUC under noisy protected group labels. In this context, fairness must be enforced throughout the optimization process. **By contrast**, post-processing methods, such as the one proposed in [Kallus & Zhou, 2019], **operate after model training**, and **do not have access to the model internals or training dynamics**. Therefore, while [Kallus & Zhou, 2019] presents an elegant post-processing method for fair ranking in clean settings, it **does not apply to the in-processing, noise-robust fairness optimization problem** that we address.", "claims_and_evidence": "No. The submission claims that their approach has fairness theoretical guarantees, but I am not able to find the convergence theory for Algorithm 1 in Section 4.4 Optimization.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs. This paper considers $m$ protected groups, but for the experiments, all datasets have only 2 protected groups. It would be better to see the experimental results on the dataset with more than 2 protected groups. Moreover, compared with tables, I think the efficiency frontier is more appropriate for evaluating the performances of different methods.", "relation_to_broader_scientific_literature": "Compared with literature, the paper considers noisy protected groups and uses distributionally robust optimization to solve AUC fairness problem.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Other Strengths:\n1. The paper is clearly written and easy to follow.\n\nOther Weaknesses:\n1. Based on Table 1, for noisy protected groups, the method proposed in this paper does not have advantage over baselines, which limits the contribution of the paper.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can the post-processing method that is used in [Kallus & Zhou, 2019] be applied to the problem in this paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1D9Vtn9jj", "reviewer": "Reviewer_4pfG", "summary": "The authors consider fair AUC optimization problem. They consider a problem where the sensitive attributes are noisy, which is quantified by a distribution shift. They employ robust optimization technique to account for possible shift in the distribution.\n\nThe optimization method is rather involved and utilises: a) Lagrangian factors b) min-max problems with unknown probabilities over the whole training dataset, in fact over all pairs of positive-negative examples (making it O(N^2) parameters) c) A smoothing technique SAM d) SGDA method.\n\nThe authors conduct extensive evaluations over 3 table datasets and 4 image dataset. The experiments show improvements compared to previous non-robust datasets.\n\nPros\n1. The problem appears to be relevant from practical point of view.\n2. The advantage of the algorithm confirmed through extensive experiments \n\nCons\n1. The formulation of the shift is a little confusing, I think it would be more intuitive to describe the change of Z' conditional on X, rather than the other way round.\n2. The method is rather expensive and I am not sure it is usable for large datasets, since the parameter size (p_{z, z'}) grows quadratically with number of training instances.\n3. Finally, it is not clear why SAM is necessary in the algorithm, in addition to an error in Eq. (11).\n\nDetailed comments:\n\nl27-right: \"mitigate fairness\" - usually one wants to mitigate bias, disparity\n\nl183-left states the gap is quantified by the absolute value |g_{z, z'}|, and the claim is constraint $g_{z, z'} \\leq 0$ suffices. Why is it so? As far as I understand $g_{z, z'}$ is not anti-symmetric, is it?\n\nl182-190-right: Why these distributions cannot be factorized? I'm pretty sure you can write $p = p^{+} \\otimes p^{-}$, what about $p_{z, z'}$? I was wondering if the conditions and formulations can be simplified then.\n\nl195-right: \"suppose the model is trained with Eq. (7)\" the equations defines an objective to be minimised and the constraints. So far no algorithm how to achieve this is discussed. Furthermore, the theorem does not relate the corresponding maximised objectives AUC(theta) under the two constraints.\n\nl240-243-right The sentence does not really make a particular statement. It would be better if the authors state it as a limitation of their approach. There is another limitation to using CLIP, for example one cannot use it to infer location or political affiliation, which is often considered as sensitive attribute. Finally, it is not clear to me, if CLIP's label is regarded as a ground truth, why not use it in the first place?\n\nl279-left Why is local minima such an issue in your case?\n\n\nl289-left Eq (11) the argmax will be $\\nu \\nabla L / \\| \\nabla L \\|$ instead.\n\nl317-left Computer-> compute\n\nl691 in the proof of the lemma, you claim that the marginal distribution of $(Z, Z')$ and $(\\hat{Z}, \\hat{Z}')$ are the same. Was this stated in the main text? Is it confirmed by computations in Eq. (10)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for providing valuable input. We are glad to answer them below. \n\n**Cons**\n1) **Formulation**. Our method models the **feature distribution conditional on group**, i.e., $X|Z$, which is standard in group fairness literature (e.g., Hardt et al., 2016; Madras et al., 2018). Since our fairness objective involves **pairwise comparisons** within and across groups, the constraints are naturally expressed in terms of predictions within each group.\n2) **Scalability and efficiency**. While $p_{z, z'}$ grows quadratically with the number of training instances, this is **inherent to the pairwise AUC formulation**, not a limitation introduced by our algorithm. We address it using an efficient mini-batch strategy (Sec. 4.4) and optimize via SGDA. As shown in Table 2, our method scales well on large datasets like FF++. Training times in Table A.1 (https://imgur.com/a/Zh6RaZ7) show our method remains computationally feasible, comparable to or faster than baselines like PG-FDD.\n3) **Role of SAM**. The SAM is used to show that our developed SGDA method integrates well with advanced optimization techniques and enhances model generalization in practice, improving its applicability and reliability. Notably, our method still outperforms baselines without SAM, as shown in Table 3. \nThank you for pointing out the typo. You are correct, there is a typo in **Eq. (11)**, it should be $\\nu\\frac{\\nabla \\mathcal{L}}{||\\nabla \\mathcal{L}||_2} $. We will correct this in the revised version of the paper.\n\n**Detailed comments**\n1) **Confused expression**. We will change “mitigate fairness” to “mitigate bias” in the revision.\n2) **Constraint design**. Yes, $g(z,z')$ **is not anti-symmetric**, since the underlying distributions of pairs (positive from group $z$, negative from group $z'$) and vice versa can differ. Despite this, we **do not directly constrain** $|g_{z,z'}(\\theta)|$ in optimization because: (1) in our robust DRO formulation, this constraint is made conservative by slack variables (TV bounds $\\gamma_{z,z'}$), which provide robust fairness guarantees even without enforcing symmetric absolute deviations. See **Theorem 4.1** for the guarantee that $g_{z,z'}(\\theta) \\leq \\gamma_{z,z'}$, which shows that fairness deviation is controlled for all group pairs. (2) In practice, absolute deviation constraints like $|g_{z, z'}| \\leq \\epsilon$ are harder to be optimized, whereas using linear constraints (as in $g_{z,z'} \\leq 0$) allows efficient optimization via dual variables and minimax reformulation (see **Eq. 9**).\n3) **Factorize Distribution**. The factorization **does not simplify the formulation in our case** due to the nature of the **pairwise AUC problem**. In our setting, AUC loss is inherently pairwise, and the optimization depends on sample interactions, not marginals. Our current formulation accurately captures the dependencies required to optimize AUC fairness across group pairs.\n4) **Training with Eq. (7) and theorem clarification**. The experimental results about training models with **Eq. (7)** are already shown in **Table 5 and 6 on page 15**. It is clear that our method is better than directly training with **Eq. (7)**. In addition, our theorem is specifically to provide a solution for addressing the fairness guarantee issue corresponding to the model training with **Eq. (7)**. According to the theorem, we **derived the final learning objective Eq. (9), which contains two constraints**.\n5) **Limitation and CLIP clarification**. A. Per your suggestion, we will state the noise estimation approach for tabular data as a limitation in the revised version. B. In addition, note that we did not use CLIP for directly predicting or correcting labels. We use its strong feature representation ability for noise estimation, which has been proven effective in Radford et al. (2021), Sun et al. (2022), Hu et al. (2023), and Li et al. (2024). C. We do **not** use CLIP to relabel or as ground truth. Using CLIP for classification would require training additional MLP heads, increasing complexity and compromising fairness guarantees. Since CLIP predictions are not perfect, they cannot support provable fairness, which is our central goal.\n6) **Local Minima**. Local minima is a common issue in DNN model training. Since our method is very general and could be used for training any DNN models, we consider this issue and use SAM to make our method more applicable in practice.   \n7) **Typo**. We will change “Computer” to “compute” in the revision.\n8) **Assumption**. Yes. The assumption for marginal distribution was stated in **line 271-272 left**. In addition, Eq. (10) is only for estimating the upper bound**. In this case, computation in **Eq. (10)** cannot be used to confirm the assumption.", "claims_and_evidence": "There is extensive evidence of improvement over non-robust algorithms.", "methods_and_evaluation_criteria": "Evaluation criteria includes AUC and the disparity measurements.", "theoretical_claims": "I didn't check proof of Th. 4.1 but it is not surprising to me.\nThe proof of Llemma 4.2. contains a condition that marginal distribution of $(Z, Z')$ and $(\\hat{Z}, \\hat{Z}')$ are the same. Was this stated in the main text?", "experimental_designs_or_analyses": "I did check, but superficially.", "relation_to_broader_scientific_literature": "They extend the existing literate on AUC fairness.", "essential_references_not_discussed": "Discussed.", "other_strengths_and_weaknesses": "Discussed in the \"Summary\" sections", "other_comments_or_suggestions": "Discussed in the \"Summary\" sections", "questions_for_authors": "Discussed in the \"Summary\" sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81RcTh7cew", "reviewer": "Reviewer_evdn", "summary": "UPDATE AFTER REBUTTAL:\nThanks for the rebuttal. Due to the additional experiments a) better visualizing the fairness/accuracy tradeoffs and b) baselining against CLIP labelling, I'm increasing my score from 2 to 3.\n\n########\n\nThis paper proposes an approach to learning models that improve on a fair-AUC metric when protected group information may be noisy. The approach leverages work in distributionally robust optimization (DRO) to optimize over worst case underlying labels given noisy ones. The authors give a bound on disparity given the noise level, and discuss a practical way to estimate the noise level. Empirically, they demonstrate an improvement on AUC fairness using this approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Claims And Evidence**.\n1) **Fairness accuracy trade-off**. The formulation Eq. (7) introduces no explicit hyperparameter balancing fairness and utility. Instead, Eq. (9) uses Lagrangian multipliers $\\lambda_{z,z'}$ learned **automatically via our minimax optimization**, eliminating the need for manual trade-off tuning. Thus, **no fixed fairness–accuracy knob** exists to generate a trade-off curve.\n2) **CLIP usage for noise estimation**. We use CLIP as a feature extractor, not for relabeling. Using CLIP for prediction would require training additional MLPs, increasing complexity, and cannot guarantee 100% label accuracy, especially on deepfake faces. Thus, fairness guarantees would not hold. Also, **noise estimation with CLIP is not our contribution**. We use it due to its empirical strength (Radford et al., 2021; Sun et al., 2022; Hu et al., 2023; Li et al., 2024), not for comparative evaluation.\n3) **CLIP for tabular data**. Our statement suggests that foundation models (like tabular foundation model), not CLIP specifically, could potentially support similar tasks for tabular data. We will revise this for clarity.\n\n**Theoretical Claims**.\n1) **Constraint design**. To clarify, we **enforce $g_{z,z'} \\leq 0$ as a constraint, not a strict inequality**, in line with prior fairness works (e.g., Cotter et al., 2018; Agarwal et al., 2018). The goal of this constraint is to **ensure that each group-pair AUC is close to the overall AUC**. Importantly, as Yang et al. (2023, Eq. 4, https://arxiv.org/pdf/2208.10451) show, overall AUC is a weighted average of group AUCs, so **not all group AUCs can fall below it simultaneously**.\n2) **Lemma 4.2**. In **Eq. 9**, $\\gamma_{z,z'}$ bounds the total variation between $p_{z,z'}$ and $\\widehat{p}_{z,z'}$. Our **Lemma 4.2** gives a method to estimate this. The assumption is only for bounding and is practical, as supported by Wang et al. (2020).\n\n**Experimental Designs Or Analyses**.\n1) See answer 1) in **Claims And Evidence**.\n2) **Label setting**. In Table 1, fairness metrics use **underlying group labels**. In Table 2, we use **noisy group labels**, which are common in datasets like FF++ where demographic attributes are inferred. Evaluating fairness under label noise is practical and follows prior work (Celis et al., 2021; Mehrotra & Vishnoi, 2022). To further address your concern, we test the model on a human-corrected FF++ test set from the latest CVPR 2025 paper (https://arxiv.org/pdf/2406.00783v3), which contains relatively clean labels. As shown in **Table A.4 (https://imgur.com/a/s4whGks)**, our method still maintains **the best performance** across all fairness metrics.\n3) **SAM** is only used to improve model generalization capability instead of solving the issue of noise labels in test sets.\n4) **Fig 3 right**. We estimate noise as approximately 0.02 using Eq. (10). In Fig. 3(right), fairness peaks near this level when sweeping adjacent noise levels, validating the estimation's practical reliability.\n\n**Relation To Broader Scientific Literature**. Thank you for the suggestion. We will expand the appendix to clarify connections between noisy and missing label setups. Noisy label: observed but potentially incorrect $\\tilde{Z} \\neq Z$. Missing label: unobserved $Z$. Noisy labels generalize the missing-label case (with max uncertainty). Our DRO-based method naturally extends to both by modeling deviations from the true label distribution.\n\n**Other Strengths And Weaknesses**.\n1) **SAM**. Yes, **SAM is applied in every iteration** via its two-step update: computing perturbation $\\epsilon^*$ and updating with the perturbed loss. This is integrated into our SGDA loop. Also, we will correct the typo: sign(grad(L)) should be replaced with $\\frac{\\nabla \\mathcal{L}}{||\\nabla \\mathcal{L}||_2} $ since the $\\epsilon$ is controlled in a $\\ell_2$ norm ball. Also, the sentence should be \"direction of the gradient\". Thanks for pointing out the typo.\n2) **AUC motivation clarification**. We will revise Line 44 for clarity. Our intention was to highlight that in deepfake detection, misclassifying fake content as real can lead to serious consequences, such as the spread of misinformation on social media. This explains why the task itself is high-stakes. AUC is preferred as it evaluates model performance across thresholds, unlike fixed-metric alternatives.\n3) **Deepfake mislabel question**. We respectfully clarify that the deepfake mislabel issue discussed in Line 70 is **more accurately characterized as a label noise problem rather than a distribution shift**. In our case, group labels (e.g., gender) are often incorrect due to heuristic annotation of synthetic faces, i.e., observed $\\tilde{Z}$ does not match true $Z$. \n\n**Other Comments Or Suggestions**. Thanks for the suggestion. We will revise “become” to “becoming”. In addition, we confirm that $\\tilde{g}_{z, z'}$ in Eq. (8) **is conditioned on group pairs** consistent with Eq. (5), which computed over the noisy distribution.", "claims_and_evidence": "Claim is pretty clear: giving a method for improving fairness in AUC optimization under noisy labels. The experiments more or less show that so this is a yes from me. There are a couple caveats however:\n- I would like to see fairness accuracy tradeoffs across hyperparameters to really see what is a fairness metric at a given accuracy level\n- The approach for estimating the level of noise (using CLIP to detect mislabeled images) is a bit odd to me - in particular, if you have something this powerful, I'm not sure why you wouldn't just relabel your entire dataset? Some baseline like this I think would be beneficial to understanding how much of the improvement is due to increased computational power from CLIP, and how much is methodological\n- CLIP-style noise estimation approaches do not really seem plausible in a tabular data setting as claimed (line 242)", "methods_and_evaluation_criteria": "yes evaluation setup is reasonable", "theoretical_claims": "L186: I'm not sure what it means to  be enforcing g < 0 for all groups - it's not clear to me this is possible that all group AUCs can simultaneously be less than the overall AUC? Needs more explanation on these dynamics\n\nLemma 4.2 - I’m not sure what this is lemma is saying. The supposition is unclear to me", "experimental_designs_or_analyses": "Table 1/2: would be more convincing to show a tradeoff over hyperparameters to really see if fairness is improving at a given accuracy\nTable 1/2: is this fairness on noisy labels or underlying labels? Should be clearer - and if it's not on the underlying labels, it's not clear how useful the result is (since that's what we care about, right?). In particular, not sure how calculating on underlying labels works in Table 2\nTable 3 - is there any property of this problem in particular you think SAM helps with?\nFig 3 right - I’m unclear how this checks the correctness of the noise ratio estimation as claimed, needs more explanation", "relation_to_broader_scientific_literature": "Would be good to clarify a bit how this \"noisy label\" setup connects to the (more common I believe) \"missing label\" setup theoretically", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "SAM: this should be clarified a bit for those unfamiliar - is this solved at every step? Also I'm not sure why sign(grad(L))  is what comes out at the end - shouldn’t this be a scalar? Cause maximizing eps will give norm(grad(L)) (I think). Relatedly I'm not sure what the “direction of the gradient norm” means in the text - isn't the norm is a scalar?\n\n\nSmaller notes:\n044: “where misclassifications can have significant implications” - this does not really explain why AUC is a useful metric given its previous description in this sentence\n070: the deepfake mislabel question is more of a distribution shift one than a label noise one I believe", "other_comments_or_suggestions": "015: becomes -> becoming\nEq 8: do you mean that g-tilde is conditioned on Z as in (5)?", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript proposes an AUC fairness approach under noisy protected groups, with theoretical fairness guarantees based on distributionally robust optimization.

Reviewers raised several major concerns, including the computational overhead of the proposed method compared to baseline approaches on large-scale datasets, the method's performance in scenarios with extremely high noise levels in protected group labels, and the need for better visualization of the fairness–accuracy trade-offs across hyperparameters. There is also a concern regarding the use of the powerful multimodal foundation model CLIP to detect mislabeled images for estimating the level of noise. The authors have provided several additional experiments during the rebuttal, and most reviewers agree that the claims, evidence, and technical contributions are sufficient.
Decision: Accept (poster)