Quantifying and Mitigating the Impact of Label Errors on Model Disparity Metrics
Authors: Julius Adebayo
Melissa Hall
Bowen Yu
Bobbie Chern
Venue: ICLR 2023 poster (2023)

Abstract: Errors in labels obtained via human annotation adversely affect a trained model's performance. Existing approaches propose ways to mitigate the effect of label error on a model's downstream accuracy, yet little is known about its impact on a model's group-based disparity metrics\footnote{Group-based disparity metrics like subgroup calibration, false positive rate, false negative rate, equalized odds, and equal opportunity are more often known, colloquially, as \textit{fairness metrics} in the literature. We use the term group-based disparity metrics in this work.}. Here we study the effect of label error on a model's group-based disparity metrics like group calibration. We empirically characterize how varying levels of label error, in both training and test data, affect these disparity metrics. We find that group calibration and other metrics are sensitive to train-time and test-time label error---particularly for minority groups. For the same level of label error, the percentage change in group calibration error for the minority group is on average 1.5 times larger than the change for the majority group. Towards mitigating the impact of training-time label error, we present an approach to estimate how changing a single training input's label affects a model's group disparity metric on a test set. We empirically assess the proposed approach on a variety of datasets and find a 10-40\% improvement, compared to alternative approaches, in identifying training inputs that improve a model's disparity metric. The proposed approach can help surface training inputs that may need to be corrected for improving a model's group-based disparity metrics.

QUANTIFYING AND MITIGATING THE IMPACT OF LA- BEL ERRORS ON MODEL DISPARITY METRICS
Julius Adebayo Prescient Design/ Genentech
Melissa Hall Meta Inc.
Bowen Yu Meta Inc.
Bobbie Chern Meta Inc.
ABSTRACT
Errors in labels obtained via human annotation adversely affect a model's perfor- mance. Existing approaches propose ways to mitigate the effect of label error on a model's downstream accuracy, yet little is known about impact on a model's s disparity metrics'. Here we study the effect of label error model's ' disparity metrics. We empirically characterize how varying levels label error, both training and test data, affect these disparity metrics. We find that group calibration and other metrics are sensitive to train-time and test-time label partr-particulaly for minority groups. This disparate effect persists even for models trained with noise- aware algorithms. To mitigate the impact of training-ting siienggtmmmm/ label error, we present approach to estimate the influence of training input's label model's group disparity metric. We empirically assess the proposed approach on variety datasets and find significant improvement, compared to alternative approaches, identifying training inputs that improve a model's disparity metric. We complement the approach with an automatic relabel-and-finetue scheme that produces updated models with, provably, improved group calibration error.
INTRODUCTION
Label error (noise) - mistakes associated with the label assigned a data point is pervasive problem in machine learning (Northcutt et al., 2021). For example, 30 percent of random 1000 samples from the Google Emotions dataset (Demszky et al., 2020) had label errors (Chen, 2022). Similarly, analysis of the MS COCO dataset found that up to 37 percent (273,834 errors) of all annotations are erroneous (Murdoch, 2022). Yet, little known about the effect label error on a model's group- based disparity metries like equal odds (Hardt et al., 2016), group calibration (Pleiss et al., 2017 7), and false positive rate (Barocas et al., 2019).
It is now common practice conduct 'fairness' audits (see: (Buolamwini and Gebru, 2018; Raji and Buolamwini, 2019; Bakalar et al., 2021)) of model's predictions to identify data subgroups where the model underperforms. Label error in the test data used conduct fairness audit renders the results unreliable. Similarly, label error in the training data, especially if the error is systematically more prevalent in certain groups, can lead to models that associate erroneous labels such groups. The reliability of faimess audit rests on the assumption that labels are accurate; yet, the sensi- tivity of a model's disparity metrics label error is still poorly understood. Towards such end, we ask:
what the effect of label error on a model's 's disparity metric?
We address the high-level question in a two-pronged manner via the following questions:
1. Research Question 1: What is the sensitivity of model's disparity metric to label errors in training and test data? Does the effect of label error vary based on group size?
2. Research Question 2: How most influence on a model's group disparity metric?
practitioner identify training points whose labels have the
'Group-based disparity metrics like subgroup calibration, false positive rate, false negative rate, equalized odds, and equal opportunity are more often known, colloquially, as fairness metrics in the literature. We use the term based disparity metrics in this work.

---

CONTRIBUTIONS & SUMMARY OF FINDINGS
features, Labets
Data
Training
Model
+
Parmase
Dispartyy
0.11
0.06
Training Lebols
A schematic of the test and train-time empirical sensitivity tests, Here we show the model training and fairness audit pipeline. Our proposed sensitivity tests capture the effect of label error, both stages, on the disparity metric. In the Test-time sensitivity test, we flip the label of a portion of the data and then compare the corresponding disparity metric (group calibration for example) for the flipped dataset to the metrics for a standard model where the test labels were not flipped. In the Train-time sensitivity test, we flip the labels of portion of the training set, and then measure the change in disparity metric to a standard model.
Figure I:
addressing these questions, we make two broad contributions:
Empirical Sensitivity Tests. We assess the sensitivity of model disparity metrics to label errors with a label flipping experiment. First, we iteratively flip the labels of samples the test set, for a fixed model, and then measure the corresponding change in the model disparity metric compared to unflipped test set. Second, we fix the test set for the fairness audit but flip the labels a proportion of the training samples. We then measure the change in the model disparity metrics for model trained on the data with flipped labels. We perform these tests across a datasets and model combinations.
We propose approach, based modification to the influence of a training example on test example's loss, to identify training points whose labels have undue effects any disparity metric of interest the test set. We empirically assess the proposed approach on a variety of datasets and find 10-40% improvement, compared to alternative approaches that focus solely on model's loss, in identifying training inputs that improve model's disparity metric.
Training Point Influence Disparity Metric.
SETUP & BACKGROUND
In this section, we discuss notation, and set the stage for our contributions by discussing the disparity metrics that we focus on. We also provide an overview of the datasets and models used in the experimental portions of the paper.?
Overview of Notation. We consider prediction problems, i.c, settings where the task is learn mapping, 0 X x A + y, where X' E F is the feature space, y {0, 1} is the output space, and A is a group identifier that partitions the population into disjoint sets e.g. race, gender. We can represent the tuple (I,, 0z 1/i) as 2;. Consequently, the training points can be written as: {z, )=11 Throughout this work, we will only consider learning via empirical risk minimization (ERM), which corresponds to: 0 = arg mingge Σ. E(2;,)) Similar to Koh and Liang (2017), we will assume that the ERM objective is twice-diffi siice-differentiable and strictly convex in the parameters. We focus binary classification tasks, however, our analysis can be casily generalized.
Disparity Metrics. We define a group disparity metric be function, GD, that gives a performance score given a model's probabilistic predictions (0 outputs the probability of belonging to the positive class) and 'ground-truth' labels. We consider the following metrics (We refer readers the Appendix for a detailed overview of these metrics):
We refer readers to the longer version of this work the arxiv. Code to replicate findings is available https:ttt.........t.. / / thub. com/ adebayoj/ uenced sparity
2

---

Dataset
Classes
71
d
Group
Source
CivilComments
2
1,
820, 000
768
Sex
Koh and Liang (2017)
ACSIncome
195, 665
10
Sex, Race
Ding et al. (2021)
ACSEmployment
2
378, 817
16
Sex, Race
Ding et al. (2021)
ACSPublic Coverage
2
138, 554
19
Sex, Race
Ding et al. (2021)
Credit Dataset
405, 032
6
Sex
De Montjoye al. (2015)
Table I: Overview of dataset characteristics for the datasets considered in this work.
1. Calibration: defined as IP (g ylp p). Vp E [o, 1]. In this work, we measure calibration with two different metrics: 1) Expected Calibration Error (ECE) (Naeini et al., 2015; Pleiss et al., 2017), and 2) the Brier Score (Rufibach, 2010) (BS).
2. (Generalized) False Positive Rate (FPR): is GD (0)) E[0(),) y = 0] (see Guo et al. (2017)),
3. (Generalized) False Negative Rate (FNR): is GD (0) B[(1 ((x)) y = 1},
4. Error Rate (ER): is the (D) = 1 acc(0)
We consider these metrics separately for each group as opposed to relative differences. For each dataset, we consider the protected data subgroup with the largest size as the majority group, and the group the smallest size is the minority group.
Datasets. We consider datasets across different modalities: tabular, and a text dataset. description of these datasets along with test accuracy is provided in Table 2. Each dataset contains annotations with a group label for both training and test data, so we are able to manipulate these labels for our empirical sensitivity tests. For the purposes of this work, we assume that the provided labels the ground-truth-a strong assumption that nevertheless does not impact the interpretation of our findings.
Model. We consider three kinds of model classes in this work: 1) logistic regression model, 2) a Gradient-boosted Tree (GBT) classifier for the tabular datasets, and 3) a ResNet-18 model. We only consider the logistic regression and GBT models for tabular data, while we fine-tune a ResNet-18 model on embeddings for the text data.
EMPIRICAL ASSESSMENT OF LABEL SENSITIVITY
In this section, we perform empirical sensitivity tests quantify the impact of label error on test group disparity metrics. We conduct tests on data from two different stages of the ML pipeline: I) Test-time (test dataset) and Training-time (training data). We use as our primary experimental tool: label flipping, i.e., we flip the labels of percentage of the samples, uniformly at random in either the test or training set, and then measure the concomitant change in the model disparity metric. We assume that each dataset's labels are the ground truth and that flipping the label results in label error for the samples whose labels have been overturned. Recent literature has termed this setting synthetic noise, i.e., the label flipping simulates noise that might not be representative of real-world noise in labels (Arpit et al., 2017; Zhang et al., 2021; 1; Jiang et al., 2020).
3.1 SENSITIVITY TO TEST-TIME LABEL ERROR
Overview & Experimental Setup. The goal of the test-time empirical test is measure the impact of label error on the group calibration error of fixed model. Consider the setting where model has been trained, and a fairness assessment is to be conducted on the model. What impact does label error, in the test set used to conduct the audit, have the calibration error on the test data? The test-time empirical tests answer this question. Given a fixed model, we iteratively flip a percentage of the labels, uniformly random, ranging from model's 's calibration using the modified dataset. Critically, we keep the model fixed while performing these tests across each dataset.
30 percent in the test data. We then estimate the
Results. In Figure 2, we report results of the label flipping experiments across tasks. On the horizontal axis, we have the percentage of labels flipped in the test dataset, while on the vertical
3

---

Ad.t Inteer: ROme
Kir Oenore depiermeee
dinert groud
paviing pone
Mortl
prop
Majurty grop sajote 2740
Moprtty
20 15
16 35 25
Figure 2: Test-time Label Flipping Results across. For each dataset, we plot the percent change in calibration error versus the corresponding percentage change in label error. Here, we plot the minority (smallest) group as well as the majority (largest) group. These two groups represent two ends of the spectrum for the impact of label error. We observe that across all datasets, the minority group incurs higher percentage change in group calibration compared to the majority group.
axis, we have the percentage change in the model's calibration. For each dataset, we compute model calibration for two demographic groups in the dataset, the majority and the minority-in We do this since these two groups constitute the two ends of the spectrum in the dataset. As shown, we observe more distinctive effect for the minority group across all datasets. This is be expected since flipping even a small number samples in the minority group can have a dramatic effect on test and training accuracy within this group. For both groups, we observe a changes the calibration error. For example, for the Income prediction task on the Adult dataset, a 10 percent label error induces at least a 20 percent change in the model's test calibration error. These results suggest that test-time label error has more pronounced effects for minority groups. Similarly, we observe for other disparity metrics (See Appendix) across all model classes that increases in percentage of labels flipped disproportionately affects the minority group.
grreegro.ps...
3.2 SENSITIVITY TO TRAINING LABEL ERROR
Overview & Experimental Setup. The goal of the training-time empirical tests is to measure the impact of label error a trained model. More specifically, given training set in which fraction of the samples' labels have been flipped, what effect does the label error have on the calibration error compared to model trained data without label error? We simulate this setting by creating multiple copies of each of the datasets where a percentage of the training labels have been flipped uniformly random. We then assess the model calibration of these different model using the same fixed test dataset. Under similar experimental training conditions for these models, we are then able to quantify the effect of training label error on a model's ' test calibration error. We conduct this analysis across all lataset-model task pairs.
Results & Implications. We show the results of the training-time experiments in Figure 3. Similar to the test-time experiments, we find minority groups are more sensitive to label error than larger groups. Specifically, we find that even 5 percent label error can induce significant changes in the disparity metrics, of a model trained on such data, for these groups
conjecture for the higher sensitivity to extreme training-time error is that a model trained on significant label error might have more difficult time learning patterns in the minority class where there are not enough samples to begin with. Consequently, the generalization performance model worsens for inputs that belong to the minority group. Alternatively, in the majority group, the proportion corrupted labels due to label error is smaller. This might mean that uniform flipping does not affect the proportion of true labels compared to the minority group. Even though the majority group exhibits label error, there still exists enough samples with true labels such that a model can learn the underlying signal for the majority class.
this
4

---

Miceth graat
Moarth
Morrt
Adliiaaio Maaas Caveage
Caal Caliet
dinerty Duliaa
Martycec
Unarthy plece
Figure 3: Training-time Label Flipping Results. For each dataset, we plot the percent change in calibration error versus the corresponding percentage change label error for the training set. Here, we plot the minority (smallest) group well as the majority (largest) groups by size. Similar to the test-time setting, we observe that across all datasets, the minority group incurs higher percentage change in group calibration compared to the majority group. However, we observe larger magnitude change for the minority groups
A second important finding is that overparameterization seems to confer more resilience to training label error. We find that for the same levels of training label error, an overparametrized model less sensitive to such change compared to a model with a smaller number of parameters. Recent work suggests that models that learn functions that are more aligned with the underlying target function of the data generation process are more resilient to training label error (Li et al., 2021). It might be that compared to linear and tree-based models, an overparametrized deep net is more capable of learning aligned function.
3.3 NOISE-AWARE ROBUST LEARNING HAS DISPARATE IMPACT
Overview Experimental Setup. We now assess whether training models with noise-aware algorithmic interventions (e.g. robust loss functions (Ma et al., 2020; Ghosh et al., 2017)) results in models whose disparity metrics have reduced sensitivity label error in the training set. We test this hypothesis on modified Cifar- 10 dataset following the setting of Hall et al. (2022). Specifically, the Cifar- 10 dataset is modified to binary classification setting along with group labels by inverting subset each class's examples. Given specified parameter
Hebunt Les Mare
Colacing Hine
[0, 1/2], inverted, while + of the positive class is inverted leading 2e fraction of one group of samples and - 2e of the other group. 0.15 for 30 percent minority group
of the negative class is
Figure 4: Effect of Noise- aware algorithms on group calibration.
In all experiments we membership. We replicate the label flipping experiment on the task with Resnet-18 model. We test the MEIDTM (Cheng et al., 2022), DivideMix (Li et al., 2020), and robust loss approach (Ghosh et al. 2017).
Results. At a high level, for the majority group, we find that group calibration remains resilient to low rates of label error (below 25 percent). At high rates (>30 percent label error), we start to see increased sensitivity. However, for the minority group (30 percent of the dataset), we observe group calibration remains sensitive to label error even at low levels. This finding suggests that noise-aware methods show are more effective for larger groups in the data. A similar observation has also been made for other algorithmic interventions like Pruning (Tran al., 2022; Hooker et al., 2019), differential privacy (Bagdasaryan et al., 2019), selective classification (Jones et al., 2020) and adversarial training (Xu al., 2021).
5

---

4 INFLUENCE OF TRAINING LABEL ON TEST DISPARITY METRIC
We now present an approach for estimating the 'influence' of perturbing a training point's label on disparity metric of interest. We consider: 1) up-weighting a training point, and 2) perturbing the training label.
Upweighting a training point. Let d_s,, be the ERM solution when a model is trained on all data points, { except 2₁. The influence, Zup.params of datapoint, 2;, the model parameters is then defined: 0 8- - 0. This measure indicates how much the parameters change when the model is 'refit' on all training data points except 2, Koh and Liang (2017) give a closed-form estimate of this quantity as:
det d0c, de
Tup Sup.params
= --H Vol(2())))
(1)
==0
where H is the hessian, i.c.. Hg E O).
The loss on a test example, {(z, 0), is function of the model parameters, so using the chain-rule, we can estimate the influence, Tup. loss (Zi, 2+), a training point, 2,, on f(2,, 0) as:
Iap,inss Ines (Zi, 21) del de(ze, Óe, = Vel(e, 0) H
( Vo 0).
(2)
de
Perturbing a training point's label. A second notion of influence that Koh and Liang (2017) study is how perturbing a training point leads to changes in the model parameters. Specifically, given training input, 2i, that a tuple (æ; Ms). how would the perturbation, 2; + 21.6, which defined as (2i, y:) + (zi, + 8), change the model's predictions? Koh and Liang (2017) give a closed-form estimate of this quantity as:
Fosrt,loss,y (2j, 21) Vol(2r,02,..-s))
7 Vol(z,,)))
Adapting influence functions to group disparity metrics. We now propose modifications that allow us to compute the influence of training point on test group disparity metric (See Appendix D for longer discussion). Let S, be set of test examples. We can then denote GD(S,, 0) as the group disparity metric of interest, e.g., the estimated ECE for the set S, given parameter setting Ó.
Influence of upweighting a training point on a test group disparity metric. A group disparity metric the test set is function of the model parameters; consequently, to Tup, params (from Equation 1) to estimate the influence, Tup disparity. of up-weighting training point the disparity metric as follows:
apply the chain rule
disparayy (zi, St) def dGD(S θ 21) - VODD(S,, 0)
d8
de
de
===
l==0
-DDDDD H 0).
(4)
We now have closed-form expression for training point's 'ss influence test group disparity metric.
Influence of perturbing a training point's label on a test group disparity metric. We now consider the influence of training label perturbation simply consider the group disparity metric function as the quantity of interest instead of the test loss. Consequently, the dosed-form expression for the influence of modification the training label disparity for a given test set is:
group disparity metric of interest. To do this,
(5)
With Equations 4 and 5, we have the key quantities of interest that allows us to rank training points, in terms of influence, on the test group disparity metric.

---

5 IDENTIFYING AND CORRECTING TRAINING LABEL ERROR
In this section, we empirically assess the modified influence expressions for calibration across these datasets for prioritizing mislabelled samples. We find that the prioritization scheme shows improve- ment, compared to alternative approaches. In addition, we propose approach to automatically correct the labels identified by our proposed approach.
5.1 IDENTIFYING LABEL ERROR
Overview & Experimental Question. We interested in surfacing training points whose change in label will induce a concomitant change in test disparity metric like group calibration. Specifically, we ask: When the training points are ranked by influence on test calibration, are the most highly influential training points most likely to have the wrong labels? We conduct our experiments to directly measure method's s ability to answer this question.
Experimental Setup. For each dataset, we randomly flip the labels of 10 - 30 percent of the training samples. We then train on this modified dataset. In this task, we have direct access to the ground-trurt of the exact samples whose labels were flipped. This allows to directly compare the performance of our proposed methods to each of the baselines on this task. We then rank training points using well as the modified influence approaches. For the top 50
number of baseline approaches examples, we consider what fraction of these examples had flipped labels in the training set. We discuss additional experimental details in the Appendix.
Approaches Baselines. We consider the following methods: 1) IF-Calib: The closed-form approximation to the influence of a training point on the test calibration; 2) IF-Calib-Label: The closed-form approximation to the influence of a training point's label on the test calibration; 3) Loss: A baseline method which is the training loss evaluated at each data point in the training set. The intuition is that, presumably, more difficult training samples will have higher training loss. We also consider several additional baselines that we discuss in the Appendix.
Addi Dataest Incane 201)
Adull Dotment Foces
Acul Untaseet Grypoyment
Precorton www.oooiieeeeee Lasete
6.8
0.8
6.0
0.6
5.0
Locz
I7-oom
F-chbb
I-Ch B-LAbe
0.2
1001
E-am
IFCal
5.0
F-Ch b-bbb
Presoraoon of Fiteeed Lateee
6.4
6.6
0.6
5.6
0.4
1901
E-umm
1200
F-ck
Figure 5: Empirical Results for Training Point Ranking Across datasets. For the top 50 most influential examples, we show the proportion of samples whose labels were flipped in the training data.
Results: Prioritizing Samples. In Figure 5, we show the performance of the two approximations that we consider this work as well as two baselines. We plot the fraction of inputs, out of the top ranked 50 ranked training points, whose labels were flipped in the training set. The higher this proportion, then the more effective an approach in identifying the samples that likely have wrong labels. In practice, the goal is to surface these training samples and have domain expert inspect them. If larger proportion of the items to be inspected are mislabeled, then a higher proportion of training set mistakes, i.c. label error, can be fixed. Across the different datasets, find a 10-40 percent improvement, compared to baseline approaches, in identifying critical training data points whose labels need to be reexamined.
7

---

We find the loss baseline to be ineffective for ranking in our experiments, A possible reason is that modern machine learning models can typically be trained to 'memorize' the training data: resulting in settings where model has low loss even outliers mislabeled examples. In such case, ranking by training loss for a sample is an ineffective ranking strategy. We find that the noise-aware approaches perform similarly to the IF-Norm baseline. We defer the results of the uncertainty-based baselines and the noise-aware methods to Appendix (Section F). We find that these baselines also underperform our proposed approaches.
5.2 CORRECTING LABEL ERROR
We take label error identification one step further to automatically relabelling inputs that have identified as critical. We restrict our focus to binary classification where the label set is {0, 1), and the corresponding relabelling function is simply 1 y/s where yi is the predicted label.
Setup& Experiment: We consider the logistic regression model across all tasks for a setting with 20 percent training label error. We consider calibration as the disparity function of interest. We then rank the top 20 percent of training points by label-disparity influence, our proposed approach. For these points, we apply the relabelling function, and then fine-tune the model for an additional epoch with the modified labels.
First, observe an improvement, in group calibration, across all groups, with larger improvement coming from the smallest group. As expected, we also observe a decrease in the average loss for the overall training set. These results point to increasing promise of automatic relabeling.
Results:
Theoretical Justification. We now present a theorem that suggests that the influence priorization and relabeling scheme described above provably leads to better calibrated models.
Theorem 1.
Given k-strongly corvex loss function ((.,.,, with k. > O, training dataset, DD where A indexes the data groups, and model, 0 I; Yt, optimized via ((..-) that maps inputs labels. Let Q be a set of test examples all belonging to group = a, where ECalo(0) is the expected calibration error of 0 on the set Q. In addition, let A=o be the set of problematic training examples, belonging to group a, prioritized based on influence, i.e., Ippettcaalib.y/ (x Q) 0. We term a model trained on a different training set (D+) where the problematic examples have been relabeled to be 2R. Analogously, the expected calibration error of this new model on the set Q is ECale (Q)). We have that:
(.Cal((0R) S RC)lo(0).
We defer the proof to the Appendix. Theorem suggests that when a model is trained on a relabeled dataset, following the influence prioritization scheme, the expected group calibration of the retrained model should be lower than that of a model trained on dataset that has not been relabeled,
6 RELATED WORK
We discuss directly related work here, and defer a longer discussion to Section A of the Appendix.
Impact of Label Error/Noise on Model Accuracy. Learning under label error falls under the category more commonly known as learning under noise (Frénay and Verleysen, 2013; Natarajan al., 2013; Bootkrajang and Kabán, 2012). Noise in learning can come from different either input features or the labels. In this work, we focus on label error-categoriization mistakes associated with the label in both the test and training data. Previous work focused primarily on the effect of label error in the training data; however, we advance this line of work to investigate the effect of label error in the test data used to conduct a fairness audit on the reliability of the audit. Model resilience to training label error has been studied for both synthetic (Arpit et al., 2017; Zhang et al., 2021; Rolnick et al., 2017) and real-world noise settings (Jiang et al., 2020). A major line of inquiry is the development of algorithmic approaches to learn accurate models given training set with noisy labels. These approaches include model regularization (Srivastava et al., 2014; Zhang et al., 2017), bootstrap (Reed et al., 2014), knowledge distillation (Jiang et al., 2020), instance weighting (Ren et al., 2018; Jiang and Nachum, 2020). robust loss functions (Ma et al., 2020; Ghosh et al., 2017), or trusted data (Hendrycks et al., 201 1018), joint training (Wei al., 2020), mixture models in semi-supervised learning (Li et al.,
8

---

2020), and methods to learn transition matrix that captures noise dependencies (Cheng et al., 2022). In contrast to this line of work, we primarily seek to identify the problematic instances that need to be relabelled, often by human labeler, and not automatically learn model that is robust to label error.
Impact of Label Error on Model 'Fairness'. This work contributes to the burgeoning area that studies the impact of label error on a model's 'fairness' (termed "group-based disparity' in this paper) metrics. Fogliato et al. (2020) studied a setting in which the labels used for model training are a noisy proxy for the true label of interest, e.g., predicting rearrest proxy for rearrest. Wang et al. (2021) considers an ERM problem subject to group disparity constraints with group-dependent label noise, and provides theoretical results along with scheme to obtain classifiers that are robust to noise. Different from their setting, we consider unconstrained ERM (no fairess constraints during learning). Similarly, Konstantinov and Lampert (2021) study the effect of adversarial data corruptions on fair learning in a PAC model. Jiang and Nachum (2020) propose a re-weighting scheme that is able to correct for label noise.
Influence Functions & Their Uses. Influence functions originate from robust statistics where it is used as a tool to identify outliers (Cook and Weisberg, 1982; Cook, 1986; Hampel, 1974). Koh and Liang (2017) introduced influence functions for modern machine learning models, and used them for various model debugging tasks. Most similar to our work. Sattigeri et al. (2022) and Li and Liu (2022) also consider the influence of a training point on model's disparity metric, and present intriguing results that demonstrate that reweighting training samples can improve a model's 's disparity metrics. Here, we focus specifically the role of mislabeled examples; however, our goal aligns with theirs. Similarly, Kong et al. (2021) propose RDIA, a relabelling scheme based on the influence function that is able provably correct for label error in the training data. RDIA identifies training samples that have a high influence on the test loss for a validation set; however, we focus on identifying training samples that influence a group-disparity metric on a test/audit set. We also rely on their technical results to prove Theorem 1.
In recent work, De-Arteaga et al. (2021) study expert consistency in data labeling and use influence functions to estimate the impact of labelers on model's predictions. Along similar direction, Brunet et al. (2019) adapt the influence function approach to measure how removing small part of a training corpus, in word embedding task, affects test bias measured by the word embedding association test Caliskan et al. (2017). Feldman and Zhang (2020) use influence functions to estimate how likely training point is have been memorized by model. More generally, influence functions are gaining widespread use as a tool for debugging model predictions (Barshan et al., 2020; Han et al., 2020; Yeh al., 2018; 8; Pruthi et al., 2020). Different from these uses of influence functions, here we isolate the effect of a training point's label on a model's 's disparity metric on a audit data.
7 CONCLUSION
In this paper, we sought to address two key questions: 1) What is the impact of label error on model's group disparity metric, especially for smaller groups in the data; and 2) How can practitioner identify training samples whose labels would also lead to a significant change in the test disparity metric of interest? We find that disparity metrics are, indeed, sensitive test and training time label error particularly for minority groups in the data. In addition, we present an approach for estimating the 'influence" of perturbing a training point's 's label on disparity metric of interest, and find a 10-40% improvement, compared to alternative approaches, in identifying training inputs that improve model's disparity metric. We present an approach to estimate the effect of a training input's s label on model's group disparity metric. Lastly, perform simple automatic relabel-and and---n---itteee scheme that produces updated models with, provably, improved group calibration error.
Our findings come with certain limitations. In this work, we focused on the influence of label error on disparity metrics. However, other components of the ML pipeline can also impact downstream model performance. The proposed empirical tests simulate the impact of label error; however, it might be the case that real-world label error is less pernicious to model learning dynamics than the synthetic flipping results suggest. Ultimately, we see our work as helping to provide insight and an additional tool for practitioners seeking to address the challenge of label error particularly in relation to disparity metric of interest.
9

---

REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International Conference on Machine Learning, pages 233-242. PMLR, 2017.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019.
Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern, Sam Corbett-Davice Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, al. Faimesss the ground: Applying algorithmic fairness approaches to production systems. arXiv preprint 1eXiv:2/03.06177 2021.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. http::/wwet fai i rm wmllook.orr
Elnaz Barshan, Mare-EEienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference Artificial Intelligence and Statistics, pages 1899-1909. PMLR, 2020.
Jakramate Bootkrajang and Ata Kabán. Label-noie noise robust logistic regression and its upplications. In Joint European conference machine learning and knowledge discovery in databases, pages 143-158. Springer, 2012.
Marc-Etienne Brunet, Colleen Alkalay- Ashton Anderson, and Richard Zemel. Understand- ing the origins of bias in word embeddings. In International Conference Machine Learning, pages 803- 81 1. PMLR, 2019.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In Conference fairness, accountability and transparency, pages 77-91. PMLR, 2018.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semanties derived automatically from language corpora contain human-like biases. Science, 356(6334) 2017.
Edwin labeled, 2022.
Chen.
30% of google's
dataset is mis- fttps://wwat https://wwwwssurgehg. /l/blog/
emotions
URL
30-
De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. mustance-dependee label-noise learning with manifold-regullaiized transition matrix estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 16630-16639, 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.000 2019.
Dennis Cook. Assessment of local influence. Journal of the Royal Statistical Society: Series (Methodological), 48(2):133-155, 1986.
R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 21 2189-2200. PMLR, 2021.
Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. Levergging expert consistency to improve algorithmic decision support. arXiv preprint arXiv:2 001409 2021.
De Montjoye, Laura Radaelli, Vivek Kumar Singh, and Alex "Sandy" Pentland. Unique in the shopping mall: On the reidentifiability of credit card metadata. Science, 347(6211); 536-539 2015.
10

---

Dorottya Demszky, Dana Movshovitz-Atiia, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset fine-grained emotions. arXiv preprint arXiv:2005.0054. 2020.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint SxKi:20008003700 2020.
Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. Fairesss evaluation in presence of biased noisy labels. In International Conference on Artificial Intelligence and Statistics, pages 2325-2336. PMLR, 2020.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845 (19888669666988 2013.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
Melissa Hall, Laurens van der Maaten, Laura Gustafson, and Aaron Adcock. systematic study of bias amplification. arXiv preprint arXiv:2201.11706 2022.
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-399) 1974.
Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676, 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29:3315-3323,, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. Advances in neural information processing systems, 31, 2018.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv:1911.05248. 2019.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statisties, pages 702-712. PMLR, 2020.
Lu Jiang. Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. International Conference on Machine Learning, pages 4804 4815. PMLR, 2020.
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. arXiv preprint arXiv:2010.. 14134, 2020.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pages 247-254, 2019.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pages
PMLR, 2017.
Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence-based data relabeling. In International Conference on Learning Representations, 2021.
Nikola Konstantinov and Christoph H Lampert. Faimess-aware pac learning from corrupted data.
arXiv preprint arXiv: 102. 060044 2021.

---

Fabian Küppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. Multivariate confidence calibration for object detection. In The IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR) Workshops, June 2020.
Jingling Li, Mozhi Zhang, Keyulu Xu, John Dickerson, and Jimmy Ba. How does neural network's architecture impact its robustness to noisy labels? Advances in Neural Information Processing Systems, 34:9788 1199788-8803, 2021.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels semi- supervised learning. arXiv preprint arXiv:2002.0739: 2020.
Peizhao Li and Hongfu Liu. Achieving faimeess at no utility cost via data reweighing with influence. In International Conference on Machine Learning, pages 12917-12930. PMLR, 2022.
Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In International Conference on Machine Learning, pages 4051-4060. PMLR, 2019.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Normal- ized loss functions for deep learning with noisy labels. In International Conference Machine Learning, pages 6543-6552 PMLR, 2020.
Edwin
Murdoch. coco, 2022.
How
i found nearly 300,000 erTOrs in ms URL
/ttps://mediiie um. com http:/jaammmem__2_0
how-1-foundh
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26:1 1911-9664444 2013.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint 10V:2/003.1749, 2021.
Gcoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. arXiv preprint arXiv: 1709.02072 2017.
Romila Pradhan, Jiongli Zhu, Boris Glavic, and Babak Salimi. Interpretable data-based explanations for fairness debugging. arXiv preprint arXiv:2112.09745. 2021.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracing gradient descent. arXiv preprint 2rXhv:2002.08484 2020.
Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 429 435, 2019.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343, PMLR, 2018.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 10694, 2017.
Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certified robustness to label- flipping attacks via randomized smoothing. In International Conference on Machine Learning, pages 8230-8241. PMLR, 2020.
Kaspar Rufibach. Use of brier score to assess binary predictions. Journal of clinical epidemiology. 63(8)998-9339 2010.
12

---

Prasanna Sattigeri, Soumya Ghosh, Inkit Padhi, Pierre Dognin, and Kush R Varshney. Fair infinites- imal jackknife: Mitigating the influence of biased training data points without refitting. arXiv preprint arXiv:2212.06680 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1299-1958 2014.
Cuong Tran, Ferdinando Fioretto, Jung-Eun Kim, and Rakshit Naidu. Pruning has a disparate impact on model accuracy, arXiv preprint arXiv:2205.13577 2022.
Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM conference on fairness, accountability. and transparency, pages 2021.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726 13735, 2020.
Huiyu Wu and Diego Klabjan. Logit-based uncertainty measure in classification. 2021 IEEE International Conference Big Data (Big Data), pages 956. IEEE, 2021.
Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust to be fair: Towards faimess in adversarial training. In International Conference on Machine Learning, pages 1 1492-11500..
PMLR, 2021.
Yeh, Joon Kim, Ian Esu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115. 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710:0710 2017.
13


Reviews:
[{"review_id": "EmaqzywPDa", "reviewer": "Reviewer_DK6U", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. This paper studies the effect of label error on a model’s group-based disparity metrics. Differences in terms of calibration error can be observed for the minority (smallest) group and the majority (largest) group.\n2. They also propose an approach (influence function) to estimate how changing a single training input’s label affects a model’s group disparity metric on a test set. Based on the influence function, they can identify training points that have a high effect on a model’s test disparity metric.\n", "summary_of_the_review": "1. The overall quality of the paper is good, and the key ideas are clear enough to make reviewers easy to follow.\n2. If as they claimed, label errors’ influence on group disparity is not yet covered by literature, this is a strong submission.\n", "strength_and_weaknesses": "Strength\n\n1. They try to answer two significant questions regarding label errors and fairness:\n    What is the sensitivity of a model’s disparity metric to label errors in training and test data? Does the effect of label error vary based on group size?\n    How can a practitioner identify training points whose labels have the most influence on a model’s group disparity metric?\n2. Various disparity metrics, models, and datasets are considered\n    model disparity metrics: expected calibration error (ECE), the Brier Score (BS), False Positive Rate (FPR), False Negative Rate (FNR), and Error Rate (ER).\n    datasets across different modalities: 4 tabular, and a text dataset.\n    models: a logistic regression model, a Gradient-boosted Tree (GBT), and ResNet-18.\n3. The closed form influence function is potentially useful for regularizing or mitigating the label error while training.\n\nWeaknesses\n1. Although multiple datasets are used, how to make sure that the conclusions and analysis drawn is not dataset-specific is still an open question.\n2. The label flipping is indeed useful for empirical results but not interesting as a technical contribution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7AWRUUgNqYe", "reviewer": "Reviewer_Xn1n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of label error on the model’s disparity metrics (e.g., calibration, FPR, FNR) on both the training and test set. Empirically, the authors have found that label errors have a larger influence on minority groups than on majority groups. To mitigate the impact of label errors, The authors have proposed a method to estimate the influence of changing a single training input’s label on a model’s group disparity metric.", "summary_of_the_review": "For me, the motivation and research problems of this paper are strong and important. My major concerns are that the technical contribution may not that strong, and the proposed method may not practical and hard to be applied to real-world machine learning datasets. ", "strength_and_weaknesses": "Strength:\n+ The research problems are important and may have many practical applications. The real-world machine learning dataset can easily contain label errors. Improving the robustness of learning models trained on noisy data is important. Existing methods mainly focus on downstream accuracy, but group-based disparity metrics have been ignored which are also important for designing a robust algorithm. \n+ The proposed method is well-motivated. Estimating the influence of a single training input on a model’s group disparity metric is important for confident example selection and dataset purification.\n\n\n\nWeakness:\n+ The technical insight may not be enough. The authors have empirically illustrated that minority groups are more sensitive to label errors than majority groups. To make the conclusion more meaningful and practical, I think it would be great to add some theoretical analysis on the influence of label errors with different minority and majority group sizes.\n\n+ The proposed method for estimating the ‘influence’ of perturbing a training point’s label on a disparity metric may not practical. The computational cost of the method seems very expensive and needs a lot of retraining processes to detect the effect of all training inputs, which can be hard to apply to a dataset with high-dimensional features. In addition, to demonstrate the performance of the proposed methods, some SOTA methods should be compared (e.g., JoCoR, CVPR’20; DivideMix, CVPR’20; MEIDTM, CVPR’22). The benchmark datasets such as CIFAR10 and CIFAR100 with different types of synthetic noise should also be compared.\n\n\n+ The experiment setting is not clear to me. For example, it is not clear how the minority group and majority group in Fig. 1 and Fig.2 are obtained. I think the authors may also need to discuss that how to apply the convolutional network Resnet-18 to tabular and text datasets. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1UlNMuZj0fp", "reviewer": "Reviewer_oCK7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers an important problem of label noise in the training data. Specifically, it studies the effect of label error on a model's group-based disparity metrics, with more focus on smaller groups in the data. Then, the authors of the paper take a step further by considering a method based on influence function to identify training samples that significantly impact the test disparity metric of interest. The authors of the paper conduct a series of experiments to answer these questions and offer valuable insight into this important problem. ", "summary_of_the_review": "Despite the interesting perspective and a well series of well-conducted experiments, I feel like the authors of the paper can provide some additional experimental insight to the paper, as suggested above. As such, I recommend a weak accept for now. ", "strength_and_weaknesses": "Strength:\n- This paper is very well organized and written in general. Most of the claims are supported by ample experimental analysis. \n- The problem of concern has a unique fairness perspective, which has great practical significance. \n\nWeaknesses: \n- Learning with noisy labels is a widely studied topic, especially in the context of neural networks. As an empirical paper, it would be nice if the authors of the paper can conduct some additional analysis to show the effect of some of the recently proposed noise-robust algorithms on such group-based disparity metrics. \n- Similar to the above point, it would be nice if the authors of the paper benchmarked the proposed \"IF-Calib-Label\" against some other recently proposed noise-robust algorithms that can potentially identify label errors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster