Learning with Instance-Dependent Label Noise: Balancing Accuracy and Fairness
Authors: Donna Tjandra
Jenna Wiens
Venue: Submitted to ICLR 2023 (2023)
Keywords: noisy labels
supervised learning

Abstract: Incorrect labels hurt model performance when the model overfits to noise. Many state-of-the-art approaches that address label noise assume that label noise is independent from the input features. In practice, however, label noise is often feature or instance-\textit{dependent}, and therefore is biased (i.e., some instances are more likely to be mislabeled than others). Approaches that ignore this dependence can produce models with poor discriminative performance, and depending on the task, can exacerbate issues around fairness. In light of these limitations, we propose a two-stage approach to learn from datasets with instance-dependent label noise. Our approach utilizes \textit{anchor points}, a small subset of data for which we know the ground truth labels. On many tasks, our approach leads to consistent improvements over the state-of-the-art in discriminative performance (AUROC) while balancing model fairness (area under the equalized odds curve, AUEOC). For example, when predicting acute respiratory failure onset on the MIMIC-III dataset, the harmonic mean of the AUROC and AUEOC of our approach is 0.84 (SD 0.01) while that of the next best baseline is 0.81 (SD 0.01).  Overall, our approach leads to learning more accurate and fair models compared to existing approaches in the presence of instance-dependent label noise.

Under review as a conference paper at ICLR 2023
LEARNING WITH INSTANCE-DEPENDENT LABEL NOISE: MAINTAINING ACCURACY AND FAIRNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Incorrect labels hurt model performance when the model overfits noise. Many state-of-the-art approaches that address label noise assume that label noise is inde- pendent from the input features. In practice, however, label noise often feature or mestance-dependett and therefore is biased (i.e., some instances are more likely to be mislabeled than others). Approaches that ignore this dependence can pro- duce models with poor discriminative performance, and depending on the task, can exacerbate issues around fairness. In light of these limitations, we propose two-stage approach to learn from datasets with instance-dependent label noise. Our approach utilizes alignment points, a small subset of data for which we know the observed and ground truth labels. On many tasks, approach leads to con- sistent improvements over the state-of-the-an art in discriminative performance (AU- ROC) while maintaining model fairness (area under the equalized odds curve, AUEOC). For example, when predicting acute respiratory failure onset on the MIMIC-III dataset, the harmonic mean of the AUROC and AUEOC of our ap- proach is 0.84 (SD 0.01) while that of the next best baseline is 0.81 (SD 0.01). Overall, our approach leads to learning more accurate and fair models compared to existing approaches in the presence of instance-dependent label noise.
INTRODUCTION
Datasets used to train machine learning models can contain incorrect labels (i.e., label noise). While label noise widely studied, the majority of past work focuses on when the noise independent from an instance's 's features (i.e., instance-independent label noise) (Song et al. (2020)]. However, label noise is sometimes biased and depends on an instance's ' features (i.e., instance-dependent) (Wei et al. (2022))). leading to different noise rates within subsets of the data. This results in model overfitting, and in tasks where the dataset contains instances from different groups corresponding to some sensitive attribute, this can also lead to disparities in performance [Liu| (2021)). For example, consider the task of predicting cardiovascular disease among patients admitted to a hospital. Com- pared to male patients, female patients may be more likely to be misdiagnosed [Maserejian et al. [2009]] and thus mislabeled, potentially leading to worse predictions for female patients. Although instance-dependent label noise has recently received more attention [Cheng et al. (2020b); Xia al. (2020): Wang et al. (2021a)), the effect of these approaches on model fairness has been relatively understudied [Luu| (2021)). Here, we address the limitations of current approaches and propose novel method for learning with instance-dependent label noise, specifically examining how model- ing assumptions affect existing issues around model fairness.
Broadly, current work addressing instance-dependent label noise falls into one of two categories: 1) that which learns identify mislabeled instances [Cheng et al. (2020a); Xia et al. (2022) y Zhu al. (202a))]. ]. and 2) that which learns to optimize a noise-robust objective function |eng et al. (2020); Wei et al. (2022at]. In the first category, instances identified as mislabeled are either filtered out [Kim et al. (202T)] or relabeled [Berthon et al. (202T)). In some settings, this approach can have a negative effect on model fairness. For example, when instances represent individuals belonging to subgroups defined by sensitive attribute, approaches that filter out mislabeled individuals could ignore a disproportionately higher number of individuals from subgroups with more label noise. While relabeling approaches use all available data, they be sensitive to assumptions around the noise distribution [Ladouceur et al. (2007)). In the second category, current approaches rely on objective functions that are less prone to overfitting to the noise, while using all of the data and

---

Under review as a conference paper at ICLR 2023
Table 1: Notation. We summarize our notation, with the notation appearing in the left column and description in the right column. Superscripts in parentheses represent specific instances (e.g.. x()). Subscripts represents indexes into a vector (e.g., X;)
Notation
Description
Notation
Description
Number of instances
d
Number of Teatures
C
Number of classes
Number groups
X
E
R"
Feature vector
5 [1, 112.....
Observed label
Predicted class probabilities
V (1,2, 2,,
Ground truth label
86
Ply == V.. x)
A
Set of alignment points
0
Main model parameters
Auxiliary model parameters
observed labels [Chen et al. (2021)). However, like the first category, many rely on assumptions like the memorization effect, and thus, potentially suffer from the same limitations (Wang et al. (202Tab).
In light of these limitations, we propose an approach to address instance-dependent label noise, makes no assumptions about the noise distribution, and uses all data during training. We leverage set of representative points in which we have access to both the observed and ground truth labels. While past work has used the observed labels as the ground truth labels for anchor points [Xia et al. (2019); Wu et al. (202I)], we consider a different setting in which the ground truth and observed labels do not agree for some points. To make the differentiation clear, we refer to these points as "alignment points'. Such a setting arises frequently in healthcare. Oftentimes, one labels an entire dataset using proxy function (obtaining observed labels) but also labels small subset of the data using manual review (obtaining ground truth labels). We use them to initialize the model's decision boundary and learn the underlying pattern of label noise in pre-training We then add the remaining data for fine-tuning, minimizing a weighted cross-entropy loss based on the learned noise pattern.
On synthetic and real data, we evaluate model faimess,, measured using the area under the receiver operator curve (AURÓC) and area un- der the equalized odds curve (AUEOC). We demonstrate that our approach improves on state-of- the-art baselines from the noisy labels and fairness literature, such as stochastic label noise [Chen] al. [00[I] and group-based peer loss [Wang al. (202Ib)). Overall, our contributions are: 1) novel approach to learn from datasets with instance-dependent noise; 2) a systematic examination of different settings of label noise, showing where approaches fail with respect to discriminative performance and fairness; 3) empirical results showing that the proposed approach is robust to both to the noise rate and amount of noise disparity between subgroups, reporting the model's ability to maintain discriminative performance and fairness; 4) a demonstration of how performance of the proposed approach changes when assumptions about the alignment set are violated.
approach in terms of discriminative performance and
2
METHODS
We introduce two-stage approach for learning with instance-dependent label noise that leverages a small set of alignment points for which we have both observed and ground truth labels.
Notation and Problem Setup Our main notation is in Table We train model using dataset D (x(), f00m Ji=1, where E Rd and y E (1.2,....c) c} to learn a function f y that can map unseen instances into one of c classes based on their feature vectors. In the presence of noisy labels, ÿ is not always equal to y. For each alignment point, we know both and y. This is in contrast to the rest of the data, where we only know J. We aim to learn model parameters, 0, such that (x)) represents the predicted class probabilities, (i.e., g). Alignment points are similar anchor points Xia et al. (2019)). but we do not assume that 7( y(d for these points.
For the rest of the paper, we focus on the following case of instance-dependent noise. Let f be the function used generate the ground truth labels (i.e., fxx y). and let m be the function used to generate an instance's risk of being mislabeled (i.e., y = ÿ if m(() is above some threshold). In the following toy example, suppose f(x) for true positive instances, and (x) 1 2 > 0.5 and 0 otherwise. Here, instances where xi > 0.5 have noisy labels. Although f and in were deterministic for simplicity, they can also be based on probabilistic functions. We denote the set of alignment points (i.e., the alignment set) set A consisting of instances i such that both g(o)
2

---

Under review as a conference paper at ICLR 2023
Training
Step Using the anchor points, salve min Cp Cg //
8. < (x)
Inference
(19 .00 2//0 2/
0' o'
( softmax
Inttal model parameters
Step Using complete dataset, atternte between
θ
Step 2a: phi and solve min C. yC
Step 2b: Freeze theta and solve C.
0, Φ
(x(). g00 of/ Jaaa
meddal parameters
y
X
(a) Proposed network. The model makes class predictions, ÿ. at training and inference time using 0. At training time, also pre- diets whether the observed label is correct using Φ. 0 and é are pre-trained using A and then fine tuned with the complete dataset.
(b) Proposed training pipeline. We first pretrain the model ing the alignment points (i.c., data with observed and ground truth labels), then train on the noisy data. Lg is the objective to train on the alignment points. La is the objective to leam the label confidence score (ße). Lg is the objective to train the noisy data, on, 02 and are hyperparameters.
Figure 1: Proposed Approach
and y are known and assuming that it includes instances where g( * y()) Although it is not always possible to obtain alignment points, the setting above does occur in practice. For example, in healthcare, automated labeling tools based on the structured contents of the electronic health record are often applied to identify cohorts or outcomes of interest [Upadhyaya et al. (2017); Norton et al. (2199;; Tjandra et al. (2020); Yoo et al. (.020)]. However, it is often overlooked that such practical definitions are not always reflective of ground truth, and thus, require validation by comparing to subset of the data that was chart reviewed by an expert. This subset can be derived from a randomly chosen individuals, and thus can be constructed to be representative of the target population.
Proposed Approach We describe the proposed network and training procedure below.
Proposed Network. Our proposed network (Figure La consists of two components. The first, 8, is feed-forward network that uses feature vector predict the class probabilities, The second component, Φ, is an auxiliary feed-forward network that uses observed label ÿ and the pre-sofimax output of 0 to compute B.- = Pyy = ūlg. x), an instance-dependent prediction for whether the observed label is correct based and ÿ. A& can be considered confidence score for the observed label, with higher values indicating higher confidence. Learning Ag learns the underlying pattern of label noise by forcing the model distinguish between which examples are correctly or incorrectly labeled, This will be helpful during training. which we describe below. By including the observed label input o, approach also applies to instance-indepenie noise by accounting for the case when the underlying pattern label noise cannot be learned from the features alone. At training time, we compute the loss using the outputs from both A and ø. At inference time, we compute the class predictions from 0 only since ÿ is not available.
((x).
label
Training Procedure. Our procedure is summarized in Figure 1h and Appendix A. In Step 1, we pre-train the 0 and é networks using the alignment points, minmmizing objective function based on cross entropy: ming.. Le + a Lg- an E IR is a scalar hyperparameter; and ø' are the initial parameters of 9 and é. La, is the cross entropy loss between the class predictions and ground truth labels, learning the weights for 0, and thus, the model's decision boundary. II is an indicator function.
-1 Lo [A
I (1
=- log (
j=1
b.s is the cross entropy loss between the predicted confidence score 3000 and the actual agreement between gitt) and (1)) It learns the weights for o, and in turn, the underlying label noise pattern.
L. Σ (1 -= = () log (350) II ()()) + ()) log (1 -3(()
In Step 2, we initialize and 6 using 6' and ø' from Step 1 and fine tune them using the complete dataset, which potentially contains noisy labels. Step can be further divided into two parts, Step
3

---

Under review as a conference paper at ICLR 2023
2a and Step 2b. Each part aims to improve specific component of the network (e.g.. f) using another component of the network (e.g., Φ). Throughout Step 2, we alternate between Step 2a and Step 2b in a manner similar to expectation maximization that continually improve both 0 and db. In Step 2a, we freeze & and train 0 by minimizing the objective ming L. + 7 Y E R is a scalar hyperparameter. In Step 2b, we freeze and train Φ using the objective min, L% + azLa- 02 E R+ is a scalar hyperparameter. La computes the cross entropy loss over the potentially noisy, non-alignment points. Each instance is weighted by the model's confidence in whether the observed label is correct via 300) & taking advantage of the model's 's learned noise pattern.
c
C. A
Σ 1=1 VEAN
Σ
3
(g - log (0)5) 6
PK
The quantity 1 1- f% is as follows. We introduce sets gt for k = 1, 2, . G to represent disjoint subgroups of interest in the data, where 9a ng. 0 for all a 1, ..., G, b = 1, 1,2, .... G with a * b and Uk=19k = D. Each group It is then associated with estimated noise rate fx Ioal Eiegu
g(0) and estimated clean (i.e., correct) rate 1- Fx T9aa Eieo 3(0) Although weighting each
instance by Ag is form of soft filtering, weighting each group by the inverse of its overall clean rate avoids the effect of de-emphasizing groups with higher predicted noise rates. As a result, the expected value of CG with respect to 3g is equal to the cross entropy loss between the model's predictions and ground truth labels (see Appendix A for proof). However, this assumes accurate estimates of 3g. Thus, we expect that the proposed approach will perform best when the alignment set is representative of the target population (i.e., there is no dataset shift from the alignment set to the remaining data), since this will produce the best estimate of Bgg. In Section 4, we test this.
Our approach encourages fairness, as measured by equalized odds [Hardt et al. (2116)), by upweight- ing groups with higher estimated noise rate so that they are not dominated/ignored compared to groups with a lower estimated noise rate. In doing so, we hypothesize that accuracy and related metrics, such as the false positive rate, will improve, thereby improving equalized odds. We focus on equalized odds since, in the domains of interest, metrics like the true and false positive rates are particularly important. For example, in healthcare, we would like correctly predict who will and will not develop a condition so that the appropriate treatment may be used.
During Step 2a, S is used train 0 by learning to predict ý such that it matches observed label on instances that are predicted to be correctly labeled. During Step 2b, L' is used to train 6. Here, since 0 is frozen and é is not, the network learns to predict the optimal A.g. Based on L' alone, there are two possible ways to learn 3z. The first is to consistently make 3r close O. The second is to learn B. such that is close to 1 when ý matches ÿ and close to 0 when ý does not match j. Since ÿ is used as proxy for y in this step, the second way aligns with what we want 3g to represent. To encourage this over the first way (i.e., consistently predicting for (gg), we include L. Step 2b, which is not minimized by consistently predicting 0 for å-- Note that, in order to expect a benefit by including the noisy data Step 2b, we rely on the cluster assumption (Singh et al. (2008)] from semi-supervised learning, which broadly states that labeled data fall into clusters and that unlabeled data aid in defining these clusters. In the context of Step 2b, 'labeled' and 'unlabeled' are analogous to whether we know if the ground truth and observed labels match (i.e., alignment point versus non- alignment point), rather than the actual class labels themselves. As a result, we rely on the alignment set being representative of the target population here as well to avoid dataset shift.
In contrast to previous filtering approaches, our approach utilizes all data during training. Moreover, it does not require a specialized architecture beyond the auxiliary network to compute Bg. Thus, can be used to augment existing architectures in straightforward manner.
3 EXPERIMENTAL SETUP
We empirically explore the performance of our proposed approach relative to state-of-the-an base- lines on five benchmark prediction tasks with two different label noise settings. Implementation details are provided in Appendices and D.
4

---

Under review as a conference paper at ICLR 2023
Datasets We consider five different binary prediction tasks on four datasets from several domains on synthetic and real datasets. While work in noisy-label learning often focuses on image datasets (e.g., MNIST or CIFAR 10/100), given emphasis on fairness, we selected datasets reflective of tasks in which harmful biases can arise (e.g.. predicting clinical outcomes, recidivism, and income).
Synthetic:
We generated dataset containing 5,000 instances as described below. Each instance had 30 features, and features were drawn from a standard normal distribution. We defined the feature at index 0 to be a synthetic sensitive attribute. Instances with values below the 20th percentile for this feature were considered as the 'minority', and the rest were considered as the 'majority.. Features 10-19 for the majority instances and features 20-29 for the minority instances were set to 0 to provide more contrast between the two groups. Coefficients describing the contribution of each feature to the outcome, w, were also drawn from a standard normal distribution, and an outcome z was computed by taking the dot product of the features and coefficients and passing it through sigmoid. A binary ground truth label, y. for each instance was assigned, based on z. Instances whose value for fell above the 50th percentile were given the label I. All other instances were labeled O. The positive rates for the synthetic majority and minority groups were 37.5% and 32.3%, respectively.
MIMIC-III: Within the healthcare domain, we leverage publicly available dataset of electronic health record data [Johnson et al. (2016)]. We considered two separate prediction tasks: onset of 1) acute respiratory failure (ARF) and 2) shock in the ICU (intensive care unit) [Oh et al. [. MIMIC-III includes data pertaining to vital signs, medications, diagnostic and procedure codes, and laboratory measurements. We considered the four hour prediction setup for both tasks as described by [Tag et al. (2020)]. This resulted in 15,873 and 19,342 ICU encounters for each task, respec- tively. After preprocessing (see Appendix B), each encounter had 16,278 and 18,186 features, for each task respectively. We used race as a sensitive attribute, with about 70% of patients being White (positive rate 4.5% [ARF]. 4.1% [shock]) and 30% being non- Whn-White (4.4% [ARF], 3.7% [shock]).
Adult: publicly available dataset of census data (Dua & Graf (017)) We considered the task of predicting whether an individual's 's income is over $50,000. This dataset includes data pertaining to age, education, work type, work sector, race, sex, marital status, and country. Its training and test sets contain 32,561 and 16,281 individuals, respectively. We used a pre-processed version of this [ We only used 1,000 randomly selected individuals out of 32,561 during training and only included features pertaining to age, education, work type, marital status, work sector, and sex to make the task more difficult (see Appendix B). We also normalized each feature to have range of 0-1. This resulted in 56 features per individual. We used sex as a sensitive attribute, with 67.5% of individuals being male (positive rate 30.9%) and 32.5% being female (positive rate 11.3%).
COMPAS: a publicly available dataset collected by ProPublica from Broward County, Florida, USA (nggwin et al. (2016)). We considered the task of predicting recidivism within two years, i.e., whether a criminal defendant is likely to re-offend. COMPAS includes data pertaining to age, race, sex, and criminal history. We used a pre-processed version of this datase and also normalized each feature to have a range of 0-1 (see Appendix B). This resulted in 6, 6,172 individuals with 11 features per individual. We used race as sensitive attribute, with 65, 65.8% of individuals being White (positive rate 39.1 1 %) and 34.2% being non- White (positive rate 44.5%)
Label Noise To test the robustness of our approach in different settings of label noise, we introduce noise in two ways. Like past work (Song et al.] (2020)), our setup is limited in that our noise is synthetic and we use the labels in the real datasets as ground truth, since we do not have access to actual ground truth labels on these public datasets. For uniform random noise, each instance had its label flipped with a fixed probability (e.g., 0.3). The noise rates were the same between subgroups. For instance-dependent noise, mislabeling was function of the features. Let wm (Wm
N (O, 0.33) P) and 2m (2m. = a(x- W W.)) denote the coefficients describing the contribution of each feature to mislabeling and the risk of mislabeling, respectively. Whether an instance was mislabeled was based on 2m and the desired noise rate (m)()) = y if Zm > 1 noise rale, - y otherwise). For example, for a noise rate of 30%, instances whose value for Zm was above the 70th percentile had their labels flipped. This allowed us to vary the noise rate within subgroups in a straightforward manner. Across datasets, we focused on cases where the noise rate in the 'minority' population was always greater than or equal to that of the 'majority' group since this is more likely to occur.
5

---

Under review as a conference paper at ICLR 2023
Evaluation Metrics We evaluate our proposed approach in terms of discriminative performance and model fairness. For discriminative performance, we evaluate using the area under the receiver operating characteristic curve (AUROC) (higher is better). For model fairesss, we measure the area under the equalized odds curve (AUROC) (de Freitas Pereira & Marcel (2000) (higher is better). We use the AUEOC rather than equalized odds avoid choosing classification threshold to mea- sure the true and false positive rates. For classification threshold т. we calculate the equalized odds (BO(T)) between two groups, called and 2, as shown below. T TP (T) and FPP denote true and false positive rates for group at threshold т, respectively. The AUBOC is obtained by plotting the EO against all possible values of 7 and calculating the area under the curve. We also compute the harmonic mean (HM) between the AUROC and AUEOC to highlight how the approaches main- tained discriminative performance and fairnsss. We compute the harmonic mean so that the overall measure is more heavily weighted by the worse performing metric. For example, if a classifier has A and AUEOC=1.0, we emphasize the poor discriminative performance.
EO(T)= = 2 - (T) - TP2(T)) - -FP(()))FP(T
2
Baselines We evaluate our proposed approach in the context of several baselines to test differ- ent hypotheses. Standard does not account for label noise and assumes that ū y is always true. SLNN + Filter [Chen et al. (202I)] combines filtering (Arpit et al. (2017) and SLN [Chen let al. (2021)] and was shown outperform state-of-the-ar approaches such as Co-Teaching [Han let al. [2018) and DivideMix (Li et al. (2020)]. It relies on filtering heuristics, which indirectly rely on uniform random label noise maintain discriminative performance and model fairness. Loss (Englesson & Azizpour (2021)] builds from semi-supervised learning and encourages model consistency when predicting on perturbations of the input features and was shown to be competitive with other state-of-the-an art noise-robust loss functions Ma et al. (2020)] It was proposed in the context of instance-inde independent label noise. Transition (Xia et al. (2020) learns to correct for noisy labels by learning transition function and was shown outperform state-of-the- art approaches such as MentorNet (Jiang et al. 2018)). It applies to instance-dependent label noise, but it assumes that the contributions of each feature to mislabeling and input reconstruction identical. CSIDN (confidence-scom)
IS Seensen-Shannow
instance- deeenddet noise) [Berthon et al. (202I) also learns a transition function and was shown outperform state-of-the-art approaches such forward correction PPatrini et al.] [- Like our approach, CSIDN uses the concept of 'confidence' in the observed label to help with training. Unlike our approach, CSIDN uses the model's class predictions directly as confidence scores (instead predicting them via an auxiliary network) and uses them to learn the transition function (as opposed re-weighting the loss). Fair GPL (Wang al. [202Tb)] builds on work addressing uniform random label noise Diang & Nachum (220)] and uses peer loss (i.e., data augmentation that reduces the correlation between the observed label and model's predictions) within subgroups defined by a sensitive attribute (Wang et al. (202 It assumes that label noise depends on only the sensitive attribute. We also train model using the ground truth labels (referred to as Clean Labels) an empirical upper bound for discriminative performance.
4 RESULTS AND DISCUSSION
We describe the results from experiments with instance-dependent noise. For each plot, we com- bined discriminative performance and fairness and plotted the HM of the AUROC and AUEOC to assess general performance with respect to both of these aspects.
Is the Approach Robust to Varying Noise Levels? Here, we investigated how robust the proposed approach and baselines were to varying amounts of instance-dependent label noise (Figure 2a). On the synthetic dataset, we considered noise rates between 10% and 60% in the majority group. For the minority group, we considered noise rates of that were consistently 20% higher than that of the majority group, to keep the noise disparity level (i.e., the difference in noise rates between sub- groups) constant. As expected, all approaches degrade as the noise rate increases, with the proposed approach experiencing the least degradation up a majority noise rate of 60%. The baseline Tran- sition was also robust to the noise rate, although overall performance was generally worse than the proposed approach. At noise rate of 60%, it performed similarly to the proposed approach. The other approaches experienced more degradation, likely because they were for instance-inded -inennee-ddeeeddeent label noise, highlighting the need for approaches that designed for instance-dependen noise.
6

---

Under review as a conference paper at ICLR 2023
Hermonic Moan of AUROC and AUEDC
Harmonic Mean of AUROC and AUEOC
0.50
Standard
SUN Filter
SLN + Filter
Loss
[5 Loss
Transition
Transition
CSIDN
0.60
CSIDN
0.70
Frir
Pruposed
Proposed
Labels
0.70
Clean
0.60
0.60
Major Naise Rate
Minonty Naise Rate (Majonity Rate 20%)
(a) As the noise rate increases, the proposed ap- proach generally shows the least degradation.
(b) As the noise disparity increases, the proposed ap- proach generally shows the least degradation.
Figure 2: Robustness to noise rate and noise disparity an instance dependent setting. We plot the
mean and standard deviation for 10 random seeds.
Synthetic
MIMIC.ARF
MIMIC-Shock
Adult
COMPAS
0.50
Stancerd SU% i fller
0.85
Ji
012
0.00
C3.DV
042
Fek CPL
Respzed
0.53
0.50
0.03
6300
0.3
0.30
(a) As decrease the alignment size (proportion of training data) performance decreases. Still, an alignment size of 5%, the proposed approach generally outperforms the basclines. Synthetic
Harrook Near df and asr
MIMIC-ARF
MIMIC-Shock
Adult
0.75
655
0.72
Sis Fiber
0.70
E.
5.50
046
Fele
2.22
665
Reppoed
Ober Looels
Papp Si01 Mentty
(b) we vary the aligmment set bias (proportion of minority instances) performance varies. The proposed approach is generally robust to changes the bias the alignment set. The dashed vertical black line shows the proportion at which the minority group occurs in the dataset (i.c., an unbiased aligmment set).
Figure 3: Robustness to varying alignment sets. Mean and standard deviation for 10 random seeds.
Is the Approach Robust Varying Noise Disparities? Here we investigated how robust the pro- posed approach and baselines were to varying levels of noise disparity (Figure(b).. With a majority noise rate of 20% throughout, we evaluated the approaches with minority noise rates between 20% and 90% (i.e., from no disparity to disparity of 70%) on the synthetic dataset. In line with our ex- pectations, the proposed approach was robust to changes in the noise disparity. This likely because the objective function C/ from Step 2 of training accounts for disparities by scaling each instance- specific loss term with the reciprocal of its estimated group clean rate (i.c., the group noise rate). Most of the baselines, such SLN Filter, which learn to filter out noisy instances, experienced a larger drop in performance as the disparity increased. This aligns with our hypothesis. Although the baseline Transition also demonstrated robustness the noise disparity and outperformed the proposed approach at minority noise rates higher than 80%, this setting is unlikely to be realistic.
Does Alignment Set Composition Matter? Our next set of experiments tests the proposed ap- proach in settings where we relax key assumptions about the alignment set. We consider all datasets with instance-dependen noise. The majority/minority noise rates were 20%/40%, respectively.
Part I: Alignment set size. We varied the size of the alignment set, testing at sizes between 1% and 15% of the training set, with the alignment set representative of the test set (Figure 3aj. As expected, when the alignment set is small, performance worsens. However, the proposed method generally

---

Under review as a conference paper at ICLR 2023
improves over the baselines an alignment set size of as small as 3% of the training data. On the real datasets, the proposed approach had a higher HM compared to the baselines at an alignment set size of < 5%, showing improved overall performance. As the size of the alignment set increased, some baseline approaches were able to match that of the proposed method (e.g., for MIMIC-Shoek, the baseline Transition had similar performance to the proposed method at alignment set >> 5%). This is in line with expectations, as this makes the overall training set cleaner.
Part 2: Biased alignment set. We hypothesized that our approach works best when the alignment points are representative of the population. Here, we test how the proposed approach performs when this assumption is violated. We varied the amount of bias in the alignment set by changing the proportion at which the subgroups were present. We kept the size of the alignment set constant at 10% of the training data (2% for MIMIC-III on both tasks). We observe that the proposed approach is beneficial over wide range of conditions, i.e., when the minority proportion is 20%-80% (Figure 3b). We hypothesize that this could be because the learned relationship between the features and noise can generalize across groups to an extent. In scenarios where the proposed approach is less beneficial compared to the baselines, one subgroup heavily dominated the alignment set. This is shown in Figure 3b| on the extremes of the x-axis, which correspond to an alignment set that is heavily over-represented for one subgroup and heavily -ndres-rpresented for the other. Our approach relied, in part, on having relatively unbiased alignment set for estimating Bo in order to avoid introducing dataset shift between the two steps of our training pipeline. Thus, these results are in line with our expectations and highlight a limitation of our approach. However, despite this reliance, we observe that the our approach is still robust in some scenarios where the alignment set biased.
Additional Experiments: We include results for experiments where we vary the proportion at which the minority group appears in the entire dataset, results for uniform random noise, an ablation study, and hyperparamter sensitivity analysis in Appendix E.
RELATED WORK
[2020)] and may be instance- independent or -dependent. Regardless of the setting, past approaches fall into two broad categories. The first aims to identify mislabeled instances via filtering heuristics by estimating the probability of mislabeling (Patrini et al. (2017); Li et al. (2020); Cheng et al.] (2020b)). The second adapts regularization techniques, such as label smoothing and contrastive learning noisy labels [Manwani] & Sastry (2013); Menon et al. (2019); Xue et al. (2022); Liu et al. (2022)] We also examine how label noise has been tackled in the fairness literature and how past work has used anchor points.
Label noise is well-studied problem in machine learning (Song et al.
Identifying Mislabeled Data Approaches that learn to identify mislabeled instances fall into two sub-categories: 1) filtering approaches and 2) relabeling approaches. Filtering approaches use heuristics to identify mislabeled instances (e.g.. MentorNet [Jiang et al. (2018). Co-teaching (Han) jet al. (2018)), FINE [Kim et al. (202T))) Many are based on the idea that correctly labeled instances easier to classify than mislabeled instances (i.e., the memorization effect) [Arpit et al.) [017 For example, mislabeled instances could be those that the model incorrectly classifies [Verbaeten (2002) Khoshgoftaar & Rebours (2004); Thongkam et al. (2008); Chen et al. (2019)) have a high loss value (Yoo et al. (2020a)), or significantly increase the complexity of the model [Gamberger] (et al. [1996]] Given the identified mislabeled instances, these approaches either ignore them during training [Zhang et al. [2020)] or treat them as 'unlabeled" and apply techniques from semi-supervised learning (e.g., DivideMix [Li et al. (2020)), J. SELF (Nguyen et al. (2020))). Overall, these heuristics have been shown to improve discriminative performance. However, depending on the setting, they can disproportionately discard subsets of data, which could hurt model fairness.
For binary classification, some approaches 'correct' (i.c., switch) the observed label for such in- stances (Han et al. (2020) /Zheng///eggggge et al.) (2020)) Building on this idea, others make use of a transition function that estimates the probability of the observed label being correct. Model predictions can then be adjusted by applying the function to the classifier's predictions for each class. Some works manually construct the transition function from expert knowledge (Patrini et al. (2017)). while oth- ers learn it (Xiao et al. (2015); ; Xu et al. (2019): Yao et al. (2020b); Xia et al. (002): Zheng et al. (2021); Berthon et al. (2021) ; Jiang al. (0022) ; Bae et al. al. (2022)) However, such approaches often make assumptions on the form of the noise distribution, and past work has shown that results are sensitive to the choice of distribution [Ladouceur al. (007)
8

---

Under review as a conference paper at ICLR 2023
To date, much of the work described above assumes instance- Instance-independen independent label noise (i.e., misla- beling is independent of the features). However, when this assumption is violated, the model may overfit to label noise (Lukasik et al. [2020]] From an emerging body of work in instance-dependent label noise [Cheng et al. ( Xia et al. 2 Wang et al. (2021c); (Zhu et al. (2022b)), current approaches remain limited in that they still rely filtering heuristies. A Although
use soft filtering, we filter based on the learned relationship between the features and noise rather than existing heuris- tics and upweight groups with higher estimated noise rate. While similar a transition function in some aspects, our approach requires fewer probability estimates on label correctness (two estimates compared to e² for a transition function) while achieving
performance.
Woise-Robus' Loss Functions Prior work examines how regularization techniques can be adapted to the noisy labels setting, addressing issues related to overfitting on noisy data [Menon et al. (2019); Lukasik et al. (2020); Englesson & Azizpour (202T). Recently, label smoothing, and in some cases negative label smoothing, were found to improve the accuracy on both correctly labeled and mislabeled data [Lukasik et al. (2020); Wei et al. (2022a)]. With this approach, the observed labels are perturbed by a small, pre-determined value, with all labels receiving the same perturbation at every training epoch. Follow-up work found that, instead of applying the same perturbation each epoch, adding a small amount of Gaussian stochastic label noise (SLN) at each epoch resulted in further improvements, as it helped to escape from local optima (Chen al. (202T)). However, these approaches were most beneficial in the context of augmenting existing methods that identify mislabeled instances (e.g., stochastic label noise applied instances that are identified as correctly labeled by filtering approaches), and thus, potentially suffer from the same limitations. Alternatively, recent work has also proposed perturbing the features instead of the labels encourage consistency in the model's predictions [Englesson & Azizpour (002 I, though mainly in the context of instance- independent label noise. Others have proposed noise-robust variations of cross entropy loss [Feng et al. (2020); Wang et al. (2021a)] that generally relied on assumptions like the memorization effect.
Label Noise and Fairness Label noise has also been addressed within the fairness literature recently. When the frequencies at which subgroups (defined by a sensitive attribute) appear are different within a dataset, past work has shown that common approaches addressing label noise can increase the prediction error for minority groups (i.e., rarer subgroups) [Liu| (202I)) Past work proposed to re-weight instances from subgroups during training where model performance is poorer [Jiang & Nachum p in the random noise setting. Others use peer loss (Liu & Guo [2020)] within subgroups Wang et al. (2021b)) but assume that noise depends only on the sensitive attribute. We also train with weighted loss, but weights are based on predicted label correctness rather than performance on the observed labels. Recently, Wu et al. (2022) addressed some of the gaps of past work by examining the instance-dependent case. Our proposed approach differs from theirs in that we do not require our features to be grouped into distinct categories, such as root and low level attributes.
Anchor Points for Addressing Label Noise Another related setting in past work uses anchor points. To date, anchor points are generally used to learn a transition function [Xia et al. (2019) 2020); Berthon et al. p202l)] or for label correction directly [Wu et al. (202I)). We use a similar concept, alignment points, to 1) pre-train the model, and 2) predict label correctness. The first part builds from work in semi-supervised learning (Cascante-Bonill et al. (20T)), which has shown improvements from pre- pre-trainn trainini on labeled data. The second part is similar to a transition function, but differs in that we use the correctness predictions to re-weight the loss rather than adjust the predictions. We also assume that, for some alignment points, the ground truth and observed labels do not match.
CONCLUSION
We introduced a novel approach for learning with dependent noisy labels. Our two-stage approach uses the complete dataset and lears the relationship between the features and label noise using a small set of alignment points. On synthetic and real datasets, we showed that the proposed approach led to improvements over state-of-the-art baselines in maintaining discriminative perfor- mance and model fairness. Our approach is not without limitations. We demonstrated that the success of the approach depends, in part, on the composition of the alignment set. Additionally, we only examined one form of fairness. Nonetheless this work brings together and builds on key aspects from the instance-dependent noisy labels literature, highlighting issues related to fairness.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Section 3 describes our experimental setup, with additional details on implementation in Sections B. C, and D of the Appendix. The code has been included as a supplemental file and will be publicly released upon publication.
ETHICS STATEMENT
We do not believe that our paper has any ethical concerns. However, past work has shown that machine learning models can enforce harmful systemic biases if they are not carefully implemented Obermeyer et al. (2019). As result, it important to consider not only overall model performance, but also how the outcome is measured (i.e., choice of labels).
REFERENCES
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias, 2016.
Devansh Arpit, Stanislaw K Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In ICMI., 2017.
HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHo Jang. Kyungwoo Song, and Il-Chul Moon. From noisy prediction to true label: Noisy prediction calibration via generative model. In Inter- national Conference Machine Learning, pp. 1277-1297. PMLR, 2022.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning, pp. 825-836. PMLR, 2021.
Paola Cascante- Bonilla,, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revis- iting ggeuud-labelligg for semi-supervised learning. In Thirty- AAAI Conference on Artificial Intelligence, 2021.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learn- ing. pp. 1062-1070, 2019.
Pengfei Chen, Guangyong Chen, Junjie Ye, Jingwei Zhao, and Pheng-Ann Heng. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Representations, 2021.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2020a.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ranamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. International Conference on Machine Learming, pp. 1789-1799. PMLR, 2020b.
Tiago de Freitas Pereira and Sébastien Marcel. Fairness in biometrics: figure of merit biometric verification systems. arXiv e-prints, pp. arXiv-2011, 2020.
Dheeru Dua and Casey Graf. Uci machine learning repository. http://anchive.iss.uccdduuu 2017.
Erik Englesson and Hossein Azizpour. Generalized jensen-shannon divergence loss for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li, and Bo An. Can eross entropy loss be robust to label noise? In International Joint Conferences Artificial Intelligence, pp. 2206-2212, 2020.
Dragan Gamberger, Nada Lavrač, and Saão Džeroski. Noise elimination in inductive concept leam- ing: A case study in medical diagnosis. In International Workshop on Algorithmic Learning Theory, pp. 212. Springer, 1996.
10

---

Under review as a conference paper at ICLR 2023
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8531 2018.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning, pp. 4006-4016. PMLR, 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29:3315-3323, 2016.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 702-712. PMLR, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentoreet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2304-233 PMLR, 2018.
Zhimeng Jiang, Kaixiong Zhou, Zirui Liu, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. An information fusion approach to learning with instance-dependent label noise. In International Conference Learning Representations, 2022.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:100335, 2016.
Taghi M Khoshgoftaar and Pierre Rebours. Generating multiple noise elimination filters with the ensemble- ensemble-parttiionnng filter. In Proceedings of the 2004 IEEE International Conference on Infor- mation Reuse and Integration, 2004. IRI 2004., pp. 369-375. IEEE, 2004.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Se- Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Martin Ladouceur, Elham Rahme, Christian A Pineau, and Lawrence Joseph. Robustness of preva- lence estimates derived from misclassified data from administrative databases. Biometrics, 63(1): 272-279, 2007.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. In International Conference on Machine Learning. PMLR, 2022.
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. Interna- lional Conference on Machine Learning, pp. 6725-6735. PMLR, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
Michal Lukasik, Srinadh Bhojanapalli,, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference Machine Learning, PMLR, 2020.
Xingjun Ma. Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. In International Conference on Ma- chine Learning, pp. 6543-6555. PMLR, 2020.
Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3): 115-1116-11,,, 2013.
11

---

Under review as a conference paper at ICLR 2023
Nancy N Maserejian, Carol L Link, Karen L Lutfey, Lisa D Marceau, and John B McKinlay. Dis- parities in physicians' interpretations of heart disease symptoms by patient gender: results of a video vignette factorial experiment. Journal of women's 's health, 18(10):661-1167, 2009.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2019.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling, In International Conference on Learning Representations, 2020.
Jenna M Norton, Kaltun Ali, Claudine T Jurkovitz, Krzysztof Kiryluk, Meyeon Park, Kensaku Kawamoto, Ning Shang. Sankar Navaneethan, Andrew Narva, and Paul Drawz. Development and validation of a pragmatic electronic phenotype for ckd. Clinical Journal of the American Society of Nephrology, 14(9): (K9))1306-133 1314, 2019.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in algorithm used to manage the health of populations. Science, 366(6466))444-4553, 2019.
Jeeheh Oh, Jiaxuan Wang, Shengpu Tang, Michael W. Sjoding, and Jenna Wiens. Relaxed parameter sharing: Effectively modeling time-varying relationships in clinical time-series. In MLHC, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1944-1952, 2017.
Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now it helps, now it doesn't 't. Advances in neural information processing systems, 21, 2008
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv preprint 20Xi:2207..88199 2020.
Shengpu Tang, Parmida Davarmanesh, Yanmeng Song. Danai Koutra, Michael W Sjoding, and Jenna Wiens. Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association, 27(12):12111194, 2020.
Jaree Thongkam, Guandong Xu, Yanchun Zhang, and Fuchun Huang. Support vector machine for outlier detection in breast cancer survivability prediction. In Asia-Pacific Web Conference, pp- 99-109. Springer, 2008
Donna Tjandra, Raymond Q Migrino, Bruno Giordani, and Jenna Wiens. Cohort discovery and risk stratification for alzheimer's disease: an electronic health record-based approach. Alzheimer's & Dementia: Translational Research & Clinical Interventions, 6(1):e 2035, 2020.
Sudhi G Upadhyaya, Dennis H Murphree Jr, Che G Ngufor, Alison M Knight, Daniel J Cronk, Robert Cima, Timothy B Curry, Jyotishman Pathak, Rickey Carter, and Daryl J Kor. Au- tomated diabetes case identification using electronic health record data at tertiary care facility. Innovations, Quality & Outcomes, 1(1):100 110, 2017."
Mayo Clinic Proceedings:
Sofi Verbaeten. Identifying mislabeled training examples ilp classification problems. In Pro- ceedings of twelfth Belgian-Dutch conference machine learning, pp. 1-8, 2002.
Deng-Bao Wang. Yong Wen, Lujia Pan, and Min-Ling Zhang. Learning from noisy labels with complementary loss functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10111-10119, 2021a.
Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Transparency, pp- 526-536 2021b.
12

---

Under review as a conference paper at ICLR 2023
Qizhou Wang, Bo Han, Tongliang Liu, Gang Niu, Jian Yang, and Chen Gong. Tackling instance- dependent label noise via a universal probabilistic model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10183-10191, 2021c.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learn- ing. PMLR, 2022a.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022b.
Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning, pp. 927-943. PMLR, 2022.
Yichen Wu, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng. Learning to purify noisy labels via meta soft label corrector. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10388-10396, 2021.
Xiaobo Xia, Tongliang Liu, Nannan Wang. Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in neural information processing systems, pp. 6838-6849, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Con- ference on Learning Representations, 2022.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L.dmi: An information-thhorede noise-robust loss function. Advances in Neural Information Processing Systems, 32, 2019.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference Machine Learning. PMLR, 2022.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin-Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In International Conference Machine Learn- ing. pp. 10789-10798. PMLR, 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. Ad- vances in Neural Information Processing Systems, 33, 2020b.
Jung Eun Yoo, Dong Wook Shin, Kyungdo Han, Dahye Kim, Seung-Pyo Lee, Su-Min Jeong, Jinkook Lee, and Sang Yun Kim. Blood pressure variability and the risk of dementia: a nationwide cohort study. Hypertension, 75(4):982-990. 2020.
Xuchao Zhang, Xian Wu, Fanglan Chen, Liang Zhao, and Chang-Tien Lu. Self-paced robust learn- ing for leveraging clean labels in noisy data. In AAAI, pp. 6853-6860, 2020.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Enror-bounded correction of noisy labels. In International Conference on Machine Learn ing, pp. 11447-11457. PMLR, 2020.
13

---

Under review as a conference paper at ICLR 2023
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training model to predict. In International Conference on Machine Learning. pp. 27412-27427. PMLR, 2022a.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference on Machine Learning. PMLR, 2022b.
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
A PROPOSED APPROACH: ADDITIONAL DETAILS
We provide additional details proposed approach, including pseudocode description and justification on the loss of the proposed objective function.
A.1 GENERAL OVERVIEW
We summarize approach with pseudocode below in Algorithm 1. We begin with the dataset and initial model parameters, and we aim to use the dataset to learn the final model paramters.
Algorithm Proposed approach for learning with instance-dependent label noise. A is the set of anchor points. 8' and ø' the initial model parameters for the O and é networks. Here, 'stopping criteria' may refer to any stopping criteria, such as early stopping. The Freeze() function takes as in- put model parameters and freezes them, and the Unfreeze() function takes input model parameters and unfreezes them.
Input: (x(0), g(() y HEA (x(), hieA. ,' Output: 0, à (final model parameters) Hyperparameters: Scalars 01, 022,
Train , ,0)) y())
heA, (x(), Fign.
1: while stopping criteria do
Step Predict label
2:
ÿ = (x)
3:
B d'(x,
> Predict label confidence
La a EiEA Ej=1 TI (y == log 11/0)
4:
C -1 EiEA I (8 - y1) log (3!1) + [7((0 + y log (1 (1- B/0))
5:
6:
Loss = Lo Lat
7:
Update model parameters
8:
Compute stopping criteria
9:
0,6 + 0',''' d'
Freeze(ò)
11: while stopping criteria do
Step 2
8' fx(x)
13:
Bo o' (x. 3)
Cs = [ EiEs Eq= [g == j) log (000)
14:
C. = -1 EvEA [8 = y) log 3((0) + (g(0) + (0) log (1- 300
15:
IA
16:
C. -1 Ig I-Fx
1 ΣiEtng Σラ=1 1 (ū(() = 5) log (0,0)
Weighted loss
A if frozen then
D Step 2a
18:
Loss Lor + Tho Unfreeze( Freeze(!)
19:
20:
21:
else
D Step 2b
22:
Loss Lgr + Lz Unfreeze(#)
23:
24:
Freeze(())
Update model parameters Compute stopping criteria
26:
return 0,
Final model parameters, but only 0 is used as inference time
15

---

Under review as a conference paper at ICLR 2023
A.2 PROPOSED AND CLEAN LABEL LOss
Here, we show that minimizing the proposed loss C. from step 2 of the proposed method is equal to minimizing cross entropy on the clean labels in expectation.
C
3(0 (g --]) log (0)5))
Tk
k= iEAngx
1=1
G
(1)
(10) == 5) log (550)
E
A=
iEAnge
j=1
Σ
I II (z(0) == -) log (0!²)
E
fk
v
iE Args
j=1
Σ
Σ
E
- (1-1x)] (1()) = log (1)5)
1-fg Pk
#= An
j=1
Σ
--)= log (1)))
jEA j=l
As reminder, each group Ok is then associated with estimated noise rate f lsv Σien. ///
and estimated clean (i.e., correct) rate 1 - f4- 1ull Exege g(3). We can express the noise and clean rates in terms of (0) since
I (-10== 00)
==
99
kEgk
Pyy U1y x) for randomly chosen instance in gk
Ply = 7 (1) x())
sEgg
where Tk and F' are the actual noise and clean rates within group k; respectively. Therefore, since Be is trained to predict P(y == Ulū. x), we estimate the noise and clean rates using Bg.
B PREPROCESSING DETAILS
Here, we provide more detail real dataset pre-processing.
B. MIMIC-III
Data were processed using the Flexible Data Driven pipeLinE (FIDDLE), (Tang et al. (2020)], publicly available pre-processing tool for electronic health record data. We used the same features as [Tang et al.] (2020)) for our tasks. More information can be found at http://wwww...oo.....oooooooacua
B.2 ADULT
Although, we used a pre-processed version of this dataset, we omitted features pertaining to edu- cation, work type, and work sector to make the task more difficult. More specifically, in the file 'headers.txt' at the repository mentioned in Footnote 1, we kept all features beginning with 'age', 'workclass'. 'education', 'marital status', and 'occupation', We also kept the Sex_Female feature. Values were normalized for each feature to have a range of 0-1 by subtracting by the minimum value observed among all individuals and dividing by the range.
16

---

Under review as a conference paper at ICLR 2023
Table 2: For each dataset, we list the range of hyperparameters considered for each dataset. For each hyperparameter, the lower bound is shown in the top row, and the upper bound is shown in the bottom row. For hyperparameters we did not tune, only one row shown.
Hyperparameter
Synthetic
MIMIC-ARF
MIMIC-Shock
Adult
COMPAS
0.00001
0.00001
0.000001
0.00001
0.0001
Learning Rate
0.01
0.001
0.001
0.01
0.05
0.0001
0.000001
0.0001
0.0001
0.0001
L2 Constant
0.1
0.01
0.1
0.1
0.01
Layer Size
10
500
500
100
10
0.40
0.50
0.50
0.50
0.50
Filter Threshold
1.00
1.00
1.00
0.90
0.90
0.00001
0.00001
0.00001
0.0001
0.0001
Noise Added
0.01
0.001
0.001
0.001
0.01
Number of Parts
10
10
T0
10
10
0.01
0.1
0.001
0.01
0.01
(GGP
1.0
1.0
1.0
1.0
1.0
0.1
0.1
0.01
0.1
0.01
0 Proposed
10.0
10.0
10.0
10.0
10.0
0.1
0.1
0.01
0,1
0.01
Proposed
10.0
10.0
10.0
10.0
10.0
0.1
0.1
0.01
0.1
0.01
d2Proppsed
10.0
10.0
10.0
10.0
1O.0
B.3 COMPAS
Although, we used a pre-processed version of this dataset, we omitted the feature 'score factor' (i.e., the risk score for recidivism from the ProPublica model) to make the task more difficult. Values were normalized for each feature have range 0-1 by subtracting by the minimum value observed among all individuals and dividing by the range.
C NETWORK AND TRAINING DETAILS
Here, our ranges of hyperparameters and implementation choices for the proposed network. All networks were trained on Intel(R) Xeon(R) CPUs, E7-4850 v3 @ 2.20GHz and Nvidia GeForce GTX 1080 GPUs. All layers were initialized with He initialization from a uniform distribution. We also include are source code as part of the submission the supplemental attachment.
C.I HYPERPARAMETER VALUES CONSIDERED
Here, we show the range of values we considered for our random search. More details are provided in Table 2 For any hyperparameters associated with the Adam optimizer not mentioned above, used the default values. We divide our training data into five batches during training. Fandom seeds (for Pytorch, numpy, and Python's 's random) were initialized with 123456789.
Not all hyperparameters were used with each approach. 'Filter Threshold' and 'Noise Added' were only used with the baseline SLN + Filter. Here, Filter Threshold refers to the minimum value of the predicted probability of the observed label for an instance to be considered 'correctly labeled'. For example, if Filter Threshold=0.5, then all examples whose predicted probability for the observed la- bel is at least 0.5 are considered 'correct" and used during training. 'Number of Parts" was only used with the baseline Transition. 'AGPL" was only used with the baseline Fair GPL, and Prropsee was only used with the proposed method. Here, Propoeee corresponds the term that was used to balance Lo and C.. We refer to it as a Proposed in this section to distinguish it from the a value used by the baseline Fair GPI.
17

---

Under review as a conference paper at ICLR 2023
Harmonic AURDC and AUROC
AUROC and Autoc
Standard
Standdrrd
0.90
Filter
0.90
SLN Filter
Loss
0.80
CSIDN GPL
Trensition
0.80
Transition
CSIDN
Proposed
0.70
Proposed
0.70
Cleen Labels
Cleen Labels
0.60
0.05
0.10
0.15
0.20
0.10
0.20
0.30
0.40
Size of Minority Group (Proporiton)
Noise Rate
(a) Comparison of the proposed approach baselines when varying the size of minority set in whole dataset. The proposed approach outperforms the base- lines on minority sizes of 10% higher.
Comparison of the proposed approach to baselines with uniform random noise. Overall, the proposed ap- proach is robust to the noise rate and is competitive with the baselines.
Figure 4: Extra experiments on different data and noise settings.
C.2 NETWORK DETAILS
For the overall architecture, we used feed forward network with two hidden layers. The auxiliary prediction component was also implemented with two feed forward layers. All layer sizes are as described in Table 2 In addition, used the ReLU activation function. The complete implemen- tation can be found in the attached code.
C.3 SOURCE CODE
See the attached zip file. We will make the code public upon publication.
D IMPLEMENTATION DETAILS
For each dataset, we randomly split the data into 80/20% training/test, ensuring that data from the same individual did not appear across splits. For the Adult dataset, we used the test set provided and randomly selected 1,000 individuals from the training set. We then randomly selected 10% of training data for all datasets except MIMIC-III from each subgroup to be alignment points, thereby ensuring that they were representative of the overall population. For the MIMIC-II dataset, 2% from each subgroup
selected aligmment points due to the larger size of the dataset. Alignment points were selected randomly to simulate our setting of focus, where we have proxy labeling function and then randomly select subset of the data to chart review in order to validate the proxy function. Then, for all datasets, half the alignment points were then set aside as a validation set to use during training for early stopping and hyperparameter selection, while the other half remained in the training set. Later, in our experiments, we evaluate when the alignment set size varies and when the alignment set is biased. All approaches (i.e., baselines and proposed) were given access to ground truth labels for data in the alignment set so that some approaches did not have an unfair advantage.
All models were trained in Python3.7 and Pytorch1.7.1 Paszke et al. (017)) using Adam Kingma Ba )) Hyperparameters, including the learning rate, 1.2 regularization constant, and ob- jective function scalars (e.g., a), were tuned using random search, with budget of 20. We used early stopping (patience=10) based on validation set performance, where we aimed to maximize the HM. We report results the held-out test set, showing the mean and standard deviation over 10 replications.
E RESULTS: ADDITIONAL EXPERIMENTS (SYNTHETIC DATA)
We examined performance when we varied the size of the minority group (Figure 4a) and found that approach is no longer beneficial minority groups smaller than 10% of the dataset. This
18

---

Under review as a conference paper at ICLR 2023
Abletion
1.0
Alignment Only
+4g'
0.9
+4z'+ydo
larronnic AUROC
Proposed
0.8
0.7
0.6
Approach
Figure 5: Ablation study of proposed approach.
3,00
Hyperparenter Anabys d
Hyperpeanter Anelyale d)
Hyperpermmtrr Anelysds
0.50
5.55
3.40
0.0
2.53
0.55
0.00
0.30
072 3074
30'4
100
30
0.75
35
38
0.75
33
23"
(a) We varied on-
(b) We varied oz.
(c) We varied y.
Figure 6: Sensitivity analysis of proposed approach objective function hyperparameters.
presents limitation for future work. We also examined uniform random noise (Figure 4b At low noise rates, all approaches achieved similar performance. At noise rate 40%, all approaches outperformed the standard approach, which did not account for label noise. Our approach was still robust to the noise rate and competitive with the baselines. However, since our approach focused on learning the relationship between the features and label noise, expected that our approach would not provide as large of benefit on random noise. This is because the only signal our approach would learn is now based solely the observed label, which may get diluted out if there many input features.
In addition, also examine approach
closely by conducting ablation study and hyperparameter sensitivity analysis. In our ablation study (Figure(3), we began with training on only the alignment points (i.e., Step only), which achieved the worst performance. We then introduced Step 2 and added the remaining training data (i.e., non-alignment points) but only trained using Co This led to an improvement in performance, but not the level of the full approach. The next two ablations build on the previous one. In the first one, we added continued supervision on the alignment points with La, and observed an improvement retention of high quality data in this step. In the second one, we added continued supervision on the alignment points using Eg, and observed an even larger improvement. This is likely because including Co prevented the model from learning a solution where 3 was small for all instances, as previously discussed. Finally, we end with our full proposed approach, which performed noticeably better than each of the ablations, showing the importance of each component.
performance, likely due to the
In our sensitivity analysis (Figure 0 we tested how performance of the (full) proposed approach varied to changes in the hyperparameters 0i, 022, and 7. For each of these hyperparameters, we measured performance values between 0.01 and 100 on logarithmic scale while keeping the other two values constant at 1. We found that and were the most robust to changes in the value. We found that az was more sensitive, with values between 0.I and 10 generally working best.
19


Reviews:
[{"review_id": "pg1YPmH1Lm", "reviewer": "Reviewer_GGCC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem that current learning methods which handle specially noisy labels may increase the unfairness in prediction. Such a problem exists because in some data sets the minority group often has more noisy annotations. For such kind of data, the paper proposes a method targeting instance-dependent label noise by assuming a group of anchor points, which is sufficient to represent the whole dataset, is available. The paper then proposes to use two models simultaneously: one does the classification, and another predicts how noisy the given label is. The proposed training phase composes of two steps: one step uses only anchor points with both clean and noisy labels to train the two models, followed by another step that alternatively optimizes the two models and uses the predicted noisy level to weigh the non-anchor data points. Finally, the paper did some experiments showing that in various anchor points situations (the number of them, and the bias of them), the proposed method can achieve a good overall performance in both classification and fairness. ", "summary_of_the_review": "The paper has proposed a solution to an important research problem. The solution is not novel enough, but simple and effective. The concern may be on the assumptions made, and whether a semi-supervised solution can easily solve the problem given such strong assumptions. The clarity on the \"fairness\" part also needs to be increased (maybe significantly). ", "strength_and_weaknesses": "I think the paper considers a combination of two existing problems: fairness in learning from noisy labels, and instance-dependent noise. These two problems have already been considered in existing works. Nevertheless, the paper tries to provide an effective solution to such a combination of problems. From the experimental results, the paper's solution is effective. It is comparable to the Transition method with slightly better performance under some settings. However, from the illustration, the improvement seems marginal and is not significant if considering the standard deviation.\n\nThe paper has claimed to solve the \"balance between accuracy and fairness\". The experimental results have used the harmonic mean of an accuracy metric and a fairness metric to demonstrate the proposed method has achieved the balance. On the other hand, the wording of balance suggests that there is some controversy between the two, such that the current work needs to \"balance\" them. But in the proposed method, I do not see quite strong evidence that a special technique is introduced for increasing the balance at the sacrificing of the classification performance. The subgroup weighting used in the training phase may be a technique to increase the balance. But I think finding the appropriate subgroup is tricky. And such subgroup techniques have been used in previous works (as shown in Section 5->Label noise and fairness).\n\nOverall, the strength of the paper is that it tries to solve a combination of two important problems in learning from noisy labels. While the combination is straightforward, the resulting problem is an important problem and may draw attention from the research community. From the metric, the performance looks good and shows the effectiveness of the solution from a different technical perspective rather than the transition matrix one. \n\nThe weakness of the paper is that despite its claim, I do not think the problem of \"fairness\" has been clearly explained in the paper, and there are few technical contributions towards the \"fairness\" end. The assumption on the representativeness of the anchor points seems too strong and I do not think it could be easily satisfied practically. If saying the anchor points are sufficient, and the data satisfies the clustering distribution, then I am afraid a semi-supervised method could solve the problem easily without knowing any noisy labels. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RZIIseQ8tDX", "reviewer": "Reviewer_3mWV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a two-stage method to solve instance-dependent label noise learning with anchor points, taking care of robustness and fairness simultaneously.", "summary_of_the_review": "This paper aims to address instance-dependent label noise learning based on anchor points, which is not very novel for this area. Promisingly, they propose to consider fairness in this setting. Fairness problem is associated with instance-dependent label noise naturally. However, this paper lacks the corresponding quantative results to justify their advantages towards fairness.", "strength_and_weaknesses": "Strength:\n1. They link instance-dependent label noise and algorithmic fairness, which is a practical and significant in real scenarios.\n\n2. Their experiments are thorough enough and achieve relativelt good performance.\n\nWeaknesses:\n1. I think that considering the fairness in instance-dependent label noise learning is a key contribution in this paper. However, I cannot find any quantative results to evaluate the fairness of their proposed method and baseline methods. These results serve as the important support for this claim.\n\n2. Their technical novelty is limited. Anchor points have been thoroughly studied in recent years. Maybe I have not realized their unique technical contributions compared with previous literatures, e.g., Xia et al., 2020. This question should be clearly explained.\n\n3. I find their proposed method often fails to outperform Transition, e.g., Figure 2(b), Figure 4(b). Does this phenomenon indicates that their proposed method fails to address the severe label noise?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BIx_YswrVO", "reviewer": "Reviewer_wG5f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to study a \"fair\" model for learning with instance-dependent label noise. The authors proposed a simple yet intuitive solution, which is first pre-train a classifier $\\theta$ and a discriminator network $\\phi$ with anchor points. And then use the trained discriminator networks to discriminate the noisy samples from clean samples, and use the predicted clean samples to train the classifier. Which can be theoretically equivalent to the model trained with clean data.", "summary_of_the_review": "Overall, while the topic this paper aims to study is non-trivial and promising, the content of this paper suggests only a very limited relation to the fairness of label noise learning methods. Also, since the authors' assumption on anchor points is too strong, I cannot agree with the premises of this paper, nor the conclusion it draws. Therefore, I vote for the rejection of this paper.", "strength_and_weaknesses": "Pros:\n1. The question of interest is interesting and vital, the fairness of label noise learning methods is an important yet under-appreciated topic.\n2. Author attempts to provide theoretical proof to justify that their method is equivalent to the model trained on the clean domain in the ideal case.\n3. Authors provide comprehensive experiment settings, which helps the reviewer understands the detail of the experiments.\n4. The empirical performance of the proposed method is significant compared with selected baseline methods.\n\nCons:\n1. The biggest issue of this paper is the authors' assumption on anchor points, where authors assumed that \"a subset of data for which we know the observed and ground truth labels\" and \"is representative of the target population\". Generally, in the field of label noise learning, only the first assumption holds true, whereas the assumption that anchor points are representative of the target population is too strong. This problem directly hinders the theoretical soundness of this paper, because, in reality, the learned $\\theta$ and $\\phi$ are bound to be biased.\n2. While the authors claimed that they are trying to study the fairness of label noise learning methods, I fail to see any relevant definition or justification regarding fairness. As a contrastive example, [1] studies the well-defined counterfactually fairness, and [1]'s relation to fairness is clearly discussed. \n\nMinor issues:\n1. Authors are suggested to use more commonly used benchmarks such as CIFAR or MNIST.\n2. Authors are encouraged to include more recent baseline methods.\n\n[1] Wu, S., Gong, M., Han, B., Liu, Y., & Liu, T. (2022, June). Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning (pp. 927-943). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5ZJn3r-pPzH", "reviewer": "Reviewer_LP3P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "-This paper considers a general label noise problem (which does not assume any noise models). It proposes a fully heuristic approach: they firstly pre-train the classification model with some clean data, and then update the classification model by alternating between minimizing a loss on learning label confidence score and another loss on weighted training using the learned label confidence weights on each instance and a fairness weight on each subgroup. They also provide some favorable experimental results in the paper.", "summary_of_the_review": "-Overall the paper considers an interesting problem, but the current presentation seems not ready, and more experimental analysis is needed.", "strength_and_weaknesses": "Pros:\n\n-This paper tries to solve a general label corruption problem by considering both the accuracy and fairness performance measures, which is very **ambitious**. In the paper, they provide a heuristic approach and empirically analyze its effectiveness on some datasets with artificial instance-dependent label noise.\n\nCons:\n\n-First of all, the presentation of the paper is **very unclear** and needs significant improvement. For example, in Section 2 problem setup, the statement “learn a model using dataset $D=${$\\textbf{x}^i,y^i$}, $i=1,...,n$, that classifies each instance into one of $c$ classes based on $\\textbf{\\textbf{x}}$” sounds like it’s transductive learning rather than inductive learning; in the last sentence of this paragraph “using $\\theta (x)$ we obtain prediction $\\hat{y}$”, how do you obtain the prediction? do you use $\\hat{y}=argmax_{i\\in [c]} \\theta_i(\\textbf{x})$ or others, this should be made explicit; in the next paragraph, what does “$x_i>0.5$” mean? In proposed approach, the optimization problem is written as $\\min_{\\theta’,\\phi’} \\mathcal{L}_\\theta+\\alpha_1\\mathcal{L}_\\phi\\cdot \\mathcal{L}_\\theta$ but what are $\\theta’,\\phi’$? I hope all authors can carefully proofread the whole draft and **write rigorously**. \n\n-Also, the proposed two-step training approach is complicated and contains many heuristics. It would be very helpful if an algorithm can be provided.\n\n-In mainstream label noise literature, anchor points are defined in the clean data domain, that is, an instance $\\textbf{x}$ is an anchor point for the class $i$ if $p(y=i\\mid\\textbf{x})$ is equal to one or very close to one. However, the anchor point concept seems to be **misused** as it means an instance $\\textbf{x}$ that we know both the observed noisy label and its ground-truth label in this paper.\n\n-The proposed method has two stages, pretraining and alternative training. The latter is further based on two objective functions which are combinations of the clean-data loss component and noisy-data loss component. To analyze why such a complicated method works well empirically and how each component contributes to good performance, an **ablation study** removing pretraining and removing each loss component is definitely needed.\n\nQuestions:\n\n-The proposed method is a combination of many loss terms, and therefore has many combination parameters such as $\\alpha_1, \\alpha_2, \\gamma$, is the performance of the proposed method sensitive to these parameters? Sensitivity analysis may be necessary. \n\n-In the proposed method, $\\Phi$ is the network for predicting the label confidence score, but given its loss $\\mathcal{L}_\\Phi$ I don’t see how the instance-dependency is modeled and learned. This needs to be clarified, otherwise, the problem setting of this paper should not be instance-dependent label noise, but general label corruption.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject