When Optimizing  $f$-Divergence is Robust with Label Noise
Authors: Jiaheng Wei
Yang Liu
Venue: ICLR 2021 Poster (2021)
Keywords: $f-$divergence
robustness
learning with noisy labels

Abstract: We show when maximizing a properly defined $f$-divergence measure with respect to a classifier's predictions and the supervised labels is robust with label noise. Leveraging its variational form, we derive a nice decoupling property for a family of $f$-divergence measures when label noise presents, where the divergence is shown to be a linear combination of the variational difference defined on the clean distribution and a bias term introduced due to the noise. The above derivation helps us analyze the robustness of different $f$-divergence functions. With established robustness, this family of $f$-divergence functions arises as useful metrics for the problem of learning with noisy labels, which do not require the specification of the labels' noise rate. When they are possibly not robust, we propose fixes to make them so. In addition to the analytical results, we present thorough experimental evidence. Our code is available at https://github.com/UCSC-REAL/Robust-f-divergence-measures.

Published as a conference paper at ICLR 2021
WHEN OPTIMIZING f /-DIVERGENCE IS ROBUST WITH LABEL NOISE
Jiaheng Wei and Yang Liu" Department of Computer Science and Engineering University of California, Santa Cruz Santa Cruz, CA 95060, USA
{j tjiahengwei, yangliu) @ucsc.
ABSTRACT
We show when maximizing a properly defined f-divergence measure with re- spect to a classifier's predictions and the supervised labels is robust with label noise. Leveraging its variational form, we derive nice decoupling property for family of f-divergence measures when label noise presents, where the di- vergence is shown to be linear combination of the variational difference de- fined on the clean distribution and bias term introduced due to the noise. The above derivation helps us analyze the robustness of different f-divergence fune- tions. With established robustness, this family of f-divergence functions arises as useful metrics for the problem of learning with noisy labels, which do not require the specification of the labels' noise rate. When they possibly not robust, we propose fixes to make them so. In addition to the analytical results, we present thorough experimental evidence. Our code is available at https: //github. cttp UCSC REAL/Robuse f d-divergence measures
INTRODUCTION
A machine learning system continuously observes noisy training annotations and it remains chal- lenge to perform robust training in such scenarios. Earlier and classical approaches rely on estima- tion processes understand the noise rate of the labels and then leverage this knowledge to perform label correction (Patrini et al.] [017 Lukasik et al. 2020). or loss correction (Natarajan et al.] 2013] Liu & Tao 2015 Patrini et al. (017)) both, among many other more carefully designed ap- proaches (please refer to our related work section for more detailed coverage). Recent works have started to propose robust loss functions or metrics that do not require the above estimation (Charoen- phakdee et al., 2019; Xu et al. 22199; Liu & Guo (2020; Cheng et al. [021). Clear advantages of the latter approaches include their casiness implementation, as well as their robustness to noisy estimates of the parameters. This work mainly contributes to the second line of studies and aimed
to propose relevant loss functions and measures that are inherently robust with label noise.
We start with formulating the problem of maximizing an fidivergence defined between classifier's prediction and the labels:
h argmax Dj (Pxxy Q
(1)
h
where in above Dy is an f-divergence function, P and Q are the joint and product (marginal) distribution of the classifier h's predictions on feature space X and label Y. Though optimizing the /-divergence measure is in general not the same as finding the Bayes optimal classifiers, we show these measures encourage a classifier that maximizes an extended definition of f-mutual information between the classifier's prediction and the true label distribution. We will also provide analysis for when the maximizer of this f-divergence coincides with the Bayes optimal classifier.
Building on a careful treatment its variational form, we then reveal a nice property that helps establish the robustness of the f-divergence specified in Eqn. ( the variational difference term defined with noisy labels is an affine transformation of the clean variational difference, subject to
"Corresponding author. y i me@ussc.edu.

---

Published as a conference paper at ICLR 2021
an addition of a bias term. Using this result, analyze under which conditions maximizing an f- divergence measure would be robust to label noise. In particular, we demonstrate strong robustness results for Total Variation divergence, identify conditions under which several other divergences, including Jenson-Shannon divergence and Pearson X divergence, are robust. The resultant f- divergence functions offer ways learn with noisy labels, without estimating the noise parameters. As mentioned above, this distinguishes our solutions from major line of previous studies that would require such estimates. When the f-divergence functions are possibly not robust with label noise, our analysis also offers new way to perform "loss correction". We'd d like to emphasize that instead of offering one method/loss/measure our results effectively offer a family of functions that can be used to perform this noisy training task. Our contributions summarize follows:
We show a certain set of f-divergence measures that are robust with label noise (some under cer- tain conditions). The corresponding f-divergence functions provide the community with robust learning measures that do not require the knowledge of the noise rates.
When the f-divergence measures are possibly not robust with label noise, our analysis provides ways correct the f-divergence functions to offer robustness. This process would require the estimation of the noise rates and our results contribute new ways to leverage existing estimation techniques make the training more robust.
We empirically verified the effectiveness of optimizing f f-divergences when noisy la- bels present. We opensource our solutions at |attps://github com/ CCo//UCSC-REAL Robust- www.----vveeeeeeeeeeeeee
1.1 RELATED WORKS
The now most popular approach of dealing with label noise is to first estimate the noise transition
matrix and then use this knowledge to perform loss or sample correction (Scott et al.
2013 3013|Nataraja Natarajan
2013 [ Patrini al. [017]] Lu al. 2018: Han et al. 2018 Tanaka et al.. [0188] Yao al. 2020
Zhu et al. 2021). In particular, the surrogate loss (Scott et al. 2013 Natarajan et al. 2013 Scott
2015 Van Rooyen et al.) 2015 Menon et al. 2015) uses the transition matrix define unbiased
estimates of the true losses. Other works include (Sukhbaatar & Fergus 2014 Xiao et al.) 2015),
which consider building a neural network facilitate the learning of noise rates or noise transition matrix. Symmetric loss has been studied and conditions have been identified for when there is no need to estimate noise rate (Manwani & Sastry 2013] Ghosh et al., 2015) 2017; Van Rooyen et al. 2015] Charoenphakdee et al. (011)). Nonetheless, remains a challenge to develop training approaches without requiring knowing the noise rates for more generic settings.
More recently. (Zhang & Sabuncu 2018 Amid et al. 2019) proposed robust losses for neural net-
works. When noise rates are asymmetric (label class-dependent), (Xu et al. 2019) proposed an information-theorete loss that is also robust to asymmetric noise rates. There are also some trials on modifying the regularization term to improve generalization ability with the existence label noise (Jenni & Favaro, 2018 Yi & Wu 2019), and providing complementary negative labels (Kim)
et al. 2019). Peer loss (Liu & Guo 2020) is a recently proposed loss function that does not require
knowing noise rates.
f-divergence is a popular information theoretical measure, and has been widely used and studied. Most relevant to us, f-GAN was proposed in (Nowozin al., 2016) to study f-divergence in training generative neural samplers. To our best knowledge, ours is the first to study the robustness of f- divergence measures in the context of improving the robustness of training with noisy labels.
LEARNING WITH NOISY LABELS USING f /-DIVERGENCE
Our solution ties to the definition of f-divergence. The f-divergence between two distributions P and Q with probability density function p and q being measures for Z E Z defined as:
p(Z)
Dy(()lQ) (())
dz
We use 7 instead of X conventionally done for good reason will be reserving X explicitly denote the features.
2

---

Published as a conference paper at ICLR 2021
f()) is a convex function such that f(1) 0. Examples include KL-divergence when f(u) log and Total Variation (TV) divergence with f(v) = 2/1 i 1|. Other examples can be found in Table
Following from Fenchel's 's convex duality, f f-divergence admits the following variational form:
sup Dy(PIIQ)= g:2-ado((f-)
Ez [g( Rz-q [f" (g(
where f is the Fenchel duality the function f(-). which is defined as (u) = supben [v)). We use dom(f") ) to denote the domain of f*
We consider the classification problem of learning a classifier - y that maps features X E X to labels Y E } == 21. K), where in above X Y denote the random variables for features and labels. X x Y jointly draw from a distribution D. For clear presentation, we will often focus on presenting the binary classification setting = {-1, +1}, }. but most of our core results extend to multi-class classification problems, and we shall provide corresponding justifications.
Instead of having access sampled training data from Xx XxY, consider setting with noisy labels where the noisy label Y generates according a transition matrix T defined between Ý and the true label Y. The (i,j) element of T is defined Tıj IP(Y j|Y i) where i, E {1,..., K}- For the ease of presentation, when we present for the binary case, we adopt the following notation:
e+ = P(Ý - +1), e == P> = +1|y = -1) + e_ < 1. Suppose we have access to noisy training dataset {In, In Jn where On generates according to Y.
2.1 LEARNING USING D
We will start with presenting our idea of training classifier using D, with the clean training data. Then we will proceed to the case with noisy labels. For an arbitrary classifier h, let's denote by Paxx the joint distribution hX)) and Y:
Joint distribution: Paxx == ((((X) = Y = =/), v.y E y.
And we use Q Y to denote the product (marginal) distribution of fxX) and Y:
Product distribution: QAXY (() y) IP y'), y. E y.
When it is clear from context we will also shorthand the above two distributions as P and Q. We formulate the problem of learning using f-divergence follows: the goal of the learner is to find classifier that maximizes the following divergence measure between P and Q:
Learning using Dy: hj argmax D, (Paxy
(3)
h
Effectively the goal is find classifier that maximizes the divergence between the joint dis- tribution and the product distribution. Define a f-mutual information based on f-divergence: My(h(X);Y) Dy (Paxy
(Paxy|lQnxx> [Qxxx) equivalently the maximization in Eqn. (3) tries to find the classifier that maximizes the f-mutual information between classifier's output distribution and the true label distribution. A notable example is when (wv = log v, the corresponding D, and My become the famous KI. divergence and the mutual information. It is important to note gen- eral maximizing (f-) mutual information between the classifier's predictions and labels does not promise the Bayes optimal classifier ht = argmax, P(h(X) = Y). Nonetheless, maximizing it of- ten returns quality one. We provide further analysis in Section 2.2
Variational representation As mentioned earlier, f-divergence admits a variational form which further allows us to focus on maximizing the following variational difference:
h argmax sup Ez~Paxy [g(Z)] - Ez~ [f*()
where we use Z to shorthand the tuple (h(X), Y]. Denote the variational difference as follows:
VD, == Ez-Paxy [o(2)] Ez~Quxy x [5 (o(Z)))
(4)
Let g" be the corresponding optimal variational function g for VD,(h.g). This variational form allows us to training dataset (xx,, ) to perform the above maximization problem listed in Eqn. (3) (Nowozin et al.) (016)). A list of f-divergence functions together with the optimal variational/conjugate functions g/ is summarized in Table
3

---

Published as a conference paper at ICLR 2021
Name
dom
("(w)
Total Variation
1 [p(a) d())dds
Nign
p(#)
g(=)
2'
p(z) log 2p(a)
2y(=)
2p(+)
Jemson Shannon
a(z) log
log
log
log
e²))
p(=) + ((=)
p(x) + +(())
p(=) + ((=)
(0(x) p(a))
Pearson
de
R
uu
p(s)
KL.
J p(=) log
pis)
log
p(s)
R
[(a)
qfa)
Table Dys, optimal variational g (g"), conjugate functions (f*). A more complete table, including Jeffrey, Squared Hellinger, Neyman X², Reverse Kl., is provided in the Appendix.
2.2 How GOOD IS hG?
As we mentioned earlier, maximizing our defined f-divergence measures (or maximizing the f- mutual information) between the classifier's predictions and labels is not always returning the Bayes optimal classifier. However, for a binary classification problem, we prove below that with balanced dataset, maximizing Total Variation (TV) divergence returns the Bayes optimal classifier:
Theorem 1. For TV, when P(Y = 11 (balanced), b is the Bayes optimal classifier.
Remark 2. The above theorem extends to the multi-class setting when we restrict attentions to confident classifiers. See Appendix for details.
The above observation is not easily true for other f-divergence. Nonetheless, denote by Y = x) the Bayes optimal label for an instance x: Y* = x) = argmax, P(Y vl X x). Denote by PaxY-, , Qxxy- the joint and product distribution P. Q defined w.r.t. h(X) and Y*. We prove:
Theorem 3. When P(Y* = +1) = P(Y = -1) (balanced), maximizing Dy [[Quxy-) returns the Bayes optimal classifier; if fv) is monotonically increasing in | 1| on dom(f).
For example, Pearson 2²2 (f(v) (u 1)2) satisfies the monotonicity condition. In practice, when the label distribution P(Y = x) has small uncertainties, the ground truth labels are approxi- mately equivalent to the Bayes optimal label. Therefore, the above theorem implies that maximizing is also likely to return high-quality classifier for other f-divergences.
2.3 LEARNING WITH NOISY LABELS
Consider an arbitrary classifier h. Denote by P Paxy the joint distribution of (xX) and Y:
Joint noisy distribution: PAxx == ((h(X) = y = v/), y.v/ E V.
Similarly, we use QAXy to denote the product (marginal) distribution of h(X) and Y:
Product noisy distribution: Qxxy P(N(X) = v) PP--P-/))) = v)) y. v/ e eV.
When it is clear from context, we shorthand using Q. We are interested in understanding the robustness in maximizing (, (1Qux))))) Using training samples {x J there exists
algorithms to compute the gradient of D, leveraging its variational form (Nowozin et al. 2016), such that one apply gradient descent ascent to optimize it. We provide details in Section 5
VARIATIONAL DIFFERENCE WITH NOISY LABELS
For an arbitrary g. define the variational difference term w.r.t. the noisy label as follows:
(h, = E2~P Rx [0(2)] Ez- [M((Z)]]
(5)
where 2 denote [K(X), VI. Denote by ÿ" the corresponding optimal variational func- tion for (D. g). In this section, we show that the variational difference term under noisy labels is closely related the variational difference term defined on the clean distributions P, Q. Define the following quantity: Ay(h, = Ex|g(E(( (g(1((),,)) - Ex [s" (g(h(X), v))], for example
4

---

Published as a conference paper at ICLR 2021
(h, = Exlg(h(X), +1) - Ex [f (o(h(X), For binary classification problem, further denote by Bias (h, g) == c- c. (h, + (h, g). We derive the following fact:
Theorem 4. For binary classification, the variational difference between the noisy distributions P and Q relates to the one defined on the clean distributions in the following way.
(h. 9) =
c- g) + Bias, (h,
(6)
The above decoupling result is inspiring: Bias (h, introduced by label noise. If this term has negligible effect in the maximization problem, maximizing the noisy variational difference term will be equivalent to maximizing (1 - e_) VD, g), and therefore the clean variational difference term. If the above is true, we have established the robustness of the corresponding f-divergence. This result also points out that when the effects from the bias term are non-negligible, finding ways to counter the additional bias term will help us retain the robustness of D, measures. Next we show that Theorem 4 extends to the multi-class setting under two broad families of noise rate models, both covering the binary setting as a special case.
(h,g) g) can be viewed the additional bias term
Multi-class extension of Theorem 4: uniform off-diagonal case We first consider the following transition matrix: uniform off-diagonal transition matrix, where ej -Tij Vi & j. that is any other classes i * j has the same chance of being flipped class j. The diagonal entry Tı (chance of correct label) becomes 1 - Σjソ; ej. We further require that E, 9j < <.. Note that the binary noise rate model is easily a uniform off off-diagonal transition matrix.
Theorem 5. (Multi-class) For uniform off diagonal noise transition model, the noisy variational difference term relates the clean one in the following way:
VD,(,,9)= = Σ c) VD, +
Σ ej (h) 3=1
If we define B (h, == 1 E
ej (h, g). reproduced the results in Theorem for binary case, relabel class 1 +1,2 -1 Then 6 F(Ý +1|Y -1) e-, P(Ý -1-Y = +1) = c+. Another case of noise model we consider is sparse noise. Mathematically, assume K is an even number, sparse noise model specifies 2
K disjoint pairs of classes (fe,Je) where
E ] and ie < <e. The labels flip between each pair. We provide details in the Appendix.
4 WHEN D, IS ROBUST WITH LABEL NOISE
Denote by H an arbitrary hypothesis space for training candidate classifier h. We will focus on H throughout this section, and with abusing notation bit, let h; = argmaaлęи Dy (Paxyl1Qbxy). We first define formally what we mean by robustness of (PhxxxllQQnx
Definition 1. Dy [[xxx)) is H-robust if h; = argmax, (((Pnxy11@nxx))
The above definition is stating that the label noise does not disrupt the optimality of hi when maxi- mizing Dy ((xx))) instead of Dy [[Quxx).
4.1 IMPACT OF THE BIAS TERMS
In this section, we take a closer look at the Bias terms and argue that they have diminishing effects as compared to the VD terms when label noise increases. Recall 9", "g" are the corresponding optimal variational functions for (h, g) and (D,
Total variation (TV) For TV, since f(v) = f/v 1/. f*(u) = u. we immediately have Vy g(h = v Ex If (g(h(X), v))l = O, Vy. and further Bias, (h.g) g) = This fact helps establish the robustness of TV divergence measure (Theorem 7.
(g(h = v/ y)) = and therefore (y Exlg(h(XX, y)]
5

---

Published as a conference paper at ICLR 2021
Other divergences The above nice property generally does not hold for other f-divergence func- tions. Next we focus on the binary classification setting and prove the following lemma:
Lemma 1. For f-divergence listed in Table 6 (Appendix), Blas,
O ((1 e e-)²).
Note the variational form will be used when optimizing Dy be using Ø*). This lemma simplifies Eqn. to VD,(h,g) (h,g) x VD (h.g) + O e+ e-). Since e e- S 1, when the noise rate e4 + e_ is high, the effect of Bias term diminishes. When the Bias term becomes negligible, we will have (D,(h.g) x (h, g) if + e- + 1, establishing the fact that optimizing (D, 9) approximately the same as optimizing VD,(h.g)
(and therefore will
4.2 How ROBUST ARE Dys?
We first prove the following result:
Theorem 6. Dy is H-robust when Bias, (h. g) satisfies either of the following conditions: (I) vh H. B (h, .) = const.; (II) vh E H. h7. Bias,(h, Bias, (h:;
Theorem 6 gives sufficient conditions when the Bias term does not get in the way of reaching the optimality hj. Intuitively, when Bh 9²) is upper bound of Bias, (h, ÿ*), the Bias term will not interfere with the convergence of the VD term. Next we provide specific examples of f- divergence functions that would satisfy these conditions.
Total Variation (TV) is robust For TV, the fact that (h, g) = 0 allows us to prove:
Theorem 7. For TV divergence, Bias, (h, g) = const. and (((PPx)(1QXxx)) is H-robust with label noise for any arbitrary hypothesis space H.
This result establishes TVV as a strong measure that does not require specifying the noise rates.
Divergences that are conditionally robust Other divergences functions do not enjoy the above nice property as TV has. The robustness of these functions need more careful analysis. Define the following measures that capture the degree a classifier fits to a particular label distribution:
Definition 2. The fitess of h to R E (Y, (Y is defined as FIT(h = R = s/) = v(
P(h(X)=))
FIT measures capture the degree of fit of the classifier to the corresponding label distribution. A high (1 = Y y) (same label) indicates potential overfit to the noisy label. Denote by
H = h H. min (I y. Y y) 2 max (IT y, Y v) 1) (h))
The 1 in the "2 1" above corresponds to the FIT for a random classifier. H contains the classifiers that are likely overfit to the noisy labels. We argue, and also observed in training, that H is the set of classifiers the training should avoid converging to, especially when the training only sees noisy labels. Suppose IP(Y +1) = P(Y -1) (balanced clean labels) and e e (symmetric noise rate), we have the following theorem for binary classification:
Theorem 8. f-divergences listed in Table 6 (Appendix, except for Jeffrey) are H -rouut...
4.3 MAKING D, MEASURES ROBUST TO LABEL NOISE
For the general case, to further improve robustness of Dy measures, we will need to estimate the noise rates (e.g., e-, and then subtract Bias, (g, h) from the noisy variational difference term to correct the bias introduced by the noisy labels. As corollary of Theorem f we have:
Corollary 1. Maximizing the following bias-corrected VD, (h, g) defined over P and Q leads to hy
h = argmax sup EZ KEH
[o(2] - E Eg~
[ Bias, (h,
becomes the same with maximizing the divergence defined on the clean distribution (1 Σ/= e₁) (h, g). The Corol- lary follows trivially from this fact. The calculation of the Bias terms will require the inputs of
By removing the Bias, term, maximizing

---

Published as a conference paper at ICLR 2021
noise rates. Our work does not intend to particularly focus noise rate estimation. But rather, we can leverage the existing results in performing efficient noise rate estimation. There are existing literature estimating noise rates (noise transition matrix) which can be implemented without the need of ground truth labels. For interested readers, please refer to (Liu & Tao 2015 Menon et al.
2013 HHarish et al., 2016; Patrini al. 2017 Arazo al., 2019 Yao et al. 2020 Zhu al. 2021 We will test the effectiveness of this bias correction step in Section
5
EXPERIMENTS
In this section, validate our analysis of D, measures' robustness via of empirical evaluations on datasets: MNIST (LeCun et al. (1998)) Fashion-MNIST (Xiao et al. (2017)), CIFAR-10 and CIFAR- 100 (Krizhevsky al. (2009), ). and Clothing IM (Xiao et al. (2015)). Omitted experiment details are available in the appendix.
Baselines We compare our approach with five baseline methods: Cross-Entropy (CE), Back- ward (BLC) and Forward Loss Correction (FLC) methods as introduced in (Patrini et al. 2017). the determinant- based mutual information (DMI) method introduced in (Xu et al. 2019) and Peer-Loss (PL) functions in (Lu & Guo 2020 BLC and FLC methods require estimating the noise transition matrix. DMI and PL are approaches that do not require such estimation.
Noise model We test three types of noise transition models: uniform noise, sparse noise, and ran- dom noise. All details of the noise are in the Appendix. Here we briefly overview them. The uniform and sparse noise are as specified at the end of Section for which our theoretical analyses mainly focus on. The noise rates of low-level uniform noise and sparse noise are both approximately 0.2 (the average probability of label being wrong). The high-levels are about 0.55 and 0.4 respectively. In the random noise setting, each class randomly flips to one of 10 classes with probability (Ran- dom p). For CIFAR-100 100, the noise rate of uniform noise about 0.25. The sparse label noise generated by randomly dividing 100 classes into 50 pairs, and the noise rate is about 0.4.
Optimizing (y hxx ( using noisy samples With the noisy training dataset {Znz: Fn Ja we optimize D (P x using gradient ascent of its variational form. Sketch is given in Algorithm| [ For the bias correction version of our algorithm, the gradient will simply include the BBias, (h,g) The variational function g can be updated progressively or can be fixed beforehand using an approximate activation function for each (see e.g.. (Nowozin et al. 2016)).
Algorithm Maximizing D, measures: one step gradient
1: Inputs: Training data (I J.) I. variational function "", conjugate f classifier h..
i 2: Randomly sample three batihe {( Bn) {( " {(x) 1 from
((In, ) { R= simulate samples ~ P; [xr,, ) to simulate Q
3: Use hr,z, [7n] to denote model prediction on x,, for label In. E ((......)).. ((2 to denote the empirical sample mean calculated using the mini-batch data.
4: At step 6, update hú by ascending its stochastic gradient with learning rate T):
he+1 = h1 1h Vn. ((o..).)) 15" (hr,,,, (7,))) -E ((tt,,0))) 15" (5 (7))
Tips: In practice, we suggest (also implemented in our experiments) using the fixed form of g which appears gf (v) in Table (appendix).
5.1 How GOOD is h2 ON CLEAN DATA
As a supplementary of Section 2.2 we validate the quality of h;; on clean dataset of MNIST, Fashion MNIST, CIFAR-I0 and C00. In experiments, since the estimation of product noisy distri- bution are unstable when trained on CIFAR-100 training dataset, we use CE as a warm-up (120 epochs) and then switch to train with D, measures. For other datasets, we train with D, measures without the warm-up stage. Results in Table demonstrate that optimizing f diivergencce on clean dataset returns high-quality h; by referring to the performance of CE. Even though D, measures
7

---

Published as a conference paper at ICLR 2021
can't t outperform CE clean dataset, do observe that the gap between CE and D, measures negligible, for example, the largest gap of Total- Vara--aaittionn (TV) is only 0.81% among four datasets.
Dataset
CE
TV
Cap
J-S
Gap
KL
Gap
MNIST 99.39(99 38-00.01)
99,37(99 34-0..2)
-0.04
99.35(99.3 +:004)
-0.07
99.31 21 +0006
-0.17
Fashion MNIST 00.44.9033410.111
89,98(89.964 0.06)
40.40
90.40/9.17110.2
-0.17
90.19689.96 10.14)
.0.38
CIFAR-I0
47.5893.4440008)
92.0092266-00133)
23000
-1.24
15.5500.84+0.15)
-3.09
CIFAR- 100
3347013399:0000
13,4373,39=000
0,00
73:4703.26+00177
-0.13
73.3873. 16 10)
-0.23
Table Experiment results comparison on clean datasets: We report the maximum accuracy of CE and each Dj measures along with (mean = standard deviation); Gap: mean performance comparison w.r.t. CE. Numbers highlighted in blue indicate the gap is less than 1%.
5.2 ROBUSTNESS OF Dj MEASURES
1.0
100
/divergence value
0.6
Test Accuracy
0.4
-40
0.2
TVcccc
_V_div
-20
J5, acc
J5. div
0.1
0.2
0.3
0.4
0.5
0.8
Noise rate
Figure 1; Robustness of TV, JS, PS divergences.
As demonstration, apply the uniform noise model CIFAR-10 dataset test the robustness of three D, measures: Variatvoi (TV), JensonShammon (JS) and Pearson (PS). We trained models with Dj measures using Algorithm [ on 10 noise settings with an increasing noise rate from 0% to approximately 81%. The visualization of the D, values and accuracy w.r.t. noise rates are shown in Figure [ Both the D, values and test accuracy are calculated on the reserved clean test data. We observe that almost all D5 measures are robust to noisy labels, especially when the percentage of noisy labels is not overwhelmingly large, e.g., S 70%. Note that the curves for other f-divergences almost the same as the curve of total variation (TV), which is proved to be robust theoretically. This partially validates the analytical evidences we provided for the robustness of other f-divergences in Section 4.1 and 4.2
5.3
PERFORMANCE EVALUATION AND COMPARISON
From Table several Dy measures arise as competitive solutions in variety noise scenarios. Among the proposed f-divergences, Total Variation (TV) has been consistently ranked as one of the top performing method. This aligns also with our analyses that TV is inherently robust. For most settings, the presented f f-divergences outperformed the baselines we compare to, while they fell short DMI (once) and Peer Loss (5 times) on several cases, particularly when the noise sparse and high. The sparse high noise setting tends to be challenging setting for all methods. We conjecture this is because sparse high noise setting creates highly inbalanced dataset, model training is more likely to converge to a "sub-optimal" carly in the training process. It is also possible that with sparse noise, the impact of Bias terms becomes non non-negligibbe. We do observe better performances with very careful and intensive hyper-paramettr tuning. but the results are not confident and we chose to not report it. Fully understanding the limitation of our approach in this setting remains an interesting on-going investigation.
In Table 4 (full details on MNIST and Fashion MNIST can be found in Appendix), we use noise transition estimation method in (Patrnni al. (2017) to estimate the noise rate. The estimates help
8

---

Published as a conference paper at ICLR 2021
us define the bias term and perform bias correction for Dj measures. We observe that while adding bias correction can further improve the performance of several divergence functions (Gap being positive), the improvement or difference is not significant. This partially justified our analysis of the bias term, especially when the noise is dense and high (uniform and random high).
Dataset
Noise
CE
BLC
FLC
DMI
PL
TV
J-
KL
Spane, Low
97211
95.23
97.37
91.76
98.59
W. 10.8)
99.1509903
0.09)
99.211199115 15 0.05)
Sparss, Nigh
45.35
5536
49.67
.9.61
00.27
28.2155-7::11.111
88
49.24699.17 07:00
Unitoem,
94.27
55.51
90.72
99.00
99.199905
0.04)
99.13999.0
Unitorm, llihh
95255
8592
$3.35
95.50
97.77
98.0997.666...1
97.8697.71
2.010
96. 66.19997.8 0.18)
Kundom (022)
98.56
97.46
97.81
98.82
09.25
99.2019.27 0..00
99 291999999 19:20.06)
Random (0.5)
97300
93.52
87.24
95.47
98.52
95.81198.73 0.06)
96,72198.63 0.08)
98.76198.65 -0.10)
84.36
50.02
58.15
55.05
88.32
89.74
0.33)
88,80158, 79 10.01
59.77189.88 42 00.44)
Fishion
MNIST
Uniform, Low
Spurse,
K238
$4.38
56.38
47.63
83.89
42.15
89.31
51.92
89,0088.75 1:001
15,860
47,46:46
18,38338. 0.110.
3265, 16: 1.TT)
0.06)
Unifoem, High
79.52
78.10
82.41
77.94
84.69
85.55/85.07
0.330
85.62185.8 39 0.33)
85.69185. 43 0.330
Random (022)
K5.47
83,10
77.61
36.21
89.75
90,22/90.09
19.0090000 1.0.9
89,75389
45.20.44
89.21189 05 0.
Randon (0.7)
R206
78.1
73.42
80389
87.22
87.79187.33 0.29
57 57.06157 00 0.00)
Sparse,
76.17
92.32
91.35
91.81091
56 10)
0.31
Sparse, High
61.81
56.30
66.12
27.94
69.70
63.90062.22 25 2--+-11.000
62.35:65.275
27: 1,34)
46.35146-43 4.000)
CIFAR 10
Uniferm, Low
85.68
72.73
71.12
90.39
91.70
V2.10.92.01
0.099
91.2291447
91.26/92.08 0.12)
Unitsem, High
71.38
54.41
6422
82.68
85.42
85.36/85244::000
84,39/84.35
84.36184.19
Kandom (0.5)
78.40
59.31
35.06
86.47
97.28887.803--0011
56.93/6685
Random (0.7)
68:26
38.39
$4.39
77.91
57.81
80.5%
1.0.10)
80.50030.277
0.15)
78.9378.99
0.30)
Uniform
63.87
51.40
0000
04.39
67.94
09.15008.901
0.117
0.37)
69.13/68.80 in 0.211
68.79168.01 0.11)
Sparse
40145
36337
43.39
40.35
44.25
38.05 30000
97744076600000
CIAAR-100
(0:2)
62.92
70,4N70.22::111
70.40/70.12
70.28170.00 0.14)
Kandeen (0L.)
56.2
$5.88
36.06
39.00
02.14613911000
61.88661 0.11))
Table 3: Experiment results comparison (w/o bias correction): is highlighted in blue. We report the maximum accuracy of each Dy measures along with (mean - standard deviation). All f-divergences will be highlighted if their mean performances are better (or no worse) than all baselines we compare to. A supplementary table including Pearson X²2 and Jeffrey (JF) is attached in Table (Appendix)
The best performance in each setting
Nosse
Cap
Gap
KL.
Gap
Gap
Sparse, Low
91
4R09 424
91.73191 57
40.11
91,45/91
00.21)
40.00
Sparse, High
46.45(46.31 4..14)
-2.88
903114990---444
4005
46.39/46.52- 111466622-00000 0.05)
+0.04
46.25/48.77
+0.500
+0.04
Tnilkam, Low
-0.44
2012
-1.34
92.1992 1168)
40.02
Unifona High
K37998361 +0.12
40.18
8198183.79-00 0.12)
40.13
Table 4: D, measures with bias correction on CIFAR-10: Numbers highlighted in blue indicate better than all baseline methods; Gap: relative performance w.r.t. their version w/o bias correction (Table 3) those red indicate better than w/o bias correction.
ClothingIM Clothing J is large-scale clothes dataset with comprehensive annotations and can be categorized as feature-dependent human-level noise dataset. Although this noise setting does not exactly follow our assumption, we are interested in testing the robustness of our f f-divergence approaches. Experiment results in Table 5 demonstrate the robustness of the D, measures. TVV and KL divergences have outperformed other baseline methods.
Dataset
Noise
CB
BLC
FLC
DMI
PL
T-V
J-S
Pear
KL
Jeffrey
Clothing
Human Noise
68. $14
69.13
69.84
72.46
72.60
73.09
72.32
72.22
72.65
72.46
Table 5: Experiment results comparison on Clothing M dataset.
6 CONCLUSION
In this paper, we explore the robustness of properly defined f-divergence measure when used to train classifier in the presence of label noise. We identified a set of nice robustness properties for a family of f-divergence functions. We also experimentally verified our findings. Our work primarily contributed to the problem of learning with noisy labels without requiring the knowledge of noise rate. Beyond this noisy learning problem, the derivation and analysis might be useful for understanding the robustness of f-divergences for other learning tasks.
Acknowledgement This work is partially supported by the National Science Foundation (NSF) under grant I1S-2007951 and the Office of Naval Research under grant N00014-20-1-22
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, pp. 14987-14996, 2019.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. arXiv preprint arXiv: 1238, 2019.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learn- ing from corrupted labels. arXiv preprint arXiv: 1901 04 14, 2019.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong. Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2021. http:::tttss/trrtteeettst www.rrewwwwweeeeoruununu
net. wtt.rummmmmnnieneeerrr
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. COLT 2010 The 23rd Conference on Learning Theory, 12:257-269, 2010. ISSN 15324435.
Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant label noise. Neurocomputing, 160:93-107, 2015.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Ramaswamy Harish, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International conference on machine learning, pp. 2052-2060, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 618-633, 2018
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Niul: Negative learning for noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 101-110, 2019.
Diederik P. Kingma and Jimmy Ba. abs/1412.69800, 2014.
Adam: A method for stochastic optimization. CoRR,
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-222 2324, 1998.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38:447 461, 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. ICML, 2020.
Nan Lu, Gang Niu, Aditya K Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint arXiv: 1808. 10585, 2018.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishma Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.02819, 2020.
10

---

Published as a conference paper at ICLR 2021
Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions cybernetics, 43(3): 1111111611111 2013.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from cor- rupted binary labels via class-probability estimation. In International Conference Machine Learning, pp. 125-134, 2015.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204 2013.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances neural information processing systems, pp. 271-29., 2016.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. AISTATS, 2015.
Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pp. 489-511, 2013. 2013.
Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv: 2080, 2(3):4, 2014.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In Advances in Neural Information Processing Systems, pp. 18, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mman a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: An information-theorrtt noise- robust loss function. NeurlPS, arXiv: 1909.0338,, 2019.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix in label-noise learning. arXiv preprint 2-Xiiv20006078000 2020.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels, 2018.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative anchor points when learning with noisy labels. arXiv preprint arXiv:2102.05291, 2021.
11


Reviews:
[{"review_id": "qJIP8n49ibc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study optimising f-divergence for supervised learning in the context of label noise. They show robustness to noise for certain families of f-divergence functions and devise a technique for minimising bias for other f-divergence functions. They evaluate their method against pre-existing methods as baselines on a variety of learning tasks including simulation and real-world data. There are some minor issues to address but overall a good submission.\n\n# pros\n\n- Easy to read paper, the authors guide the reader well through the theory\n- proofs are sound\n- good set of experiments\n- results are interesting and applicable in many domains, of broad interest to the community\n\n# cons\n\n- The experiments are not described in sufficient detail. It is unclear what h is used for experiments and what variational function was used exactly. The authors state g* can be updated progressively or fixed beforehand, but neglect to say what they did for experiments. \n- The authors mention the BLC and FLC methods require estimating the noise transition matrix: how was this done?\n\n# minor things\n\n- typo on page 3: f(v)= log v for KL divergence", "extra_scores": {}, "extra_text": {}}, {"review_id": "f7M25IBb8aZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nThe manuscript identify robustness properties of the f-divergence optimization when noisy labels presents. The f-divergence measure is defined between the joint distribution and the marginal distributions of classifier’s predictions and labels. In this case, the f-divergence measure can be interpreted as the f-mutual information and thus the optimal classifier can be estimated by maximizing the f-divergence measure. The authors examine a family of f-divergence functions (generators in f-divergence measures) and derive several interesting robustness properties. For example, total variation is robust for the label noise problem (the label noise does not affect the optimality of the maximizer; it does not require the knowledge of noise rate), and when the f-divergence functions are not robust, we can still correct the solution to be the optimal by calculating the bias term introduced by the noisy labels.\n   \n##########################################################################\nReasons for score: \n \nI vote for accepting. The manuscript provides theoretical analyses of the f-divergence optimization with noisy labels. Although there are several research works on training with f-divergence measures recently, I can see the novelty of this work regarding the robustness of the f-divergence optimization when noisy labels presents. The effectiveness of optimizing f-divergence with label noise is also verified by a set of numerical experiments. I think that this work can be a good reference about robust training approaches dealing with label noise. \n \n##########################################################################\nSuggestions \nI have just few minor suggestions.\n1)\tAt first glance, maximization of f-divergence measures sounds confusing in the abstract and introduction as we generally minimize any divergence measures to train a model to fit the training data. I think that it would be better if the authors include additional explanations (in the abstract and introduction) that the f-divergence measure can be understood as the f-mutual information in this problem formulation.\n2)\tBelow equation 2), f(v)=log v should be f(v)=v log v   \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HyWaV06BPrz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the problem of learning a classifier when the labels are noisy. They consider an approach where they optimize the mutual information between the predicted label and the noisy label (or more generally, a variant of mutual information defined by an f-divergence). To optimize over mutual information, they use the variational form of f-divergence. They show that their loss function is a biased version of the mutual information between the predictions and the clean labels. They argue that the bias term is small and that, for many choices of f-divergence, that the true optimal solution is optimal even in the noisy case over a specific subset. They then experimentally demonstrate that their approach performs better than the baselines for a large number of datasets and noise settings.\n\nI found the paper interesting, however the theoretical justification is not fully clear to me. The authors should clarify the following two questions.\n\n(1) Is f-mutual information a good loss function on clean data, for any choices of f-divergence? I see in Section 2.2 special cases for when this is true, but can f-mutual information return a very bad solution when these conditions are not true?\n\n(2) In Theorem 8, why is $\\mathcal{H}^*$ the only set that we need to worry about?\n\nThe authors use the variational form of f-divergence to optimize their loss function. Algorithmically, how is the term $g^*$ calculated and updated? Since $P$ and $Q$ are changing as the model is optimized, does $g^*$ need to be recalculated over time?\n\nThe experimental section shows compelling results compared to state-of-the-art approaches. I think including no noise as an additional comparison can improve the understanding of the theoretical questions from above.\n\nOverall, I found the paper interesting with solid experimental results, though I still have some questions of the theoretical principles of the work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8q8UPptPrhr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This studies when maximizing a proper f divergence measure between the predicted labels and the observed labels would make the model robust with label noise. The authors derive that the f divergence can be decoupled into variational differences defined on distributions without noise and a bias term introduced by the noise. With this measure, we can learn with noisy labels without knowing the noise rate. They also consider the situations when the proposed f divergence measure is not robust against noise in labels, they propose a correction to mitigate this issue. Comprehensive experiments are done across several datasets with three noise models. Results show the proposed methods outperform baselines in most cases.\n\nIf the authors could explain the assumptions on the label noise that the work is based on early in the introduction and Section 2, the presentation would be improved. Now I have to read until Section 1.1 and Section 2.3 to know noise labels are generated based on the transition matrix.\n\nIn Table 2, in the sparse and high noise situation, the PL model outperforms the proposed method consistently. It would be better if the authors can discuss the difference between PL and the proposed method and provide reasons about why PL is better. Currently, the discussion focuses on why the proposed method cannot do well. In addition, the current explanations are conjectures. I would suggest the authors run experiments to verify such conjectures. For example, analyze the input of the g function to verify the first conjecture.\n\nIn Table 2, there are no results on random noise for MNIST and fashion MNIST. In addition, it did not distinguish Sparse Low and High and Uniform Low and High for CIFAR 100, could the authors explain this?\n\nIn Table 3, the J-S model with sparse high noise produces a gap value -20.88, which is way larger in magnitude than others. Could the authors further explain this result? It also lacks discussion on the meaning of the gap value, I wonder why such a large negative value has a trivial impact on the performance of the model. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles a very important problem. The formulation of the paper is sound as under lightweight assumptions, the supervised loss follows an f-divergence formulation (see "Information, Divergence and Risk for Binary Experiments" by Reid and Williamson (JMLR 2011), in particular Section 4.7). It would make sense to dig in the loss in the context of label noise; the variational formulation provides an interesting direction along those lines. The rebuttal on the experimental concerns of reviewers is appreciated (Cf authors’ rebuttal summary).

Decision: Accept (Poster)