Logistic-Normal Likelihoods for Heteroscedastic Label Noise
Authors: Erik Englesson
Amir Mehrpanah
Hossein Azizpour
Venue: Accepted by TMLR ()

Abstract: A natural way of estimating heteroscedastic label noise in regression is to model the observed (potentially noisy) target as a sample from a normal distribution, whose parameters can be learned by minimizing the negative log-likelihood. This formulation has desirable loss attenuation properties, as it reduces the contribution of high-error examples. Intuitively, this behavior can improve robustness against label noise by reducing overfitting. We propose an extension of this simple and probabilistic approach to classification that has the same desirable loss attenuation properties. Furthermore, we discuss and address some practical challenges of this extension. We evaluate the effectiveness of the method by measuring its robustness against label noise in classification. We perform enlightening experiments exploring the inner workings of the method, including sensitivity to hyperparameters, ablation studies, and other insightful analyses.

Published in Transactions on Machine Learning Research (08/2023)
Logistic-Normal Likelihoods for Heteroscedastic Label Noise
Erik Englesson
engless@kth.se
KTH Royal Institute of Technology
Amir Mehrpanah
amirme@kth.se
KTH Royal Institute of Technology
Hosscin Azizpour
azizpour azizpour@kth..co @kth. se
KTH Royal Institute of Technology
Reviewed OpenReview: https: opeeerve//perreveee. net/ forum? id= 7wA65xL33B
Abstract
natural way of estimating heteroscedastic label noise in regression is to model the observed (potentially noisy) target as sample from normal distribution, whose parameters can be learned by minimizing the negative log-likelihood. This formulation has desirable loss attemuation properties,
it roduces the contribution of high-crror examples. Intuitively, this behavior can improve robust.nss against label noise by reducing overfitting. We propose an extension of this simple and probabilistic approach to classification that has the same desirable loss attenuation properties. Furthermore, we discuss and address some practical challenges this extension. We evaluate the effectiveness of the method by measuring its robust ness against label noise classification. We perform enlightening experiments exploring the inner workings of the method, including sensitivity to hyperparameterrs, ablation studics, and other insightful analyses.
1 Introduction
Supervised learning relies datasets with input-label pairs, in which some labels are likely to be wrong, e.g., due to annotation mistakes in a classification problem or measurement devices' precision in regression problem. Even the systematically annotated datasets, like ImageNet, contain noisy labels (Beyer et al. 2020 It is, therefore, crucial to be able to effectively handle label noise.
Heteroscedastic Noise in Regression: Motivation. A natural way deal with mislabeled examples in regression is model the observed target (v) as the true target (n) with additive noise (c) (Nix & Weigend
1994 Kendall & Gal 2017 Lakshminarayanan et al. 2017):
(()) u(x) + e(w))
(1)
Assuming a normally-distribated e with zero mean and variance o², the likelihood of the observed target becomes plylæ, μ, o²) = u()), u(x), (x)). Then, neural networks can be used to estimate the input- dependent parameters of the distribution: μ(x) ss (z()), d²(x) = oz(x). The parameters of the noural network, O, are typically learned via some type of maximum likelihood estimation with N data samples:
(yi 22 (x)
1
+ log (z + const 2
arg max
This loss is of an interesting form: a label-dependent loss (mean square error) divided by the noise variance 2 and regularizing term for d² (log-partition). Hence, a² acts inverse importance weight of the mean

---

Published in Transactions on Machine Learning Research (08/2023)
squared error loss. For example, for high-residual examples, the penalty of the mean squared error loss be reduced by increasing o², leading to higher freedom for po to deviate from y-
We aim to obtain an analogously simple loss function to attenuate erroneous labels for classification tasks.
Heteroscedastic Noise in Classification with Attenuation. Kendall & Gal (2017) argued such a loss attenuation property desirable for classification, and proposod to learn the mcan and covariance of a normal distribution over the pre-softmax logits by maximizing categorical likelihood. This results in a form of loss attenuation, but interestingly not the same as in regression.
Contributions.
The main contributions of our work ard
We propose a natural extension of the above regression noise model to classification and show that it leads to the target following a Logistic-Normal distribution (Atchison & Shen 1980): ; see Section 2.2
We propose using the Logistic-Normal likelihood in maximum posteriori estimation setting and show its negative log-likelihood has the same desirable loss attenuation properties as in the regression case, e.g., reducing the contribution of high-residual examples; see Sections 2.3 & 2.4 Furthermore,
we address implementation challenges and propose practically important techniques; see Section S
We empirically study the proposed loss several datasets with synthetic and natural noise, where we show improved robustness to label noise compared to recent works; see Section 5
2
Method
Our goal is to extend the simple and probabilistic loss attenuation approach from regression to classification. Here, we give high-level overview of the problem and describe the main idea of our method. We give important details of our approach in Section 3
2.1 Background and Problem
In this section, we define label noise, identify ils source, and highlight the challenges it poses for deep learning.
Dataset Generation. In classification, we consider a dataset as samples from an unknown joint distribution: D ya y (z) p(x. y) = p(y)=))(zz). The generation process can be interpreted as, first, sampling the input, and then the label: (i) æ; p(ar), (ii) yi p(ylar,). This is, in fact, how many multiclass datasets are constructed, i.e., first collecting a large set of inputs and then, automatically or manually, annotating each input with a single output label.
Noisy Labels. The label y, can be seen as a categorical distribution via one-hot encoding, 8,, which is equal to one in component c, and zero otherwisc. As y; is a single sample from p(y|æ;), we see p(y]æ;;) = 5y as a crude estimate of the true label distribution p(ylæ:). Occasionally, sampling p(y[æ) gives an unlikely sample y, e.g., when probable errors are made in the annotation, causing large difference between p(y|æ) and p(ylæ)). We aim to model this label noise (difference between p(ylæ)) and p(ylæ=) in the pro-softmax logit space.
Learning with Label Noise. Specifically, are interested in a probabilistic model with parameterized output distribution pe(viz) and aim to optimize 0 so that the prediction is close to the true distribution, po(y)z;)) x p(ylæ;); using only the (noisy) training data (I,, ((ylz:)) Although we see both pe (y/z²) and p(yl=)) as approximations of p(y x), note that they differ in that the former is predicted, and the latter is directly determined by the given label. The challenge learning from noisy labels with deep nerssrks their susceptibility to overfitting to p(ylæx,) 8y,, for which we propose the following method and noise model.
2.2 Modelling Label Noise with Logistic-Normal Likelihoods
The core idea of our work is: Given invertible mapping from logit space to the probability simplex, can define a regression noise model in logit space and use the map to get noise model for classification. Here, we go over the map (softmax centered), the noise model, and the corresponding classification likelihood.
'Qur code available at: https://githmn: http::/hhttt.ccoooeee.emeonaaaaa
2

---

Published in Transactions on Machine Learning Research (08/2023)
Ys
Sc
Prediction
Ma true label
P2 label
P5
22
-2
y:
Y2
Figure 1: Logit Space and Probability Simplex Equivalence, An example showing how a normal distribution in R²² corresponds to a Logistic-Normal distribution in Á when using the softmax centered function as the bijective transformation.
Softmax Centered. The softmax function S(z) S([
[e e Σ e is map from logit space RK to the probability simplex of K classes AK-1_ This map is not invertible, as S(=) S( + e) for any e in B-K where all components of e are equal. An alternative is the bijec- tive softmax centered function Sc(z) = S((21...,,
from K- to the interior of the simplex, R- {p AK-1 P 0, Vi E (1,2....,KK) with inverse S5 (q) log (1.9)
22 1) "x
Noise Model. Consider a multivariate version of Equation where y(æ), ((x). <(x) E R- are vectors in logit space. We can apply the softmax-centered transformation this noise model to get a label noise model for classification:
Sc(y(z)) So(u(w) + <(x)).
(3)
With a deterministic e, analogous to regression, model the noisy label distribution p(ylz) Sc (z) the true label distribution p(y|æ) Sc(u(z)) with additive noise E(x) in logit space" Hereafter, we occasionally omit the dependence of variables μ,, E to æ for notational convenience.
Likelihood Function. Assuming a normally distributed <(x:) in Equation 3 with zero mean and covariance [x),, the likelihood of the observed target p(y|z) is:
1
( w) (+()))))-))
(4)
IL (1(()) ((2T) - m 4
As now pxxx) + <(xx) a Gaussian random variable, transforming with the softmax bijection leads a transformed random variable with a density proportional to (a) Σ) for q E ÁK-1 (derivation in Appendix B.I.. Importantly, this corresponds to a well-studied probability density function, called Logistic-Normal distribution (Atchison & Shen 1980), which is defined for categorical distributions in Á K-1,
As the map is bijective, this model gives rise to dual interpretations: (i) a regression problem with Gaussian likelihoods with targets, S-1 ()(ylz)), in logit space, or (ii) a classification problem with Logistic-Normi Normal likelihoods with targets, p(yla)), in the probability simplex. This duality is visualized in Figure]
2.3 Estimation with Logistic-Noria Normal Likelihoods
We use deep networks with parameters 0 to predict true logit vectors μ, ~ (0( and per-example (heteroscedastic) noise covariance matrices Σ, x Ex;);) We use separate linear layers for and Σ that share the same backbone. The network parameters are found by minimizing the following negative ligliielihood of
Regarddigg #(y/=) (x)() note that f(ylz) = 6 K-1 corresponds therefore not in the co-domain of the softmax centered function. We solve this issue by slightly diffusing it with label smoothing: Myla)) & (1 t)6y, respectively. We use 0.01 all our experiments, except. sensitivity analysis in Appendix G.
corner of the probability simplex, and is
KK-1, where >>000 and 8y and are the delta and uniform distributions over classes,
3

---

Published in Transactions on Machine Learning Research (08/2023)
the dataset (inputs æ, and observed targets Se (p(y(z)) y:) in addition to the negative log-prior over A:
Σ EE (y: (y: ui) + log E + const.
y m4)
(5)
As the first factor in Equation is independent of it is part of the constant term. Our goal to find classification loss with similar loss attenuation to the loss for regression. We see that our loss in Equation is almost identical to a multivariate version of the regression loss in Equation 2 Hence, the loss has the same attenuation effect: It can be decreased by learning Σg such that (y, p4)) Σ; (y, p4) is smaller than (y: 14) (y: p4), leading to higher freedom for μ4, to deviate from yi as the penalty is reduced.
We now established the capability for Logistic-Normal likelihood to attenuate high-residual samples and potentially predict their correct mean. Next, we do gradient analysis whereby we show how such behavior is in fact encouraged when learning with realization of such effect not only through the final performance noisy training data but also with targeted analyses of the training behavior.
methods. In Section 5 empirically verify the
2.4 Loss Attenuation: A Gradient Perspective
Let L be the negative log- lieelhoos in Equation 5 then the gradients w.r.t. f of example j are:
ac 0p, - (リ, M1y).
(6)
This reveals an interesting form where the gradients related to the difference between the target logit (y,) and the predicted mean (14)), and this difference is scaled by the inverse covariance matrix (Σ,1). This scaling is a major difference compared to the gradients of the negative log- ligliielihoodd of a categorical distribution (CE) w.r.t. its logits: -66,, S(z;)), where 2j is the logits and S(.) is the standard softmax function.
To better understand what per-example Σ matrices the network tries to predict, we look the optimal covariance matrix, i.e., when ac 8E,
(details in Appendix B.2
E = (y,- - 14) 13) - -,,)².
The geometric interpretation that the optimal density is a thin hyperellipsoid with its center on js and highest variance in the direction of the noisy target y, see Figure left.
Plugging the optimal covariance matrix from Equation 7 in Equation we get (details in Appendix B.3
ac
(y) 14)) [1 12))
0p, 22
That is, for given pa; the corresponding optimal covariance matrix divides the difference between the label and the logits by its squared 12-norm, which is exactly the loss attenuation property of our method. Clearly, the role of [(x) is to increase and decrease the gradients for low- and high-residual examples, respectively.
As gradients are not affected by constants, and as the LN and multivariate normal likelihoods are the same up to constants, the gradients here also apply to the regression case. This reflects the analogy of LN for classification, with the loss-attenuating properties of the regression case, that we were after in this work.
3 Important Details
While the essence of our proposed noise model and the induced likelihood function are simple the high level, it involves important details that discuss in this section.
4

---

Published in Transactions on Machine Learning Research (08/2023)
A
loopt
0.1
4//o
-2
-4
2
1.0
A
0.1
1
-8
0.5
1.0
0
-10
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
y -
y y-u
Figure 2: Controlling Loss Attenuation with X. In higher dimensions, À is necessary to ensure invertibilityy of Σ, but it is also a way to control the loss attenuation for low residual examples. We plot the loss (left) and gradients w.r.t. (right) with the optimal obtainable a (binary classification) against varying residuals. Clearly, the value of À determines the residual threshold for where loss attenuation starts, We have loss attenuation when the residual is larger than X, resulting in a loss of one and a gradient of -1/(y p)) and otherwise the loss is the mean-squared error loss divided by X².
3.1 Estimating Per- Eer-Example Covariance Matrices
Note that the output covariance matrix E(x) has O(K?) parameters, needs to symmetric and semi- cositivsi definite. Next, to feasibly predict E(x) for a large number of classes, we make use of the analysis for the optimal [x) propose a structured reparametrization requiring only O(K) parameters.
Parametrization of E. Hinging the rank deficiency of the optimal covariance matrix, we parametrize it as Ex(x) = E (z) (x)) where:
Eg (xx) co()))o(()) + AI,
with co (z) E KR1 and a hyperparameter À E R Note that such decomposition reduces the parametriza- tion to a rank-1 matrix, co to(x)col cx))))(()) and positive scalar, x. Therefore, it gains computational efficiency by acknowledging the singular structure of opp while crucially remaining full rank for numerical stability
À and Loss Attenuation. Interestingly, affects the loss beyond stability. Considering binary classification for simplicity, with X = 0, the optimal variance per-example is scalar o (y: 14)22 Using the optimal variances in the label-dependent term of the loss in Equation 5 results in each term being 1, (y: /o (y (/)a)///// /t/ -242 = 1. Furthermore, the gradients for μi in Equation g become Ju, 1o-m -y; 14/// = - - u4)(w; - (4,)² = - 11/(1: - n4). This, again, clearly shows the loss attemaation properties of the LN likelihood, i.e., it increases and reduces the contribution of low and high residual examples, respectively.
3C
How does À affect this behaviour? It acts a threshold (on residuals) for where loss attenuation occurs. To this, note that the minimum value for any dg is X?, de o3. o and the minimum value for of is X, due to the parametrization in Equation Thus, uf cannot be smaller than X² the desired a. = (y:: - =x:32 2 is unattainable when (y m2) X2. Instead, the optimization gets as close as possible, resulting in 0,
opt X2 (details in Appendix B.4. In general, we have o. max (X² (y) - pz)2). Therefore,
when (y, 22,) < X², the label-dependent loss becomes (yt: 14) /o (v. //2, and the gradients become (ys p:)/o?" = - p2)/ We show the effect of the loss allenuation threshold X in Figure
SSimple and efficient implementations normal distributions with low-rank covariance matrices be done in, e.g., the distribution packages FeosorFlow Diilon al. 2017 and PyTorch.

---

Published in Transactions on Machine Learning Research (08/2023)
3.2 The Softmax Centered Function
Softmax--entered with Temperature Sg. We incorporate temperature T in the softmax-centered function by seeing it as bijective scale function:
(T(2) Sc Scale1 Sc(z/T),
(10)
(5S) (q) = Scale 1/7 o ST" (0)=
- -
(11)
where z E RR-1 and E ÁK-1, By introducing temperature in the softmax centered bijection, the target logit in the log-likelinn log-likelihood of Equation 5 changes from to Tys, i.e., T controls the magnitude of the target logit voctor. We believe this has two major effects on learning: i) a low 5 moves the target logit closer to the origin, making casier for the network to match I to it. Conversely, a large T imposes a challenge as the network must output a with large magnitudes, which it) The chosen T controls the range of residuals in the mean squared error loss. To illustrate, consider a sample with noisy target logit Tyt, for which the network predicts the true target μ4 ty TSE ((1 t)ō, + tu) with ys * j. Then the negative log-likel likelihood for this example with an identily covariance is:
penalized from doing by the log prior (weight decay).
(Ty: (((xz))) (² (ry, uo(x)))
= (ry: - (WV--Ty) Try) - -Ty)
(12)
= (()) - yy) = =²²
where C== (y: u))((, - y) the loss without temperature scaling. Hence, the temperature determines how much the mean squared crror part of the loss penalizes deviations from the obscrved target. In this work, we treat the temperature T as hyperparameter.
Softmax Centered with Dummy Class. An issue with the softmax centered function is that it treats the last class differently from the rest. To see this, we look at the target logits (y) for target categoricals (Sc(y) for different classes (v), for details see Appendix B.5 If is not the last. class, then y is zero in all components except for component where it is a constant (C) that depends on the number of classes K. However,
is the last class, then all components of y are -C. See Figure [ (left) where yı = [C, 0], y/2 = [0,C] C] and M3 [-C. C.. This becomes a bigger problem for large K, as the squared 12-norm for an observed target logit. is (K 1)22 for the last class, and C² otherwise.
As it is only the last class that is treated differently, we mitigate this by introducing dummy class. We make Sy) be in Á by having the delta and uniform distribution be over K + 1 classes, and the network therefore has to output mean and covariance R and RKxK respectively. Then, all the first classes that we care about are treated equally.
3.3 Predictions on Unseen Data
With our noise model in Equation 3 relate the observed noisy larget with the true larget via an additive normally-distribtted noise in the logit space. For unseen data x², however, we would like to predict the true target, not the noisy target, and, therefore, we use Sc(ue(x*)) instead, i.e, setting E(x*) = 0. This means that we can discard the network's head predicting Σ after training.
4 Related Work
Heteroscedastic Noise Estimation. Nix & Weigend (994] tackle the problem of input-dependent noise, for regression, in a maximum likelihood estimation framework. They assume additive Gaussian label noise and oplimize two different networks to predict the mean and the variance of the output distributions. Kendall & Gal 2017) importantly note that such a framework provides a model with the capacity to effectively attenuate the loss induced by samples which are hard to model (Equation 2 and thus renders it possibly robust label noise. Furthermore, they argue that such attenuation properties are also desirable in classification and propose method termed Heteroscedastic Classification NNs (Het). They place normal distribution over

---

Published in Transactions on Machine Learning Research (08/2023)
the logits to model heteroscedastic noise, which is then marginalized out to obtain categorical distribution:
(e(c:))- (m:) + <(x:))]
(13)
where μ E RK and Σ E RK* is a diagonal covariance matrix, both predicted by a neural network, and 5 is the standard softmax. The negative log-likelihodd of this calegorical distribution is then used as the loss. Collier et al. (2020) extended this by tempering the softmax (Het-r), and evaluated the robustness of the method to label noise. In another work, [Collier et al. (2021) proposed an efficient low-rank parameterization for the covariance matrices (Het-r-EyuR)) which is similar to ours in Equation g To better understand these works, we analyze the gradients w.r.t. p for sample i with label yi = c (derivation in Appendix B.6
S(4 + +c)c - (5. - S(u; + 6) [[ + +
(14)
Comparing the expectation in Equation 14 with the expectation in Equation 13 we see it is modified by
a scalar factor to increase the contribution of each sampled categorical distribution S(m; + +)) that has a high confidence in the given class, S(H; + c)c. Clearly, this is doing loss attenuation as the network could learn to add noise to increase the confidence in the given class, making the expected categorical be closer to the target onc-hot distribution. However, the loss attenuation of this method different from the one in maximum likelihood estimation with Normal (regression) and Logistic-Normal (classification) likelihoods, see Section 2.4 We empirically compare Logistic Normal with all the variants in this line of work.
Loss Correction. Closest to our work, are the loss correction methods that estimate the true categorical distribution and transform it to the observed noisy one (Sukhbattar et al., 2014 Patrini et al.] 2017]: p(ylz) = Tp(ylz), where T is a matrix with elements Tly estimating the probability that the noisy class is j, given that the true class is i. This is clearly related to our noise model, as we transform the true categorical to the noisy one by adding noise in logit space. Importantly, we estimate a per-example covariance matrix, while these works estimate single matrix T per dataset.
Loss Reweighting. These methods propose to weight the per-example losses to reduce the contribution of noisy examples. These weights can be estimated through density estimation (Liu & Tao 2015), or predicted by the same network (Wang et al. 2017 Thulasidasan et al. 2019), another network (Jiang et al. 2018), or via meta-learning (Ren et al. 2018). These methods need to avoid the trivial solution of all weights being 0, which is typically done by engincering an extra regularization term. Our method's loss attenuation is form of reweighting. However, in contrast, we naturally extend a classic noise model that leads to a likelihood, which combined with a standard MAP estimation framework, directly leads to our loss with inherent regularization. This makes our design choices more interpretable and more conducive to further extensions.
Memorization Effects. These methods rely on the observation that neural networks learn easy (correctly labeled) examples first (Arpit et al., 2017). Examples with small loss can thorefore be assumed to be correctly labeled and selected for learning. Some notable methods making use of this are: MentorNet (Jiang el al. 2018 with predefined curriculums, the Co-teaching methods (Han et al. 2018 Yu et al. 2019), and DivideMix Li ltt al. 2020a). Incorporating similar small-loss tricks in our method could further improve robustness.
Robust Loss Functions. Ghosh et al. 2017 proved that, for certain (symmetric) loss functions, the globally optimal classifier is the same when trained with noise-free data as when trained with symmetric or asymmetric noise, under certain assumptions. Based on this theory, several new loss functions have been proposed (Zhang & Sabuncu, 2018 Ma et al. 2020: Englesson & Azizpour 2021) and even extensions of the theory (Zhou et al. 2021). These theoretical works are commendable, however, most assume access to unlimited data, which makes unclcar how the results translate to standard finite classification datascts.
Regularization. Several standard regularization methods have also been studied when training with label noise, e.g., label smoothing (Lukasik et al. 2020). dropout (Rusiecki 2020 Goel & Chen 2021), and early stopping (Li et al. 2020b Bai et al.) 2021). Furthermore, some methods regularize the predictions of the network to be consistent with predictions from earlier in training (Liu el al. 2020 Laine & Aila 2017), while others add noise to the gradients, e.g., by adding noise to the oot-hot/o labels (Chen et al. 2020). Although many of these mothods only implicitly tackle the underlying problem of label noise via standard regularization techniques, they show impressive empirical robustness. We believe that targeted methods like ours that explicitly tackles the problem could be naturally combined with these methods.
7

---

Published in Transactions on Machine Learning Research (08/2023)
41
M6
(a) CE
(b) LN
(c) CE
(d) LN
Figure 3: Synthetic Datasets. To gain insights into the behavioral differences between our method (I.N) and the cross-entropy (CE) loss, we study two binary datasets: the Two Moons dataset (left), and a dataset of two circles of different sizes (right). The color indicates the probability of the blue triangle class. Training with CE makes the network fit all the examples, resulting in a complex decision boundary. In contrast, the loss attenuation of our method reduces the need to fit all examples, leading to smooth decision boundary.
5 Results
Here, we empirically verify that our theoretically motivated method demonstrates robustness label noise, We describe the common training sctup (Section 5.1 and the basclines (Section 5.2) and present results synthetic datasets (Section 5.3). synthetic label noise (Section 5.40, and natural label noise (Section 5.5
Finally, we conduct additional insightful experiments studying different aspects of our method (Section 5.6
5.1 Experimental Setup
We implement method and the baselines in the same code base and compare on the following datasets: Two Moons & Circles, MNIST (Deng) 2012), CIFAR-10 & CIFAR-100 (Krizhevsky et al., 2009), CIFAR-1ON & CIFAR-100N WWi et al. 2022), and ClothingIM (Xiao et al. 2015). For all methods, and on all datasets, we search for method-specific hyperparameters based on noisy validation accuracy at the end of training, We report the mean and standard deviation the test accuracy at the end of training for five different random seeds with the optimal pyrrparrameters. The seeds affect the network initialization, data loaders, and the generation of the synthetic label noise. For details about the experimental setup, see Appendix C
5.2 Baselines
In addition to the standard cross-entropy (CE) loss, we compare our method with methods sharing the same motivation and goal having a method that doals with hetcrosoedastic label noise similar the proba bilisicc method for regression: Het (Kendall Gal 2017). Het (Collier et al., 20200, and Het (Collier al., 2021). Our method and these baselines model the pre-softmax logit vector as being normally distributed and use deep neural networks to output the mean and covariance of this distribution. We also compare with the loss correction method Forward (Patrini et al. 2017), and label smoothing (LS) regularization (Lukasik et al. 2020). Additionally, for completeness, we consider more distant baselines: Generalized Cross Entropy
(GCE) (Zhang & Sabuncu 2018). and Noise Against Noise (NAN) (Chen et al. 2020
5.3 Synthetic Datasets: Two Moon & Circles
In Figure we compare the behavior of our method (LN) with the cross entropy (CE) loss on two synthetic binary classification datasels: a datasel where the classes correspond to two half moons, and another with two circles of different radii. We find that training with the CE loss makes the network classify almost all examples according to their observed targets, resulting in complex decision boundary that does not generalize well. In contrast, the network trained with the log-likelihood of the Logistic-Normn Normal distribution has seoother decision boundary, as the network is not classifying some examples as their given targets.
8

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: Synthetic Noise on MNIST, CIFAR-10 and CIFAR-100. We implement our method and all baselines the same shared code based and do scarch for the best hyperparamcters on a noisy validation set for all methods. We report the mean and standard deviation of the test accuracy from five runs with different random sceds. Our method (LN) shows strong performance compared to basclines, especially the predictable asyimmetric noise, and the challenging CIFAR-100 dataset.
No Noise
Symmetric Noise Rate
Asymmetric Noise Rate
Method
0%
20%
40%
60%
20%
30%
40%
CE
99.27 0.07
88.41 0.34
70.67 + 1.30
51.04 1: 1.19
91.09 + 0.79
86.31 1: 1.25
80.31 1.81
GCE
99.22 = 0.06
98.85 = 0.18
98.60 + 0.11
97.45 = 0.31
98.52 = 0.31
86.36 = 0.86
79.81 = 1.46
NANN
98.44 + 0.24
97.51 + 0.37
90.03 = 0.95
74.00 = 2.92
96.46 = 2.23
95.43 + 1.09
88.95 + 1.63
MNIST
Forward
99.27 0.03
87.46 0.70
69.96 = 2.10
50.43 + 1.43
91.95 = 0.40
86.31 + 0.43
80.97 1.23
LS
99.35 + 0.06
89.92 = 0.85
69.07 = 0.93
47.77 = 2.00
92.06 = 0.95
86.50 = 0.44
80.00 = 0.93
Het
99.28 = 0.05
87.09 = 0.70
70.30 = 1.10
50.37 # 1.02
91.12 = 1.29
86.35 = 0.62
80.04 = 0.87
HetT
99.25 0.07
88.10 0.70
70.19 = 1.58
51.95 + 1.13
91.13 +
85.95 + 0.69
81.37 1.14
Het
99.25 + 0.15
89.16 + 0.44
70.34 = 1.20
50.89 + 1.79
91.06 + 0.24
86.25 + 0.88
79.93 = 0.44
LN
99.38 T 0.06
98.53 1 0.27
97.21 T 0.38
90.93 1 2.29
99.19 1 0.10
99.01 1 0.19
96.54 T 1.20
CE
90.67 T 0.80
73.54
1.01
56.56 1 1.44
39.44 1 1.87
81.35 1 1.26
76.01 T 2.67
71.89
1.67
GCE
90.83 1 0.44
87.55 1 0.41
84.72 1 0.82
64.28 1 1.42
85.68 1 0.69
83.97 1 0.52
72.90 + 1.61
CIFAR-10
NAN
89.61 0.93
83.86 + 1.03
79.80 = 0.59
73.58 + 0.41
84.32 = 1.05
76.79 = 2.28
72.90 = 1.92
Forward
90.69 0.38
74.39 1.49
59.60 1.40
40.06 + 2.16
82.10 = 1.09
77.02 + 2.38
72.77 1.43
LS
89.78 = 0.39
79.09 + 0.96
64.27 = 1.50
43.57 + 3.13
81.99 = 1.22
76.49 + 1.17
71.66 = 1.78
Het
90.41 + 0.69
74.67 = 1.06
58.53 = 1.96
39.51 + 2.53
81.72 = 1.65
76.97 = 1.17
72.88 = 1.53
Het
91.18 0.41
76.90 1.79
63.55 2.27
44.73 # 1.67
8L.00 0.96
77.41 = 2.53
72.53 1.83
Het Hea
90.82 + 0.42
77.16 + 0.94
62.85 + 1.88
44.20 + 3.05
81.55 + 0.80
77.05 + 0.33
72.69 + 0.89
LN
90.17 + 0.55
86.13 + 1.03
81.37 + 1.97
76.08 + 0.63
87.64 +: 0.78
86.91 + 1.03
82.18 + 1.30
CE
64.87 + 0.88
47.39 + 0.43
33.62 = 0.79
20.04 + 0.58
50.98 = 0.88
44.04 + 0.73
36.95 = 0.58
GCE
64.33 1 0.83
61.67 1 0.67
53.96 1 1.40
42.85 1 0.79
59.63 1 1.28
49.21 1 0.53
36.78 1 0.50
CIFAR-100
NAN
64.25 0.64
56.93
0.77
50.03 = 0.62
40.45 i 0.41
56.40 + 1.07
52.78 1
40.59 0.84
Forward
64.33 + 0.73
47.90 + 0.93
32.28 = 1.10
20.00 + 0.75
50.82 = 0.57
43.87 + 0.47
37,02 = 0.72
LS
65.39 + 0.40
57.08 + 0.70
44.03 = 1.20
26.13 1 1.45
55.47 + 0.76
44.70 + 0.73
38.56 + 0.66
Het
64.48 0.31
48.40 1 1.32
34.26 1 0.37
20.33 1 0.31
51.44 1 1.15
45.09 1 0.40
37.43 0.66
Het
64.20 + 0.37
54.17 0.79
42.03 = 0.84
22.33 + 0.57
59.89 = 0.54
53.75 = 1.08
41.14 = 0.98
Het
65.18 + 0.90
54.83 + 0.46
41.49 = 1.53
22.42 + 0.95
61.29 = 0.46
56.44 - 0.53
45.75 = 1.02
IN
64.88 + 0.98
60.58 + 1.07
55.55 + 1.30
46.43 + 1.15
64.31 + 0.98
64.07 + 0.77
61.20 + 1.22
5.4 Synthetic Noise
Noise types. Here, we study our method on two types of synthetic class-dependent label noise (asymmetric and symmetric). For each training sample, there is a risk (according to the noise rate) that its label is randomly re-sampled from a uniform distribution over the classes (symmetric noise) or changed to another, often perceptually similar, class (asymmetric). The asymmetric noise changes the labels as follows: MNTST: 7 + 1,2 > 7,5 +> 6,3 + 8, CIFAR-10: bird - airplane, cat. <> dog, deer -> horse, CIFAR-100: cyclically to the next class, e.g., + and 99 -> 0.
Table shows the results using symmetric and asymmetric label noise. Compared to the most related works (Forward and Het methods), we find that the test accuracy of our method is degraded the least for all noise types and rates. Out of the more general set of basclines, we find that the robust GCE loss performs remarkably well on symmetric noise. Our method shows largest improvements in robustness for the more challenging CIFAR- 100 dataset, as well as for asymmetric noise. For example, on CIFAR-106 with 40% asymmetric noise, our method achieves a mcan test error of ~61% compared to 446% of the best bascline. We find that the generalization of the networks trained with our method is barely affected when increasing asymmetric noise rates from 20% to 30% on all datasets. This is likely due to the predictable st succtue of the
Results.
9

---

Published in Transactions on Machine Learning Research (08/2023)
Table 2: Natural Noise: CIFAR-ION, CIFAR-10ON, and ClothingIM. For all methods, we search for method-specific hyperparamcters based on a noisy validation set and report the mean and standard deviation for the best setting. Our method performs as well as other baselines, or is a close second to the robust GCE (Zhang & Sabuncu, 2018 loss.
CIFAR-I00
LIFAR-10ON
ClothingIM
Method
Random
Random
Random 3
Aggregate
Worst
CE
77.75 + 0.74
75.52 # 1.08
76.25 + 1.26
83.59 + 0.98
59.01 + 0.98
42.75 + 0.93
71.04 = 0.15
GCE
85.66 0.73
85.58 + 0.65
84.78
0.62
86.66 + 0.68
77.48 1.22
48.81 = 0.46
71.95 = 0.21
NAN
81.85 + 1.13
83.40 + 0.84
82.77 t 0.78
85.53 + 0.83
75.47 + 0.76
50.00 + 0.72
71.50 = 0.41
Forward
77.97 + 0.80
77.13 + 0.74
77.51 + 1.21
83.58 + 1.50
58.91 + 0.57
42.53 + 0.41
70.75 = 0.25
LS
80.07 0.82
79.81 1 0.62
79.41 0.68
85.08 1 0.54
63.07 1.93
45.98 1.44
72.04 0.42
Het
76.38 1 0.97
75.85 + 1.45
76.18 + 1.60
83.87 + 1.02
58.86 + 1.27
42.90 1 0.48
70.87 + 0.38
Het"
78.83 + 1.65
78.29 + 1.61
78.27 + 0.86
84.34 + 0.48
62.01 + 2.03
45.82 = 0.53
72.24 + 0.30
Het
78.87 0.47
76.24 + 0.96
77.68
1.93
84.45 + 0.57
63.27 2.62
45.58 + 0.80
72.41 0.15
LN
83.70 + 0.80
83.65 + 0.82
84.07 0.71
85.35 + 1.33
74.31 1.08
50.37 + 0.50
72.03 + 0.52
asymmetric noise, which could also explain why our method fall behind the robust GCE loss on symmetric we use neural network to predict the noise variance
noise. The predictability of the noise is important, of the LN likelihood. As the optinal Σ in Equation 7 depend on the residual, which in turn depend on the label, the network needs to predict the noisy label for the mislabeled examples. However, for uniform noise, the labels are inherently unpredictable, and therefore the network has to resort to memorization.
5.5 Natural Noise
Datasets. Synthetic label noise is excellent for studying robustness to noise under controlled noise rales. However, this comes at the cost the structure of the noise (clese-dep)ndent) potentially being different from what one would observe in practice (input-dependent), e.g., due to mistakes in the annotation process. this section, we study the robustness of our method on natural noise by using the recently proposed CIFAR-N datasets (Wei al. 2022) and ClothingIM (Xiao et al. 2015), see Appendix D for more information.
Results. Table 2 shows the test accuracy of our method naturally noisy datasets. Compared to the most relevant baselines (Forward and Het methods), we find that networks trained with our method generalize better on the CIFAR-N datasets, and as good as Het" and Het on Clothing For the more general set. of baselines, NAN, and especially GCE, have strong performance in this setting.
60%
60%
0%
28%
63%
0%
20%
60%
100
100
100
100
AycuracyyM
80
Accuracy(%)
80
cyCu
80
Train Accuracy(%)
80
60
60
60
40
40
40
40
20
20
20
20
100
200
300
100
200
300
100
200
300
100
200
300
Epochs
Epochs
Epochs
Epochs
(a) Het
(b) Het
(c) LN
(d) LN
Figure 4: The Evolution of Validation and Training Accuracy. We plot the noise-free validation accuracy and noisy training accuracy of Het (left) and our method (right) on varying symmetric noise rates during training on CIFAR-10. We report the training accuracy of noisc-frec (full) and noisy (dashed) examples separately. We observe that the generalization degrades (a) for networks trained with Het as it fits the noisy examples (b). Our method results in better generalization (c) and is more robust against fitting the noisy examples of the training set (d).
10

---

Published in Transactions on Machine Learning Research (08/2023)
Etrai Reya!
Zoras
Eg. Reval -
Evalo Ep. Zevol = -gg
0.8
Clean Nolsy
Clean Noisy
80
Nolsy
Test Accuracy
60
Density
0.4
A
40
0.1
0.25
0.2
20
0.05 0.1 0.25 0.5 0.75 1.0 2.0 4.0 Temperature (T)
0.001
0.1
10
0.001
0.1
10
0.001
0.1 10
(
Figure 5: Sensitivity to Hy- perparameters. The mean and standard deviation of test accuracy for various hyperpa- rameters on CIFAR-10 with 40% symmetric noise. A too small T leads to overfitting to noise, while a too large T and X leads to slow convergence.
Figure 6: Histogram of Residuals. We train two LN networks with different covariance matrices: Etrain = I, and Etrain = Eg. At the end of training, we calculate the distributions of the label-dependent term of the loss, (y 1) EEval pe), the CIFAR-10 training set with 20% asymmetric noise. We use Eiaain Ecvel I (left), and for the network with Etrain Eg, we use Leaal I (middle), and Eevat = Eo (right). We find that μ fits less of the noisy examples when learning Σ (cf. left and middle), and that Eg increases and decreases the contribution of low- and high-residual examples, respectively (ef. middle and right)).
5.6 Empirical Study of LN
How does the accuracy evolve during training? Figure 4 shows the clean validation and noisy training accuracy (for clean and noisy examples separately) on CIFAR-10 with symmetric noise for various noise rates for Het Seun and our method. The generalization of networks trained with Het improves early in training, and degrades when the networks start fitting the noisy examples. In contrast, we find that networks trained with LN fit significantly fewer noisy examples and thus show a smaller decrease in generalization.
How sensitive is LN to hyperparameters? See Figure 5 A small makes the network overfit, likely due to the target logit being close to the origin (Equation 11 , thus casy to fit. A large J with a small À leads to slow convergence, likely due to some clean examples being loss altenuated as increases residuals (Equation 12 making examples more likely to be above the loss attenuation threshold, see last paragraph in Section 3.
How is Ee(x) (x) affecting the network? In Figure i network trained with an identity matrix has, as expected, more noisy examples with lower residuals (left), compared to the network that learns the covariance matrix (middle). Interestingly, the residuals in the middle figure are bimodal, which we believe due to some clean examples being below the Icarnable loss attemuation threshold (Section 3.1 and some above. Comparing Figure 6 middle and right, we find that Σg is increasing the residuals of some clean examples, while also reducing the residuals of noisy ones mixed with some (hard) clean samples. These results are for the CIFAR-10 10 training set with 20% asymmetric noise, see Appendix G.3 for more noise types.
How important is learning a full Ee(x)? In Table 3 we train with LN on CIFAR-1ON (noise type "worst") ) and CIFAR-100N with different per-example covariance matrices: (Identity), o²I (Tsotropic), diag((o?....,.. I) (Diag), and the parametrization in Equation 9 (Full). We observe that using an identity matrix generalizes significantly worse than all the other learnable ones, highlighting the importance of loss attenuation. Furthermore, we find our proposed parametrization (Full) significantly outperform the rest.
Table 3: Ablation Study. We analyze the effect of learning E, different parameterizations, and the dummy class. Learning a full Σ performs the best, and the dummy class is crucial for datasets with many classes.
Σ
Dummy Class
CIFAR-1ON
CIFAR-10ON
Identity
65.64 1.53
47.60 1.46
Isotropic
71.12 1 1.85
46.44 1 0.67
Diagonal
/
69.87 + 1.11
46.98 + 0.89
Full
74.31
1.08
50.37 = 0.50
Full
X
74.72 + 1.14
26.31 + 1.43
11

---

Published in Transactions on Machine Learning Research (08/2023)
How important is the dummy class? Our analysis in Section 3.2 highlighted that the softmax centered treated the last class differently, and that this difference increased with the number of classes. In Table 3 we evaluate the importance of our solution (dummy class) to this problem. As expected, on CIFAR-10, it makes no significant difference, howover, on CIFAR-100 with its many more classes, it becomes crucial.
6 Limitations and Future Work
Interleaved in the previous sections, we have addressed or discussed limitations of our work, e.g,, asymmetry of softmax centered + dummy class, full covariance matrices do not scale to high number of classes - low-rank approximation, the Logistic-Normal distribution is not defined on the borders of the probability simplex + label smoothing. Although our method is conceptually simple, these minor designs to make the extension work in practice are surprising to us. We believe it could explain why such natural extension from regression has not happened before despite the cstablishment of the regression approach and the more practicality and wide applicabilityy of the classification settings, especially for modern deep networks. Purthermore, in Section 5.4 we discuss the limitation of using neural networks to predict the noise variance in the LN likelihood. We believe an interesting future direction is improving this estimation. Next, we discuss more future work.
Soft Labels. We model the observed larget as the softmax of a logit vector, Sc(y). However, this leads a limitation that the target cannot be on the borders of the probability simplex. We proposed label smoothing a simple solution for this issue. However, future work, there are many other interesting possibilities of obtaining soft label, e.g., using the categorical prediction of another network, as in Knowledge Distillation (Hinton et al. 2014), or temporal ensembling (Taine & Aila 2017), etc.
Neural Network Point Estimates. We assume the observed target logit can be modelled as the true target with additive noise in logit space: y = t, e (O NN E). That is, we explain the residual y/ - μl, as zero-mean normally distributed noise e. Hence, high-residual examples will be attenuated. In this work, as μ and Σ are unknown, we estimate them with neural network. Therefore, a limitation of our work is that if estimates of I are poor, then will incorrectly model the residual as noise, which could lead to e.g., slower convergence. To improve these estimates, we believe exciting directions for future work are to extend our method to incorporate epistemic uncertainty and/or /or distance awarcness, which has been done for the most related work to us (Het) by Kendall & Gal (017)) and Fortuin et al. (2022). respectively.
Gaussian Process Classification. Due to the duality between the logit space and the probability simplex, see Figure [ we can interpret our method as: i) converting the classification labels to regression labels (S (f(y)(())))) ())),, ii) training a regression neural network with a Gaussian likelihood loss, and iii) making classification predictions
unseen examples by turning the regression predictions (μ)) to categorical distributions (Sc( via the softmax centered bijection. Gaussian Processes (GPs) for classification typically use the categorical likelihood, which requires approximate methods to find the approximate posterior predictive distributions. However, given the interpretation of our method above, we can get closed-form predictive posterior distributions even in classification. Similarly to above, the procedure would be: i) turn a classification dataset into a regression dataset using the softmax centered bijection, ii) train a regression GP on this dataset, iii) get closed-form posterior predictive distributions (Normal distributions) from the GP, and finally transform these predictions to classification predictions (Logistic-Normal distributions) by applying the softmax centered function. We expect this method to be much faster than the approximate methods, and it would be interesting to compare the quality of the predictions, especially in settings with label noise.
7 Conclusion
The goal of this work was to extend the simple and probabilistic approach of doing loss attenuation in regression to classification. We successfully achieved this by proposing noise model that lead to the Logistic-Normal distribution. We proposed to learn the parameters of the distribution with neural networks through maximum likelihood estimation and formally presented the loss attenuation effects obtained when optimizing such models. Finally, we empirically verified that LN is effectively robust to label noise. As our method has the same loss attennation as in the regression case, it can serve as a simple alternative to the methods of Kendall & Gal (2017); Collier et al. 2020
2021
12

---

Published in Transactions on Machine Learning Research (08/2023)
Acknowledgement. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. All experiments were performed using the supercomputing resource Berzelius provided by the National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg foundation.
References
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Kruoger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in doep networks. In International conference on machine learning, pp. 233-242. PMLR, 2017.
Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):
201-272, 1980.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24392 2440332--4443, 2021.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? Xiv preprinl arXiv:2006. 07159, 2020.
Pengfei Chen, Guangyong Chen, Junjie Ye, Pheng-Ann Heng, et al. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference Learning Representations, 2020.
Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. A simple probabilistic method for deep classification under input-dependent label noise. arXiv preprint XXtx:2003.06776 2020.
Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. Correlated input- dependent label noise in large-scale image classification. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 1551-1560, 2021.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEFE Signal Processing Magazine, (9())1141 142, 2012.
Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matthew D. Hoffman, and Rif A. Saurous. Tensorflow distributions. CoRR, abs/1711.10604, 2017. URL httt:://aaxxtii org hatp//ab//1111.111
Erik Englesson and Hossein Azizpour. Generalized jensen-shannon divergence loss for learning with noisy labels. Advances in Neural Information Processing Syslems, 34:30284 4-229,, 2021.
Vincent Fortuin, Mark Collier, Florian Wenzel, James Urquhart Allingham, Jeremiah Zhe Liu, Dustin Tran, Balaji Lakshminarayanan, Jesse Berent, Rodolphe Jenatton, and Effrosyni Kokiopoulou. Deep classifiers with label noise modeling and distance awareness. Transactions on Machine Learning Research, 2022. URL htttss://ppnreviiee htt.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Purvi Goel and Li Chen. On the robustness of monte carlo dropout trained with noisy labels. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp. 2219 2228, 2021.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
David A Harville. Matrix algebra from statistician's perspective, 1998.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge neural network. In NIPS 2014, Deep Learning Workshop, 2014.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304 2313. PMLR, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. Xix preprint arXiv:2002.073)) 2020a.
Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pp. 4313-4324 PMLR, 2020b.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Grandd. Early-learning regular- ization prevents memorization of noisy labels. Advances in neural information processing systems, 33: 20331-20342, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, IEEE Transactions on pattern analysis and machine intelligence, (8(3):447-460) 2015.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference Machine Learning, pp. 6448 6458. PMLR, 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pp. 6543-6553. PMLR, 2020.
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 deee international conference on neural networks (ICNN 94), volume 1, pp. 55 60. IEEE, 1994.
Giorgio Patrini, Alessandro Rozza, Adilya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1944 1952, 2017.
Kaare Brandt Peterson, Michael Syskind Pederson, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334-4343. PMLR, 2018.
Andrzej Rusiecki. Standard dropout remedy for training deep neural networks with label noise. In Wojciech Zamojski, Jacek Mazurkiewicz, Jaroslaw Sugier, Tomasz Walkowiak, and Janusz Kacprzyk (eds.), Theory and Applications of Dependable Compuler Systems, pp. 534-542, Cham, 2020. Springer International Publishing. ISBN 978-3-030-48256-55
Jack Sherman and Winifred J. Morrison. Adjustment of an Inverse Matrix Corresponding a Change in One Element of Given Matrix. The Annals of Mathematical Statistics, 21(1):124 127, 1950. doi: 10.1214 20114///m//17772000 URL https:///oi..rgg//// http://o....r//....///////////// 22200///292///////7779999393
14

---

Published in Transactions on Machine Learning Research (08/2023)
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolu- tional networks with noisy labels. arXiv preprint arXis:1406.200 2014.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chenmupati, and Jamal Mohd-Yusof. Combating label noise in deep learning using abstention. In International Conference Machine Learning, pp. 6234-6243. PMLR, 2019.
Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In International Conference on Machine Learning, pp. 3646-3655. PMLR, 2017.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annolations. In International Conference on Learning Representations, 2022. URL http:s///ppparrrreeee.eerrrrrrre
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691 2699, 2015.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagrcement help generalization against label corruption? In International Conference Machine Learning, pp. 7164-7173. PMLR, 2019.
Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. International conference on machine learning, pp. 12846 12856. PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (08/2023)
A Appendix
In the Appendix, we provide derivations of important theoretical results (Section B describe details regarding hyperparameters (Section C and the natural datasets (Section D Furthermore, provide implementation
details of the method (Section
E. discuss additional related work (Section and present additional
experiments (Section
B Derivations
B.1 The Probability Density Function of Logistic-Norme Normal Distribution
Assume Y ~ ~N,N Σ) for some la E _k--- E IR (K (X) and S = Sc then the probability density
function (pdf) of can be written as fs(s) abs([ (())) (S (s)). In which, [J(s],, the determinant of the Jacobian of Sx²(s) (log Sh >ENK- is given by Dillon et al. (2017
1
[J(8] D
8K
D 8 8 K
D
DD
8K
D
8 D
D +
(15)
8 K
D 8-K
(ID - 8 k 8 _ xl)
-1
(DD $ S xxDD18kk)) -1
K
-1
II
8k
where D diag(s k) and s-K (8k) (8k) The first equality is obtained by computing derivatives
while 8к Σ k=1 8k, the fourth equality is based on the Sherman Morrison identity Sherman &
Morrison 1950 and the fifth equality holds based on the matrix determinant lemma Harville (1988). As
(s(s) (s))
(8)), the pdf of the Normmal distribution is
fs(s)
1
1 I Sc(v), (2T) K 1
x-'
(16)
l sk 1(2T) 1
where (5) / n.
B.2 Optimal Covariance Matrix
We want to show that the optimal E matrix for example j is: Σg ( m4) -. First, we note that the gradients of the negative log likelihood in Equation with respect to E, is:
ac
N
Ti log E, 0Σ,
1 1r, I rr log Σ,
σΣ
2
σΣ,
all other terms of the loss are unaffected by Eg and are therefore zero. Computing the gradients with á respect to Σj while applying two identities øH log Σ = I-1 and ax I-1p = --- (see Petersen
et al. 2008) Equations 57 and 63), we have
T D r3 + log E,1l = - 1 D プ
(17)
16

---

Published in Transactions on Machine Learning Research (08/2023)
where the latter identity being justified by properties of symmetry and trace of matrices as follows
0 - πΣ
a dm -
=---"=====----""
(18)
Setting the term the right-hants side of Equation 17 to zero then left and right multiplying by Ej yields the maximizer of the likelihood Σ =r
B.3 Gradients for the Mean with Optimal Covariance
The optimal covariance 2nnh (uj p1))(u) 12²)² rrrr" is rank and is therefore not invertible. However, for invertible Ej have
OC
OC ,,
(19)
0p
This is a linear system of the form Ax = b, which can be solved with exact (or approximate) methods. If we assume Dop is invertible (in practice this is done by adding a diagonal matrix), and solve the linear system with = gg instead, one solution is aC ope
Tr as is an eigenvector of Inth with eigenvalue ||ry|β:
E - - ||, 12r;
(20)
Side-note: In practice, this is not how the gradients computed. Typically the label-dependent part of the loss is computed with a similar rewrite as above, i.c., rr (E E \ r)) where E \ is the (exact or approximate) solution to the linear system Σx = r, which we then do backpropagation through. This is more numerically stable than calculating the inverse and multiplying it with ry.
B.4 The Effect of Lambda on the Optimal Variance
In this section, we provide derivations for the behavior in Figure 2 Our goal is to show how affects the optimally learned a². First, we note that Equation for binary classification becomes
yi
1=)a + +log of +C of
C =
(21)
and we want to look at Ne? P = μ and let r denote the residual. With this notation, the gradient of the loss with respect to o² for example i
c = 0 for particular example i. To simplify notation, we let oā =²= = and
2 1 20
2
2 log 0²1
(22)
0
2
20
2
Solving for when the gradient is zero, gives dapt 2 However, as our predicted variance is oe (cg + A) + X) eg + 2X02 + X2, it cannot. be smaller than X2, and therefore o2 is not always obtainable If then o² obtainable with og = r- A. This corresponds to when the loss is one in Figure 2 However, the optimal variance is not obtainable if, X, as implics og O, which is impossible. To better understand what the network does in this case, look at. the gradient of the loss with respect co
1: Oco 20
a 1
72 + log a dog 12 + log log o² = dog 20% 2
(ağ - ->) + 4Ace)
(23)
dco
o
where the first equality follows from the chain rule. Hence, in the case that oB X2 > 222 the numerator of the gradient can only be zero co zero, making oe X². This corresponds to the (scaled) squared error behavior for small residuals (r- < X2) in Figure
17

---

Published in Transactions on Machine Learning Research (08/2023)
B.5
Target Logits
From Section 2.2
the larget categorical for i is:
t/K (1-
i
S(y) ((1 -t); + tu); ........P...
(24)
i=
and (p) log (IP1,.... .......... PK pk-11/pk). Hence, if + K, then the observed target logit is:
0 i
y= (SC(())) = -5 t)6; + tu);
(25)
C i=
as log (+/+)= = log and where C = log (1-t)K+1 If i = K, then we have for all j:
1
y Sc ((1 - t) + tu); = -CC
(26)
B.6 Gradients for the Heteroscedasitc NN methods
In Heteroscedasitc NN methods, the prediction is the mean of M samples T. = N7 Em ye, in which we define y." --== = /5 and Dx Σx= tet Therefore, the gradient of sofumax oulput w.r.t logits is given by:
0
M
1
dy
M Ô c=ite" M
Σ y
0zj M
M
0zj
dzj M
1
M
8ije=ite +e
t C#i+t"
M
(2 )
m=1
1
M
zitte"
8.g ER
e -e
M
m=1
1
y (8.g 4²²h)
M
m=1
Lm=1 y.ah (1 v i i
i i=j
Σ y y.
M
Using the last equation, we can compute the derivatives of the log- likelihood with respect logits as follows:
a
t logí
1
Σ
y
K
3z;
m==1
M y
Oz
M
M
m=
K
M
U!"h (8ig y
1
ΣM V.th
1=1
M
y ym
1/2
m=
>M m= y
i$j
m= (171)2
+ t2
Σ M
M= 37
K
> M
>m V y
-t,
t
y
m=1
18

---

Published in Transactions on Machine Learning Research (08/2023)
Assuming label t de is given, we can rewrite the above equation in simpler form:
K
M
K
M
0
t log(
1
Σ yi
)
tj
m==
yi yj
dzj
M
m==1 y
1=1
m=1
K
Σ M
m==1 y y3
-t3
t;
> y,"
Σ M
ye y"
-13
mm
Σ mm= y
M
ye
-t3
y3
y
m=1
m=
M
V
-t-
M
yj
M
1
m==
M
ye
The derivatives of the bigliielihood, in vector notation, are given by:
M
t"
log(
Σ )d
=
1
yd
ye
0x
M
M
M
m=1
M
V-
M
S(z + + k )c Σ 2 S( + tTm)
= 6
S(z + E
M
m=1
M
C Hyperparameters
C.1 Method--ndeeendent Hyperparameters
All methods are implemented in the same code base, using the same network architectures, optimizers, hyperparameter scarches, etc. We use a learning rate of 0.0001 for synthetic datasets, 0.001 for MNIST and ClothingIM, and 0.01 for the CIFAR datasets. We show that our method performs well under different optimizers by using gradient descent for the synthetic datasets, Adam for MNIST (batch size 256), and SGD with Nesterov moment-um of 0.9 for ClothingIM (batch size 32) and the CIFAR datasets (batch size 128). We use weight decay of 1e-3 and 5e-4 for Clothing IM and the CIFAR datasets, respectively, but no such regularization for the other datasets. We use an MLP with two hidden layers with 2000 hidden units for the synthotic datasets, a convolutional network (LeNet-5) for MNIST, and residual networks for the CIFAR datasets (WideeRessNet-2822) and Clothing1M (ImageNet pre-trained ResNet-50). We train for 2000, 100, 10, and 300 epochs for the synthetic datasets, MNIST, ClothingIM, and CIFAR, respectively. We use 10% of the training set of MNIST and CIFAR as noisy validation set.
C.2 Method-dependen Hyperparameter Search
In this section, we go over our thorough hyperparameter search we did for the results in Tables and 2 For each method, search for method-specifie hyperparameters for each noise rate and noise type per dataset. For Het-7 and Het-r-Eful, we search for temperatures in [0.1, 0.5, 1.0, 10.0, 20.0], while Het- also scarches over R of the covariance matrix in [1, 2, 2.. We choose the range of values to scarch over is based on the original papers. Our method searches over temperatures and As in [0.1, 0.5, 1.0] for MNIST, but As in [0.5, 1.0] for the CIFAR datasets and ClothingIM. We treat the label smoothing parameter t for LN as fixed, and set to 0.01 all experiments. For GCE, we search over q in 00.10.3. 0.5, 0.9. For NAN, the search is over a in [0.1, 0.2, 0.5, 0.75, 1.0]. For LS, we search for values in [0.1, 0.5, 0.7, 0.9). All searches are done with a single seed, and the hyperparameters with the highest noisy validation accuracy at the end
19

---

Published in Transactions on Machine Learning Research (08/2023)
Symmetric: Epg
Asymmetric E
Aggregate =1, Teye
Random:
Worst: Tua
0.8
Clean
Cloan Nolsy
Clean Nolsy
Clean Noisy
Clean Noley
Nolsy
Density
0.6
0.4
0.0
Symmetric: Zues In Zoye
Asymmetric: Zray Ez. Are
Aggregate: Zrzn Zz. Erce
Random: Zoyn Ex Eay
Worsh: Krran Es, Xeyo
Clean
Clean Noisy
Clean Noisy
Clean Noisy
Clean Noisy
Noisy
Density
0.6
0.4
0.2
0.0
Symmetric: Euys Ea Zeza
Asymmetric: Ress Ee Eecy
Aggregate: Ergs Eg. Epyel
Random: Roas
E. Evc
Worst: Esuo Eg. Exyal Ee
0.8
Clean Noisy sy
Clean Nolsy
Clean Nolsy
Clean Noisy
Clean Nolsy
Density
0.6
0.2
0.0
0.001
0.1
10
0.001 0.1 10 1 Ey 11)
0.001 0.1
10
0.001
D.1
1.0
0.001
0.1
10
1y
11) Eydu W)
(y
MI Eevol (y 1
W1] Ey
Figure 7: Loss Attenuation: The Effect of the Covariance Matrix. For a network trained on MNIST with 40% symmetric noisc, we show the distribution of the label-dependent term of the loss with (left) and without (right) the scaling of the covariance matrix for clean and noisy examples separately. The x-axis correspond to the log-scaled loss. We find that the covariance matrix increases the loss of clean examples and reduces the loss of noisy examples.
of training used train four more networks with different seeds. The hyperparameters used for different. noise rates, noise types, and datasets are shown in Tables and
D Natural Datasets
The CIFAR-1ON dataset has five new sets of labels for the CIFAR-10 training set, which was generated by having each training image labeled by three different humans. Naturally, this gives rise to three different labeled sets (Random 1-3). The fourth set is generated through majority voting, where ties are broken at random (Aggregate). Finally, the last set of labels (Worst) was created by randomly picking one the labels that are different from the original training label, if no such a label exists then the original is used. The noisy labels are 18%, 9%, and 40% of all labels for Random, Aggregate and Worst, respectively. CIFAR-10ON was created similarly, but with single human annotator per image, resulting in noise rate of 40%. ClothingIM is dataset of one million images of clothes from 14 different classes, automatically labeled based captions. As there is large imbalance between the classes, we follow the balancing strategy of Li et al. 2020a We use the provided validation and test sets. The noise rate is estimated to be 38%.
E Implementation Details
We implement our method using the TensorFlow Probability Dillon el al. 2017 library. The Logistic-Normall distribution is implemented as a transformed distribution (the TransormedDistiibtttio class) comprised of a distribution and transform. For the distribution, we use multivariate normal distribution (the AutivariaeVormaaiigPPPl....
class). For the transform, we combine a softmax centered and scale (for temperature) bijector using the Chain bijector class. Conveniently, the loss can then be implemented by
20

---

Published in Transactions on Machine Learning Research (08/2023)
using the built-in method (log (log_prob) of the transformed distribution class to calculate the logarithm of the pdf.
F
Connections to Additional Related Work
F.1
Relationship with Calibration Methods
Both calibration methods and our method are interested in capturing a true probability distribution p(y[æ) and in that sense some standard calibration techniques, such as Lemperature scaling, might be applicable for both. However, there are two key fundamental differences:
Probabilistic noise models are interested in obtaining the true probability distribution of sampled data (only dependent on data and irrespective of the machine learning method). Calibration methods are by defimition tied to a machine learning method and want their probabilistic output to be true reflection of the "probability of correctness" (only dependent on the machine learning method and irrespective of the data). This can be seen, for instance, by considering that a model that always outputs the uniform distribution calibrated but minimally accurate.
Calibration is for unseen (test) data, while noise models are typically for the training data. As such, many calibration methods are often applied as a post hoc approach, tuning calibration metrics on unseen (calibration) data, and then evaluated on unseen (test) data. Furthermore, as the temperature scaling typically done after training, has no effect on the training dynamics, while the temperature we use directly affects the observed larget locations (see first paragraph in Section 3.2 As noise models, such as ours, are primarily concerned with modelling of the training data, the full distributions (that includes variance due to label noise) can in fact be discarded at test time, see Section 3.3
F.2 Relationship with Linear Regression
There is close relationship between our work and (Multinomial) logistic regression. In fact, what we do with the Logistic-Normal distribution can be seen a generalization of doing logistic regression as set of binary regressions, where the generalization is to incorporate label noise. That is,
Logistic regression Sc(y) = Sc(Wx)
(27)
Our case: Sc(v) = Sc(Wa + + e> Sc(y)
(28)
where W are the parameters of the single linear layer. In the logistic regression case, we have no noise, and the resulting likelihood is categorical distribution. However, in our case, due to the noise being normally distributed, the resulting likelihood is a Logistic-Normal distribution, which is a distribution over categorical distributions. This makes it possible for us to explain differences between potentially noisy labels p(v x), with our approximation of the true categorical distribution po po(y)z) as label noise, akin the regression case.
F.3 Relationship with Linear Discriminant Analysis
We see standard LDA as a fundamentally different method as it puts an explicit distribution on the samples belonging to each class, which we do not.
To simplify the analysis, let's consider fixed mapping from the input space x logit space, i.e., LDA operates on h(x) instead of z, and LN has u(z) h(z), and let's call these the "features".
LDA assumes that the features of all examples of a particular class are samples from the same normal distribution. The variance of the features of examples of the same class, could be interpreted as per-class label noise. Hence, we expect LDA to not. be robust to input/feature/ heput//eeauur//eeetooccclaalll /etessaccsoaaatcccccccc...s noise, as a single example with a noisy label, could dramatically affect the learnt location for the distribution for that class.
In LN, we have fixed positions for where the class clusters should be, i.c., the target logit locations. LN assumes each feature corresponds to the true logit position for that example, and the difference between
21

---

Published in Transactions on Machine Learning Research (08/2023)
80
Test Accuracy
75
1
70
0.1
65
0.25
1.0
60
0.001
0.01
0.1 0.2
0.3 0.4 0.5
Label Smoothing (t)
Figure 8: Effect of Label Smoothing on Robustness of LN. The mean and standard deviation of test accuracy for various label smoothing parameters / on CIFAR-10 with 40% symmetric noise. For small X², label smoothing has little effect on robustness. Interestingly, for X2 I the networks overfit the noisy labels more for increasing values instead of improving robustness.
the feature and observed target location is due to normally distributed label noise. Hence, we treat each observed target sample from a normal distribution centered its corresponding feature. If we think of the features of all examples of particular class, as did for LDA, then some might form cluster around the given larget logits, while some features might. be closer to other target logit locations. Consider such a feature that is far away from the observed (noisy) target logit location. Our method would then have large variance to make the target logit more likely, while still keeping the correct fcature location the same.
summarize, LDA estimates per-class normal distributions such that the features of the examples of the corresponding class distribution are likely samples. In contrast, LN estimates per-example normal distributions (up to constants) centered on cach feature, such that the corresponding target logit is likely sample. We single example with noisy label could completely
expect LDA to not be robust against noisy labels, shift the location of the per-class distribution for that particular noisy label. Our model is affected less, noisy examples could still have the correct location, while having a larger variance instead. Furthermore, at test time, our method predicts per-example distributions, while LDA evaluates the per-class distributions for each input.
G Additional Experiments
If not otherwise stated, all the results in the tables in this section reports the mean and standard deviation for the test accuracy over five runs with different seeds.
G.1 On the Robustness of the Logistic- -ormmal Likelihood Due to Label Smoothing
As the Logistic-Normal distribution is not defined on the border of the probability simplex, we proposed use label smoothing to solve this issue. Interestingly, Lukasik et al. (2020) showed that label smoothing itself helps with robustness against label noise. Therefore, a natural question is how important label smoothing is for the robustness of our method. In Figure g show the mean and standard deviation of test accuracy for various settings of the label smoothing parameter (defined in Section 2.2) for our method trained on 40% symmetric noise on CIFAR-10. We find that for X of 0.1 and 0.25, the label smoothing paramelers have little effect on the test accuracy, suggesting that label smoothing is not having a big effect on the robustness of our method. Interestingly, for X2 1, increasing degrades the test accuracy instcad of improving believe this is similar to the temperature, that too high value for t makes most of the residuals be below the loss attenuation threshold and therefore the loss behaves like standard mean squared error loss and overfits.
We
22

---

Published in Transactions on Machine Learning Research (08/2023)
Table 4: Calibration on the CIFAR-1ON Training Set. CIFAR-ION provides three labels per training example, which we see as three samples from the true p(y|z)) Instead of measuring calibration with a single (potentially noisy label), we propose to measure calibration on the training set by taking the average negative log-likelihood (NIL) of each of the provided labels. We observe that LN has significantly lower average NLL and is therefore better calibrated.
Method
NLL
Het-r-Erall
1.66 0.03
CE
1.60 0.01
GCE
1.58 + 0.04
NAN
1.34 + 0.05
LN
1.18 + 0.01
G.2 Training Set Calibration
Calibration metrics are typically used as scalar surrogates to measure the difference between the true p(ylz) and the prediction po(y)=) on unseen data. This is typically done using, P(yla) as we don't usually have access ply|z). However, interestingly, CIFAR-N provides three labels from human annotators per image in the CIFAR-10 dataset, we see these as three samples from p(yla) and have better estimate of the true label via:
M Σ
1 (onchot(y())) onchot(y(2) onchot(y = 5(v1=)) (29)
plylz)= lim M 000
onchot(y(m)
3
where y y/m) p(y(z). Hence, we propose to measure the training set calibration of models trained on the CIFAR-10 dataset (no added noise), by evaluating the standard negative log-likelihood of the predicted catogorical distribution using the three labels provided by CIFAR-N. More specifically, the per-example NLL is calculated as } 3(-logpo(2 = (z) log = -,(2 - po = ())))=))) and all per-example NLLs of the training set are then averaged. See the table below for the results, which the mean and standard deviation of the five networks of models trained with no synthetically added label noise in Table
As with the standard NLL calibration metrics, lower values indicate the model has put more confidence into these classes, which is therefore desired. From the results in Table 4 we find that our method has significantly lower NLL on the training set than the other methods.
G.3 Residual Histograms
In this section, we show similar histograms as in Figure but for more noise types: symmetric noise and aggregate, random 2, and worst from CIFAR-1ON, The setup is the same, to train two networks by minimizing the negative log-likelihood of Logistic-Normal likelihoods with different covariance matrices (Etrain) ) and evaluate the label-dependent term Equation with either Eevot equal to I or Eg at the end of training. See Figure 7 Comparing Etroin = Eeval I (top row) with Egrain Eg, Eeval I (middle row) that the former reduces the residuals of more of the noisy examples than the latter, which indices learning Σ is more robust. Furthermore, comparing Etrain Eg, Eevel I (middle row) with Eraain Eg, Eeval Eg (bottom row), we find that Eg increases the loss for some clean examples and reduces the loss of high-residual examples, both clean and noisy ones.
23

---

Published in Transactions on Machine Learning Research (08/2023)
Table Hyperparameters used for Synthetic Noise on MNIST and CIFAR datasets. A hyperparameter search over method-specific hyperparameters done, and the best values are shown here. For Het-T, we report the temperature, for Het-r-Enan the temperature and the number of factors (R), for GCE q, for NAN or for LS and for LN the temperature (r) and X.
No Noise
Symmetric Noise Rate
Asymmetric Noise Rate
Dataset
Method
20%
10%
60%
20%
30%
40%
Het-r
0.5
0.1
20
0.5
0.5
0.1
MNIST
Heb---Enail
[1.0,
[0.1, 4]
[0.1,
[10, 4]
[10,
[0.5,
[10,
NAN
1.0
1.0
0.75
1.0
GCE
0.7
0.9
0.7
0.5
0.3
LS
[1.0, 1.0]
[1.0, 1.0]
[1.0, 0.5]
[0.5, 0.1]
[0.5, 0.1]
[0.5, 0.1]
[0-5, 0.1]
Het-r
20
10
10
20
CIFAR-10
Het-r-Epal
[0.5,
(20,
[20,
[10, 2]
[20,
[0.1,
[0.5, 2]
NAN
0.5
GCE
LS
0.9
0.9
0.5
0.9
LN
[0.1, 0.5]
[0.5, 0.5]
[1.0, 0.5]
[1.0, 0.5]
[0.5, 0.5]
(0.5, 0.5]
[0.5, 0.5]
Het-r
10
20
CIEAR-100
Hot-r-Epau
[0.5, 4]
[10, 4]
[10, 4]
[10, 1]
[20,
[20, 1]
[20, 1]
0.2
0.1
GCE
0.5
0.5
0.7
0.7
0.5
LS
0.1
0.9
0.7
0.7
0.9
0.7
0.9
LN
[0.1, 0.5]
[1.0, 0.5]
[1.0, 0.5]
[1.0, 0.5]
[0.5, 0.5]
[0.5, 1.0]
[0.5, 1.0)
Table 6: Hyperparameters used for Natural Noise A hyperparameter search method-specific hyperparameters is done, and the best values are shown here. For Het-T, we report the temperature, for Het-r- Hetl the temperature and the number of factors (R), for GCE q, for NAN a, for LS t, and for LN the temperature (r) and X.
CIFAR-ION
CIFAR-100N
Clothing M
Method
Random
Raudom 2
Random 3
Aggregate
Worst
Het-r
10
20
20
20
20
0.1
Het-r- Eron
[10,
[0.1,
[20,
[20, 1]
[10, 4]
[10, 1]
[0.1,
NAN
0.75
0.5
0.5
0.5
0.75
0.2
0.2
GCE
0.9
0.7
0.9
0.5
0.9
0.5
0.9
LS
0.5
0.9
0.5
0.9
0.7
0.9
0.5
LN
(055, 0.5]
[1.0, 0.5]
[0.5, 0.3]
[1.0, 1.0j
[0.5, 0.5]
0.55, 0.5]
[1.0, 1.0]
24


Reviews:
[{"review_id": "3rVHFUhDb4", "reviewer": "Reviewer_Nxf4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The approach of using a distribution over the simplex is reasonable.\n- The proposed method is simple and easy to implement.\n\n## Weaknesses\n\n- The choice of logit-normal distribution was not well justified. Even Atchison and Shen said in their paper that \"Most statisticians ... would start and end with ... Dirichlet distributions.\" Why not Dirichlet? The author did not explain why it is reasonable to assume that the logits are normally distributed nor verify whether real data follows the logit-normal distribution.\n- The estimation of covariate matrices could be unreliable. Since there is usually one label for each input, it is unlikely that the \"per-sample heteroscedastic noise covariate matrices\" can be accurately estimated from the data. Besides, if one uses neural networks to predict them, the outputs might be too smooth. The author only used the test accuracy as a metric but did not directly verify if the covariate matrices can be correctly estimated (not even on synthetic data).\n- The author provided the code, which is nice, but the code is not well documented and quite unreadable. I cannot pin down the location the proposed method is defined, so I cannot verify the implementation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studied the logit-normal distribution for modeling the probability of noisy labels.\nThe proposed method is based on a neural network prediction of logit/covariance and maximum likelihood.\nThe method was evaluated on synthetic data and some image classification datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The author said the method is a maximum a posteriori estimation, but in fact, the proposed method is simply maximum likelihood. No prior was given. Even if we can say ML is MAP given an uninformative prior, their focuses are still different.\n- \"As yi is a single sample from p(y|xi), we see it as a noisy version, p̃(y|xi) = δyi , of the true label distribution p(y|xi)\" This sentence is not understandable for me.\n- ϵp(x) = p̃(y|x) - p(y|x): No, the substraction is not defined. A simplex is included in a vector space, but not all vector operations are defined in a simplex. The difference of probabilities is meaningless in this context.\n- \"our loss is almost identical to a multivariate version of the regression loss\": There is no _the_ regression loss. Besides the Gaussian model, we can also use, e.g., Laplace or Gumbel. This is also why the choice of distribution should be justified for a certain application.\n- Why does the proposed method use gradient-based optimization for the covariance matrix but not the optimal one directly (Eq. (7))?", "broader_impact_concerns": "The author did not discuss broader impacts in the paper.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FSTomupoRp", "reviewer": "Reviewer_6hsn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The method is reasonably simple to understand and implement.\n- Writing is generally good and the description of the method can be followed.\n\n### Weaknesses: \n- Several claims are made that are slightly wrong or at least not well-defined.\n- The method introduces several hyper parameters $t, \\tau, \\lambda$ that must be tuned.\n- It isn’t clear on what sorts of real datasets I would prefer using this method to GCE. In practice, it seems that GCE often works as well or better, and there isn't a concrete description of types of real problems where this method should be preferred. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose performing regression on the logits in classification with a neural network, with heteroskedastic noise that is also the output of a neural network. \nThis is done by adding uniform noise to the labels and pulling-back the (noisy) class labels via the (centered) soft-plus. The authors compare this approach to prior approaches involving\nheteroskedastic noise for classification on several real and synthetic tasks, with most of the real data tasks being image tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The simplex ($\\Delta^K$) should be defined, with some care around the boundary. In particular, it is not true that the map $S_C$ is bijective between the closed simplex and $R^{K-1}$, but from the interior of the simplex to $R^{K-1}$. This is an important detail since, as you later note $\\tilde{p}(y|x)$ is on the boundary of the simplex. Discussion of this point should come earlier. Relatedly, the density in equation 4 isn’t well-defined on the boundary of the simplex, because of the term with $\\tilde{p}(y|x)$ in the denominator. I think these issues should be fixed for the paper to be accepted.\n- The legend in figure 4 is too small.\n", "broader_impact_concerns": "I don’t think a broader impact statement is needed for the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xthMfpHcGF", "reviewer": "Reviewer_vzPd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper shows an interesting extension of loss attenuation in regression to classification.\n- The paper studies the loss attenuation from investigating the gradients.\n- The paper provides experimental evidence about how the proposed method is more robust to label noise compared with baselines for synthetic (class-dependent) noise and natural noise.\n- The paper provides code used in the experiments (note: I did not read the code carefully as part of my review.)\n\nWeaknesses/questions:\n- I found it challenging to locate the values of the label smoothing hyperparameter $t$ utilized in the experiments. It would be helpful if the authors can explain where it is shown. \n- Given its vital role in the proposed method, perhaps an ablation study of this hyperparameter could provide more clarity and will help readers understand the method better.\n- The usual cross-entropy loss minimization has a similar issue of this, where we try to fit the output of the softmax to a one-hot classification label. One way to avoid this is to use label smoothing. The technique referenced in Section 3.2 appears to align with this approach. With reports indicating label smoothing's usefulness in ordinary classification problems and label-noise problems (e.g., Lukasik et al., ICML 2020), it's not immediately evident whether the loss attenuation based on the proposed method enhanced generalization, or if the label smoothing component did, or if both contributed. Thus, a comparison with another baseline (label smoothing + cross entropy) could be helpful.\n- The bi-tempered method is used in experimental results shown in Tabel 5, 6, and 7. However, this is not explained in the paper. While it's possible that I overlooked it, I did not find it explicitly mentioned among the baselines in Section 5.2. Perhaps this could be clarified?\n- The notation $S$ occurs in Eq.13, Eq.14, Eq.24, and in subsequent places. I'm curious if $S$ is equivalent to $S_C$ as defined in Section 2.2? Could you provide some clarification on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Drawing inspiration from how regression methods naturally account for heteroscedastic noise by treating the problem as a maximum likelihood estimation with Gaussian noise, this paper introduces a classification counterpart: estimation with logistic-normal likelihoods. The proposed method exhibits an interpretation of loss attenuation, grounded in the covariance matrix, similar to the regression case where each sample's loss is weighted based on the variance of the considered normal distribution. The paper acknowledges and addresses some unique challenges, such as the inability to consider $y_i$ that is mapped to a one-hot vector, and the handling of classes due to the softmax centering issue. The experimental findings showcase the competitiveness of the proposed method, not only for conventional classification problems but also for those involving noisy-labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n- Add explanation of the hyper-parameter $t$ used in experiments.\n- Add definition of $S$ if it is different from $S_C$.\n- Add explanation of the bi-tempered baseline.\n\nAdditionally, in my opinion, adding experiments and discussions about label smoothing will be interesting. I wrote my detailed comment in the \"Strengths and Weaknesses\" section.\n", "broader_impact_concerns": "I do not have strong ethical concerns of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose performing regression on the logits in classification with a neural network, with heteroskedastic noise that is also the output of a neural network. This is done by adding uniform noise to the labels and pulling-back the (noisy) class labels via the (centered) soft-plus. The authors compare this approach to prior approaches involving heteroskedastic noise for classification on several real and synthetic tasks, with most of the real data tasks being image tasks.

The method is reasonably simple to understand and implement. Writing is generally good and the description of the method can be followed. However, the authors are missing some important references in Memorization Effects category (e.g., MentorNet and Co-teaching series). Therefore, based on three qualified reviews, this paper can be accepted with minor revision and the authors are encouraged to merge the comments into their update versions.
Decision: Accept (poster)