Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
Authors: Vincent Mai
Waleed Khamies
Liam Paull
Venue:  (2021)
Keywords: Regression
Noisy labels
Supervised Learning
Uncertainty
Variance
Heteroscedastic
Privileged Information

Abstract: In model learning, when the training dataset on which the parameters are optimized and the testing dataset on which the model is evaluated are not sampled from identical distributions, we say that the datasets are misaligned. It is well-known that this misalignment can negatively impact model performance. A common source of misalignment is that the inputs are sampled from different distributions. Another source for this misalignment is that the label generating process used to create the training dataset is imperfect. In this work, we consider this setting and additionally assume that the label generating process is able to provide us with a quantity for the role of each label in the misalignment between the datasets, which we consider to be privileged information. Specifically, we consider the task of regression with labels corrupted by heteroscedastic noise and we assume that we have access to an estimate of the variance over each sample. We propose a general approach to include this privileged information in the loss function together with dataset statistics inferred from the mini-batch to mitigate the impact of the dataset misalignment. Subsequently, we propose a specific algorithm for the heteroscedastic regression case, called Batch Inverse-Variance weighting, which adapts inverse-variance weighting for linear regression to the case of neural network function approximation.  We demonstrate that this approach achieves a significant improvement in network training performances compared to baselines when confronted with high, input-independent noise.

Under review as a conference paper at ICLR 2021
BATCH INVERSE- VARIANCE WEIGHTING: DEEP HET- EROSCEDASTIC REGRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
In model learning, when the training dataset which the parameters opti- mized and the testing dataset on which the model is evaluated are not sampled from identical distributions, we say that the datasets are misaligned. It is well- known that this misaligmment can negatively impact model performance. A com- mon source of misalignment is that the inputs are sampled from different distri- butions. Another source for this misalignment is that the label generating process used to create the training dataset is imperfect. In this work, consider this set- ting and additionally assume that the label generating process is able to provide us with quantity for the role of each label in the misalignment between the datasets, which we consider to be privileged information. Specifically, we consider the task of regression with labels corrupted by heteroscedastic noise and we assume that we have access to an estimate of the variance over each sample. We pro- pose general approach to include this privileged information in the loss function together with dataset statistics inferred from the mini-batch to mitigate the im- pact of the dataset misalignment. Subsequently, we propose a specific algorithm for the heteroscedastic regression case, called Batch Inverse- Variance weighting, which adapts inverse- variance weighting for linear regression the case of neural network function approximation. We demonstrate that this approach achieves a significant improvement in network training performances compared to baselines when confronted with high, input-independent noise.
INTRODUCTION
In supervised learning, a central assumption is that the samples in the training dataset, used to train the model, and the samples in the testing set, used to evaluate the model, are sampled from identical distributions. Formally, for input x and label y. this assumption implies that Piraia (x, (x, y) Pteet (x, y). This assumption can be decomposed as the product Pursin (x) Pirain (y/x) = Ptest Peest (g/2x) x)) which is true if two conditions are respected:
1. The features in both datasets are sampled from the same distribution: Pusin(x) = Piest (x).. When this is condition is violated, the training dataset is not representative.
2. The labels in both datasets are sampled from the same conditional distribution: Ptrain (v|x) Prest (y) x). If this condition is violated, the training labels noisy.
In practice, these assumptions are not always respected because gathering representative and precise data (including labels) can be arduous. In this case, the training and testing datasets are misaligned, and the performance of the deployed model may decrease since the training process did not actually optimize the model's parameters based the correct data (Arpit al. 2017 Kawaguchi et al. 2020). One possible reason for misalignment is that there is some uncertainty about the labels in the training set as result of the labeling process. Since our objective is to optimize the performance of the model compared to ground truth labels, we should consider that the labels in test dataset have no uncertainty, even though it may be impossible to collect such a dataset in practice. As result, Prest (ylx) is sampled from a Dirac delta function, whereas Ptrain x)) is not since it encapsulates the uncertainty in the labelling process, which leads to misalignment.
In this paper, we propose an algorithm for more efficient model training in the case where we have some information about the sample-wise misalignment. More specifically, we examine the case of

---

Under review as a conference paper at ICLR 2021
regression with deep network where labels are corrupted by heteroscedastic noise. We assume that we have access at least an estimate of the variance of the distribution of the noise that corrupted each label, information that is available if the labels are being generated by some stochastic process that is capable of also jointly reporting uncertainty. We examine how the knowledge of the estimate of the label noise variance can be used to mitigate the effect of the noise the learning process of a deep neural network. We refer to our method as Batch by information theory, performs re-weighting using both the the sample-wise variance but also statistics over the entire mini-batch. BIV shows a strong empirical advantage over L.2 loss as well as over a simple filtering of the samples based on threshold over the variance
Veersse- (BIV), which, inspired
Our claimed contributions are threefold:
1. A definition of the problem of learning with information quantifying the misalignment between datasets for the case of heteroscedastic noisy labels in regression.
2. A general formulation of how to use the mini-batch to infer statistics of the dataset and incorporate this information in the loss function when training neural networks.
3. We present Batch Inverse-Variance as an instantiation of this framework and show its use- fulness when applied to regression tasks with labels corrupted by heteroscedastic noise.
The outline of the paper is as follows: In section 2 describe the task of regression with het- eroscedastic noisy labels and its parallels with learning with privileged information, and we explain the challenges of applying classical heteroscedastic regression methods stochastic gradient de- scent. In section we position our work among the existing literature on learning with noisy labels. In section 4 we present general framework to incorporate information regarding dataset misalign- ment in the mini-batch loss. We introduce BIV within this framework to tackle heteroscedastic we describe the setup for the experiments we made validate the benefits of using BIV, and we present and analyze the results in section 6
regression. In section 5
2 BACKGROUND
2.1 HETEROSCEDASTIC NOISY LABELS IN REGRESSION
Here, we introduce how heteroscedastic noisy labels can be generated in regression and how the variance can be known. Consider an unlabelled dataset of inputs }x, To label it, one must apply to each input x; an instance of a label generator which should provide its true label y;. This label generator has access some features z, correlated to x; We define LG, Z + R. When the labelling process is not exact and causes some noise on the label. the noisy label of x, provided by LG, defined as ÿi.z- Noise on measured or estimated value is often represented by a Gaussian distribution, based on the central limit theorem, as most noisy processes are the sum of several in- dependent variables. Gaussian distributions are also mathematically practical, although they present some drawbacks as they can only represent unimodal and symmetric noise (Thrun et al.) 2006). We model:
Ú = yo + dy., with dy.. N N(O,07.)
of can be a function of 2, and I.G,, without any assumption on its dependence on one or the other. We finally assume that the label generator is able to provide an estimate of o²,j- therefore being re- defined as LG,: Z + R x Ezo. The training dataset is formed of triplets (x,, (x,, Ft.s), renamed (xg, 0 Dv) for triplet is for simplicity. This setup describes many labelling processes, such as:
Crowd-sourced labelling: In the example case of age estimation from facial pictures, labellers Alice and Bob are given z, = x, the picture of someone's face and are asked to estimate the age of that person. Age is harder to estimate for older people come (5 and 15 years of age are harder to confuse than 75 and 85) suggesting a correlation between of, and z;. But Alice and Bob may also have been given different instructions regarding the precision needed, inducing correlation between of, and I.G,. Finally, there may be some additional interactions between z, and I.G,, as for example Alice may know Charlie, recognize him on the picture and label his age with lower
OOr code is available in supplemental material and will be publicly released after the reviewing process.
2

---

Under review as a conference paper at ICLR 2021
uncertainty. Both labellers can provide estimation of the uncertainty around their labels, for example with a plus-minn minus range which can be used as a proxy for standard deviation.
Labelling from sensor readings, population studies, or simulations: Imagine you want build a dataset of pictures x; from a camera on the ground labelled with the position yi of a drone in the sky. To estimate the position of the drone at the moment the picture was taken, you could use state esti- mation algorithms based on the Bayes' filter (Thrun et al.) (006). These algorithms take as an input z; the measurements of the drone's sensors, and provide a full posterior distribution over the state, sometimes under a Gaussian assumption for Kalman filters for example. The uncertainty depends, among others, on the precision of the sensors, the observability of a given state, the precision of the dynamic model, and the time since sensor signals were received. Similarly, studies based on pop- ulation such as polling or pharmaceutical trials have quantified uncertainties based on the quantity and quality of their samples. It is also possible to train on simulators, as in climate sciences (Rasp) et al. 2018) or epidemiology (Alsdurf et al., 2020), and some of them provide their estimations' uncertainty based the simulation procedure and the inclusion of real measurements in the model.
Using predictions from a neural network in complex neural architectures: In deep reinforce- ment learning for example, the critic network learns to predict value from a state-action pair under the supervision of the heteroscedastic noisy output of a target network plus the reward (Mnih et al.) 2015 Haarnoja al., 2018). While the estimation of the uncertainty of the output of a neural net- work 18 not an easy task, it is an active field of research (Gal & Ghahramani] 2016 [Peretroukhin ct al. 2019). There, z, is the state-action pair at the next step, and I.G,, the target network being updated over time. The prediction is mix of aleatoric and epistemic uncertainties as defined by Kendall & Gal (2017) which are dependent on both z; and I.Gy-
We could not find any current dataset that provides such label uncertainty information for regression. However, as it is precious information, we argue that it should actually be provided when possible. In classification, Xie et al. (2016: 2020) took step in this direction by providing a "confidence" score from 0 to 255 for each pixel in the KITTI-360 dataset.
2.2 LEARNING USING PRIVILEGED INFORMATION
Training with dataset of triplets (x,, xi, 3i), where x; is only given at training time and not avail- able at test time, fits in the framework of learning using privileged information (LUPI), defined in Vapnik & Vashist (2009) and mainly applied to SVMs. In most works in this field, this privileged information makes the task easier on a sample-to-sampe be improved by adding segmentation masks (Feyereisl et al., (2014) or depth images (Hoffman et al., 2016). Another interpretation of LUPI is use privileged information as a vector for knowledge transfer between a teacher and student (Vapnik & Izmailov| (015).. Hernández-Lobate et al. (2014) and Lambert et al. (2018) have made a link between privileged information and uncertainty. using to evaluate the conlidence of the model for a training sample. The former applied the approach to Gaussian processes through latent noise, and the latter to neural networks through Gaussian dropout.
basis. For example, object detection can
More formally, at training time the neural network has access to triplets (Xx: xi yk) where x is the input, yı its corresponding label, and X the additional information with respect this sample.
The objective in LUPI is the same as classical supervised learning: train the network parameters so that, at test time, and without access information x the expected loss is minimized, i.e.:
gept_ = arg min B(x....EDD... I (I 9:))]
where L(f(x;, 0), y:) is the objective loss function based on the true label and on the network's prediction f(x;, 0), for example the L2 distance in the task of regression.
In our work, we have x = of- In contrast with the usual LUPI setting, x; does not help the task on sample-to-sample basis, but instead informs about the role of each sample on the misalignment between the datasets due to the noise in the labelling process. The objective, however, is the same: use this privileged information during training to minimize the expected loss at test time.
3

---

Under review as a conference paper at ICLR 2021
2.3 HETEROSCEDASTIC REGRESSION FOR LINEAR MODELS
The task of heteroscedastic linear regression, where the model is linear, is solved by optimizing weighted mean square error (WMSE) with unversevarriace weights, which the optimal solution as per the Gauss-Markov theorem (Shalizi 2019):
Σ yi 8 of
(3)
where 3 is the vector of parameters used as linear coefficients. This is also the solution to maximum
likelihood estimation for B (Fisher 1957).
While the solution to such an optimization is known for linear regression (3* = (x" wx )xx wy), several problems appear when attempting to adapt it gradient-based methods neural networks, such as stochastic gradient descent: (1) the learning rate in gradient-based methods impacts the optimization process in multiple ways and should be controllable by the practitioner regardless of the amount of noise in the samples to prevent very small or large gradients from destabilizing the learning process (2) similarly, near ground-truth samples should not have disproportionate learning rate with respect to the others, as they risk to cause overfitting.
In our work, we propose method to apply such weights to neural networks while addressing these issues.
3 RELATED WORK
Noise on labels amounts to a loss of information. When the noise is significant enough, leads to overfitting and lower model performance (Liu & Castagna 1999 Zhang al. 2017). This effect is more prevalent in small data settings (Van Hor et al. 2015). Four possible strategies exist in the literature to tackle this problem: detection, correction, robustness, consists of identifying noisy labels and ignoring them in the learning process. These methods are often based on the observation that neural networks first fit on consistent, non-noisy data (Arpit al.
re-weighting. Detection
2017). thus converging to a higher loss on the noisy samples (Reed et al. [015 Shen &Sangaavi 2019 Other methods use several neural networks co-teach each other (Han et al.) [2188] Yu et al. 2019) or dropout to estimate the consistency of the data (Reed et al., 2015). However, in the case of imbalanced training datasets, higher loss also be the signature of non-noisy but rare sample. [Cao et al. (2020) address this ambiguity by regularizing different regions of the input space differently. Correction strategies go further: once noise is detected, the noisy labels changed to probability distributions. Such an operation requires a noise model. Goldberger & Ben- Reuven (2017): Kremer al. (2018): Ma al. (2018); Tanno et al. (2019); Yi & Wu (2019) learn it jointly with the parameters, assuming a correlation between the noise and the input, the labels, or both. Robust loss functions are less sensitive noise. Liu & Castagnal (9999) proposed to avoid overfitting due to noise by ignoring samples during the training when the prediction error is reasonable. Natarajan et al. (013) compute the loss assuming knowledge of xxample-independal mislabelling probabilities in binary classification, and then optimize these hyperparameters with cross-validation. More recent works are based reverse cross-entropy (Wang al.) 2019) or curriculum loss (Lyu & Tsang, 2020). Others leverage a distillate of the information gathered from a subset of clean labels guide training with noisy labels (Li et al. 2017). Re-weighting the samples is another efficient method for mitigating noise in datasets. Liu & Tao (2016) estimate the effective label probabilities
well noise rates for a given input and use these estimates to weigh the samples using importance sampling. Shu et al. (2019) go one step further by learning the weighting function through a meta-learning method. Jenni & Favaro (2018) control overfitting by adjusting sample weights in the training and validation mini-batches, increasing robustness to overfitting on noisy labels.
While most works that address noisy labels consider classification tasks (Song et al. 2020). only some of these strategies be generalized to regression. Heteroscedastic regression occurs when each label's 's noise is sampled from a different distribution. [Ni & Weigend (1994) tackle this prob- lem in neural networks by jointly training variance estimator based on the maximum likelihood of an underlying Gaussian model. Kendall & Gall (2017) use the same idea to estimate the aleatoric (input-dependant) uncertainty of the network's prediction, while using dropout Bayesian ap-
4

---

Under review as a conference paper at ICLR 2021
proximation for the epistemic uncertainty (due to the learning process) in (aal & Ghahramani 2016).
Our method tackles heteroscedastic regression in neural networks using re-weighting approach. The main distinction between our work and most the related literature is that, while we do not require that the noise variance is a function of the input of the label, we do assume that have access to the noise variance, or at least an estimate of it. In addition, we do not seek to regress the variance of the model's prediction. This is significant compared to the previous works in both regression and classification as changes the loss function and removes the need for a regularizer for the variance prediction.
4 INCORPORATING PRIVILEGED INFORMATION IN THE LOSS FUNCTION
In this section, we first present a general operator incorporate privileged information and infer dataset statistics on the loss computed at the mini-batch level. Then, we describe our solution, BIV, an instance of this operator for heteroscedastic regression. Finally, we introduce more basic filtering function which we will use as a baseline our experiments.
4.1 GENERAL OPERATOR FOR TRAINING ON THE MINI BATCH LEVEL
The value of privileged information about misalignment between the training and testing datasets often higher when combined with statistics about the datasets. For example, in the case of noisy labels, the uncertainty on each label is relevant when compared the information carried by the other samples, similarly to (3).
We propose both incorporate the privileged information and infer dataset statistics during the training of neural networks over a mini-batch, as opposed to the individual samples. There are two main advantages in doing First, if the mini-batch samples sampled from the dataset, they can be used to infer some statistics of the whole dataset. Working on the mini-batch level allows whole dataset, which is important for many tasks such as continuous learning. Second, by focusing on the single step of computing the loss, this approach does not interfere with any of the variety of other methods used to optimize the learning process, such as regularization, batch normalization, annealing learning rates, etc.
independently and identically
such approach without any pre-pre pre-processing step over the
In general, this approach can be expressed by defining an operator G applied on the objective loss function. For a mini-batch D, of K sample triplets, we define the loss as:
Lratch (D.,0) 0) = 9 (XI:K; xi.K, YIKK & (.,))
Note that without any privileged information, operator G is usually the unweighted average of the loss computed over each sample of the batch, which is equivalent empirical risk minimization.
4.2 BATCH INVERSE- VARIANCE WEIGHTING FOR HETEROSCEDASTIE NOISY LABELS
To tackle the problem of heteroscedastic regression in neural networks, we follow the intuition of equation (3) and describe as a weighted average with weights wkk = 1/ (o) + c). We introduce the Batch Inverse-Variance (BIV) loss function:
((((x,)),, 0)
Lhatch (D;,
Σ
of +6
Here, the inverse of the sum has two major roles. is a normalization constant for the mini-batch, allowing to keep consistency in the learning rate. Note that consistency is verified as, when of is identical for each sample, this formulation leads to empirical risk minimization.
The hyper-parameter is effectively lower bound on the variance. This allows samples with ground-truth labels without completely ignoring the other samples. The choice of e trade-off between regulating the weights of tra-grouun-truth labels and using BIV its full capacity. We found that it can be set between 0.01 and 0.1 and use e = 0.1. More details on can be found in appendix B.1
incorporate
5

---

Under review as a conference paper at ICLR 2021
These two elements, added to the advantages of computing the loss function in the mini-batch only, allow us to overcome the challenges related to inverse variance weighting applied to gradient de- scent, as described in section 2.3
4.3 CUTOFF: FILTERING HETEROSCEDASTI NOISY LABELS
As we do not assume that there a correlation between X5 and of most correction and re-weighting algorithms as presented in section 3 are not applicable. Most robust loss function are specifically designed for classification problems. We thus compare BIV a detection and rejection strategy.
In heteroscedastic regression, an important difference from classification with noisy labels is that all labels are corrupted, albeit not at the same scale. Defining which labels to ignore is therefore matter of putting threshold on the variance. As we have access to this information, strategies such as the ones used in section 3 are not necessary. Instead, we simply use an inverse Heaviside step function as a weight in the loss function:
(6)
where the threshold C' is a prper-parameter. Similarly equation (5), we normalize the loss in the mini- batch by the sum of the weights, equal here to the number of samples considered as valid. As this filterng is equivalent to cutting off part of the dataset, refer to this method 'Cutoff", and consider it to be a relevant baseline to compare BIV against.
5 EXPERIMENTAL SETUP
To test the validity of the BIV loss (5) approach, we compared its performance with the classical L2 loss well as cutoff loss O on two datasets. We refer to ground-truth (GT) labels when training with L2 on noise-less data as the best performance that could achieved on this dataset.
Unfortunately, did not find any existing dataset for regression where label uncertainty is associ- ated to the samples. We therefore used two UCI datasets (Dua & Graff 2017) for regression cases, and artificially added noise to them. UTKFace Aligned& Cropped (Song & Zhang 2017) (UTKF), a dataset for image-based age prediction. In the Bike Sharing dataset (Fanaee-1
Gama 2013). the task is to predict the number of bicycles rented in Washington D.C., from structured data con- taining the date, hour, and weather conditions. For UTKE, a convolutional neural network was used to predict the age, while simple multi-layer perceptron was used for BikeSharing. More details about the datasets and models can found in appendix A
5.1 NOISE GENERATION
To produce the datasets {xi, o } with noise as described in section 2.1 we use two-step process which does not assume any correlation between the noise and the state.
1. the noise variance of is sampled from distribution P(a?) which only has support for g² 2
2. U1. is sampled from normal distribution (B ot)
P(o²) has strong effect the impact of BIV Cutoff. For example, if it is Dirac delta and all variances are the same, BIV becomes L2. We evaluate BIV on three different types of P(o²). The average noise variance μP chosen empirically that the lowest test loss achieved by L2 is doubled compared to the ground-trutte label case: μ = 2000 for UTKF and 20000 for BikeSharing.
Uniform distribution The uniform distribution is characterized by its bounds b. Its expected value pP is the average of its bounds, and its variance V
a/2/12 As P only has support for x² > O, the maximum variance Vmaa is when a = 0 and b = 2pp. While such a distribution is not realistic, it is simple conceptually and allows for interesting insights.
"Binary uniform" A more realistic distribution, which also help us understand the effects of BIV, is when the data is generated from two regimes: low and high noise. We call the "binary uniform" distribution mixture of two uniform distributions balanced by parameter p.

---

Under review as a conference paper at ICLR 2021
With probability p. the label is in low noise regime: o² (0, 1), with expected value 14 0.5. With probability 1 1-p, the label is in a high noise regime: o² ~ U (an, b,). ) an and by are chosen such that the average is μ4 and the variance of the high-noise distribution is determined by Vy, E [0, Vmaxl- Note that, if we want a given expected value of the whole distribution μAp, the value of μt changes depending p. μА (uP pp4)/(11 p)
Therefore, the high-noise expected value μ4, of the noise variance o² in a distribution with high p will be higher than the one for low for the same value of pp. In other words, a higher means more chance to in the low-noise regime, but the high-noise regime noisier.
Gamma distributions While the mixture of uniform distributions ensures support in the low noise region, it is not continuous. We therefore also propose to use a Gamma distribution with shape parameter a. If we want to control the expected value (4p, we adjust 8 a/pr-
For a fixed expected value μup, lower a and 3 mean that there is a stronger support towards low variance noise, but the tail of the distribution spreads longer on the high-noise size. In other words, a lower a means more chances to have low-noise samples, but when they are noisy, the variance is higher. When a < I, the highest support is at o² O.
5.2 EVALUATING THE PERFORMANCE OF THE MODEL
The objective of BIV is to improve the performance at predicting the true label yi, as mentioned in equation ( While non-noisy test dataset may not be available in a real application, we aimed here at determining if BIV performs better than L2, and therefore measured the performance of the network using ground-true -truth test data.
EXPERIMENTAL RESULTS AND ANALYSIS
6.1 FOR L2 LOSS, MEAN VARIANCE IS ALL THAT MATTERS
Before looking at the results for BIV, we share an interesting insight for L.2 loss with noisy labels which helps simplifying the analysis of the results. Under the unbiased, heteroscedastic Gaussian- based noise model presented in section 5.1 the only parameter of distribution P(a) that mattered to describe the performance of the L2 loss is its average μu,, which is also the variance of the overall noise distribution. Independently of the distribution type, and the values of V, P and Vh, or a, as long p is equal, the L2 loss trained neural networks had the same performance. This is shown in Figure [[] For the sake of clarity, all the curves in this section were smoothed using moving average with 35 steps window, and the shaded area represents the standard deviation over 10 runs.
Test lass on true labels
aveddged 10 runs)
Uniform a
Zep Binary uniform p=0.1, V,
0.8
Sinrry uniform pmo.9. V2 Vaza
0.6
0.0008
0.4
Gamma 0.25,
Labels
0.2
200
400
600
800
1000
1200
Training steps (batch size: 256)
Figure 1: Performance of the neural network on UTKF trained with L2 loss, for different P(a²) with constant pp 2000. No matter the distribution type parameters, the performance is similar.
6.2 HIGH AND LOW VARIANCE NOISE REGIMES: BIV ACTS AS A FILTER
With the binary uniform distribution, the noise is split in two regimes, with high or low variances. In this case, our results show that BIV performs better than L2, and actually similarly to the cutoff loss presented in section 4.3 with a threshold C' 1.
Figure 2 compares the test losses UTKF with different values of p for V, 0. While the L.2 curves are strongly impacted by the noise, both the BIV and cutoff losses lead to better and very similar performances for given p. When p 0.3, there are not lot of information that can be

---

Under review as a conference paper at ICLR 2021
Test Test loss on true labels
0.7
p-0.3, V,-0-12
p-0.3 V,=0
0.6
V,-0 -utoff
(avereqed 10
y-0.A V,=0-L2
p-0. v, -IV
p-0.5 v,-0
p-0.9, V,-0-12
pmb.9. V, BIV
p-0.0, V,-O
200
400
600
800
1000
1200
Training steps size: 256)
Figure 2: Comparison between BIV, Cutoff and L2 losses for binary uniform distributions of vari- ance with different ps for μp 2000 and V,, on UTKE.
used, and the performance is still impacted by the noise. When p 0.9, there is nearly as much near-ground-truth data as in the noiseless case, and the performance comparable.
In the case of binary uniform distributions, BIV is acting as a filter, cutting off labels which are too noisy contain any useful information.
6.3 CONTINUOUS, DECREASING NOISE VARIANCE DISTRIBUTION: THE ADVANTAGE OF BIV
On Gamma distributions, there is no clear threshold to define which information to use. When S BIV shows strong advantage compared to both L2 and cutoff. Figure shows the results in both the BikeSharing and the UTKF datasets for Gamma distributions with a 1.
UTKF PP 2000
BikeSharing "p 20000
Test loss on true labels
(averaged on 10 nuns)
0.18
C Cy Cutoff
0.16
C Cma 12 Cutof
0.7
Pp Cutoff
0.14
0.6
0.12
2pp
Spp
0.5
0.1
BMV
0.4
0.08
No noise L2
0.06
20C
400
600
800
1000
1200
500
1000
1500 2000 25000
Training steps (batch size: 256)
Training steps (batch size: 256)
Figure 3: Comparison between the performances of BIV, 1.2, and different cutoff values on both datasets where the noise variance follows Gamma distribution with 1.
In both datasets, when the cutoff parameter C' is too low (up/4 /4 and pP/2), there is not enough data train the model. When C is high (2μp and 5μp), the data too noisy and the curves go close to the original L2 loss. Even at the best case (C = μPP), cutoff is not better than BIV. This is because, in contrast cutoff, BIV able to extract some information from noisier samples while avoiding to overfit on them.
In Table present the lowest value of the test loss curves for the different methods with other parameters for the Gamma distributions over both datasets. BIV consistently leads to the best performances, regardless of P(a²). The plots showing these runs be found in the appendix B.3.1 BIV is less sensitive to hyperparameters than cutoff, as it avoids the need to choose the right cutoff parameter for each distribution P(o²). BIV's own hyperparameter can be set between 0.01 and 0.1 for any dataset with a normalized output, as shown in appendix [B.]] and be ready to use. As can be seen as minimal variance, scaling it for other label distributions is straightforward: suffices to multiply it by the variance of the label distribution.
The benefit of BIV over 122 clearly higher when a is lower. This is due increase in the support for low-variance noise P(o²). The more BIV can count on low-noise elements and differentiate them from high noise ones, the better it can perform. This is consistent with results from section
8

---

Under review as a conference paper at ICLR 2021
Table 1: Lowest loss for different a on two datasets, for BIV, L2 and several cutoff losses. The test loss with standard deviation is computed as the average over 10 runs. In every case, BIV loss led the lowest value. The best C' value differs based on a.
0.5
0.25
UTKF
Bike
UTKF
Bike
UTKF
Bike
C
/20
0.79#.08
0.327#.056
0,48#.04
0.125#.010
0.38 =.05
0.092
C = 4p/4
0.4
0.135=.016
0,45
0.097#.006
0.39
0.085
C pr
0.50+.04
0.111+.009
0.48+.04
0.097+.008
0.43+.03
0.088
C' 5pp
0.55#.06
0.1201.009
0.54
0.1112.012
0.51
0.1071.009
L2
0.56:E.05
0.122+.010
0.56:.05
0.1164.0111
0.55
0.119 =.012
BIV (ours)
0.47+.03
0.096
0.41
0.084#.006
0.34
0.079+#000
GT labels
0.25 .2
0.0661..04
0.25#.02
0.0661.004
0.25
0.066 ..066:1.004
6.2 and with other experiments we have run. For example, when a > 1, the highest support of not at o² = 0. BIV was less able to improve the performance compared to L2.
We also ran the experiment with uniform distributions: the performance is better when variance V is closer to Vmar (and to 0). But even when V = Vmazz as there is less support in low noise variance than for Gamma distributions with a S 1, the improvement is less important.
In all cases, BIV was performing consistently better than L,2 and at least better than cutoff in all the experiments we ran. More details on these results can be found in appendix [B2
6.4 ROBUSTNESS
We identified two elements that could impact the performances of BIV: the size of the mini-batches and the accuracy of the noise variance estimation. We tested the robustness of BIV when these factors are different than during our experiments.
Size of the mini-batches In equation S each weight is normalized based on the assumption that the distribution of noise variances in the mini-batch is representative of the one in the whole training dataset. While this is less the case with smaller mini-batches, our results show that BIV still performs very well in these cases, presented in section B.4.1
Noisy variances Because the noise variance of is often estimated, the method needs to be robust to errors in o A model for the noise of af can he a Gaussian for which the variance is proportional to of. In this case, results show that the effect of moderate to high levels of noise BIV is not significant. More details can be seen section B.4.2
7 CONCLUSION
We have proposed a mini-batch based approach to incorporate in the loss function privileged infor- mation which quantifies the participation of each sample the misalignment between the training and testing datasets. We described how such a setup can oceur in the case of regression with het- eroscedastic noisy labels. To tackle this problem, variance weights stochastic gradient descent. BIV is able to extract more information from the noisy dataset than L2 loss threshold-based filtering approaches, and consistently outperforms them on both structured and unstructured datasets. BIV can improve the performance supervised learning in many heteroscedastic regression scenarios, where the label is generated by a process such as browd-labelling, sensor- sensor-based state estimation, simulation, or complex neural architectures. More generally, the framework for including privileged information quantifying the datasets misalignment in the loss function on a mini-batch level could be used to account for other types of misalignment, such under-represennte hidden features correlation between samples.
introduced BIV, method to apply inverse-

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hannah Alsdurf, Yoshua Bengio, Tristan Deleu, Prateek Gupta, Daphne Ippolito, Richard Janda, Max Jarvie, Tyler Kolody, Sekoul Krastev, 'Tegan Maharaj, and al. Covi white paper. arXiv:2005.08500 [cs], May 2020. URL 2005.08502.
arXiv:
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and et al. A closer look at memorization in deep networks. SkKV:1706.053399 [cs, stat), Jul 2017. URL
http://xxivv.rrg.go. http://aax..../////////.6.... 05394
arXiv: 1706.05394.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Arechiga, Adrien Gaidon, and Tengyu Ma. Het- eroskedastic and imbalanced deep learning with adaptive regularization. arXiv:2006.1577 [cs, stat], Jun 2020.
2006. 15766 arXiv: 2006. 15766.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://arrri
ics. www.cci.cco www.:uiii..um.....
Hadi T and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, pp. 1-15, 2013. ISSN 2192-6352. doi: 10.1007/ $13748-013-0040-33
Jan Feyereisl, S. Kwak, Jeany Son, and Bohyung Han. Object localization based on structural svm using privileged information. Advances in Neural Information Processing Systems, 1:208-216, 01 2014.
G. R. Fisher. Maximum likelihood estimators with heteroscedastic errors. Revue de I'Institut Inter- national de Statistique / Review of the International Statistical Institute, 25(1/3):52, 1957. ISSN 03731138. doi: 10.2307/1401674.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML' 16, pp. 1050-1059. JML..org. 2016.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// openreview wtt.:/tteeeoommmdd111mleaa...
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. /.01v0/0001./221 [cs, star), Aug 2018.
org/ abs / 1801. 01290 arXiv: 1801.01290.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. arXiv e-prints, art. 1.Xi::8804006777 April 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv: 1512.03382 [cs], Dec 2015. URL httppp//aarxt..ooo//b///5552 arXiv: 1512.03385.
Daniel Lernández-Lohato, Viktoriia Sharmanska, Kristian Kersting, Christoph H. Lampert, and Novi Quadrianto. Mind the nuisance: Gaussian process classification using privileged noise. TrXiiv1107.0111 [cs, stat), Jul 2014. URL RRRLittp::/aaxxivorr//aa//////// 1407.0179.
0179 arXiv:
Judy Hoffman, Saurabh Gupta, and Trevor Darrell. Learning with side information through modality hallucination. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 826-834. IEEE, Jun 2016. ISBN 978-1-4633-88811- doi: 10.1109/CVP..2016.966 URL
http:// http::///eeeeppllec
ieee. org /tc.ee..ogg/ccccnnee///000000000
Simon Jenni and Paolo Favaro. Deep bilevel learning. arXiv: 080.0019990505 [cs, stat), Sep 2018. URL http://arxiv wttp::/wwwwwwww./..oeee//
arXiv: 1809.01465.
10

---

Under review as a conference paper at ICLR 2021
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learn- ing. arXiv: 1710.05468.
[cs, stat]. Jul 2020. URL /
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for com- puter vision? In Advances in Neural Information Processing Systems 30, pp. 5574-5584, Curran Associates, Inc.. 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 [cs], Jan 2017. URL http::/ttttttt//
wwwg/as////a//// 6980 arXiv: 1412.6980.
Jan Kremer, Fei Sha, and Christian lgel. Robust active label correction. volume 84 of Proceedings of Machine Learning Research, pp. 308-316, Playa Blanca, Lanzarote, Canary Islands, 09-11 Apr 2018. PMLR.
mml. s
www.srressseeeee he.////rreeerrrr aa html
John Lambert, Ozan Sener, and Silvio Savarese. Deep learning under privileged information using heteroscedastic dropout. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 8886 8895, IEEE, Jun 2018. ISBN 978-1-53886 -388--6-38-642 00926. URL
doi: 10.1109/CVPR.201 leee org/ wttp:o////.cccmmmt//////////
Yuncheng Li, Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. W:Xiv:/703.02:30 [cs, stat), Apr 2017. URL http:://axtt
org/abs/ 1103.. 02391 arXiv: 1703.02391.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):447-461, Mar 2016. ISSN 0162-8828, 2160-9292. doi: 11100//AM...2055522244
arXiv: 1411.7718.
Z. P. Liu and J. P. Castagna. Avoiding overfitting caused by noise using a uniform training mode. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No. No.99CH36339) volume 3, pp. 1788-1793 vol.3, 1999.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. arXiv: 10045 [cs, stat], Feb 2020. URL http://axxxxx..g.//aa/ 10045 arXiv: 1905.10045.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Su- danthi Wijewickrema, and James Bailey. Dimensionality-driver learning with noisy labels. arXiv:1806. 02002 [cs, stat], Jul 2018. URL arXiv: 1806.02612.
absrgg/bbb/8866.6 02612
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, and et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529-533 Feb 2015. ISSN 0028-0836, 1476-4687, doi: 10. 11008/natur1122555
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In C. J. C. Burges, I.. Bottou, M. Welling, 7. Ghahramani, and K. Q. Weinberger (cds.), Advances in Neural Information Processing Systems, volume 26, pp. 1196-1204. Cur- Associates, Inc., 2013, URL http:/ts//roccepaiiissssoo/o//es www.eeerseeeennnnee.ee.......... www.wnnww.i...m wtwp:ripps..cppppe//////0/ F11::391166122200333344440011333 .al
D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEF International Conference on Neural Networks (ICNN' (94), volume 1, pp. 55-60 vol.1, 1994.
Valentin Peretroukhin, Brandon Wagstaff, and Jonathan Kelly. Deep probabilistic regression of elements of so(3) using quaternion averaging and uncertainty injection. In CVPR Workshops, 2019.
Stephan Rasp, Michael S. Pritchard, and Pierre Gentine. Deep learning represent subgrid pro- cesses in climate models. Proceedings of the National Academy of Sciences, 118(39)9684-9689, Sep 2018. ISSN 0027-8424, 1091- doi: 10. 2o.1073/pnan aataaa...112000 1810886111. 15.
11

---

Under review as a conference paper at ICLR 2021
Scott Reed, Honglak Lee, Dragomir Anguelov. Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv:1412.6596 [cs], Apr 2015.
httpg///ab////1// 6596 arXiv: 1412.6596.
Cosma Rohilla Shalizi. Advanced Data Analysis from Elementary Point of View, 2019. URL
https:/ /www. stat. cmu. cessed November 13th, 2020).
pdf
(Ac-
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss min- imization. arXiv:/8100 10.001011 1874 [cs, star], Feb 2019. URL htt.tttppp/aaii..or////////////p htspg//bs/1110..
11874 arXiv: 1810.11874.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zonghen Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. arXiv: /902.07379 [cs, stat], Sep 2019.
URL http:httt::/ttete org/ abw//as///2 07379
arXiv: 1902.07379.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv:2007.08) 18999 [cs, stat], Jul 2020. URLhttp://arxxt
org/ 2007. 081 99
arXiv: 2007.08199.
Yang Song and Zhifei Zhang. UTKFace, Large Scale Face Dataset, 2017. URL
https:
7/susanqq. git huu. / /
(Accessed June 11, 2020).
Ryutaro Tanno, Ardavan Sacedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Sil-
berman. Learning from noisy labels by regularized estimation of annotator confusion. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11236-11245. IEEE, Jun 2019. ISBN 978- 778-1-72213-29341 doi: 10.1 1w1.11///VR...1...0111
URL https:
7/ieeexplore. leee. org dwcument 89534067
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics. Intelligent robotics and autonomous agents. MIT Press, 2006.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVRR), pp. 595 604. IEEE, Jun 2015. ISBN 978-1 doi: 10.1 1019/C/PR.201..799665. URL http:: ieee.
7298658/
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(61):2023-206) 2015.
Vladimir Vapnik and Akshay Vashist. new learning paradigm: Learning using privileged infor- mation. Neural Networks, 22(5-6):544-5555 Jul 2009. ISSN 08936080. doi: 10. 10.1016/j.neunet. 2009.06.042.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. arXiv: 11080998811111 12 [cs, stut], Aug 2019. URL
http::/arxiv.coz
http::///aaa..ooo//////9888..0. wwtp://////////....//./// 06112
arXiv: 1908.06112.
Jun Xie, Martin Kiefel, Ming-Ting Sun, and Andreas Geiger. Semantic instance annotation of street scenes by 3d to 2d label transfer. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Jun Xie, Martin Kiefel, Sun Ming-Ting, and Andreas Gieger. Kitti-300, 2020. URL ht
www. cvl ibs https//tttaaaasee//tttt-----
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. arXiv:
arXiv: 1903:0/00000788 [es], Mar 2019. URI. 1903.07788.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? PeXiiv1901.000 101.11 [cs, stat), May 2019. URL
04215
arXiv: 1901. 10425.5
12

---

Under review as a conference paper at ICLR 2021
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arxiv:1611.033: [cs], Feb 2017. htthtt::
03310 arXiv: 1611.03530.
A APPENDIX DATASETS AND NEURAL NETWORKS
A.1 UTKFACE
A.1.1 DATASET DESCRIPTION
The UTKFace Aligned&Cropped dataset (Song & Zhang 2017) consists of 20,000 pictures of faces labelled with their age, ranging from 0 to 116 years. We use it in a regression setting: the network must predict the age of person given the photo of their face. Unless described otherwise, 16,000 images were used for training, and 4,000 for testing.
Some images are in black and white and some are in color. The pixel dimension of each image is 200x200.
Both the pixels and the labels were normalized before the training, so that their mean is 0 and standard deviation is over the whole dataset. The noise variances were correspondingly scaled, as well as the cutoff threshold if applicable.
A.1.2 NEURAL NETWORK AND TRAINING SAPEE-PARAMMTERS
The model that we used was a Resnet- 18 (He et al. 2015). not pretrained. It was trained with an Adam optimizer (Kingma& Ba 2017). learning rate of 0.001 over 20 epochs. A batch size of 256 was used in order to ensure the best performance for the L2 method with noisy labels as well as to reduce the time necessary the training process.
A.2 BIKE SHARING DATASET
A.2.1 DATASET DESCRIPTION
The Bike Sharing Dataset (Fanaee-T & Gama 2013) consists of 17,379 samples of structured data. contains, for nearly each hour of the years 201 and 2012, the date, season, year, month, hour, day of the week, a boolean for it being a holiday, a boolean for being a working day, the weather situation on a scale of 4 (I: clear and beautiful, 4: stormy or snowy), the temperature, the feeling temperature, the humidity, and the wind speed, in the city of Washington DC. It also contains the number of casual, registered, and total bike renters for each hour as recorded on the Capital Bikeshare system.
We use it in regression setting: the network must predict the total number of bike renters given the time and weather information. Unless described otherwise, 7,000 samples were used for training, and 3,379 for testing. We used less samples than available for training because the low-data situation, noise has a stronger effect on the performance. The minimal test loss achieved with 7000 noiseless samples was very close to the one with 14000 samples, hinting that the additional samples did give lot of additional information.
We applied some pre-pro pre-processing the data to make it easier for the network to learn. First, the date was normalized from scale between day 1 to day 730 to scale between O and 4x. Then, provided the network with the cosine and the sine of this number. This allowed to have the same representation for the same days of the year, while having the same distance between any two consecutive days, keeping the cyclic nature of year. A similar idea applied to hours, normalized from 0 to 2n instead day of the week, being category, the season and the month as it was redundant information with the date.
to 24, and with the cosine and sine given to the network. The given one-hot vector of dimension 7. We also removed
Overall, the number of features was 19:
Year
2-4 Date (sine and cos)
13

---

Under review as a conference paper at ICLR 2021
4-5 Hour (sine and cos)
6 to 12 Days of the week (one-hot vector)
13 Holiday boolean
14 Working day boolean
15 Weather situation
16 Temperature
17 Felt temperature
18 Humidity
19 Wind speed
We observed that the network was learning significantly faster and better provided with this format for the data.
Both the features and the labels were normalized before the training, that their mean is 0 and standard deviation is over the whole dataset. The noise variances were correspondingly scaled, as well as the cutoff threshold if applicable.
A.2.2 NEURAL NETWORK AND TRAINING HYPER-PAR SEPER-PARAMEE PARAMETERS
The model that we used was a multi-layer perceptron with 4 hidden layers, the first one with 100 neurons, then 50, 20, and 10. The activation function ReLU. We did use any additional technique such as batch normalization as it did not improve the performances.
The model was trained 100 epochs mini-batches of size 256 for similar reasons than plained in section A.1.2 using the Adam optimizer with learning rate 0.001.
B APPENDIX - ADDITIONAL EXPERIMENTS
B.1 THE INFLUENCE OF
In this section, provide experimental results justifying recommendation of the range of 2002210011 for e, In the experiments presented in this article, we have used = 10-1 must be chosen as part of trade-off between mitigating the effect of BIV with near ground-truth la- bels while keeping to remember that is added to the variance that is used in the loss function. When the labels are normalized which the case in our work -, the noise and its variance each label is normalized too. The value we recommend for should therefore be valid for any normalized set of labels.
effect with noisy labels. To better understand these results, is important
B.1.1 C FOR BIV WITH NEAR GROUND-TRUTH LABELS
One of the main problems BIV induces is in the presence of ground-truth or near ground-truth (NGT) labels. In this case, a near-zero variance can induce very strong weight, effectively reduce the mini-batch to this single sample, and thus ignore the other potentially valid samples.
In this case, hyperparameter is key, it allows effect on the NGT labels than on noisier ones. We tested on the UTKF dataset with a uniform distribution of variance from but, by allowing NGT labels, can already showcase the influence of e on the BIV performance. The resulting graph can be seen figure
maximal weight and has stronger relative
b 1, which is effectively very little noise for the L2 loss
clear that a very small 6, such as 10 or 10-5 leads to a significant loss of performance. good the L.2 loss. Note that it is not surprising when
However, 10-2, the performance e is high such as 102 as this case the weights are very close to similar and BIV is effectively the same as 1.2.
14

---

Under review as a conference paper at ICLR 2021
Test on true labrels
(averaged on 3 runs)
10
<- 10"
< 10
c=1
10
10
10
10
10
0.8
0.6
Training steps (batch size: 256)
Figure 4: Impact of when training with NGT labels using BIV loss on UTKF dataset. The variance was sampled through a uniform distribution bound between 0 and 1. Very low shows drastic loss of performance compared L2 loss, as NGT labels monopolize the weights in the mini-batch.
B.1.2 A HIGH REDUCES THE ADVANTAGES OF BIV ON NOISY LABELS
As discussed in the previous section, higher value of makes the weights more similar for each of samples, and therefore reduces the effect of BIV. We tested BIV with different values of c with binary distribution with 0.5 and μP 2000 on UTKF. In this setup, BIV should have enough data to train correctly when filtering out labels the noisy regime, as shown section 6.2 The results are shown in figure
3VV - 10
1.2
10
Test lss on true labels
laveraged on 3 runss
BIV 10
3VV 10"
alv--
10
No
0.6
0.4
200
400
600
800
1000
1200
Training steps (batch size: 256)
Figure 5: Impact of E when training with highly noisy labels using BIV loss on UTKF dataset. The variance was sampled through a binary uniform distribution with = 0.5, μP = 2000, and V/, = 0. Very high shows loss of performance BIV approaches the L2 results.
As expected, when is very high, the results are very similar to L2. The first effects of BIV can be seen when 10, but until 1 it still optimal. From 0.1, the algorithm is correctly filtering the data.
Considering the results from sections and B.1.2 and 10-1 to optimize the balance between regulating the importance of near ground-truth labels and benefiting from the effect of BIV.
we consider that should be between 10-2
B.2 BIV ON DIFFERENT DISTRIBUTIONS
B.2.1 UNIFORM DISTRIBUTIONS
We present in figure the results of the experiment with uniform distributions in more details.
As explained in section 6.3 we observe that BIV and L2 have the performances when V (and a = = μp)). This is to be expected, as all samples have the exact same noise variance and thus the same weights. When V Vmar (a and b 2,r)), BIV has an advantage, differentiate the samples and use the support of low-noise labels. When V = Vimaz /2 (a = 0.293 and b = 1.707 the difference between the samples is less important, and BIV only does bit
it is able
15

---

Under review as a conference paper at ICLR 2021
BikeSharing
UTKF
0.2
(averaged on 3 runs?
V=0
D BIV
0.15
V= y
V= ? DM
V-V max
0.1
V-V max 3IV
0.4
No nose- L.2
0.05
0
500
1000
1500
2000
2500
0
500
1000
Training steps (batch size: 256)
Training steps (batch size: 256)
Figure Test loss for L2 and BIV learning uniform with different variances V.
m
0.2
0.2
max
Test oss on true labels
L2
0.15
Cutoff #p
C R.
loveraged
Cutoff Pp
BN LaLeelss-
0.1
0.1
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training steps (batch size: 256)
Training steps (batch size: 256)
Figure 7: On BikeSharing with μr 20000, using cutoff is not helpful in the uniform setting. This due to the significant loss of information induced by such a strategy.
better than 1.2 BikeSharing. On UTKE, the process has more variability and is difficult to detect this effect.
In all cases, the benefit from using BIV is less important than with Gamma distributions with S 1, where the support on low-noise samples is higher.
In this setting, also show shown in figure thhat cutting off the noisy data is not good strategy, as it always performs worse than L2.
B.3
BIV ON GAMMA DISTRIBUTIONS
B.3.1
as1
As described in section 6.3 the smaller a, the better the performance of BIV and cutoffs. We show in figures { and [ the curves that led to the numbers in Table [ BIV consistently outperforms the other methods. The performance of cutoff methods strongly depends C, and the best value of C is not the same for every distribution P.
B.3.2 <>1
When 1, the highest support of P(o²) shifts towards μę. This makes the samples less distin- guishable for BIV and therefore the benefits of using it are reduced. This is shown Figure I0 UTKF.
B.4 ROBUSTNESS OF BIV
B.4.1 SIZE OF THE MINI- BATCHB
In equation (5).
the normalization constant is computed from the samples in the mini-batch. the distribution the noise variances in mini-batch is representative of the whole training dataset, the relative weight given to each sample with respect to the others the same than if the normalization was made the whole dataset. The larger the mini-hatch, the more representative it is. In
16

---

Under review as a conference paper at ICLR 2021
0.5
0.25
0.2
0.2
C-0g20 Cutof
Test loss on true labels
(averaged on 10 runs)
0-0,4-Co Cutor
0,22 Cutof
0.15
0.15
c Cutoff
2/p Cutalt
Stpp- Cutoff
3VV
GT Labels -2
0.05
0.05
500
1000
2000
1500
2000
Training steps (batch size: 256)
Training steps (batch size: 256)
Figure 8: Test loss on the BikeSharing dataset, with a S
0.5
0.25
1
Test loss on true labels
0.9
0.9
oy
66 10
0.8
0.8
opt0 Catoff
op1 Cutoff
0.7
0.7
op
0.6
0.6
Zep Cutoff
Spy Cutoff
0.5
Labes
200
400
600
800
1000
1200
200
400
600
600
1000
1200
Training ste os (batch size: 256)
Training steps (batch size: 256)
Figure 9: Test loss on the UTKF dataset, with a
Test true labels (averaged on 3 runs)
0.8
BIV
0.4
GT Labels L2
0.2
0
200
400
600
800
1000
1200
Training steps (batch size: 2561
Figure 10: Test loss on the UTKF dataset for L.2 and BIV learning on Gamma function with a 2 1.
17

---

Under review as a conference paper at ICLR 2021
UTKF 2000
UTKF 20000
0.14
lest loss on true labels
(averaged on 10 runs)
0.7
0.13
12 (16)
G7 Labels- L2 (256)
BV (16)
0.6
0.11
(64)
0.5
BMV(64)
0.1
L2 (128)
BMV (128)
0.4
0.09
(258)
BM (256)
0.08
0.3
Samples (steps batch size) 2200 x00
x10
Figure 11: BIV with different batch sizes both UTKF and BikeSharing datasets.
experiments, used size of 256, which is arguably high. We tested algorithm with lower batch sizes, from 16 to 128, to see if it was a critical factor in the performances.
The results presented in figure 11 In UTKF, the batch size does not make any significant difference in the performance with respect to the amount of samples seen, except for a slightly steeper overfitting once the best loss has been achieved. In BikeSharing, smaller batch size makes the training faster with respect to the amount of samples, but with higher minimal loss, for both L2 and BIV. While larger batch size leads to lower loss function, the effect of BIV compared to the corresponding 1.2 curve is not compromised by smaller batch-sizes.
Two main factors may explain this robustness. First, mini-batch of size 16 seems be already representative enough of the whole dataset for the purpose of normalization. Second, the fact that the mini-batches are populated differently at every epoch improves the robustness sample who would have been in a non renereeeeenttiiee batch at one epoch may not be at another epoch. In any case, the size of the mini-bated is not critical factor for BIV.
B.4.2 NOISY VARIANCE ESTIMATION
In many scenarios, the variance a² from which the noise sampled is estimated, or inferred from a proxy, and therefore prone to be noisy itself. We tested the robustness of our method to such variance noise. In this experimental setup, the value given to the BIV algorithm is disturbed by noise 8z2. We modelled this noise on of to be sampled from a normal distribution whose standard deviation is proportional to of with coefficient of variance disturbance Dy:
809 NNO,,
Dividing of by 9 allows to scale D, so that, when D. = 1, d09 < -o? is at 3 standard deviations from the mean of the distribution.
We then compute the noisy variance, which needs to be positive, as 5? o? + 8,p The noise is therefore biased, but when D, < 1. this is negligible as it happens with probability less than 0.15%.
The results presented in figure 12 show that, when D. < 1. BIV is robust to such noise. While higher D, leads to lower performance, the impact small compared to the effect of BIV. However, when D. = 2, which is an arguably high level of noise and leads to bias explained previously, the beneficial effect of BIV significantly affected in BikeSharing, and completely disappears in UTKF.
18

---

Under review as a conference paper at ICLR 2021
UTKF "p 2000
BikeSharing "P 20000
0.7
0.14
Test loss on true labels
(averaged on 10 rums)
0.65
0.13
BI =
0.6
BM-D, =1
0.12
BV-D. 0.5
0.55
0.11
BM-D, D, 0.25
0.5
BM D,
0.45
0.1
L-
0.4
0.09
0.35
0.08
0.3
500
1000
2000
2500
Training sleps (batch size: 256)
Training steps (batch size: 256)
Figure 12: Robustness of BIV to noise in the variance with different disturbance coefficients D...
19


Reviews:
[{"review_id": "hBcSXDthu8R", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a reweighting technique is proposed to suppress the impact of heteroscedastic label noise in regression model training. The objective function of the regression model training process is composed of a weighted combination of instance-wise training loss. The instance-wise weight is determined by the estimated noise variance based on prior information of the label generation process. The weighting formulation is inspired by the best possible estimator of noisy measurements reaching the Cramer-Rao bound. \n\nThe paper is clearly written. It explains well the problem definition and the methodological formulation. However, we think the innovation in this work is limited. The downside of this paper is as follows: \n1. It is a strong and usually impractical assumption to know a priori knowledge of label noise in the regression model training process. Especially, in the proposed method, the estimate of the noise variance needs to be accurate enough, so as to help suppress the noise impact accordingly. It is better to incorporate jointly the learning of noise distribution and the regression/classification \nmodel, so as to optimize the tolerance against the data-dependent noise. \n2. It only considers the noisy learning process for regression. However, in classification scenarios, noise (with respect to labels) is usually presented in the form of label flipping. The proposed reweighing technique is not directly applicable in that case. Please refer to the following work for further reading: \nLearning with Noisy Labels, Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari, NIPS 2013. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HSmOH7DumWn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose to address the heteroscedastic regression problem using deep neural networks. It assumes the variance of heteroscedastic noise is known as privileged information and suggests to reweight the samples by their noise variance in the loss.\n\nThe major issue to me is the lack of novelty. Heteroscedastic regression is a classic problem in statistics. And reweighting using the inverse variance is a textbook method. See Chapter 10 of \n\nhttp://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ADAfaEPoV.pdf\n\nThis paper failed to cite any relevant reference and clarify the novelty.\n\nTo apply the method to a deep learning setting, some interesting problem can be how to estimate the variance with deep network in a reliable way (this was done previously using classic models). However, this paper did not tackle this harder (and more interesting) problem. Instead, it assume the variance is simply given during training. This is not very realistic in real world setting. The experiments are all synthetic and are not particularly convincing. \n\nFinally, the paper claims a lot of connection with privileged information (LUPI). But I found it hard to consider this variance a similar concept as privileged information, which is realistic and interesting.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A4bZxRsMaEP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nPaper Summary:\nThis work targets regression tasks with noisy labels, and proposes to incorporate knowledge about the variance of the gaussian noise corrupting the observed labels to weight the loss function, at training time. The proposed method is evaluated in a series of experiments involving deep networks trained according to the weighted loss function, and compared to a baseline method that omits training samples that have a label noise variance larger than a threshold.\nResults indicate the proposed method is more robust to noisy labels when compared to alternatives that do not exploit the information on the noise affecting labels.\n\n##########################################################################\n\nReasons for score: \nOn the one hand, the paper is extremely well written and somehow pedagogic, in that it provides compelling motivations for considering heteroscedasticity, and its possible sources, and general, intuitive descriptions of the proposed method before specialising them to the instance they evaluate. On the other hand, I think the prose lacks sufficient technical depth, on the model they use, its relation to “text book” material on heteroscedasticity, e.g. for Maximum Likelihood Estimation (MLE), and on the properties of the proposed method. The experimental evaluation, while representing a reasonable starting point, is not sufficient to fully understand the behaviour and the properties of the proposed method.\nFor these reasons, I think this work cannot be accepted as is.\n\n##########################################################################\n\nPositive points: \n\n1) The editorial quality of this paper is high, and the overall motivations given to support the problem statement are compelling and well discussed. This also relates to the fundamental assumption underlying this work: access to privileged information, taking the form of knowledge of the stochastic noise variance affecting observed labels, at training time.\n\n2) The proposed method appears to be well positioned w.r.t. the recent literature on statistical modelling with noisy labels, especially concerning neural network based methods. It is unfortunate though that the literature scan doesn’t cover well-known approaches to tackle heteroscedastic noise in simple linear models, or in general MLE frameworks, which may be considered text-book material.\n\n3) The experimental evaluation considers two regression tasks on their respective UCI/Bike sharing, and UTK Face datasets, considering several variants of noise generation processes, affecting labels in different and sufficiently realistic manner.\n\n \n##########################################################################\n\nNegative points:\n1) My main concern is the “thin” contribution of this paper. The technical details of the proposed method are not sufficiently developed. Drawing inspiration from Fisher information calls for an appropriate discussion on the likelihood model, its noise model, to begin with. Then, I think the relation of the proposed idea to simple linear models, for which heteroscedastic regression has been studied in great detail (e.g. [1], for a general reference), and for MLE (e.g., [2]), would become more clear and would give the opportunity for the authors to develop what are the merits of their proposed method. For example, the weighted least square method is very similar to what is proposed in this paper. \n[1] Econometric analysis, Greene, William H, 2003, Pearson Education India\n[2] Maximum Likelihood Estimators with Heteroscedastic Errors, G. R. Fisher, Review of the International Statistical Institute\nVol. 25,  1957\nSee also “Pattern Recognition and Machine Learning”, Bishop, 2006, (chapter 5 and 6)\n\n2) The experimental evaluation is not sufficient to appreciate the virtues of the proposed method. For several noise distributions (including the additional ones considered in the supplement), the proposed method BIV does not seem to behave much better than the proposed baseline, that uses a simple threshold. Additionally, the figures are  cropped and do not allow to get a sense of what happens for all competing methods (Fig1 and Fig2). In Fig3, the figures report test loss and as there seems to be overfitting kicking in.\nAlso, it is mentioned the baseline method requires to set a cutoff parameter, but the proposed method also depends on a hyper parameter to optimise (done in the appendix). As a consequence, it is difficult to appreciate the main advantage of BIV w.r.t. the baseline.\nFinally, in Fig.3, there are clear signs of overfitting: why did the authors suggest (end of Sec.3) that their work does not require regularisation?\n\n\n#########################################################################\n\nMain criticism:\nI think, overall, the main criticism I have for this work is that the contribution is not sufficient. The main idea proposed in the paper fits sec 4.2, and it is based on well known results from text books. In eq.(5), the summation term is MLE with heteroscedasticity. The loss is scaled by a coefficient that collects statistics on the sample batch: if this batch is very small, or its elements not sufficiently diverse, I am afraid it could have a negative impact on the optimisation process (this is why, in the experiments, the authors chose a batch size of 256).\nOne possible way to overcome this criticism is to clarify the likelihood model, and compare the proposed method to existing approaches to address heteroscedastic Gaussian noise.\nA possible advice would be to reduce (or move to the appendix) the discursive parts on heteroscedasticity, and the general formulations (e.g., sec. 2.2, sec. 4.1), and gain more space to explain how BIV is different from what is known.\n\nAdditional comments:\nA note on experiments using the UTKFace dataset. In this case, the MLP used with 4 layers may be a bit “too simple”, in light of the high test loss on GT labels. Did you try with convolutional architectures, even simple ones?\nJust to clarify, the test loss reported in the figures, as a function of training steps: what is the *test* batch size? Same as training batch size? The test loss is computer according to \\sum_{k \\in \\text{test batch}} \\mathcal{L}(f(\\mathbf{x}_k, \\theta), \\tilde{y}_k) right?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript presents a deep network approach for heteroscedastic regression problem. It assumes the variance of heteroscedastic noise is known as privileged information and suggests to reweight the samples by their noise variance in the loss.

Three reviewers agreed that the manuscript is not ready for publication. The major issue is the lack of novelty. Heteroscedastic regression is a classic problem in statistics. And reweighting using the inverse variance is a textbook method. 

R2 and R4 confirmed that they have read author response. The rebuttals are useful to clarify some points, especially related to experimental settings and results. However, they are not convinced by the authors' argument on novelty and whether the assumption is realistic. 

Decision: Reject