Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression

1 INTRODUCTION . In supervised learning , a central assumption is that the samples in the training dataset , used to train the model , and the samples in the testing set , used to evaluate the model , are sampled from identical distributions . Formally , for input x and label y , this assumption implies that ptrain ( x , y ) = ptest ( x , y ) . This assumption can be decomposed as the product ptrain ( x ) · ptrain ( y|x ) = ptest ( x ) · ptest ( y|x ) , which is true if two conditions are respected : 1 . The features in both datasets are sampled from the same distribution : ptrain ( x ) = ptest ( x ) . When this is condition is violated , the training dataset is not representative . 2 . The labels in both datasets are sampled from the same conditional distribution : ptrain ( y|x ) = ptest ( y|x ) . If this condition is violated , the training labels are noisy . In practice , these assumptions are not always respected because gathering representative and precise data ( including labels ) can be arduous . In this case , the training and testing datasets are misaligned , and the performance of the deployed model may decrease since the training process did not actually optimize the model ’ s parameters based on the correct data ( Arpit et al. , 2017 ; Kawaguchi et al. , 2020 ) . One possible reason for misalignment is that there is some uncertainty about the labels in the training set as a result of the labeling process . Since our objective is to optimize the performance of the model compared to ground truth labels , we should consider that the labels in test dataset have no uncertainty , even though it may be impossible to collect such a dataset in practice . As a result , ptest ( y|x ) is sampled from a Dirac delta function , whereas ptrain ( y|x ) is not since it encapsulates the uncertainty in the labelling process , which leads to misalignment . In this paper , we propose an algorithm for more efficient model training in the case where we have some information about the sample-wise misalignment . More specifically , we examine the case of regression with a deep network where labels are corrupted by heteroscedastic noise . We assume that we have access at least an estimate of the variance of the distribution of the noise that corrupted each label , information that is available if the labels are being generated by some stochastic process that is capable of also jointly reporting uncertainty . We examine how the knowledge of the estimate of the label noise variance can be used to mitigate the effect of the noise on the learning process of a deep neural network . We refer to our method as Batch Inverse-Variance ( BIV ) , which , inspired by information theory , performs a re-weighting using both the the sample-wise variance but also statistics over the entire mini-batch . BIV shows a strong empirical advantage over L2 loss as well as over a simple filtering of the samples based on a threshold over the variance.1 Our claimed contributions are threefold : 1 . A definition of the problem of learning with information quantifying the misalignment between datasets for the case of heteroscedastic noisy labels in regression . 2 . A general formulation of how to use the mini-batch to infer statistics of the dataset and incorporate this information in the loss function when training on neural networks . 3 . We present Batch Inverse-Variance as an instantiation of this framework and show its usefulness when applied to regression tasks with labels corrupted by heteroscedastic noise . The outline of the paper is as follows : In section 2 , we describe the task of regression with heteroscedastic noisy labels and its parallels with learning with privileged information , and we explain the challenges of applying classical heteroscedastic regression methods to stochastic gradient descent . In section 3 , we position our work among the existing literature on learning with noisy labels . In section 4 , we present a general framework to incorporate information regarding dataset misalignment in the mini-batch loss . We introduce BIV within this framework to tackle heteroscedastic regression . In section 5 , we describe the setup for the experiments we made to validate the benefits of using BIV , and we present and analyze the results in section 6 . 2 BACKGROUND . 2.1 HETEROSCEDASTIC NOISY LABELS IN REGRESSION . Here , we introduce how heteroscedastic noisy labels can be generated in regression and how the variance can be known . Consider an unlabelled dataset of inputs { xi } . To label it , one must apply to each input xi an instance of a label generator which should provide its true label yi . This label generator has access to some features zi correlated to xi . We define LGj : Z −→ R . When the labelling process is not exact and causes some noise on the label , the noisy label of xi provided by LGj is defined as ỹi , j . Noise on a measured or estimated value is often represented by a Gaussian distribution , based on the central limit theorem , as most noisy processes are the sum of several independent variables . Gaussian distributions are also mathematically practical , although they present some drawbacks as they can only represent unimodal and symmetric noise ( Thrun et al. , 2006 ) . We model : ỹi , j = yi + δyi , j with δyi , j ∼ N ( 0 , σ2i , j ) ( 1 ) σ2i , j can be a function of zi and LGj , without any assumption on its dependence on one or the other . We finally assume that the label generator is able to provide an estimate of σ2i , j , therefore being redefined as LGj : Z −→ R× R≥0 . The training dataset is formed of triplets ( xi , σ2i , j , ỹi , j ) , renamed ( xk , σ 2 k , ỹk ) for triplet k for simplicity . This setup describes many labelling processes , such as : Crowd-sourced labelling : In the example case of age estimation from facial pictures , labellers Alice and Bob are given zi = xi the picture of someone ’ s face and are asked to estimate the age of that person . Age is harder to estimate for older people come ( 5 and 15 years of age are harder to confuse than 75 and 85 ) suggesting a correlation between σ2i , j and zi . But Alice and Bob may also have been given different instructions regarding the precision needed , inducing a correlation between σ2i , j and LGj . Finally , there may be some additional interactions between zi and LGj , as for example Alice may know Charlie , recognize him on the picture and label his age with lower 1Our code is available in supplemental material and will be publicly released after the reviewing process . uncertainty . Both labellers can provide an estimation of the uncertainty around their labels , for example with a plus-minus range which can be used as a proxy for standard deviation . Labelling from sensor readings , population studies , or simulations : Imagine you want build a dataset of pictures xi from a camera on the ground labelled with the position yi of a drone in the sky . To estimate the position of the drone at the moment the picture was taken , you could use state estimation algorithms based on the Bayes ’ filter ( Thrun et al. , 2006 ) . These algorithms take as an input zi the measurements of the drone ’ s sensors , and provide a full posterior distribution over the state , sometimes under a Gaussian assumption for Kalman filters for example . The uncertainty depends , among others , on the precision of the sensors , the observability of a given state , the precision of the dynamic model , and the time since sensor signals were received . Similarly , studies based on population such as polling or pharmaceutical trials have quantified uncertainties based on the quantity and quality of their samples . It is also possible to train on simulators , as in climate sciences ( Rasp et al. , 2018 ) or in epidemiology ( Alsdurf et al. , 2020 ) , and some of them provide their estimations ’ uncertainty based on the simulation procedure and the inclusion of real measurements in the model . Using predictions from a neural network in complex neural architectures : In deep reinforcement learning for example , the critic network learns to predict a value from a state-action pair under the supervision of the heteroscedastic noisy output of a target network plus the reward ( Mnih et al. , 2015 ; Haarnoja et al. , 2018 ) . While the estimation of the uncertainty of the output of a neural network is not an easy task , it is an active field of research ( Gal & Ghahramani , 2016 ; Peretroukhin et al. , 2019 ) . There , zi is the state-action pair at the next step , and LGj the target network being updated over time . The prediction is a mix of aleatoric and epistemic uncertainties as defined by Kendall & Gal ( 2017 ) which are dependent on both zi and LGj . We could not find any current dataset that provides such label uncertainty information for regression . However , as it is precious information , we argue that it should actually be provided when possible . In classification , Xie et al . ( 2016 ; 2020 ) took a step in this direction by providing a “ confidence ” score from 0 to 255 for each pixel in the KITTI-360 dataset . 2.2 LEARNING USING PRIVILEGED INFORMATION . Training with a dataset of triplets ( xi , x∗i , yi ) , where x ∗ i is only given at training time and not available at test time , fits in the framework of learning using privileged information ( LUPI ) , defined in Vapnik & Vashist ( 2009 ) and mainly applied to SVMs . In most works in this field , this privileged information makes the task easier on a sample-to-sample basis . For example , object detection can be improved by adding segmentation masks ( Feyereisl et al. , 2014 ) or depth images ( Hoffman et al. , 2016 ) . Another interpretation of LUPI is to use privileged information as a vector for knowledge transfer between a teacher and a student ( Vapnik & Izmailov , 2015 ) . Hernández-Lobato et al . ( 2014 ) and Lambert et al . ( 2018 ) have made a link between privileged information and uncertainty , using it to evaluate the confidence of the model for a training sample . The former applied the approach to Gaussian processes through latent noise , and the latter to neural networks through Gaussian dropout . More formally , at training time the neural network has access to triplets ( xk , x∗k , yk ) where xk is the input , yk its corresponding label , and x∗k the additional information with respect to this sample . The objective in LUPI is the same as classical supervised learning : train the network parameters θ so that , at test time , and without access to information x∗i , the expected loss is minimized , i.e . : θopt = argmin θ E { xi , yi } ∈Dtest [ L ( f ( xi , θ ) , yi ) ) ] ( 2 ) where L ( f ( xi , θ ) , yi ) is the objective loss function based on the true label and on the network ’ s prediction f ( xi , θ ) , for example the L2 distance in the task of regression . In our work , we have x∗i = σ 2 i . In contrast with the usual LUPI setting , x ∗ i does not help the task on a sample-to-sample basis , but instead informs about the role of each sample on the misalignment between the datasets due to the noise in the labelling process . The objective , however , is the same : use this privileged information during training to minimize the expected loss at test time .

Summary: In this paper, a reweighting technique is proposed to suppress the impact of heteroscedastic label noise in regression model training. The objective function of the regression model training process is composed of a weighted combination of instance-wise training loss. The instance-wise weight is determined by the estimated noise variance based on prior information of the label generation process. The weighting formulation is inspired by the best possible estimator of noisy measurements reaching the Cramer-Rao bound.