Deep Classifiers with Label Noise Modeling and Distance Awareness

1 INTRODUCTION . While deep learning has led to impressive advances in predictive accuracy , models often still suffer from overconfidence and ill-calibrated uncertainties ( Ovadia et al. , 2019 ) . This is particularly problematic in safety-critical applications ( e.g. , healthcare , autonomous driving ) , where uncertainty estimation is crucial to ensure reliability and robustness ( Filos et al. , 2019 ; Dusenberry et al. , 2020 ) . Predictive uncertainties generally come in two flavors : model uncertainty ( also known as epistemic ) and data uncertainty ( also known as aleatoric ) ( Murphy , 2012 ) . Model uncertainty measures how confident the model should be based on what it knows about the world , that is , how much it can know about certain test data given the training data it has seen . Data uncertainty measures the uncertainty intrinsic in the data itself , for example due to fundamental noise in the labelling process . Good model uncertainty estimates allow for out-of-distribution ( OOD ) detection , that is , for recognizing data examples that are substantially different from the training data . On the other hand , good data uncertainty estimates allow for in-distribution calibration , that is , knowing which training ( or testing ) data examples the model should be more or less confident about . Many recently proposed models for uncertainty estimation excel at one or the other of these uncertainty types . For instance , the spectral-normalized Gaussian process ( SNGP ) ( Liu et al. , 2020 ) uses a latent Gaussian process to achieve distance-aware model uncertainties and thus affords excellent OOD detection . Conversely , the heteroscedastic classification method ( Collier et al. , 2020 ; 2021 ) offers excellent calibration and improved accuracy thanks to modeling input- and class-dependent label noise in the training data . However , there have not been many attempts to combine the complementary benefits of these two types of uncertainty modeling ( e.g. , Depeweg et al. , 2018 ) ( see related work in Appendix A.1 ) . In this work , we propose the heteroscedastic SNGP ( HetSNGP ) model , which allows for joint modeling of model and data uncertainties based on a hierarchy of two latent random variables . We show that HetSNGP gives good in-distribution and OOD accuracy and calibration , yielding a model with uncertainties suitable for deployment in critical applications . Our main contributions are : • We propose a new model , the heteroscedastic spectral-normalized Gaussian process ( HetSNGP ) , which provides both distance-aware model and data uncertainties . • We describe an efficient approximate inference scheme that allows training HetSNGP with a computational budget that is comparable to standard neural network training . • We show empirically on different benchmark datasets that HetSNGP offers a favorable combination of model and data uncertainty . It generally preserves the SNGP ’ s OOD performance and the heteroscedastic in-distribution performance and even outperforms these baselines on some datasets , where OOD and heteroscedastic uncertainties are helpful . • We propose an ensembled version of our model , the HetSNGP Ensemble , which additionally accounts for parameter uncertainty and outperforms other ensemble baselines . 2 BACKGROUND . 2.1 MODEL UNCERTAINTY AND SPECTRAL-NORMALIZED GAUSSIAN PROCESS . Model uncertainty ( or epistemic uncertainty ) captures all the uncertainty about whether a given model is correctly specified for a certain task , given the training data . Before any training data has been observed , this uncertainty depends only on the prior knowledge about the task , which can for instance be encoded into distributions about the model parameters or into the architecture of the model ( i.e. , the model class ) ( Fortuin , 2021 ) . After training data has been observed , one should expect the model uncertainty to decrease within the support of the training data distribution , that is , on points that are close to the training data in the input space . We will generally call these points in-distribution ( ID ) . Conversely , on points that are far away from the training points and thus out-ofdistribution ( OOD ) , we should not expect the model uncertainty to decrease , since the training data points are not informative enough to make any assertions about the correctness of the given model on these points . In this sense , the optimal model uncertainty should be distance-aware , that is , it should grow away from the training data ( Liu et al. , 2020 ) . The spectral-normalized Gaussian process ( SNGP ) ( Liu et al. , 2020 ) provides such distance-aware model uncertainties by specifying a Gaussian process prior ( Rasmussen & Williams , 2006 ) over the latent data representations in the penultimate layer of the neural network . Distance-awareness is ensured by using spectral normalization on the hidden layers ( Behrmann et al. , 2019 ) , which encourages bi-Lipschitzness of the mapping from data to latent space , constraining the model from collapsing any input feature dimensions in the latent representations and approximately preserving the distances between data points in the latent space ( Liu et al. , 2020 ) . Note that this approach only partially captures the model uncertainty , namely in form of the uncertainty over the latents . It does not however capture the uncertainty over the model parameters , such as for instance Bayesian neural networks ( MacKay , 1992 ; Neal , 1993 ) or ensemble methods ( Lakshminarayanan et al. , 2016 ) . 2.2 DATA UNCERTAINTY AND THE HETEROSCEDASTIC METHOD . As opposed to the model uncertainty described above , data uncertainty is intrinsic in the data and thus irreducible with increasing amounts of data . In the case of continuous data ( e.g. , regression problems ) , data uncertainty often comes in the form of random noise on the measurements . For discrete data ( e.g. , classification problems ) , it usually comes as label noise , that is , a certain number of data points will be assigned the wrong label in the training data . This label noise can be class- and input-dependent ( Beyer et al. , 2020 ) . For instance , the Imagenet dataset ( Deng et al. , 2009 ) , contains 100 different classes of dog breeds , which are often harder to tell apart for human labelers than the other classes in the dataset . Modeling this type of data uncertainty can improve the calibration and robustness of predictive models ( Collier et al. , 2020 ) . A model that does explicitly handle the input- and class-dependent label noise is the heteroscedastic method ( Collier et al. , 2021 ) . The heteroscedastic method models input- and class-dependent noise by introducing a latent multivariate Gaussian distribution on the softmax logits of a standard neural network classifier . The covariance matrix of this latent distribution is a function of the input ( heteroscedastic ) and models inter-class correlations in the logit noise . 3 METHOD . 3.1 SETUP AND NOTATION . Let us consider a dataset D = { ( xi , yi ) } Ni=1 of input-output pairs , where xi ∈ Rd and yi ∈ { 1 , . . . , K } , that is , a classification problem with K classes . The data examples are assumed to be sampled i.i.d . from some true data-generating distribution as ( xi , yi ) ∼ p∗ ( x , y ) . 3.2 GENERATIVE PROCESS . To jointly model the two different types of uncertainty , we propose a hierarchical model of two latent random variables , which we denote by f and u. f is a latent function value associated to the input x ( as in the Gaussian process literature ) and is designed to capture the model uncertainty , while u is a latent vector of logits ( or utilities ) that capture the data uncertainty , similar to the setup in Collier et al . ( 2020 ) , which was inspired by the econometrics literature ( Train , 2009 ) . Similarly to Liu et al . ( 2020 ) , we place a latent Gaussian process ( GP ) prior over f , as p ( f ) = GP ( 0 , kθ ( · , · ) ) , where kθ is a parameterized kernel function with parameters θ . Evaluating this kernel on all pairwise combinations of datapoints in x yields the kernel matrix Kθ ( x , x ) ) . We then define u as a ( heteroscedastically ) noisy observation of f . Stacking the variables across the whole dataset gives us the matrices F , U ∈ RN×K . We refer to their respective rows as fi , ui ∈ RK and their columns as fc , uc ∈ RN . The columns are independent under the GP prior , but the rows are not . Conversely , the columns are correlated in the heteroscedastic noise model , while the rows are independent . A hierarchical model using both uncertainties therefore leads to logits that are correlated across both data points and classes . The full generative process is fc ∼ N ( 0 , Kθ ( x , x ) ) ( 1 ) ui ∼ N ( fi , Σ ( xi ; ϕ ) ) ( 2 ) p ( yi = c |ui ) = 1 [ c = arg max k uik ] ( 3 ) The predictive can then be computed my marginalizing , that is p ( y |x ) = Eu [ p ( y |u ) ] =∫ p ( y |u ) p ( u |x ) du . Intuitively , f captures the model uncertainty , that is , the uncertainty about the functional mapping between x and y on the level of the latents . It uses the covariance between data points to achieve this distance-awareness , namely it uses the kernel function to assess the similarity between data points , yielding an uncertainty estimate that grows away from the data . On the other hand , u captures the data uncertainty , by explicitly modelling the per-class uncertainty on the level of the logits . ϕ can learn to encode correlations in the noise between different classes ( e.g. , the dog breeds in Imagenet ) . It does not itself capture the model uncertainty , but inherits it through its hierarchical dependence on f , such that the resulting probability p ( y |u ) ultimately jointly captures both types of uncertainty . In practice , we usually learn the kernel using deep kernel learning ( Wilson et al. , 2016 ) , that is , we define it as the RBF kernel kθ ( xi , xj ) = kRBF ( hi , hj ) = exp ( ‖hi − hj‖22/λ ) with length scale λ and hi = h ( xi ; θ ) ∈ Rm with h ( · ; θ ) being a neural network model ( e.g. , a ResNet ) parameterized by θ . This kernel is then shared between classes . Moreover , following Liu et al . ( 2020 ) , we typically encourage bi-Lipschitzness of h using spectral normalization ( Behrmann et al. , 2019 ) , which then leads to an approximate preservation of distances between the input space and latent space , thus allowing for distance-aware uncertainty modeling in the latent space . Additionally , Σ ( · ; ϕ ) is usually also a neural network parameterized by ϕ . To ease notation , we will typically drop these parameters in the following . Note that the prior over fc ( and thus also uc ) has zero mean , thus leading to a uniform output distribution away from the data . Note that this is also reminiscent of multi-task GPs ( Williams et al. , 2007 ) , where separate kernels are used to model covariances between data points and tasks and are then combined into a Kronecker structure .

Summary: This paper studies the combination between model uncertainty and data uncertainty based on spectral-normalized Gaussian process. Theoretical results show that heteroscedastic SNGP allows for joint modeling of model and data uncertainties. The paper then proposes an approximate inference scheme for efficient model training. Experiment results support these claims on synthetic datasets and typical classification benchmarks.