Bayesian Metric Learning for Robust Training of Deep Models under Noisy Labels

1 INTRODUCTION . Deep learning has been shown as a dominant learning framework in various domains of machine learning and computer vision . One of the major limitations of deep learning is that it often requires relatively clean data sets that do not contain label noise naturally caused by human labeling errors , measurement errors , subjective biases and other issues ( Frénay et al. , 2014 ; Ghosh et al. , 2017 ; Algan & Ulusoy , 2019 ) . The performance of a machine learning method can be significantly affected by noisy labels both in terms of the reduction in the accuracy rate and the increase in sample complexity . Particularly for deep learning , a deep neural network ( DNN ) can generalize poorly when trained with noisy training sets which contain high proportion of noisy labels since a DNN can over-fit those noisy training data sets ( Zhang et al. , 2016 ; Algan & Ulusoy , 2020 ) . Developing deep learning methods that can perform well on noisy training data is essential since it can enable the use of deep models in many real-life applications . There have been several approaches proposed to handle learning issues caused by label noise , for example : data cleaning ( Angelova et al. , 2005 ; Chu et al. , 2016 ) , label correction ( Reed et al. , 2014 ) , additional linear correction layers ( Sukhbaatar et al. , 2014 ) , dimensionality-driven learning ( Ma et al. , 2018 ) , bootstrapping ( Reed et al. , 2014 ) , curriculum learning-model based approach such as MentorNet ( Jiang et al. , 2018 ) or CoTeaching ( Han et al. , 2018 ) , loss correction ( or noisetolerant loss ) ( Masnadi-Shirazi & Vasconcelos , 2009 ; Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Thulasidasan et al. , 2019 ; Ma et al. , 2020 ) , or a combination of the techniques above ( Li et al. , 2020 ; Nguyen et al. , 2019 ) . Relevant to this paper is an existing theoretically sound approach : Bayesian large margin nearest neighbor classification ( BLMNN ) ( Wang & Tan , 2018 ) that employs Bayesian inference to improve the robustness of a point estimation-based linear metric learning method . BLMNN then introduces a method to approximate the posterior distribution of the underlying distance parameter given the triplet data by using the stochastic variational inference . More importantly , BLMNN ( Wang & Tan , 2018 ) also provides a theoretical guarantee about the robustness of the method , which says that it can work with non-uniform label noise . Although BLMNN has been mathematically shown to be robust against label noise , it only focuses on a simple linear Mahalanobis distance that can not capture the nonlinear relationships of data points in deep metric learning ( Lu et al. , 2017 ) . In this paper , we introduce a Bayesian deep metric learning framework that is robust against noisy labels . Our proposed method ( depicted in Fig . 1 ) is inspired by the BLMNN ( Wang & Tan , 2018 ) , deep metric learning ( Hoffer & Ailon , 2015 ; Hu et al. , 2015 ; Wang et al. , 2017 ; Lu et al. , 2017 ; Do et al. , 2019 ) , and Bayes by Backprop ( Blundell et al. , 2015 ) . Compared to the BLMNN that only considers a linear metric learning , our framework can handle non-linear deep metric learning , which is useful for many real-life applications . Moreover , directly applying the variational Bayes learning ( Wang & Tan , 2018 ) in deep learning is challenging since it requires sampling from a distribution of the neural network parameters . Instead , we adapt the variational inference by Blundell et al . ( Blundell et al. , 2015 ) , which allows to efficiently sample the parameters of a Bayes neural networks by using a backpropagation-compatible algorithm . We also theoretically show the robustness of our proposed method when working with label noise . The experimental results on several noisy data sets show that our novel proposed method can generalize better compared to the linear BLMNN ( Wang & Tan , 2018 ) and the point estimation-based deep metric learning ( Hoffer & Ailon , 2015 ; Lu et al. , 2017 ) , especially when the noise level increases . It is important to emphasize that the motivation of our method is to produce a better calibrated model that is more robust to noisy label training , and , as a result , less likely to overfit the training set than the linear BLMNN ( Wang & Tan , 2018 ) and the point estimation-based deep metric learning ( Hoffer & Ailon , 2015 ; Lu et al. , 2017 ) . Therefore this is a paper that introduces a new theoretical framework to solve noisy label learning instead of presenting method that is competitive against the best approaches of the field ( such as Mentornet ( Jiang et al. , 2018 ) or Co-teaching ( Han et al. , 2018 ) ) in large-scale datasets ( e.g. , webvision ( Li et al. , 2017 ) and Clothing 1M ( Xiao et al. , 2015 ) ) . Furthermore , deep metric learning has been in fact considered in the Bayesian settings before ( Ishfaq et al. , 2018 ; Karaletsos et al. , 2015 ) , and recently , in ( Lin et al. , 2018 ) , but not in the context of noisy labels . Consequently , our proposed framework can be used by other methods that can deal with noisy label learning , but the extension of those methods using our proposed approach is out of the scope of this paper . 2 RELATED WORK . 2.1 POINT ESTIMATION-BASED DISTANCE METRIC LEARNING . The goal of distance metric learning ( or metric learning ) is to learn a distance function to measure the similarity between training samples . Metric learning has been shown to have great success in many visual applications such as face recognition , image classification , visual search , visual tracking , and person re-identification ( Lu et al. , 2017 ) . In principle , a supervised metric learning method aims to learn a distance metric which pulls together samples from the same class while pushing away those from different classes . Based on the complexity of the distance , metric learning can be classified into two types : linear , focusing on linear distance ( e.g. , Mahalanobis ) , which often suffers from the nonlinear relationship of data points ( Lu et al. , 2017 ) ; and non-linear , which nowadays is mostly based on deep learning . Deep metric learning ( DML ) is motivated by the fact that deep learning is an effective solution to a non-linear transformation of input samples ( Lu et al. , 2017 ; Do et al. , 2019 ) . The key idea of DML is to explicitly learn a set of hierarchical non-linear transformations to map input data points into a feature space that is used for comparing or matching these data points in a more effective manner . DML unifies feature learning and metric learning into a joint learning framework . DML is shown to be more advantageous compared to traditional models , for example , with respect to classification performance since DML does not rely on the classification layer of a trained model that often depends on the type of problems ( Lu et al. , 2017 ; Do et al. , 2019 ) . Relevant to this paper is the triplet loss-based metric learning ( Lu et al. , 2017 ; Schroff et al. , 2015 ) , in which the original training data set is represented by a set of independent triplets , formed by an anchor sample , one sample of the same class and another sample from a different class . The training process is then performed by minimizing the triplet loss to simultaneously approximate the anchor to the sample with the same class and separate the anchor from the sample of different class . 2.2 ROBUST LINEAR DISTANCE METRIC LEARNING VIA BAYESIAN INFERENCE . One drawback of the point estimation DML is that it is likely to over-fit the noisy labels ( Wang & Tan , 2014 ; 2018 ) . Wang et al . ( 2016 ) introduces the Deep Stochastic Neighbor Compression ( DSNC ) method that aims to jointly learn a nonlinear transformation that preserves the neighborhood of the data , and a compressed version of the training data set ( Ahmed et al. ) . DSNC is also robust against label noise . Motivated by the fact that Bayesian learning is a good choice for robust learning ( Zhu et al. , 2014 ; Yang et al. , 2012 ) , Wang & Tan ( 2018 ) introduced the theoretically sound Bayesian large margin nearest neighbor classification ( BLMNN ) to improve the robustness of the linear metric learning under the presence of label noise . The BLMNN framework represents the large margin nearest neighbor classification using a linear metric learning in the form of a variational Bayes method that takes the prior distribution of the transformation matrix into account and estimate the posterior distribution via stochastic variational inference ( SVI ) . BLMNN provides mathematical definitions of the noisy label triplet ( a type of non-uniform label noise ) and the β-robust algorithm against noisy label triplets . More importantly , a theoretically guarantee of the robustness of BLMNN is also provided in ( Wang & Tan , 2018 ) . Although BLMNN efficiently addresses the training issue caused by the noisy label , one key limitation of BLMNN is that it is based on a linear metric transformation that can not capture the nonlinear relationships of data points in deep metric learning . However , the extension of BLMNN to the non-linear case is not straightforward due to the complexity of the posterior distribution estimation as well as the sampling process of high-dimensional parameters , and such extension is the main target of our paper . 2.3 BAYES BY BACKPROP . Bayesian neural networks ( MacKay , 1995 ; Neal , 2012 ; Gal & Ghahramani , 2015 ) aim to estimate the posterior distribution of the network parameters given the training data . However , that inference framework is often intractable , especially when working with high dimensional parameters ( Blundell et al. , 2015 ) . Moreover , exactly calculating the posterior of the weights is challenging since it requires the integration that is known to be computationally expensive . Blundell et al . ( Blundell et al. , 2015 ) introduced the “ backpropagation-compatible ” Bayes by Backprop method for estimating the posterior distribution of the network parameter . That method is inspired by the variational free energy inference in which the exact posterior is approximated by a variational distribution by solving an optimization problem . In principle , Bayes by Backprop can directly work on the network parameter by minimizing a compression cost function ( Blundell et al. , 2015 ) . Our proposed method is inspired by ( Wang & Tan , 2018 ) , where we replace the linear Mahalanobis metric by a non-linear deep metric . Our novel proposed method therefore leverages the good performance of the deep metric learning ( Lu et al. , 2017 ) and the robustness to label noise of a Bayesian framework ( Yang et al. , 2012 ; Wang & Tan , 2018 ) . We first represent a triplet-based deep metric learning using Bayesian inference . By replacing the softmax loss by a triplet loss ( and so learning a metric ) , we impose more strict constraints , where points of the same class are forced to collapse into a small region of the feature space instead of just belonging to regions within the class boundaries . We argue that this more strict constraints has the potential to introduce more robust feature spaces for classification under noisy label training . To approximate the posterior of the network parameters , we then employ the efficient variational framework ( Blundell et al. , 2015 ) that allows to sample from a Bayesian neural network using the backpropagation framework . We theoretically show that our proposed method is robust against noisy triplet relying on a Bayesian inference framework .

Summary: This paper proposes a robust Bayesian deep metric learning framework against noise label inspired the BLMNN (Wang & Tan, 2018), deep metric learning (Hoffer & Ailon, 2015; Hu et al., 2015; Wang et al., 2017; Lu et al., 2017; Do et al., 2019), and Bayes by Backprop (Blundell et al., 2015). Directly applying the variational Bayes learning (Wang & Tan, 2018) in deep learning is challenging since it requires sampling from a distribution of the neural network parameters. Instead, this paper adapts the variational inference by Blundell et al. (Blundell et al., 2015), which allows to efficiently sample the parameters of a Bayes neural networks by using a backpropagation-compatible algorithm. The experimental results on several noisy data sets show that our novel proposed method can generalize better compared to the linear BLMNN (Wang & Tan, 2018) and the point estimation-based deep metric learning (Hoffer & Ailon,