ROBUST DISCRIMINATIVE REPRESENTATION LEARNING VIA GRADIENT RESCALING: AN EMPHASIS REGULARISATION PERSPECTIVE

1 INTRODUCTION . DNNs have been successfully applied in diverse applications ( Socher et al. , 2011 ; Krizhevsky et al. , 2012 ; LeCun et al. , 2015 ) . However , their success is heavily reliant on the quality of training data , especially accurate semantic labels for learning supervision . Unfortunately , on the one hand , maintaining the quality of semantic labels as the scale of training data increases is expensive and almost impossible when the scale becomes excessively large . On the other hand , it has been demonstrated that DNNs are capable of memorising the whole training data even when all training labels are random ( Zhang et al. , 2017 ) . Therefore , DNNs struggle to discern meaningful data patterns and ignore semantically abnormal examples1 simultaneously ( Krueger et al. , 2017 ; Arpit et al. , 2017 ) . Consequently , it becomes an inevitable demand for DNNs to hold robustness when training data contains anomalies ( Larsen et al. , 1998 ; Natarajan et al. , 2013 ; Sukhbaatar & Fergus , 2014 ; Xiao et al. , 2015 ; Patrini et al. , 2017 ; Vahdat , 2017 ; Veit et al. , 2017 ; Li et al. , 2017 ) . Recently , great progress has been made towards robustness against anomalies when training DNNs ( Krueger et al. , 2017 ) . There are three appealing perspectives in terms of their simplicity and effectiveness : 1 ) Examples weighting . For example , knowledge distilling from auxiliary models is popular for heuristically designing weighting schemes . However , it is challenging to select and train reliable auxiliary models in practice ( Li et al. , 2017 ; Malach & Shalev-Shwartz , 2017 ; Jiang et al. , 2018 ; Ren et al. , 2018 ; Han et al. , 2018b ) . 2 ) Robust loss functions ( Van Rooyen et al. , 2015 ; Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ; Wang et al. , 2019b ) ; 3 ) Explicit regularisation techniques ( Arpit et al. , 2017 ; Zhang et al. , 2018a ) . Although designing robust losses or explicit regularisation is easier and more flexible in practice , the performance is not the optimal yet . 1One training example is composed of an input and its corresponding label . A semantically abnormal example means the input is semantically unrelated to its label , which may come from corrupted input or label . For example , in Figure 3 in the supplementary material : 1 ) Out-of-distribution anomalies : An image may contain only background or an object which does not belong to any training class ; 2 ) In-distribution anomalies : An image of class a may be annotated to class b or an image may contain more than one semantic object . Regarding examples weighting , there is a core research question which is not well answered yet : What training examples should be focused on and how large the emphasis spread should be ? In this work , we present a thorough study of this practical question under different settings . For better analysis , we propose two basic and necessary concepts : emphasis focus and spread with explicit definition in Sec . 3.2 . They are conceptually introduced as follows : Emphasis focus . It is a common practice to focus on harder instances when training DNNs ( Shrivastava et al. , 2016 ; Lin et al. , 2017 ) . When a dataset is clean , it achieves faster convergence and better performance to emphasise on harder examples because they own larger gradient magnitude , which means more information and a larger update step for model ’ s parameters . However , when severe noise exists , as demonstrated in ( Krueger et al. , 2017 ; Arpit et al. , 2017 ) , DNNs learn simple meaningful patterns first before memorising abnormal ones . In other words , anomalies are harder to fit and own larger gradient magnitude in the later stage . Consequently , if we use the default sample weighting in categorical cross entropy ( CCE ) where harder samples obtain higher weights , anomalies tend to be fitted well especially when a network has large enough capacity . That is why we need to move the emphasis focus towards relatively easier ones , which serves as emphasis regularisation . Emphasis spread . We term the weighting variance of training examples emphasis spread . The key concept is that we should not treat all examples equally , neither should we let only a few be emphasised and contribute to the training . Therefore , when emphasis focus changes , the emphasis spread should be adjusted accordingly . We integrate emphasis focus and spread into a unified example weighting framework . Emphasis focus defines what training examples own higher weights while emphasis spread indicates how large variance over their weights . Specifically , we propose gradient rescaling ( GR ) , which modifies the magnitude of logit vector ’ s gradient . The logit vector is the output of the last fully connected ( FC ) layer of a network . We remark that we do not design the weighting scheme heuristically from scratch . Instead , it is naturally motivated by the gradient analysis of several loss functions . Interestingly , GR can be naturally connected to examples weighting , robust losses , explicit regularisation : 1 ) The gradient magnitude of logit vector can be regarded as weight assignment that is built-in in loss functions ( Gopal , 2016 ; Alain et al. , 2016 ; Zhang et al. , 2018b ) . Therefore , rescaling the gradient magnitude equals to adjusting the weights of examples ; 2 ) A specific loss function owns a fixed gradient derivation . Adjusting the gradient can be treated as a more direct and flexible way of modifying optimisation objectives ; 3 ) Instead of focusing on harder examples2 by default , we can adjust emphasis focus to relative easier ones when noise is severe . GR serves as emphasis regularisation and is different from standard regularisers , e.g. , L2 weight decay constraints on weight parameters and Dropout samples neural units randomly ( Srivastava et al. , 2014 ) ; GR is simple yet effective . We demonstrate its effectiveness on diverse computer vision tasks using different net architectures : 1 ) Image classification with clean training data ; 2 ) Image classification with synthetic symmetric label noise , which is more challenging than asymmetric noise evaluated by ( Vahdat , 2017 ; Ma et al. , 2018 ) ; 3 ) Image classification with real-world unknown anomalies , which may contain open-set noise ( Wang et al. , 2018 ) , e.g. , images with only background , or outliers , etc . ; 4 ) Video person re-identification , a video retrieval task containing diverse anomalies . Beyond , we show that GR is notably better than other standard regularisers , e.g. , L2 weight decay and dropout . Besides , to comprehensively understand GR ’ s behaviours , we present extensive ablation studies . Main contribution . Intuitively and principally , we claim that two basic factors , emphasis focus and spread , should be babysat simultaneously when it comes to examples weighting . To the best of our knowledge , we are the first to thoroughly study and analyse them together in a unified framework . 2 RELATED WORK . Aside from examples weighting , robust losses minimisation and explicit regularisation techniques , there are another two main perspectives for training robust and accurate DNNs when anomalies exist : 2 An example ’ s difficulty can be indicated by its loss ( Shrivastava et al. , 2016 ; Loshchilov & Hutter , 2016 ; Hinton , 2007 ) , gradient magnitude ( Gopal , 2016 ; Alain et al. , 2016 ) , or input-to-label relevance score ( Lee et al. , 2018 ) . The input-to-label relevance score means the probability of an input belonging to its labelled class predicted by a current model . The difficulty of an example may change as the model learns . In summary , higher difficulty , larger loss , larger gradient magnitude , and lower input-to-label relevance score are equal concepts . 1 ) Robust training strategies ( Miyato et al. , 2018 ; Guo et al. , 2018 ; Li et al. , 2019 ; Thulasidasan et al. , 2019 ) ; 2 ) Noise-aware modelling , and alternative label and parameter optimisation are popular when only label noise exists . Some methods focus on noise-aware modelling for correcting noisy labels or empirical losses ( Larsen et al. , 1998 ; Natarajan et al. , 2013 ; Sukhbaatar & Fergus , 2014 ; Xiao et al. , 2015 ; Vahdat , 2017 ; Veit et al. , 2017 ; Goldberger & Ben-Reuven , 2017 ; Han et al. , 2018a ) . However , it is non-trivial and time-consuming to learn a noise-aware model , which also requires prior extra information or some specific assumptions . For example , Masking ( Han et al. , 2018a ) is assisted by human cognition to speculate the noise structure of noise-aware matrix while ( Veit et al. , 2017 ; Li et al. , 2017 ; Lee et al. , 2018 ; Hendrycks et al. , 2018 ) exploit an extra clean dataset , which is a hyper-factor and hard to control in practice . Some other algorithms iteratively train the model and infer latent true labels ( Wang et al. , 2018 ; Tanaka et al. , 2018 ) . Those methods have made great progress on label noise . But they are not directly applicable to unknown diverse semantic anomalies , which covers both out-of-distribution and in-distribution cases . 2.1 REMARKS ON ROBUSTNESS THEOREMS CONDITIONED ON SYMMETRIC LOSSES . We note that ( Ghosh et al. , 2017 ) proposed some theorems showing that empirical risk minimization is robust when the loss function is symmetric and the noise type is label noise . However , they are not applicable for deep learning under arbitrary unknown noise : 1 ) We remark that we target at the problem of diverse or arbitrary abnormal examples , where an input may be out-of-distribution , i.e. , not belonging to any training class . As a result , the symmetric losses custom-designed for label noise are not applicable . 2 ) GR is independent of empirical loss expressions as presented in Table 1 . Therefore , one specific loss is merely an indicator of how far we are away from a specific minimisation objective . It has no impact on the robustness of the learning process since it has no direct influence on the gradient back-propagation . Similar to the prior work of rethinking generalisation ( Zhang et al. , 2017 ) , we need to rethink robust training under diverse anomalies , where the robustness theorems conditioned on symmetric losses and label noise are not directly applicable . 3 EMPHASIS REGULARISATION BY GRADIENT RESCALING . Notation . We are given N training examples X = { ( xi , yi ) } Ni=1 , where ( xi , yi ) denotes i−th sample with input xi ∈ RD and label yi ∈ { 1 , 2 , ... , C } . C is the number of classes . Let ’ s consider a deep neural network z composed of an embedding network f ( · ) : RD → RK and a linear classifier g ( · ) : RK → RC , i.e. , zi = z ( xi ) = g ( f ( xi ) ) : RD → RC . Generally , the linear classifier is the last FC layer which produces the final output of z , i.e. , logit vector z ∈ RC . To obtain probabilities of a sample belonging to different classes , logit vector is normalised by a softmax function : p ( j|xi ) = exp ( zij ) / ∑C m=1 exp ( zim ) . ( 1 ) p ( j|xi ) is the probability of xi belonging to class j . A sample ’ s input-to-label relevance score is defined by pi = p ( yi|xi ) . In what follows , we will uncover the sample weighting in popular losses : CCE , Mean Absolute Error ( MAE ) and Generalised Cross Entropy ( GCE ) ( Zhang & Sabuncu , 2018 ) .

Summary: This paper presents Gradient Rescaling (GR) for robust learning to combat label noise. They propose to treat each data sample with different significance scores: some samples are important to learning, and some examples are insignificant (or even detrimental) to learning. So they desire to weight each samples according to their significance. They propose the notion of emphasis focus (When learning, whether we should put emphasis on learning “hard” examples or “easy” examples) and emphasis spread (the variance of these significance weights). The authors propose that this “difficulty” of samples are proportional to their network output logit values.