Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing

1 INTRODUCTION . Ever since Szegedy et al . ( 2016 ) introduced label smoothing as a way to regularize the classification ( or output ) layer of a deep neural network , it has been used across a wide range of tasks from image classification ( Szegedy et al. , 2016 ) and machine translation ( Vaswani et al. , 2017 ) to pre-training for natural language generation ( Lewis et al. , 2019 ) . Label smoothing works by mixing the one-hot encoding of a class with a uniform distribution and then computing the cross-entropy with respect to the model ’ s estimate of the class probabilities to compute the loss of the model . This prevents the model being too confident about its predictions — since the model is now penalized ( to a small amount ) even for predicting the correct class in the training data . As a result , label smoothing has been shown to improve generalization across a wide range of tasks ( Müller et al. , 2019 ) . More recently , Müller et al . ( 2019 ) further provided important empirical insights into label smoothing by showing that it encourages the representation learned by a neural network for different classes to be equidistant from each other . Yet , label smoothing is overly crude for many tasks where there is structure in the label space . For instance , consider task-oriented semantic parsing where the goal is to predict a parse tree of intents , slots , and slot values given a natural language utterance . The label space comprises of ontology ( intents and slots ) and natural language tokens and the output has specific structure , e.g. , the first token is always a top-level intent ( see Figure 1 ) , the leaf nodes are always natural language tokens and so on . Therefore , it is more likely for a well trained model to confuse a top-level intent with another top-level intent rather than a natural language token . This calls for models whose uncertainty is spread over related tokens rather than over obviously unrelated tokens . This is especially important in the few-shot setting where there are few labelled examples to learn representations of novel tokens from . Our contributions . We present the first rigorous theoretical analysis of label smoothing by obtaining PAC Bayesian generalization bounds for a closely related ( upper-bound ) loss function . Our analysis reveals that the choice of the smoothing distribution affects generalization , and provides a recipe for tuning the smoothing parameter . Then , we develop a simple yet effective extension of label smoothing : low-rank adaptive label smoothing ( LORAS ) , which provably generalizes the former and adapts to the latent structure that is often present in the label space in many structured prediction problems . We evaluate LORAS on three semantic parsing data sets , and a semantic parsing based question-answering data set , using various pre-trained representations like RoBERTa Liu et al . ( 2019 ) and BART Lewis et al . ( 2019 ) . On ATIS ( Price , 1990 ) and SNIPS ( Coucke et al. , 2018 ) , LORAS achieves average absolute improvement of 0.6 % and 0.9 % respectively in exact match of logical form over vanilla label smoothing across different pre-trained representations . In the few-shot setting using the TOPv2 data set ( Chen et al. , 2020 ) 1 , LORAS achieves an accuracy of 74.1 % on average over the two target domains — an absolute improvement of 2 % over using vanilla label smoothing and matching the state-ofthe-art performance in Chen et al . ( 2020 ) despite their use of a much more complex meta-learning method . Lastly , in the transfer learning setting on the Overnight data set ( Wang et al. , 2015 ) , LORAS improves over vanilla label smoothing by 1 % on average on the target domains . Furthermore , LORAS is easy to implement and train and can be used in conjunction with any architecture . We show that unlike vanilla label smoothing , LORAS effectively solves the neural network overconfidence problem for structured outputs where it produces more calibrated uncertainty estimates over different parts of the structured output . As a result , LORAS reduces the test set expected calibration error by 55 % over vanilla label smoothing on the TOPv2 data set . We present an efficient formulation of LORAS which does not increase the model size , while requiring only O ( K ) additional memory during training where K is the output vocabulary size ( or the number of classes in the multi-class setting ) . 2 PRELIMINARIES . We consider structured prediction formulated as a sequence-to-sequence ( seq2seq ) prediction problem . We motivate our method through semantic parsing where the input x is an natural language utterance and the output y is a serialized tree that captures the semantics of the input in a machine understandable form ( see Figure 1 for an example ) . Specifically , given input output pairs ( x , y ) where x = ( xi ) mi=1 and y = ( yi ) n i=1 are sequences , let φ ( x , y1 : t−1 ) be the representation of the input and output tokens up to time step t − 1 modeled by a neural network . At time step t the probability of the t-th output token is given by : softmax ( Wφ ( x , y1 : t−1 ) ) , where W ∈ RK×d are the output projection weights ( last layer ) of the neural network and K is the vocabulary size . The representation and the output projections are learned by minimizing the negative log-likelihood of the observed samples S. 1TOPv2 data set is a newer version of the TOP data set introduced in ( Gupta et al. , 2018 ) containing 6 additional domains , which is particularly suitable for benchmarking few-shot semantic parsing methods . Label Smoothing . The idea behind label smoothing is to uniformly smooth the one-hot vector before computing the cross-entropy with the learned distribution . Let yt = ( 1 [ yt = j ] ) Kj=1 denote the one-hot encoding of t-th output token and pt = ( p ( yt = j | x , y1 : t−1 ) ) Kj=1 denote the distribution over the vocabulary modeled by the neural network . Then , setting yLSt = ( 1 − α ) y + α1/K , we compute H ( yLSt , pt ) , the cross-entropy between y LS t and pt , as our loss function : H ( yLSt , pt ) = − ( 1− α ) K∑ j=1 yt , j logpt , j − α K K∑ j=1 logpt , j . ( 1 ) 3 THEORETICAL MOTIVATION FOR LABEL SMOOTHING . In this section we look at why training neural networks with soft-targets can help with generalization . To simplify exposition we will consider a multi-class classification setting where we have input output pairs ( x , y ) and y ∈ [ K ] . As first described by Müller et al . ( 2019 ) , label smoothing encourages the representation of the input φ ( x ) to be close to projection weight ( w ) for the correct class and at the same time be equidistant from the weights of all other classes . We will formalize this by obtaining rigorous generalization bounds for label smoothing . Towards that end we will fix the input representation φ ( x ) ∈ Rd with ‖φ ( x ) ‖2 ≤ 1 and focus on the classification layer weights W ∈ RK×d . For a noise distribution n ( · ) , which is uniform for standard label smoothing , an upper-bound on the loss ( 1 ) is given as follows : L̃ ( S , W ) = ∑ ( x , y ) ∈S l ( x , y ; W , α ) + α 2 ∥∥n−Wφ∥∥2 2 ( 2 ) where n = ( n ( i ) ) Ki=1 is the vectorized noise distribution and φ = ∑ x∈S φ ( x ) is the sum of input representations in the training set , and l ( x , y ; W , α ) is the re-scaled negative log-likelihood of the observed data S where the linear term is scaled by α . The upper bound is obtained by ignoring the norm of Wφ from the objective ( see Appendix A.1 for a derivation ) . The objective given by ( 2 ) is essentially penalized negative log-likelihood with a penalty term that encourages the aggregated ( un-normalized ) class scores to be close to the noise distribution . Unlike standard weight regularization , however , the regularization term depends on both the weight W and the inputs x ∈ S. Therefore , existing theory for regularized loss minimization doesn ’ t apply to this case and we invoke PAC-Bayesian theory to analyze the above . As is standard in PAC-Bayesian analysis , we consider minimizing the empirical loss L̃ ( S , W ) around a small neighborhood of the weight W. For a posterior distribution Q on the weights W , which depends on the sample S , we consider the following empirical risk and the expected risk and their respective minimizers . R̂ ( Q ( W ) ) = EW′∼Q ( W ) [ L̃ ( S , W ) ] ( empirical risk ) R ( Q ( W ) ) = EW′∼Q ( W ) [ E ( x , y ) [ l ( x , y ; W′ , α ) ] ] ( expected risk ) Ŵ ∈ argmin W∈RK×d R̂ ( Q ( W ) ) ( empirical minimizer ) W ∈ argmin W∈RK×d R ( Q ( W ) ) ( true minimizer ) The following theorem , whose proof we defer to Appendix A.2 , bounds the risk of the minimizer of the label smoothing loss ( on the sample S ) in terms of the risk of the minimizer of the expected negative log-likelihood ( under the data distribution ) . Theorem 1 ( PAC-Bayesian generalization bound ) . Set the distribution Q ( W ) , parameterized by W with bounded induced norm , over the weights W′ to be such that each column W′∗ , i is sampled i.i.d . from the Gaussian distribution N ( Wφ , I ) . If α = 2d/√N , where N = |S| is the number of samples , then with probability at least 1− δ the generalization error is bounded as follows : R ( Q ( Ŵ ) ) −R ( Q ( W ) ) ≤ 2d√ N ∥∥n−Wφ∥∥2 2 + 1√ N log 2e b2 8 δ . It is important to note that the generalization error depends on the number of classes K through the term ∥∥n−Wφ∥∥ 2 which grows as Θ ( K ) . This is due to the fact that the label smoothing objective regularizes the class scores ( Wφ ) as opposed to regularizing the output layer weights ( W ) directly which would result in the generalization error depending on ‖W‖2F which is Θ ( dK ) . The above result also prescribes how to set the smoothing constant α which is typically chosen to be the constant 0.1 in practice — as the number of samples N →∞ , α→ 0 and less smoothing of the hard targets is needed to achieve generalization . Furthermore , the generalization error also depends on how close the aggregated un-normalized class scores ( Wφ ) of the true minimizer on the training set S are to the noise distribution . Therefore , choosing a more informative smoothing distribution , as opposed to a uniform distribution , should lead to better generalization .

Summary: This paper theoretically analyzes "label smoothing” (LS)  with PAC-Bayesian bound and motivated from their analysis, proposes a new method: LORAS.  In their theoretical analysis, they identify that the generalization error depends on the smoothing distribution. So they propose to learn the smoothing distribution in LORAS. In doing so, to overcome the computational issues & overfitting to diagonal smoothing, they propose Low-rank approach which seems to be successful. The authors show experimental results on semantic parsing dataset (ATIS, SNIPS, and TOPv2) where the LORAS shows better performance than no LS, LS, and some other SOTA model (with one exception of LORAS  vs. no LS on ATIS)