Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing

1 INTRODUCTION . Ever since Szegedy et al . ( 2016 ) introduced label smoothing as a way to regularize the classification ( or output ) layer of a deep neural network , it has been used across a wide range of tasks from image classification ( Szegedy et al. , 2016 ) and machine translation ( Vaswani et al. , 2017 ) to pre-training for natural language generation ( Lewis et al. , 2019 ) . Label smoothing works by mixing the one-hot encoding of a class with a uniform distribution and then computing the cross-entropy with respect to the model ’ s estimate of the class probabilities to compute the loss of the model . This prevents the model being too confident about its predictions — since the model is now penalized ( to a small amount ) even for predicting the correct class in the training data . As a result , label smoothing has been shown to improve generalization across a wide range of tasks ( Müller et al. , 2019 ) . More recently , Müller et al . ( 2019 ) further provided important empirical insights into label smoothing by showing that it encourages the representation learned by a neural network for different classes to be equidistant from each other . Yet , label smoothing is overly crude for many tasks where there is structure in the label space . For instance , consider task-oriented semantic parsing where the goal is to predict a parse tree of intents , slots , and slot values given a natural language utterance . The label space comprises of ontology ( intents and slots ) and natural language tokens and the output has specific structure , e.g. , the first token is always a top-level intent ( see Figure 1 ) , the leaf nodes are always natural language tokens and so on . Therefore , it is more likely for a well trained model to confuse a top-level intent with another top-level intent rather than a natural language token . This calls for models whose uncertainty is spread over related tokens rather than over obviously unrelated tokens . This is especially important in the few-shot setting where there are few labelled examples to learn representations of novel tokens from . Our contributions . We present the first rigorous theoretical analysis of label smoothing by obtaining PAC Bayesian generalization bounds for a closely related ( upper-bound ) loss function . Our analysis reveals that the choice of the smoothing distribution affects generalization , and provides a recipe for tuning the smoothing parameter . Then , we develop a simple yet effective extension of label smoothing : low-rank adaptive label smoothing ( LORAS ) , which provably generalizes the former and adapts to the latent structure that is often present in the label space in many structured prediction problems . We evaluate LORAS on three semantic parsing data sets , and a semantic parsing based question-answering data set , using various pre-trained representations like RoBERTa Liu et al . ( 2019 ) and BART Lewis et al . ( 2019 ) . On ATIS ( Price , 1990 ) and SNIPS ( Coucke et al. , 2018 ) , LORAS achieves average absolute improvement of 0.6 % and 0.9 % respectively in exact match of logical form over vanilla label smoothing across different pre-trained representations . In the few-shot setting using the TOPv2 data set ( Chen et al. , 2020 ) 1 , LORAS achieves an accuracy of 74.1 % on average over the two target domains — an absolute improvement of 2 % over using vanilla label smoothing and matching the state-ofthe-art performance in Chen et al . ( 2020 ) despite their use of a much more complex meta-learning method . Lastly , in the transfer learning setting on the Overnight data set ( Wang et al. , 2015 ) , LORAS improves over vanilla label smoothing by 1 % on average on the target domains . Furthermore , LORAS is easy to implement and train and can be used in conjunction with any architecture . We show that unlike vanilla label smoothing , LORAS effectively solves the neural network overconfidence problem for structured outputs where it produces more calibrated uncertainty estimates over different parts of the structured output . As a result , LORAS reduces the test set expected calibration error by 55 % over vanilla label smoothing on the TOPv2 data set . We present an efficient formulation of LORAS which does not increase the model size , while requiring only O ( K ) additional memory during training where K is the output vocabulary size ( or the number of classes in the multi-class setting ) . 2 PRELIMINARIES . We consider structured prediction formulated as a sequence-to-sequence ( seq2seq ) prediction problem . We motivate our method through semantic parsing where the input x is an natural language utterance and the output y is a serialized tree that captures the semantics of the input in a machine understandable form ( see Figure 1 for an example ) . Specifically , given input output pairs ( x , y ) where x = ( xi ) mi=1 and y = ( yi ) n i=1 are sequences , let φ ( x , y1 : t−1 ) be the representation of the input and output tokens up to time step t − 1 modeled by a neural network . At time step t the probability of the t-th output token is given by : softmax ( Wφ ( x , y1 : t−1 ) ) , where W ∈ RK×d are the output projection weights ( last layer ) of the neural network and K is the vocabulary size . The representation and the output projections are learned by minimizing the negative log-likelihood of the observed samples S. 1TOPv2 data set is a newer version of the TOP data set introduced in ( Gupta et al. , 2018 ) containing 6 additional domains , which is particularly suitable for benchmarking few-shot semantic parsing methods . Label Smoothing . The idea behind label smoothing is to uniformly smooth the one-hot vector before computing the cross-entropy with the learned distribution . Let yt = ( 1 [ yt = j ] ) Kj=1 denote the one-hot encoding of t-th output token and pt = ( p ( yt = j | x , y1 : t−1 ) ) Kj=1 denote the distribution over the vocabulary modeled by the neural network . Then , setting yLSt = ( 1 − α ) y + α1/K , we compute H ( yLSt , pt ) , the cross-entropy between y LS t and pt , as our loss function : H ( yLSt , pt ) = − ( 1− α ) K∑ j=1 yt , j logpt , j − α K K∑ j=1 logpt , j . ( 1 ) 3 THEORETICAL MOTIVATION FOR LABEL SMOOTHING . In this section we look at why training neural networks with soft-targets can help with generalization . To simplify exposition we will consider a multi-class classification setting where we have input output pairs ( x , y ) and y ∈ [ K ] . As first described by Müller et al . ( 2019 ) , label smoothing encourages the representation of the input φ ( x ) to be close to projection weight ( w ) for the correct class and at the same time be equidistant from the weights of all other classes . We will formalize this by obtaining rigorous generalization bounds for label smoothing . Towards that end we will fix the input representation φ ( x ) ∈ Rd with ‖φ ( x ) ‖2 ≤ 1 and focus on the classification layer weights W ∈ RK×d . For a noise distribution n ( · ) , which is uniform for standard label smoothing , an upper-bound on the loss ( 1 ) is given as follows : L̃ ( S , W ) = ∑ ( x , y ) ∈S l ( x , y ; W , α ) + α 2 ∥∥n−Wφ∥∥2 2 ( 2 ) where n = ( n ( i ) ) Ki=1 is the vectorized noise distribution and φ = ∑ x∈S φ ( x ) is the sum of input representations in the training set , and l ( x , y ; W , α ) is the re-scaled negative log-likelihood of the observed data S where the linear term is scaled by α . The upper bound is obtained by ignoring the norm of Wφ from the objective ( see Appendix A.1 for a derivation ) . The objective given by ( 2 ) is essentially penalized negative log-likelihood with a penalty term that encourages the aggregated ( un-normalized ) class scores to be close to the noise distribution . Unlike standard weight regularization , however , the regularization term depends on both the weight W and the inputs x ∈ S. Therefore , existing theory for regularized loss minimization doesn ’ t apply to this case and we invoke PAC-Bayesian theory to analyze the above . As is standard in PAC-Bayesian analysis , we consider minimizing the empirical loss L̃ ( S , W ) around a small neighborhood of the weight W. For a posterior distribution Q on the weights W , which depends on the sample S , we consider the following empirical risk and the expected risk and their respective minimizers . R̂ ( Q ( W ) ) = EW′∼Q ( W ) [ L̃ ( S , W ) ] ( empirical risk ) R ( Q ( W ) ) = EW′∼Q ( W ) [ E ( x , y ) [ l ( x , y ; W′ , α ) ] ] ( expected risk ) Ŵ ∈ argmin W∈RK×d R̂ ( Q ( W ) ) ( empirical minimizer ) W ∈ argmin W∈RK×d R ( Q ( W ) ) ( true minimizer ) The following theorem , whose proof we defer to Appendix A.2 , bounds the risk of the minimizer of the label smoothing loss ( on the sample S ) in terms of the risk of the minimizer of the expected negative log-likelihood ( under the data distribution ) . Theorem 1 ( PAC-Bayesian generalization bound ) . Set the distribution Q ( W ) , parameterized by W with bounded induced norm , over the weights W′ to be such that each column W′∗ , i is sampled i.i.d . from the Gaussian distribution N ( Wφ , I ) . If α = 2d/√N , where N = |S| is the number of samples , then with probability at least 1− δ the generalization error is bounded as follows : R ( Q ( Ŵ ) ) −R ( Q ( W ) ) ≤ 2d√ N ∥∥n−Wφ∥∥2 2 + 1√ N log 2e b2 8 δ . It is important to note that the generalization error depends on the number of classes K through the term ∥∥n−Wφ∥∥ 2 which grows as Θ ( K ) . This is due to the fact that the label smoothing objective regularizes the class scores ( Wφ ) as opposed to regularizing the output layer weights ( W ) directly which would result in the generalization error depending on ‖W‖2F which is Θ ( dK ) . The above result also prescribes how to set the smoothing constant α which is typically chosen to be the constant 0.1 in practice — as the number of samples N →∞ , α→ 0 and less smoothing of the hard targets is needed to achieve generalization . Furthermore , the generalization error also depends on how close the aggregated un-normalized class scores ( Wφ ) of the true minimizer on the training set S are to the noise distribution . Therefore , choosing a more informative smoothing distribution , as opposed to a uniform distribution , should lead to better generalization .

Summary: The paper proposes a label-smoothing method upon the low-rank assumption of the output dimension, especially when the output dimension is large. The contribution of this work is two folds: first, highlighted the importance of informative label smoothing through better bound, and second, proposed one label smoothing with low-rank assumption. It is overall a good paper but there are a few concerns: