Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing
Authors: Asish Ghoshal
Xilun Chen
Sonal Gupta
Luke Zettlemoyer
Yashar Mehdad
Venue: ICLR 2021 Poster (2021)
Keywords: label smoothing
calibration
semantic parsing
structured prediction

Abstract: Training with soft targets instead of hard targets has been shown to improve performance and calibration of deep neural networks. Label smoothing is a popular way of computing soft targets, where one-hot encoding of a class is smoothed with a uniform distribution. Owing to its simplicity, label smoothing has found wide-spread use for training deep neural networks on a wide variety of tasks, ranging from image and text classification to machine translation and semantic parsing. Complementing recent empirical justification for label smoothing, we obtain PAC-Bayesian generalization bounds for label smoothing and show that the generalization error depends on the choice of the noise (smoothing) distribution. Then we propose low-rank adaptive label smoothing (LORAS): a simple yet novel method for training with learned soft targets that generalizes label smoothing and adapts to the latent structure of the label space in structured prediction tasks. Specifically, we evaluate our method on semantic parsing tasks and show that training with appropriately smoothed soft targets can significantly improve accuracy and model calibration, especially in low-resource settings. Used in conjunction with pre-trained sequence-to-sequence models, our method achieves state of the art performance on four semantic parsing data sets. LORAS can be used with any model, improves performance and implicit model calibration  without increasing the number of model parameters, and can be scaled to problems with large label spaces containing tens of thousands of labels.

Published as a conference paper at ICLR 2021
LEARNING BETTER STRUCTURED REPRESENTATIONS USING LOW-RANK ADAPTIVE LABEL SMOOTHING
Asish Ghoshal, Xilum Chen, Sonal Gupta, Luke Zettlemoyer & Yashar Mehdad aaghoshal, xilun, sonalgupta, 1s2, mehdad] (fm.ccmm Facebook AI
ABSTRACT
Training with soft targets instead of hard targets has been shown to improve per- formance and calibration of deep neural networks. Label smoothing is popular way of computing soft targets, where one-hot encoding of a class is smoothed with uniform distribution. Owing its simplicity, label smoothing has found wide- spread use for training deep neural networks on wide variety of tasks, ranging from image and text classification machine translation and semantic parsing. Complementing recent empirical justification for label smoothing, we obtain PAC- Bayesian generalization bounds for label smoothing and show that the generaliza- tion error depends on the choice of the noise (smoothing) distribution. Then we propose low-rank adaptive label smoothing (LORAS): simple yet novel method for training with learned soft targets that generalizes label smoothing and adapts to the latent structure of the label space in structured prediction tasks. Specifi- cally, we evaluate our method on semantic parsing tasks and show that training with appropriately smoothed soft targets can significantly improve accuracy and model calibration, especially in low-resource settings. Used in conjunction with pre-trained sequence-to-sequene
models, our method achieves state of the art performance on four semantic parsing data sets. LORAS can be used with any model, improves performance and implicit model calibration without increasing the number of model parameters, and can be scaled to problems with large label spaces containing tens of thousands of labels.
INTRODUCTION
Ever since Szegedy et al. (2016) introduced label smoothing as way to regularize the classification (or output) layer of deep neural network, has been used across wide range of tasks from image classification (Szegedy et al. 2016) and machine translation (Vaswani et al. 2017) to pre-training for natural language generation (Lewis et al., 2019). Label smoothing works by mixing the one-hot encoding of class with uniform distribution and then computing the cross-entrope with respect to the model's estimate of the class probabilities to compute the loss of the model. This prevents the model being too confident about its predictions amount) even for predicting the correct class in the training data. As a result, label smoothing has been shown improve generalization across a wide range of tasks (Müller al. 2019]. More recently, Müller et al. (2019) further provided important empirical insights into label smoothing by showing that encourages the representation learned by neural network for different classes to be equidistant from each other.
since the model is now penalized (to small
Yet, label smoothing is overly crude for many tasks where there is structure in the label space. For instance, consider task-oriented semantic parsing where the goal is to predict a parse tree of intents, slots, and slot values given a natural language utterance. The label space comprises of ontology (intents and slots) and natural language tokens and the output has specific structure, e.g., the first token always a top-level intent (see Figure []. the leaf nodes are always natural language tokens and on. Therefore, it is more likely for well trained model to confuse top-level intent with another tep-level intent rather than a natural language token. This calls for models whose uncertainty spread over related tokens rather than over obviously unrelated tokens. This is especially important in the shw-shot setting where there are few labelled examples to learn representations of novel tokens from.

---

Published as a conference paper at ICLR 2021
TN:GRT DIRECTTONS
Driving directions to
SL:DESTINATION
18:GET EVENT
DVENT
SL:CAT EVENT
DVENT
Eagles
game
("n:get_,diroctions isl:desrination [in:get__evatt (al tal::nnm__veee Eagles)
[al reat_event gume
Figure 1: Top: Semantic parse tree of the utterance "Driving directions the Eagles game". Bot- tom: Serialized tree. IN: stands for intents while SL: stands for slots (See Gupta et al. 2018).
Our contributions. We present the first rigorous theoretical analysis of label smoothing by ob- taining PAC Bayesian generalization bounds for closely related (upper-bound) loss function. Our analysis reveals that the choice of the smoothing distribution affects generalization, and provides a recipe for tuning the smoothing parameter.
Then, we develop a simple yet effective extension label smoothing: low-rank adaptive label smoothing (LORAS), which provably generalizes the former and adapts to the latent structure that often present in the label space in many structured prediction problems. We evaluate LORAS on three semantic parsing data sets, and semantic parsing based question-answering data set, using various pre-trate treitra representations like RoBERTa Liu et al. (2019) and BART Lewis Lewis et al. (2019). On ATIS (Price 1990) and SNIPS (Coucke et al. 2018). LORAS achieves average absolute im- provement of 0.6% and 0.9% respectively in exact match of logical form over vanilla label smooth- ing across different pre-trained representations.
the few-shot setting using the TOPv2 data set Chen et al. 2020 LORAS achieves an accuracy of 74. 1% on average over the two target domains an absolute improvement of 2% over using vanilla label smoothing and matching the state-of- the-art performance in Chen et al.] (020)) despite their use of much more complex meta-learning method. Lastly, in the transfer learning setting on the Overnight data set (Wang et al., 2015). LO- RAS improves over vanilla label smoothing by 1% average the target domains. Furthermore, LORAS easy to implement and train and can be used conjunction with any architecture.
We show that unlike vanilla label smoothing, LORAS effectively solves the neural network over- confidence problem for structured outputs where produces more calibrated uncertainty estimates over different parts of the structured output. As result, LORAS reduces the test set expected calibration error by 55% over vanilla label smoothing on the TOPv2 data set.
We present efficient formulation of LORAS which does not increase the model size, while quiring only O (K) additional memory during training where is the output vocabulary size (or the number of classes in the multi-class setting).
PRELIMINARIES
We consider structured prediction formulated as a requence-to-sequeee (seq2seq) prediction prob- lem. We motivate method through semantic parsing where the input utterance and the output is a serialized tree that captures the semantics of the input in a machine understandable form (see Figure 1 for example). Specifically, given input output pairs (a, y) where & = (x: (m and y (y: == are sequences, let Φ(x, 11+-1) be the representation of the input and output tokens up time step ( modeled by a neural network. At time step the probability of the 1-th output token is given by: softmax W (0 V/:t-1)), where W E R k xd are the output projection weights (last layer) of the neural network and is the vocabulary size. The representation and the output projections are learned by minimizing the negative log- likeliild of the observed samples S.
natural language
'TOP22 data set is newer version of the TOP data set introduced in (Gupta et al. 2018 containing 6 additional domains, which is particularly suitable for benchmarking few-shot semantic parsing methods.
2

---

Published as a conference paper at ICLR 2021
Label Smoothing. The idea behind label smoothing is uniformly smooth the one-hot vector be-
fore computing the cross-entropy with the learned distribution. Let Y: = [/c = j denote the
one-hot encoding of t-th output token and P: (p(ue = j z, 11:11:1111 11: denote the distribution
the vocabulary modeled by the neural network. Then, setting y (1 a)y + al/K, compute Hyy LS Pt), the cross-entropy between y! and pt, as our loss function:
k
H .P:) =-11 a)
Σ
Vig log
a
Σ
log Pt..
K
THEORETICAL MOTIVATION FOR LABEL SMOOTHING
In this section look why training neural networks with soft-targets can help with general- ization. To simplify exposition we will consider a multi-class classification setting where we have input output pairs (I, y) and encourages the representation of the input (xx) to be close to projection weight (w) for the correct
[K]. As first described by Müller et al. (2019). label smoothing
class and the same time be equidistant from the weights of all other classes. We will formalize this by obtaining rigorous generalization bounds for label smoothing. Towards that end we will fix the input representation (xx) E IR with [|o(2)]l2 I 1 and focus on the classification layer weights w E R xxx_ For a noise distribution n(.). which is uniform for standard label smoothing, an upper-bound on the loss (1 is given as follows:
L(S, w)
Σ (a.y:WWal a n we 1 (x.))E
(2)
2
where n = (n(i)) is the vectorized noise distribution and Φ = Exes ((x) is the sum input representations in the training set, and ((x, y: W, a) the re-scaled negative oigliiilihood of the observed data S where the linear term is scaled by
The upper bound is obtained by ignoring the norm of W from the objective (see Appendix A. for derivation). The objective given by (2) is essentially penalized negative log-likelihood with penalty term that encourages the aggregated (un-normalized) class scores to be close to the noise distribution. Unlike standard weight regularization, however, the regularization term depends on both the weight W and the inputs x E S. Therefore, existing theory for regularized loss mini- mization doesn't 't apply to this case and we invoke PAC-Bayesian theory to analyze the above. As standard PAC- -Cyaayysi analysis, we consider minimizing the empirical loss L(S, W) around small neighborhood of the weight W. For a posterior distribution Q on the weights W, which depends on the sample S, we consider the following empirical risk and the expected risk and their respective minimizers.
R(Q(W)) EwwQ(() L(S. w)]
(empirical risk)
R(Q(W)) = E Ew
(a [1x, yi W a)]]
(expected risk)
W argmin R(((())
(empirical minimizer)
WeRKxd
W& E argmin R(Q(W)) WERKKá
(true minimizer)
The following theorem, whose proof we defer to Appendix A.2 bounds the risk of the minimizer of the label smoothing loss (on the sample S) in terms of the risk of the minimizer of the expected negative log- likelihooo (under the data distribution).
Theorem 1 (PAC-Bayesian generalization bound). Set the distribution Q(W), parameterized by W with bounded induced norm, over the weights W to be such that each column W is sampled i.i.d. from the Gaussian distribution ((W sb, I). a 24/ where N |S] is the number of samples, then with probability at least 1 the generalization error is bounded as follows:
R(Q(W)) R(Q(W)) <
2d
n Woll
1
log
2e
8
3

---

Published as a conference paper at ICLR 2021
It is important to note that the generalization enror depends the number of classes K through the term || W which grows as ( This is due to the fact that the label smoothing objective regularizes the class scores (We) as opposed to regularizing the output layer weights (W) directly which would result in the generalization error depending on ||] which is ((dK). The above result also prescribes how to set the smoothing constant which is typically chosen to be the constant 0.1 in practice as the number of samples N -> oc, a - 0 and less smoothing of the hard targets is needed to achieve generalization. Furthermore, the generalization error also depends on how close the aggregated un-normalized class scores (W Φ) of the true minimizer the training set S are to the noise distribution. Therefore, choosing a more informative smoothing distribution, as opposed to uniform distribution, should lead to better generalization.
4 LOW-RANK ADAPTIVE LABEL SMOOTHING
informative noise distribution than the uniform distribution to smooth the hard targets. The idea behind low-rank adaptive label smoothing
Motivated by the above result, we propose
(LORAS) is learn the noise distribution that is to be mixed in with the one-hot target jointly with the model parameters. Specifically, we consider noise distributions n(- y, S) parameterized by the true label y and a symmetric matrix S RK* Then, in LORAS we set y y!.OAAS
(1 - a)y, + an S) where n(V,, S) = softmax(y, and compute the cross-entropy as follows:
H (yt L Pr)
Σ
LORAS log Pr
Yt
1=1
expl [S (S
= - a)
Σ j=1
Vt. log Pt.j
log Pt.i
Z (),,,))
j=1
where Sy,.- denotes the yr-th row of the matrix S and g( - xx.(S is the partition function. Thinking of n, for all j E [K] as Lagrange multipliers, the matrix S serves to relax the constraint that the representation of an output token be equally close to the weight vectors for all other tokens. For instance, in the semantic parsing task where the vocabulary comprises of ontology tokens (intents and slots) and utterance tokens, is unreasonable to treat everything on equal footing since certain groups of tokens (e.g. ontology tokens) might more similar to each other than others. For the i-th output token S determines "how close" the representation of the i-th token should be to the weight wj of the j-th token, with larger (resp, smaller) values of St.j forcing the representation to be closer (resp. farther). With this interpretation, the matrix S can also be thought of as imposing notion of similarity over vocabulary tokens with St,, determining how similar the i-th token is to the j-th token.
Low-rank assumption. The size of the vocabulary in many NLP applications is often large con- taining tens of thousands of tokens. Since the size of S is quadratic in the size of the vocabulary, it prohibitive to work the matrix directly. Instead, we assume that S has low rank structure, i.e., there exists an L E R xr with & K such that S LL" This is reasonable assumption in many settings, especially in our semantic parsing application, where a natural group structure exists in the vocabulary comprising of ontology and utterance tokens. Next, formulate (3) in terms of thereby eschewing the matrix S altogether. Under the above assumption we have:
I. L.ORAS = (1 a)y; + asoftmax(Ly....
y:
Note that Ly,, L R and we don't need explicitly construct the matrix S which can be large.
Entropy constraint and dropout. To encourage discovery of latent structure in the label space, we minimize (y Pp) subject to an entropy constraint on softmax(Sx..). This forces the noise distribution to be farther away from the uniform distribution. Also note that the low-rank assumption ensures that S doesn't 't become close to a diagonal matrix which would reduce adap- tive label smoothing to the case of training with hard targets. To see this, assume that for some i S., >> S,,, Vj # i. Then softmax(S...) x e, where e; is a vector of all zeros except for a at the i-th index, thereby reducing adaptive label smoothing to no label smoothing. The low rank
4

---

Published as a conference paper at ICLR 2021
assumption ensures that matrix S does not become diagonally dominant. Lastly, to avoid the matrix from overfitting to the training data, we add dropout on L. The final objective that adaptive label smoothing optimizes is given in the following, where ((-,,) .) denotes cross-entropy while H (.) denotes entropy:
L dropout(L;;)
lpl
I ILLORAS
Σ
Hly .Pt) ()((ssttaxx LT))
LORAS
(#.g)ES
lol
Σ
Σ
Hye
LORAS
.pi) - -I( softmax(Iy.
(x.)) 121
n (Ly
(4)
where
K Yt. log Pt.j
K
exp we. (L),)) Ptys
LORAS Pr) = -11 -a) a) Σ H
(5)
Z(L LT)
2 0 is hyper-parameter that controls how far the noise distributions are from the uniform distri- bution, with larger values encouraging more peaked (low-entropy) noise distributions, and [0, 1) is the dropout parameter. The matrix L is initialized to be a matrix of all ones, and we jointly learn the matrix L along with the model parameters. After training, the matrix L is discarded. Next, we show that the above formulation of adaptive label smoothing strictly generalizes label smoothing.
Proposition 1. Setting 7) = 0 and L 1, where 1 is the K-dimensional vector of ones, T,LORAS (S) (S) E (x.))s Ela H Pc).
Since setting 0, and the rank parameter 1, we have that L = 1 is in the solution path of minimizing the LORAS loss given Therefore, adaptive label smoothing strictly generalizes label smoothing.
EXPERIMENTS
We evaluate LORAS on three semantic parsing data sets: ATIS (Price) 1990), SNIPS (Coucke al. 2018 and TOPv2, (Chen et al. (220)), and question answering data set: Overnight (Wang jet al. 2015) where the goal is to predict executable logical form that can be executed against a database to answer a natural language query. On TOPv2 we evaluate LORAS in the few-shot setting in (Chen et al.) 2020 and for Overnight consider a transfer learning setting which is detailed in the next paragraph. All data sets were pre- pre-procesed exactly as in (Chen et al. 2020). Similar to Chen al. (2020) main model. While for ATIS, SNIPS, and TOPv2 we use ontology only generation where the model only allowed generate ontology tokens while copying everything else from the utterance, for Overnight data set we place no such constraints on the model since many of the output tokens are neither part of the ontology nor are part of the utterance. We experiment with different pre- trained language models for initialization: (encoder only) RoBERTa (Liu et al. 2019) and (encoder and decoder) BART (Lewis et al. 2019). For Overnight we only use BART.
the state-of-the-art model from (Rongali al. 2020)
We compare the performance of the model with different pre-trained representations with no label smoothing, vanilla label smoothing, and LORAS. For vanilla label smoothing we experiment with (0.1, 0.2, 0.3) and report the best accuracy. An a value of 0. is typically used in the literature, while an value of more than 0.2 produces inferior results. For LORAS, we experiment with a E (0.1, 0.2, 0.3). ). few different rank and dropout parameters, set ア) 0.1 for all the experiments and report the best accuracy. We found that for BART, LORAS dropout parameter best, while for RoBERTa dropout of 0.6 worked best. For all three data sets rank parameter of 25 worked best for LORAS. We did not scale beyond rank 25 since the BART model is fairly memory intensive leaving little room use large label embedding matrices during training.
0.5 worked
5

---

Published as a conference paper at ICLR 2021
ATIS
SNIPS
Pre-training
No
LS
LS
LORAS
No LS
LS
LORAS
FA
SA
FA
SA
FA
SA
FA
SA
FA
SA
FA
SA
RoBERTa
84.8
88.6
84.5
88.5
85.2
88.0
85.1
93.3
85.7
92.3
86.3
93.6
BART
87.6
88.6
87.7
88.5
88.2
89.8
90.0
98.0
89.4
98.0
90.6
98.3
BERT*
87.5
87.9
88.0
90.0
92
92.3
Table I: Test set frame accuracy (FA) and semantic accuracy (SA) on ATIS and SNIPS data set of our seq2seq models with copying mechanism. Numbers in bold represent the best performing method for a given metric and pre-trained model. *Refers to the BERT based joint intent classification and slot tagging model of Chen et al. (2019) where frame accuracy is the sentence level exact match accuracy.
Few-shot and transier- setting. In addition to the standard supervised setting, we also evaluate LORAS on the few-shot and transfer learning setting order to verify the hypothesis that having a more informed smoothing distribution would help when training data is limited. In particular, for TOPv2 we follow the same few-shot domain adaptation setting as in Chen et al. (2020), where domains in the TOPv2 data set used source domains while the remaining 2 (reminder and weather) serve as target domains. For Overnight we consider the four largest domains source domains and the four smallest domains target domains. For Overnight, the target domain training set was about one third the size the source domain training set. We first train our model on the combined training data from source domains and then fine-tune the model combined training data from target domains. For TOPv2 the training and validation data are limited to 25 samples per intent and slot (25 SPIS) for each target domain. More details about our experimental setup and other data set and training details can be found in Appendix A.4
6 RESULTS
We use the exact match accuracy, referred to frame accuracy (FA) where the predicted quence is matched exactly with the target sequence - as the primary evaluation metric following common practice in literature (Rongali al. (020). In addition to FA, we also report the semantic accuracy (SA), where we remove the slot values and only compare the resulting target and pre- dicted sequence. For instance, the target sequence [s] :artist adele ] ], and the predicted sequence is: mon:paaymusi [sl:yeat eighties m [sl:artiss] adele ] 1. then the frame accuracy would be zero while the semantic accuracy would be one, since removing the slot values makes the two sequences identical. Since often times the model copies over extraneous characters for slot value, semantic accuracy provides a measure of semantic understanding of the input that is robust to these errors. For Overnight we consider the logical form exact match accuracy as was done in Damonte et al. (201)).
[in:playmusic [s] : year eighties
Even though motivate LORAS for structured prediction problems, for ATIS and SNIPS which only contain flat hon-hierarchical) frames, we evaluate vanilla label smoothing and LORAS on top of BERT-based joint intent and slot tagging model (Chen et al. 2019). Chen al. (2019) do not use any label smoothing and report state-of-the-ae results for ATTS and SNIPs. For the BERT-based joint intent and slot tagging model we added label smoothing (vanilla and LORAS) only for the slot tagging component since the number of intents were too small for label smoothing to be useful. We ran each model three times with three different random initialization and report the test set exact match (EM) accuracy for the model that had the best validation set EM performance. Table ! shows the performance of our method on ATTS and SNIPS data sets. LORAS consistently out-perfone out-performs vanilla label smoothing and training with hard targets (no label smoothing) in almost all cases, while using label smoothing sometimes results poorer performance over using hard targets. Since the ATIS data set contains simple, non-hierarchical queries, we observe that with BART the SA of the models is pretty close to the FA. On SNIPS we observe that the BART based models achieve almost perfect SA score with LORAS further improving SA by 0.3% over standard LS. We also observe 1% drop in performance in using standard label smoothing over using hard targets. As we will show in the subsequent section, uniformly smoothing the labels hurts performance of the models

---

Published as a conference paper at ICLR 2021
Method
Reminder
Weather
Average
Test set frame aceu- racy (FA) and semantic accu-
Table 2:
FA
SA
FA
SA
FA
SA
BART + Reptile"
70.5
76.1
77.7
80.7
74.1
78.4
racy (SA) on TOPv2 data set.
Numbers in bold represent the
BART + No LS
69.8
75.7
75.1
78.8
72.5
77.3
best performing method for
BART +LS
67.6
72.6
76.6
79.9
72.1
76.3
given metric. *(Chen et al.
BART + LORAS
70.1
75.7
78.1
81.4
74.1
78.6
2020
Domain
Damonte et al.
No LS (seq2seq)
LS (seq2seq)
LORAS (seq2seq)
2019)
(Parsing)
Publications
40.4
64.0
64.0
66.5
Calendar
48.2
65.5
61.9
63.1
Housing
38.1
56.6
60.9
62.4
Recipes
63.0
80.6
80.1
78.7
Avg.
47.4
66.7
66.7
67.7
Table 3: Logical form exact match accuracy on target domains in the Overnight data set (Wang et al., 2013). The numbers for the neural cunsition-basee parser Damonte et al. (2019) are obtained from their paper (Table 3).
parts of the logical form for which the model is fairly accurate on. Lastly, from the results for the slot tagging we see that LORAS helps even in the multi-class classification setting H To further investigate if the improvements come from LORAS discovering meaningful structure in the label space and not due to chance, we visualize the noise distributions learned by LORAS on the SNIPS data and observe that learns very distinct noise distributions for B and I tags (see Appendix B}
Table 2 shows the few-shot performance of LORAS on the TOPv2 data set and Table f shows the results Overnight. For TOPv2, also include the -tate-of-the-art BART+Reptile method (Chen iet al. 2020) for comparison. On TOPv2, we see that LORAS comfortably out-performs label smoothing on both target domains. On the reminder domain, which contains logical forms with many hierarchical and nested structures, LORAS improves by 2.5%. Note that using label smooth- ing hurts performance as compared to training with hard (true) targets. Müller al. (2019) also re- port that using label smoothing hurts knowledge distillation performance. What is surprising is that the performance of our simple LORAS technique can match the performance the sophisticated meta-learning method in Chen al. (2020) which is specifically designed for domain adaptation. For the more complex reminder domain with many nested structures in the parse tree, we observe that label smoothing degrades performance by as much as 3.1% as compared to LORAS. Finally, we also observe that, overall, LORAS provides the highest SA performance among all methods. On Overnight, LORAS improves upon both vanilla label smoothing and no label smoothing by 1 1% on an average and on the housing domain, which contains the most complex logical forms, LORAS improves by around 6% over the model with no smoothing. Note that our seq2seq model signifi- cantly outperforms (avg. improvement around 20%) the previous state-of-ther the--att on Overnight data set (Damonte al. 2019).
Exploring the structure recovered by LORAS. In this section, we explore the structure in the label space that is recovered by LORAS the TOPv2 and Overnight data sets by looking at the matrix S = LL learned from data. Since the matrix S is larger than 50k x 50k, we visualize an informative sub-matrix in Figure 2 For each target token k, we visualize the noise distribu- tion softmax(SR..) over subset of vocabulary tokens. To select the most informative set of target tokens, we select the top 10 target tokens with lowest noise distribution entropy, and target to- kens with the highest noise distribution entropy. These noise distributions and bottom 5 rows of Figure respectively. Let the set denote these 15 target tokens, Figure 2 visualizes (softmax (S)) (S)),,r r,r. where the softmax taken over the second dimension of S.
shown in the top 10
2
"Our numbers for the slot tagging model without smoothing is slightly lower that those reported in Chen et al. (2019). We suspect this is because of our use of validation set to select the best model initialization rather than directly selecting the model with the highest test set numbers. We use this procedure for all our models.
7

---

Published as a conference paper at ICLR 2021
0.24
19 dene time
[5 ocaten
(14l pePiipedyy
Bageer event
611
66
pedestnation
(Steeteeent
Dal 1p0000
68
Inirreane alam
0.2
prssend message
(st mumec__pp bype
French
decmuction
0.06
depressed
02
compat
Strmme
ca
perk
bocation
lrage decirratiio
1ul pacppiy
finsend, fromai 1ype Fiensn
Start
lina
grmppr
Paot
0.00
Figure 2: A sub-matrix of the noise distribution matrix learned from the target domains (left) the TOPv2 data in the sho- setting, (right) the Overnight data set in the transfer learning setting. Row labels indicate the target token, and each row shows the noise distribution over subset of the vocabulary tokens (column labels) given the target token. (Best viewed in color.)
LB
L5
LORAS
culouda
Q44
0.0115
0.8
Asurnay
0.6
0.0
0.0
0.1
0.2
033
0.4
0.5
0.6
0.7
0.8
0.9
0.0 0.1
0.2
0.3
0.4
0.5
0.5
0.7
0.8
0.9
2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Confidence
Figure 3: Calibration plots for model trained on hard targets (no label smoothing), standard label smoothing, and LORAS on the test set of target domains in the TOPv2 data set.
From Figure 2 we immediately observe that the noise distributions with the highest entropy cor- respond to utterance tokens and those with the lowest entropy correspond to ontology or special tokens like end of sentence (
). While the noise distribution corresponding to utterance target tokens are close to uniform, the noise distribution for ontology tokens places most of the mass on two tokens: and 
, Thus expected, LORAS groups the vocabulary into distinct groups: ontology and special tokens, and utterance tokens.
we observe three distinct noise distributions corresponding to special tokens 1. 
, and the rest of the ontology tokens. The noise distributions are also pretty informative, for instance, when decoding, the model has to decide between terminating logical form with a closing bracket or a end-of-sentence tag. Therefore, these two tokens are most similar is reflected in the noise distribution. Due using rank parameter of 25 in experiments, are not able to further discern patterns within the ontology tokens. However, it must be noted that learning more finer-grained relationships between labels would come at the cost of higher sample complexity.
Further, among the non-utterance tokens
To summarize, the low rank constraint coupled with the entropy constraint, helped discover re- lationships between ontology tokens and special tokens while learning an almost uniform noise distribution over the utterance tokens.
In this section we dive into the issue of model calibration under no label smoothing, standard label smoothing, and LORA AS. Intuitively, a probabilistic model is cal- ibrated if the model's 's posterior probabilities are aligned with the ground-truth correctness likelihood (Guo et al.) 2017). We use the framework of (Guo et al. 2017) to measure (mis- mnis-acclibration of the models using the expected calibration error. Label smoothing has been shown improve calibration in image classification and machine translation (Müller et al. 2019), and on text classification using pre-trained transformer models (Desai & Durrett 2020). We evaluate the test set model calibration
Model calibration with LORAS.
8

---

Published as a conference paper at ICLR 2021
in the few-shot semantic parsing setting on the TOPv2 data set. Similar to (Müller et al. 2019), evaluate calibration of the conditional probabilities p(y = 2, 1/1--1) on the test set of com- bined target domains those examples where the predicted prefix y12-1 is correct. Figure shows shows the calibration plots along with the expected calibration error. We see that label smoothing does not improve model calibration in the structured prediction setting while LORAS produces fairly well-calibrated model reducing the expected calibration error by almost half. This is due to the fact that label smoothing makes the model equally uncertain about all tokens including top-level intents, closing brackets among others. While as observed in Rongali et al. (2020) and in our experiments, seq2seq models are good at learning the grammar of the parse trees, l.e. they produce well formatted trees with balanced bracket almost always. So the models should naturally be more confident about the location of 
 and tokens, while being less confident about novel intents and slots in the target domain. However, by uniformly perturbing the true targets, label smoothing makes the model equally uncertain about all kinds of target tokens thereby hurting calibration.
7 RELATED WORK
The main idea behind LORAS is to mix non-uniform distribution, that is learned jointly with the model parameters, with the one-hot encoding of the true target to compute smooth labels or soft targets. Mixing non-uniform distributions with the one-hot encoding of the true target has been explored in differentt contexts using various approaches in the computer vision literature.
In the context of addressing overconfident predictions of deep neural networks Guo et al. (01 Pereyra al. (2017) propose penalizing the models output distributions using hinge loss. While Reed et al. (2014) propose mixing the models predicted class probabilities with the true targets to generate soft targets to train deep networks on noisy labels. Lastly, knowledge distillation (Hinton eet al., 2015), which involves using the output probabilities of a teacher model to train a student model. is another way in which a model (student) is trained with smooth labels. However, none of these approaches exploit structure in the label space. As opposed to the aforementioned papers. our approach involves learning a noise distribution parameterized by a low-rank symmetric similarity matrix. We then penalize the entropy of the noise distribution rather than the model's 's output distri- bution. As a result our approach scales large label spaces with size exceeding fifty thousand. A part of our loss function is similar to that used in Bertinetto et al. (2020), however in Bertinetto et al.) (2220) the authors assume a known hierarchical structure on the label space while we do not make such assumptions.
Our method is also superficially similar to label embedding approaches (Akata et al., 2015 which involves computing embeddings of labels and learning mapping or compatibility function between input representation and the label embedding. Some notable work in this area is that of Frome et al.. (2013) who initialize the label embeddings from langauge model trained data from Wikipedia. Xian et al. (2016) use different label embeddings like Glove and word2vec and learn a bilinear map between input image embeddings and class embeddings for zero-shot classification. In the NLP literature, WWang et al.] (2018) compute label embeddings by using pre-trained word embeddings of the words in the labels and propose an attention mechanism to compute compatibility between input word embeddings and the label embeddings. However, unlike our method, these label embed- dings are a part of the model and are needed during inference. Due to this limitation, such label embeddings cannot be readily incorporated into any existing architecture. Furthermore, in struc- tured prediction problems where the label space contains special or abstract tokens, initiating good embedding matrices is not trivial task.
8 CONCLUSION AND FUTURE WORK
In this paper, we developed a novel extension of label smoothing called low-rank adaptive label smoothing (LORAS) which improves accuracy by automatically adapting to the latent structure in the label space of semantic parsing tasks. While we evaluated LORAS tasks and a slot tagging task, we believe that it will be useful for other seq2seq and multi-class classification tasks over structured label spaces and can always be used in place of standard label smoothing since it strictly generalizes the latter. would also be interesting to initialize (part of) the embedding matrix used for computing the noise distribution using pre-tr models.
semantic parsing

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):
1425-1438, 2015.
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooci, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12506-12555 12515, 2020.
Qian Chen, Zhu Zhuo, and Wen Wang. Bert for joint intent classification and slot filling. arXiv preprint 1rXiv 10909, 2019.
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. Low-resource domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, al. Snips voice plat- form: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv: 1805. 1019.0000..0 2018
Marco Damonte, Rahul Goel, and Tagyoung Chung. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Industry
Papers), pp. 16-23, 2019.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. arXiv preprint arXiv:2003.0789. 2020.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: deep iisual-semantic embedding model. In Advances neural information processing systems, pp. 2121-2129 2013.
2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330 2017.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2787-2792, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint M.Xiv:1503302531, 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-t--squeeee pre- training for natural language generation, translation, and comprehension. arXiv preprint 19X.:11910.13461, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
David McAllester. Simplified pac-bayesian margin bounds. In Learning theory and Kernel ma- chines, pp. 203-215. Springer, 2003.
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06544 2017.
10

---

Published as a conference paper at ICLR 2021
Patti Price. Evaluation of spoken language systems: The atis domain. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. Don't parse, generate! a se- quence to sequence architecture for task-oriented semantic parsing. In Proceedings of The Web Conference 2020, pp. 2962-2968, 2020.
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1073-1083, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. arXiv preprint 1rXiv:1805.0411 2018.
Yushi Wang, Jonathan Berant, and Percy Liang. Building semantic parser overnight. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp- 1332-1342,2015
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEFE Conference on Com- puter Vision and Pattern Recognition, pp. 69-77, 2016.
A APPENDIX
A. LABEL SMOOTHING LOSS REFORMULATION
Starting with (.. representing the outputs y as one-hot vectors, and representing the classification layer weights as a matrix W E R xd we have the following label smoothing loss for the multi-class classification setting:
L(S, W) = -11
Σ (u.v)s
logp(y xx, W) a Σ Enlsslogpte xx, W). V
where n(.) is the noise distribution, which is uniform for standard label smoothing. Writing the noise distributions vector (n(i)) K denoting & Exre Φ(x), and with ply x, W)

---

Published as a conference paper at ICLR 2021
exply w where Z(W,z) 2)) = Ev exp(y Wo(+)) we have:
L(S, W) = -(1 a)
Σ {u T Wo(() - log
(T.))
a Σ n(y') { We(x) log z(W
Σ
-1 a)y Wo(r) + log Z(W x)) x) -a Σ n Wo(()
(x,))ES
(2.0)es
Σ
-(1 a)y We(() + log (W,
(T.Y)ES
a 1 2
n
w I 1
w w
Denoting the re-scaled negative log-likelihood by 1(x, ys W log Z(W, 2), and since ||nl|2 is a constant term for minimizing L(S, W) with respect to W, and dropping the term
-(1 a)y We(()) +
we arrive at the following upper bound on the label smoothing loss: W 2
I(S. Σ 1a, W. W.a)
n w
(6)
(x.y)ES
A.2 PAC-BAYESIAN GENERALIZATION BOUND PROOF
We st-state the theorem for convenience.
Theorem 2 (PAC-Bayesian generalization bound). Set the distribution Q(W), parameterized by W with bounded induced norm, over the weights W to be such that each column W.,; is sampled i..d from the Gaussian distribution (WW61,, I). If 2d/ where N = |S] is the number of samples, then with probability at least 1 the generalization error is bounded as follows:
2e 6²
R(Q(W)) R((Q(W))
2d VN
In w
1
log 8
8
Proof. Denote I(S; W, a. = ExE,)ss (x, y; W, a.) as the (re-scaled) negative log-likelihood of the data S. Let R(Q(W)) = Ew--(w) (SS W ax) be the empirical negative log- lieliikelihood.. Let R(Q(W)) Q(W)))) [B<e,,) (<x, ys W ,',a)]] be the expected negative log-likelihood. Choose the prior distribution to be the following product distribution parameterized by the noise distribution: P [_ Mnn I). Since Q(W) = - ((W I),
aa]]
we have that
d
KL (Q(
Σ KL (N(WO.D)((n.I)
n W
From the PAC-Bayesian theorem McAllester 2003 we have that with probability at least 1 - 6,
& = -V/vT) log 2- 1//8 and for all W TR. Kxd with bounded incduced norm:
(R(Q(W) R(Q())) <
(KI((((W))
|| P) +
N
{In- w +
where b is the Lipschitz constant of the loss which is bounded under our assumption that ||o(-))l2 5 1 and W have bounded induced norm. Setting 2d/vN and since W E argming R(Q(W)) +
af V
well we have that
R(Q(W)) 4/v7
W
(((((W)) + -
(8)
12

---

Published as a conference paper at ICLR 2021
Therefore, from and B have that:
R(Q ()) R(Q(W)) R(Q(W)) W
-RQQ(W))+4M) w +
S R(Q(W))+ +///////// n W :
-(Q(W)) + +/VV n W o + +
24/VN n We 1
A.3 PROOF OF PROPOSITION
We re-state the proposition for convinience.
Proposition Setting q 7) and where 1 the K-ddimensoonal vector of ones, LLORAS (S) L (S) E(.,)) Ee Hly!' LS Pt).
Proof of Proposition 7 With we have that
lvl Σ Σ
I.LORAS(S)
H(y Ps)
LORAS
(2.0) 1=1
With L 1 have that Ly, (I,,,)) 1, Vyr. j. Therefore, HyyOORAS .P.) Hly p.). n
Since setting =1 0, and the rank parameter 1, we have that L 1 is in the solution path of minimizing the LORAS loss given Therefore, adaptive label smoothing strictly generalizes label smoothing.
A.4 EXPERIMENT DETAILS
Domains
Avg. depth
# Train examples
Blocks
4.64
1596
Social network
5.52
3535
Basketball
5.83
1561
Restaurants
4.65
1325
All
8017
Housing
4.60
752
Calendar
4.48
669
Publications
4.50
640
Recipes
4.32
864
All
2925
Table Statistics the Overnight data set (Wang et al. 2015). The top four domains are chosen as source domains while the bottom four are chosen target domains. Average depth is the average of the maximum depth of logical forms (trees) in the test set.
Overnight data set. Statistics of the Overnight data set (Wang et al. 2015) is shown in Table For our transfer learning setting we chose the four smallest domains target domains and the rest as source domains. As was done in prior work (Wang et al. (2015). Damonte et al. (2019) we randomly select 20% of training data from each domain the validation which used for model selection and hyper parameter tuning. We used the test set provided by the data set authors (Wang et al. 2015) for reporting the final performance of all the models.
13

---

Published as a conference paper at ICLR 2021
B-party size number
6.r 617
dar 0.1
B-playlist owner 0.7 617
002
B-city
0.150
B-state
0
002
B-artist
03r 037 535
034
002
0.125
B-rating value
216 n15 555
511
10 007
52
B-music item
238
u3e
014
0I 902
5m
B-raling unit
0.100
B-object Type
Hrstaurant lype
H-plyllsst owner
012 M32 031
0:11
011
031
004
104
582 510
11
0.075
Egenre
212 012
031
0.11
011
012
2.0 804
221 5.02
23
I-object
Macility
212 012
012
904
5/2
53
0.050
8-poi
I-usiic item
H-Ulssnee
12 A12
012
011
A11
001
004
0.025
UNK
12 012 ds:
cu
004
223 0.02
Saabjct__art_____ serieseree__see type
812 ox2
ex:
ntt
on no1
B-party size number 8-playlist_ owner
B-city B-state
Brraing value
B-arlist
B-music Irem
B-ralin unit
B-object type
Heretauurnt type
I-playiis owner
1-object select
I-gerre I-fcclity
1-object part of series type
B-pai I-music Item Loulsine
UNK
Figure 4: A sub-matrix of noise distribution matrix learned over the SNIPS data set. The top ten rows shows labels with the lowest entropy noise distribution while the bottom ten rows shows labels with the highest entropy noise distributions.
After some minor pre-processing, an example utterance and it's corresponding logical form from the data is shown below:
Utterance: meetings that start later than the weekly standup meeting
Logical form:
(call: mellllissValuee (call;filter (call:getProperty (call:singleto) en meeting (string !type
(cail:eenuureuumerric ropee
(string start_time ) ) (string > ) (call: seall:eessrelmercccEntt..
:call:getProperty en en.meeiing wwweeee...weelyy...mmme.......ee seeekly_staddu (string end_time ) )
We add the function names like (call:1istValue, variable types like (string, and closing bracket at the end of the GPT2 vocabulary as ontology tokens. The final vocabulary size was 50284.
Model details. The model from Rongali et al. (2020) comprises of a seq2seq Transformer model and a Pointer Generator Network (See et al. (017). When using RoBERTa, only the encoder is initialized with the pre-trained embeddings while 3-layer 256-hidden-unit decoder trained from scratch. Unlike RoBERTa. BART provides both pre-traue pre-trained encoder and decoder which makes it an ideal option for initializing seq2seq models. Lastly, in our implementation, the model generates ontology tokens from the vocabulary while the slot values are always copied over from the utterance using the copy mechanism. For ATIS and SNIPS we did not canonicalize the data (Chen et al. 2020), i.e. alphabetically order the slots the same level, during pre-processing.
Training details. For all our experiments used the following early stopping criteria: we stop training after the validation accuracy does not improve for 20 epochs. All the models were trained on Nvidia Telsa GPUs with 16GB of RAM. For ATIS and SNIPS, the models were trained on single GPU while for TOPv2 models were trained on 2 GPUs. We used Adam optimizer with default settings, inverse square root learning rate schedule, and a batch size of 32 for all our experiments.
14

---

Published as a conference paper at ICLR 2021
B STRUCTURE RECOVERED BY LORAS IN THE SNIPS DATA SET
Similar to Figure 2 Figure visualizes the noise distributions of labels with the lowest entropy (top rows) and the those with the highest entropy (bottom ten rows). We observe that top- 10 low entropy noise distributions correspond to mostly "B" tags (beginning of slot) while top- 10 high entropy noise distributions (close uniform) are learned for "T" tags. This is intuitive, since the model is more likely to confuse one B tag with another B tag (e.g. B-city vs B-state) rather than one tag with another tag. Therefore. by smoothing over closely related labels (tags) LORAS is able to force the model to learn better representations.
15


Reviews:
[{"review_id": "iv0gb0EjFcy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a label-smoothing method upon the low-rank assumption of the output dimension, especially when the output dimension is large. The contribution of this work is two folds: first, highlighted the importance of informative label smoothing through better bound, and second, proposed one label smoothing with low-rank assumption. It is overall a good paper but there are a few concerns: \n\n1. I didn't go through the theoretic proof in detail but it is not obvious how this theoretic motivation is correlated with the low-rank assumption. It will be better for the authors to highlight more about the theoretic result and the empirical algorithms. \n\n2. The experiments are only conducted on parsing tasks, but the authors have claimed the possibility in other areas. I would suggest the authors conduct one/two more experiments on other tasks, like language modeling with large vocab. If additional experiments are presented, I will increase my score. \n\n3. Low-rank output dimension is a common assumption in many papers and it seems there are some mission citations. Few examples are listed below: \n\nhttps://arxiv.org/abs/1711.03953\nhttp://papers.nips.cc/paper/9723-mixtape-breaking-the-softmax-bottleneck-efficiently\nhttp://papers.nips.cc/paper/7312-sigsoftmax-reanalysis-of-the-softmax-bottleneck\n\n4. In most cases, the low-rank assumption is applied when the vocab size is huge. However, the proposed method requires to have to NxN matrix - S to calculate the L. If the N is 1M, like in language modeling, will it be a bottleneck here? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dZ0PLZuKc4d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve upon label smoothing (LS) by adapting the noise distribution used in LS from uniform to a distribution that better represents the correlation/similarity between the candidate space (types in vocabulary). \n\n-- It proposes to learn the appropriate noise distribution (similarity matrix) during training. This matrix is parametrized via a low-rank approximation which prevents it from collapsing to a diagonal matrix and become ineffective.\n\n-- This approach  introduces more hyperparameters and the tuning of these hyperparameters appears to be non-trivial\n\n-- This approach is empirically compared to vanilla label smoothing and no LS on the task fo semantic parsing which has natural groups of types in the vocabulary.\n\n-- The approach seems to be learning appropriate correlations among the vocabulary items (as seen in the visualization).\n\n-- The improvement over LS is consistent but small on the two semantic parsing datasets. Additionally, uniform LS seems to be hurting performance in many cases when compared to no LS, hence the improvements of the proposed approach over no-LS are modest.\n\n-- The utility of the proposed approach becomes apparent in the few-shot setting where the gains seem significant. More analysis of this would strengthen the paper.\n\n-- The approach also seems to improve model calibration.\n\n-- The theoretical analysis is reasonable but it is unclear about its contribution toward understanding the effect of label smoothing, mainly because of the assumption that the embeddings and most of the network is frozen except the last linear layer which is never the case during training of these models. It is a straightforward extension of analysis provided in prior work and is limited in terms of explaining the effect of the noise distribution/ proposed approach on the optimization of the models.\n\n-- Another straightforward experiment would have been to use prior knowledge to hand-design the noise distribution. For example, a simple baseline would be: manually cluster types based upon their membership as an intent, slot , or a word and use uniform distributions within these groups. It would be interesting to compare how the learned similarity matrix is different form hand-designed prior matrices. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VC_RJjWtQoj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper theoretically analyzes \"label smoothing” (LS)  with PAC-Bayesian bound and motivated from their analysis, proposes a new method: LORAS.  In their theoretical analysis, they identify that the generalization error depends on the smoothing distribution. So they propose to learn the smoothing distribution in LORAS. In doing so, to overcome the computational issues & overfitting to diagonal smoothing, they propose Low-rank approach which seems to be successful. The authors show experimental results on semantic parsing dataset (ATIS, SNIPS, and TOPv2) where the LORAS shows better performance than no LS, LS, and some other SOTA model (with one exception of LORAS  vs. no LS on ATIS)\n\n**Quality**:\n* The writings and organizations of the paper are very clear. Didn’t go into the details of proof steps in the appendix, but seems right. \n* This analysis sets up clear motivations for proposing new method: LORAS. \n* The experiment results are convincing. It might have been even more convincing if LORAS was applied to other tasks to show that it generally works.\n\n**Clarity**:\nI think most parts of the paper was clearly written. Addressing the following questions might paper even more clearer:\n* In section 3, if the authors could simply write the analytical form of probability $p_{t,j}$ as written in Appendix A.1, I think the context of how  $\\lvert\\lvert n-W \\bar\\phi\\rvert\\rvert$ came in eq.(2) will be clearer.\n* [suggestion] When expressing absolute changes in accuracy, perhaps say 2 percent point to differentiate with accuracy performing 2 percent better? (e.g. I am assuming calibration error became 55% smaller (almost half) rather than 55 percent point?)  \n* In section 6. Results, the paper says that \"LORAS consistently out-performs\" vanilla LS & hard targets in “all cases”, but isn’t SA worse than (No LS, LS) in case of RoBERTa & ATIS combination in Table 1? I wasn’t sure whether this was the wrong description or whether the table numbers were wrong. Please fix this later. \n* For Figure 3, ALS on the rightmost figure should become LORAS? \n\nSignificance\n* Pros \n    * The deliveries were clear where I was able to see the motivation of this work: that generalization error (on the upper bound of loss) is bounded by the difference of $n$ and $W\\bar\\phi$. And this also seems to be a contribution of this work.\n    * The low-rank approximation seems like a clever way to resolve both computational issues & overfitting to diagonal matrix. \n    * Experimental results seem to support the success of the proposed method.\n* Cons\n    * This is not a big complaint, but if the paper could have provided strong experiments with other tasks as well, then I think the paper would have become stronger. \n    * Likewise, it would have been great if the authors could have tested on much larger label size (and much smaller label size to check as well) to see whether LORAS can actually handle very large label space as claimed. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1b3xpUWpTtf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes LORAS (low-rank adaptive label smoothing) that learns with soft targets for better generalizing to the latent structure in the label space, and the experiments on three semantic parsing datasets show the improvement over the no smoothing methods.\n\nThe idea is intuitive and reasonable, and the proposed method and proof seem sound. \nAlso, as mentioned in the paper, the approach can better consider the latent structure in the label space and is more suitable for structure prediction tasks.\n\nHowever, there are some concerns about the experiments, so that it is not very convincing about the effectiveness of the proposed method. \nThe conducted experiments include three datasets, ATIS, SNIPS, and TOPv2, all of which are about semantic parsing/natural language understanding.\nThis paper formulates the task as a seq2seq task, where the output sequence contains the intent and slot-value information, but most prior work for ATIS and SNIPS formulated the task as classification and tagging problems and the achieved frame accuracy is higher than the performance shown in this paper. \nFor example, simply using BERT for joint training intent classification and slot filling (https://arxiv.org/pdf/1902.10909.pdf) achieved the frame accuracy of 88.2 in ATIS and 92.8 in SNIPS, which are better than the scores using LORAS reported in Table 1.\nBecause the proposed label smoothing method can be utilized for not only sequence generation tasks but also classification tasks, it would be more convincing if the results can be directly compared with the prior work. \nAuthors are suggested add the proposed LORAS on the existing SOTA models in order to better convince the readers.\n\nAnother concern is about the evaluation metric used in the paper. Frame accuracy is common for the task, but semantic accuracy seems not to be used in the prior work. \nFrom my perspective, evaluating the performance based on semantic accuracy simplifies the task, and the classification models should easily achieve better performance compared to the generation based methods.\nThis also implies that the authors should perform classification models in addition to generation-based models.\n\nMoreover, in TOPv2, the paper adds LORAS on BART and show the small improvement and mentions that the results are comparable to the meta-learning method designed for domain adaptation. Is it possible to add the proposed LORAS above the model proposed by Chen et al. (2020) and further improve the performance?\n\nAnother issue to be addressed is that the datasets this paper uses are not well-known structure prediction task, because ATIS and SNIPS only contain very flat structures for the semantic frames, which may not be suitable to demonstrate the effectiveness of the proposed method for structure prediction.\nOther structure prediction datasets can be included in the experiments, and probably the improvement can be more significant due to the complex latent structure in the label space.\n\nIn sum, this paper proposes a sound method for label smoothing and claims that it can benefit the generalization capability based on learning the latent structure in the label space. The experiments on semantic parsing are not very convincing, because the results cannot directly compare with the prior work's or the improvement is relatively subtle.\nTo better align with the claim, structure prediction datasets should be considered in the paper.\n\nAfter reading the responses and checking the additional experiments, I changed the score for this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes LORAS (low-rank adaptive label smoothing) for training with soft targets with the goal of improving performance and calibration for neural networks. The authors derive PAC-Bayesian generalization bounds for label smoothing and show that the generalization error depends on choice of the noise (smoothing) distribution. Empirical results demonstrate the effectiveness of the approach. All reviewers recommend acceptance.
Decision: Accept (Poster)