Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to improve upon label smoothing (LS) by adapting the noise distribution used in LS from uniform to a distribution that better represents the correlation/similarity between the candidate space (types in vocabulary). 

-- It proposes to learn the appropriate noise distribution (similarity matrix) during training. This matrix is parametrized via a low-rank approximation which prevents it from collapsing to a diagonal matrix and become ineffective.

-- This approach  introduces more hyperparameters and the tuning of these hyperparameters appears to be non-trivial

-- This approach is empirically compared to vanilla label smoothing and no LS on the task fo semantic parsing which has natural groups of types in the vocabulary.

-- The approach seems to be learning appropriate correlations among the vocabulary items (as seen in the visualization).

-- The improvement over LS is consistent but small on the two semantic parsing datasets. Additionally, uniform LS seems to be hurting performance in many cases when compared to no LS, hence the improvements of the proposed approach over no-LS are modest.

-- The utility of the proposed approach becomes apparent in the few-shot setting where the gains seem significant. More analysis of this would strengthen the paper.

-- The approach also seems to improve model calibration.

-- The theoretical analysis is reasonable but it is unclear about its contribution toward understanding the effect of label smoothing, mainly because of the assumption that the embeddings and most of the network is frozen except the last linear layer which is never the case during training of these models. It is a straightforward extension of analysis provided in prior work and is limited in terms of explaining the effect of the noise distribution/ proposed approach on the optimization of the models.

-- Another straightforward experiment would have been to use prior knowledge to hand-design the noise distribution. For example, a simple baseline would be: manually cluster types based upon their membership as an intent, slot , or a word and use uniform distributions within these groups. It would be interesting to compare how the learned similarity matrix is different form hand-designed prior matrices. 
review_title: Official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes LORAS (low-rank adaptive label smoothing) that learns with soft targets for better generalizing to the latent structure in the label space, and the experiments on three semantic parsing datasets show the improvement over the no smoothing methods.

The idea is intuitive and reasonable, and the proposed method and proof seem sound. 
Also, as mentioned in the paper, the approach can better consider the latent structure in the label space and is more suitable for structure prediction tasks.

However, there are some concerns about the experiments, so that it is not very convincing about the effectiveness of the proposed method. 
The conducted experiments include three datasets, ATIS, SNIPS, and TOPv2, all of which are about semantic parsing/natural language understanding.
This paper formulates the task as a seq2seq task, where the output sequence contains the intent and slot-value information, but most prior work for ATIS and SNIPS formulated the task as classification and tagging problems and the achieved frame accuracy is higher than the performance shown in this paper. 
For example, simply using BERT for joint training intent classification and slot filling (https://arxiv.org/pdf/1902.10909.pdf) achieved the frame accuracy of 88.2 in ATIS and 92.8 in SNIPS, which are better than the scores using LORAS reported in Table 1.
Because the proposed label smoothing method can be utilized for not only sequence generation tasks but also classification tasks, it would be more convincing if the results can be directly compared with the prior work. 
Authors are suggested add the proposed LORAS on the existing SOTA models in order to better convince the readers.

Another concern is about the evaluation metric used in the paper. Frame accuracy is common for the task, but semantic accuracy seems not to be used in the prior work. 
From my perspective, evaluating the performance based on semantic accuracy simplifies the task, and the classification models should easily achieve better performance compared to the generation based methods.
This also implies that the authors should perform classification models in addition to generation-based models.

Moreover, in TOPv2, the paper adds LORAS on BART and show the small improvement and mentions that the results are comparable to the meta-learning method designed for domain adaptation. Is it possible to add the proposed LORAS above the model proposed by Chen et al. (2020) and further improve the performance?

Another issue to be addressed is that the datasets this paper uses are not well-known structure prediction task, because ATIS and SNIPS only contain very flat structures for the semantic frames, which may not be suitable to demonstrate the effectiveness of the proposed method for structure prediction.
Other structure prediction datasets can be included in the experiments, and probably the improvement can be more significant due to the complex latent structure in the label space.

In sum, this paper proposes a sound method for label smoothing and claims that it can benefit the generalization capability based on learning the latent structure in the label space. The experiments on semantic parsing are not very convincing, because the results cannot directly compare with the prior work's or the improvement is relatively subtle.
To better align with the claim, structure prediction datasets should be considered in the paper.

After reading the responses and checking the additional experiments, I changed the score for this paper.
review_title: Sound approach but not very convincing experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a label-smoothing method upon the low-rank assumption of the output dimension, especially when the output dimension is large. The contribution of this work is two folds: first, highlighted the importance of informative label smoothing through better bound, and second, proposed one label smoothing with low-rank assumption. It is overall a good paper but there are a few concerns: 

1. I didn't go through the theoretic proof in detail but it is not obvious how this theoretic motivation is correlated with the low-rank assumption. It will be better for the authors to highlight more about the theoretic result and the empirical algorithms. 

2. The experiments are only conducted on parsing tasks, but the authors have claimed the possibility in other areas. I would suggest the authors conduct one/two more experiments on other tasks, like language modeling with large vocab. If additional experiments are presented, I will increase my score. 

3. Low-rank output dimension is a common assumption in many papers and it seems there are some mission citations. Few examples are listed below: 

https://arxiv.org/abs/1711.03953
http://papers.nips.cc/paper/9723-mixtape-breaking-the-softmax-bottleneck-efficiently
http://papers.nips.cc/paper/7312-sigsoftmax-reanalysis-of-the-softmax-bottleneck

4. In most cases, the low-rank assumption is applied when the vocab size is huge. However, the proposed method requires to have to NxN matrix - S to calculate the L. If the N is 1M, like in language modeling, will it be a bottleneck here? 


review_title: Boost the parsing through low-rank label smoothing

final_score: rating: 7: Good paper, accept
review_content: This paper theoretically analyzes "label smoothing” (LS)  with PAC-Bayesian bound and motivated from their analysis, proposes a new method: LORAS.  In their theoretical analysis, they identify that the generalization error depends on the smoothing distribution. So they propose to learn the smoothing distribution in LORAS. In doing so, to overcome the computational issues & overfitting to diagonal smoothing, they propose Low-rank approach which seems to be successful. The authors show experimental results on semantic parsing dataset (ATIS, SNIPS, and TOPv2) where the LORAS shows better performance than no LS, LS, and some other SOTA model (with one exception of LORAS  vs. no LS on ATIS)

**Quality**:
* The writings and organizations of the paper are very clear. Didn’t go into the details of proof steps in the appendix, but seems right. 
* This analysis sets up clear motivations for proposing new method: LORAS. 
* The experiment results are convincing. It might have been even more convincing if LORAS was applied to other tasks to show that it generally works.

**Clarity**:
I think most parts of the paper was clearly written. Addressing the following questions might paper even more clearer:
* In section 3, if the authors could simply write the analytical form of probability $p_{t,j}$ as written in Appendix A.1, I think the context of how  $\lvert\lvert n-W \bar\phi\rvert\rvert$ came in eq.(2) will be clearer.
* [suggestion] When expressing absolute changes in accuracy, perhaps say 2 percent point to differentiate with accuracy performing 2 percent better? (e.g. I am assuming calibration error became 55% smaller (almost half) rather than 55 percent point?)  
* In section 6. Results, the paper says that "LORAS consistently out-performs" vanilla LS & hard targets in “all cases”, but isn’t SA worse than (No LS, LS) in case of RoBERTa & ATIS combination in Table 1? I wasn’t sure whether this was the wrong description or whether the table numbers were wrong. Please fix this later. 
* For Figure 3, ALS on the rightmost figure should become LORAS? 

Significance
* Pros 
    * The deliveries were clear where I was able to see the motivation of this work: that generalization error (on the upper bound of loss) is bounded by the difference of $n$ and $W\bar\phi$. And this also seems to be a contribution of this work.
    * The low-rank approximation seems like a clever way to resolve both computational issues & overfitting to diagonal matrix. 
    * Experimental results seem to support the success of the proposed method.
* Cons
    * This is not a big complaint, but if the paper could have provided strong experiments with other tasks as well, then I think the paper would have become stronger. 
    * Likewise, it would have been great if the authors could have tested on much larger label size (and much smaller label size to check as well) to see whether LORAS can actually handle very large label space as claimed. 

review_title: Official Blind Review #4

Meta-review: The paper proposes LORAS (low-rank adaptive label smoothing) for training with soft targets with the goal of improving performance and calibration for neural networks. The authors derive PAC-Bayesian generalization bounds for label smoothing and show that the generalization error depends on choice of the noise (smoothing) distribution. Empirical results demonstrate the effectiveness of the approach. All reviewers recommend acceptance.
Decision: Accept (Poster)