Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Remarking that language models can assign high probability to low “quality” outputs, this paper proposes “sequence likelihood calibration” (SLiC) as a solution. With SLiC, “decoding heuristics become unnecessary” and “quality significantly improves.” The claims are empirically supported by experiments on abstractive summarization, question generation, abstractive question answering, and data-to-text generation, with results that “exceed or match SOTA results.”

In more detail, the high-level idea proceeds in two stages. In the first stage, for each training instance, an initial model is used to sample m candidates (a variety of decoding methods are compared, including beam search and nucleus sampling). In the second stage, the initial model is further optimised using a combination of two losses:

A “calibration” loss which aims to align the sequence likelihood with the similarity to the reference target sequence. Four such losses are considered, including a ranking loss, margin loss, listwise ranking loss, and expected reward. All losses rely on a notion of similarity of a candidate sequence to the reference sequence, for which a variation of BERTScore is proposed (S2.1).
A “regularization” loss which penalizes deviations from the token-level confidences from the stage-one model. The cross-entropy to the reference labels and the KL divergence from the stage-one model predictive distribution are both considered.

Overall, beam search using the rank “calibration” loss with KL divergence “regularization” loss are found to work best.
strength_and_weaknesses: Overall, it is notable that the proposed approach yields models that do not appear to require decoding heuristics; this suggests that the calibration loss is doing something useful. Figure 3 is also fairly convincing in this respect, as it shows a distinct difference between the vanilla model and the calibrated one when further decoding samples are used (consistent improvement for the calibrated model). However, the experiments overall are not entirely convincing, since in several cases the degree of improvement over the baselines is marginal, while the proposed model is larger than the baselines in terms of # of parameters (2B). So it’s not clear that it’s the new calibration scheme that’s responsible for the improvement in these cases.

The scoring function (S2.1) is a key element of the proposal. It’s stated that “compared to using external metrics, [...] it differs from the metrics we evaluate the generation systems with and mitigates the risk of directly optimising towards imperfect metrics.” However, the proposed similarity is related to both ROUGE and BERTScore, and so it’s not clear there isn’t some bias towards better performance on the target metrics.

Finally, I would have liked to see probabilistic calibration addressed somewhere. Does the proposed calibration loss actually lead to more meaningful probability estimates? This could be measured at the token-level or on other events (cf https://papers.nips.cc/paper/2015/file/52d2752b150f9c35ccb6869cbf074e48-Paper.pdf)
clarity,_quality,_novelty_and_reproducibility: Regarding the presentation, although the approach itself is quite straightforward, I found it difficult to follow the narrative of the paper. At various points, such as the introduction of the calibration losses, many options are considered without proper motivation or discussion of relative merits to justify their inclusion in the experiments. I found the use of “TL;DR;” blocks at the end of certain chapters to be out-of-place; it would have been better to provide this kind of information at the beginning of the sections, ideally framed as research questions.

On related work, the approach is conceptually similar to knowledge distillation, and would benefit from comparing and contrasting both the original Hinton paper (when it comes to the KL regularization loss) and the sequence-level extension (https://arxiv.org/abs/1606.07947) when it comes to the use of sequence-level training. Some other relevant work is not cited, such as papers in non-autoregressive decoding where it is common to train on decoded outputs from an initial “teacher” model.

On reproducibility, the appendix provides many details about hyperparameter choices. On the other hand, there’s no obvious intent to release code, and some of the model sizes considered are prohibitive in terms of compute costs for many researchers.

In Figure 3, what is the “score” in the y-axis?
summary_of_the_review: There are some good ideas here but the presentation could be improved and the experimental results are lackluster.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To tackle the issue of uncalibrated sequence likelihood, this paper proposes a new sequence likelihood calibration(SLiC) stage, which extends the common paradigm of pretraining and finetuning. The specific method is to calibrate with KL divergence and low rank after selecting the checkpoint of finetuned model by perplexity and decoding candidates with beam search. 

With SLiC, decoding candidates’ quality significantly improves regardless of the decoding method, without showing any sign of diminishing returns with model scale, and former decoding heuristics will be never used any more. Furthermore, this paper proposes a novel calibration similarity metric between model decodes and targets, which is much better than the existing external metrics. 
strength_and_weaknesses: Strengths:

- The contributions of the SLiC are important, and do not exist in prior works. Besides proposing the SLiC, the authors also proposed a novel calibration similarity metric and used many past outstanding works as candidate decoding methods.

- Also, calibration benefits persist as model sizes scale up, so it has great potential for models plagued by former approaches which shows diminishing by scaling up. Also, small models can add calibration step to outperform larger ones. length normalization has minimal effect on calibrated models.

- Calibrated models do not require decoding heuristics and more. Only with the SLiC, the model reaches SOTA result and even better than the model with decoding heuristics, and such heuristics has minimal effect on calibrated models. 

- Extensive experiments have been conducted to validate the effectiveness and relevance of the proposed method. Also, there are a lot of experiments in ablation study to get a simple and effective recipe for training.


Weaknesses:

- (1) typo in page 14, “WebNLG-en” paragraph, “data inputs in the from of sets” -> “data inputs in the form of sets” (2) typo in page 14, “CommonGen” paragraph, “introduces a tak” -> “introduces a task”

- The experimentation section mainly convinces me that this architecture is good for application. However, this work lacks of explanation on the reason of the choice of calibration loss, regularization loss, candidates decoding method, etc.

- Since this paper is highly theoretical and original. However, it suffers the problem that it does not make sense to many details in the method chapter. For example, as for the calibration loss and regularization loss, the literal part seems only notation, and lack of the reason for why taking these methods and the advantages of them. (Maybe it is included in the experiment part. However, I do think that it should also be mentioned in the method part.) 

- Lack introduction of some notations. For example, I haven't found out what theta means so far. Although it is default to represent model parameters, I think it should be explained. What’s more, due to the rich content, this paper involves large amounts of parameters. Therefore, I think it is a good idea to explain the parameters in a chart. Maybe it can be appended to the appendix. In this way, the readers can follow the authors more easily.

- The description of MLE method and its related works occupied a little too much space in introduction, while the motivation to proposed method was little.

- In the section of ABLATION STUDIES OF CALIBRATION, the authors show us the improvement compared with fine-tuned models,but without the comparison between SLic and the fine-tuned model using the mentioned effective heuristics or solutions.

-  The authors propose the Calibration stage after Pre-train and Fine-tune(MLE) stage. Since fig. 3 gives the inference compute trade-off between fine-tuned only and calibrated models, the  comparison of the running time and speed between fine-tuned only and calibrated in Table 3 is lacked.

Suggestions:

- The experiment results of the proposed method should be in bold.

- The conclusion part lacks the description of the deficiencies and the prospects for future improvements.

Questions:

- There are in total 3 hyperparameters: β in calibration，lr as learning rate and λ as regularization strength. To simplify the training process, β and lr∗λ are set empirically, making the process tedious, and how the values are set in the paper?
clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: The paper proposes a novel stage called sequence likelihood calibration after the pretraining and fine-tuning stages for conditional language generation. It seems that this method has not been used in prior works and with such stage the model can outperform the SOTA results without concerning about the diminishing problems. Also the entire paper is very well-written and comprehensible, and the experimentation in this paper are quite rich.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes sequence likelihood calibration (SLiC) for conditional language generation. Their proposal includes a first stage where candidates are generated from a fine-tuned model, and a second one, where they calibrate the previous model by continuing training it with a new objective that aims to align candidates’ sequence likelihoods by similarity to the target sequence in the model’s latent space. They perform experiments on abstractive summarization, question answering, and data-to-text generation tasks. 
strength_and_weaknesses: **Strengths**

* The motivation is clear, the paper is well written and is easy to follow.
* The proposed method does not directly optimize an evaluation metric (e.g., ROUGE), and thus it’s more likely to actually improve quality rather than exploiting fragilities in the evaluation metrics.
* SLiC seems to work well without beam size optimization and length normalization.  

**Weaknesses**

* I’m assuming that you’re using your proposed similarity function for all the experiments, excluding the ablation studies in $\S3.3$, where you also experiment with ROUGE. First, I think you should clarify that in the paper. Since the ablation studies appear before everything else, the “simple recipe” of that section could include the similarity function to use. Second, Table 1 suggests similar performance for both ROUGE and your method, and thus I’d be interested in seeing a more extensive comparison of these approaches. 

**Minor comments/suggestions**

* For neural machine translation, specifically, there are several works proposing alternative decoding methods that try to generate high quality outputs, related to the approach of Lee et al. (2021), discussed in $\S4.2$, which I think are relevant for your work. Some of them are [1], [2], [3], and [4]. Besides, related to the work of Edunov et al. (2018), discussed in $\S4.3$, is [5] and [6]. 
* In $\S2.2.$, you should explicitly say which one “is the contrastive loss used in BRIO (Liu et al., 2022).”

**Other questions**

* Is there any reason to vary the hyperparameters between groups in Table 1? In particular, I see in App. C that the loss you used is different (sometimes *reward*, sometimes *rank*). 
* How does your method compare to common approaches for conditional language generation in terms of memory consumption and training time? The paper would benefit from such analysis. 

[1] Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation, Eikema and Aziz, COLING 2020

[2] Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models, Sumanta Bhattacharyya et al., ACL 2021

[3] Quality-Aware Decoding for Neural Machine Translation, Fernandes et al., NAACL 2022

[4] High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics, Freitag et al., TACL 2022

[5] Beyond BLEU: training neural machine translation with semantic similarity, Wieting et al., ACL 2019

[6] Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation, Kiegeland and Kreutzer, NAACL 2021. 


clarity,_quality,_novelty_and_reproducibility: **Clarity.** The paper is well organized and clear. 

**Quality.** See Strengths/Weakness section above. 

**Novelty.** The method is related to the work of Liu et al. (2022) but the proposed similarity function appears to be novel.

**Reproducibility.** I believe there is no supplementary material/code. Are you planning to release the code?

summary_of_the_review: The paper addresses an important issue with conditional language generation models, that of uncalibrated sequence likelihood: for a given context, they are often trained with a single target sequence, and thus they lack supervision to compare (high probability) plausible hypotheses. The proposed method is interesting and the authors perform experiments that validate its applicability on different conditional language generation tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The authors propose to use latent decoder embeddings of proposed candidates in comparison to target embeddings so as to fine-tune model likelihoods. This "calibration" step operates on the same fine-tuning data and comes after the regular pre-training and fine-tuning steps. The motivation is to improve decoding qualify as well as to dispense with the need for (many) standard decoding heuristics such as length normalization or beam size selection. The proposed calibration loss consists of alignment scores that involve either rank, margin or reward based scores, each seeking to enforce better correspondence between similarity and likelihood. The calibration loss is accompanied with regularization terms, either cross-entropy or a sample-KL,  so as to keep the model anchored on the basic fine-tuned model. The technical contribution of the paper is somewhat low for ML conferences. The proposed fix, while intuitive, is somewhat heuristic itself and doesn't come with any crisp theoretical or technical insights / arguments in support. That said, empirical validation seems comprehensive across challenging problems, reaching SOTA. The authors explore how proposed alignment methods (e.g., whether rank, margin, or reward based) or the choice of the regularization term impact evaluation scores. For example, after alignment, beam size consistently improves performance. The proposed "calibration" recipe also seems superior to fine-tuning in terms of compute times. The value of the paper rests primarily in its empirical results. 

justification_for_why_not_higher_score: The technical contribution of the paper is somewhat low for ML conferences. The proposed fix, while intuitive, is somewhat heuristic itself and doesn't come with any crisp theoretical or technical insights / arguments in support.
justification_for_why_not_lower_score: Empirical validation seems comprehensive across challenging problems, reaching SOTA. The authors explore how proposed alignment methods (e.g., whether rank, margin, or reward based) or the choice of the regularization term impact evaluation scores. For example, after alignment, beam size consistently improves performance. The proposed "calibration" recipe also seems superior to fine-tuning in terms of compute times. The value of the paper rests primarily in its empirical results.
Decision: Accept: poster