Calibrating Sequence likelihood Improves Conditional Language Generation
Authors: Yao Zhao
Mikhail Khalman
Rishabh Joshi
Shashi Narayan
Mohammad Saleh
Peter J Liu
Venue: ICLR 2023 poster (2023)
Keywords: Natural Language Processing
conditional language models
sequence-to-sequence
text generation

Abstract: Conditional language models are predominantly trained with maximum likelihood estimation (MLE), giving probability mass to sparsely observed target sequences. While MLE trained models assign high probability to plausible sequences given the context, the model probabilities often do not accurately rank-order generated sequences by quality.  This has been empirically observed in beam search decoding as output quality degrading with large beam sizes,  and decoding strategies benefiting from heuristics such as length normalization and repetition-blocking. In this work, we introduce sequence likelihood calibration (SLiC) where the likelihood of model generated sequences are calibrated to better align with reference sequences in the model’s latent space.  With SLiC, decoding heuristics become unnecessary and decoding candidates’ quality significantly improves regardless of the decoding method. Furthermore, SLiC shows no sign of diminishing returns with model scale, and presents alternative ways to improve quality with limited training and inference budgets. With SLiC, we exceed or match SOTA results on a wide range of generation tasks spanning abstractive summarization, question generation, abstractive question answering and data-to-text generation, even with modest-sized models.

Published as a conference paper at ICLR 2023
CALIBRATING SEQUENCE LIKELIHOOD IMPROVES CONDITIONAL LANGUAGE GENERATION
Yao Zhao
Misha Khalman khal Taalmmn@g e com
Rishabh Joshi
yaczhacyy6@ooglee.com
ri rishabhjos joshi @googl com
Shashi Narayan
Mohammad Saleh ms msalchegoogle. com
Peter J. Liu
shathinarayan@goolle..om
peterjliu@google.. com
Google Research, Brain Team
ABSTRACT
Conditional language models are predominantly trained with maximum likelihood estimation (MLE), giving probability mass to sparsely observed target sequences. While MLE trained models assign high probability to plausible sequences given the context, the model probabilities often do not accurately rank-order generated sequences by quality. This has been empirically observed in beam search decod- ing as output quality degrading with large beam sizes, and decoding strategies benefiting from heuristics such as length normalization and iopetition-blokiii In this work, we introduce sequence likelihood calibration (SLiC) where the like- lihood of model generated sequences are calibrated to better align with reference sequences the model's latent space. With SLiC, decoding heuristics become unnecessary and decoding candidates' quality significantly improves regardless the decoding method. Furthermore, SLiC shows no sign of diminishing returns with model scale, and presents alternative ways to improve quality with limited training and inference budgets. With SLiC, we exceed or match SOTA results wide range of generation tasks spanning abstractive summarization, question generation, abstractive question answering and data-to-text generation, even with modest-sized models.
INTRODUCTION
Conditional language generation aims generate text based input context, and includes many useful and hard tasks such as abstractive summarization (Mani] [001]] Nenkova and McKeown 2011), generative question answering (Bajaj al., 2016). question generation (Zhou al.) 2017) and data-to-text (Wiseman et al.) [0117] Gardent et al. [2017). Pretraining large Transformer encoder- decoder models and fine-tuning them on downstream tasks is the common paradigm address these
tasks (Raffel et al. 2020 Lewis et al. 2019: Tay et al. 2022 Zhang et al. 2019a).
Conditional language generation tasks are modeled by learning the probability of a target se- quence given context sequence x. Since directly modeling sequence probability P(y|x) over all possible generated text sequences intractable, the canonical solution to auto-regressively factor the probability and share the parameters at all token prediction steps as Po(y)x) (o "'..."""-1xx,, where is the sequence length. These models are often trained with maximum likelihood estimation (MLE) over observed target sequences. The learning objective thus becomes L Σ" -log (x,))x,, EN E'= log Po Po(v/lv/ 1/ x4), where N is the num- ber of training instances. It is also referred to as next token prediction loss.
In the ideal setting of MLE training, a large number of target sequences are observed for each con- text, and the relative frequencies of output sequences can calibrate the assigned model probabilities. However, in practice most language generation training datasets have only single target sequence given the context. While the subsequent MLE trained models learn to assign relatively high proba- bility plausible sequences, they lack the direct supervision compare such sequences, and solely

---

Published as a conference paper at ICLR 2023
37
Fineluned
Calbrated
35-
Celibration
Pretrain
Fine-tune (MLE)
decode candidates
align likelihad
Average
33-
using
SOM
200M
500M
20
Model Size
Figure 1:
Calibrating sequence likelihood improves language generation across model scales. Scores are averaged ROUGE across 4 datasets (Rm, in subsection 3.2)
rely on models' generalization capability. We refer to this phenomenon models' sequence like- lihood not being calibrated. Prior works (Liu and Liu 2021 I Liu et al., 2022) has shown that the correlation between sequence probability and its quality for MLE trained models can be low. Liu itt al. (2022) attributed this similarly as the deterministic (one-point) target distribution problem. Ex- posure bias (Ranzato al. 2016) further aggravates the problem, sequence likelihood estimation is noisier when models' decoded sequences shift from exposed training data distribution.
Many effective heuristics have been proposed during training and decoding combat the problem of uncalibrated sequence likelihood. Label smoothing (Szegedy et al. (2016) prevents the network from becoming over-confident towards the observed target. This is particularly necessary language generation, since the gold target represents just one of many possibilities. It has been observed that increasing number of decoding candidates past certain point leads to worse quality for beam search decoding (Yang et al. 2018 Koehn and Knowles (017] and sampling (Adiwardana et al., 2020). An optimal number of decoding candidates is often determined empirically by decoding models on the validation set and measuring their performance. Using length normalization is also essential for beam search decoding (Wu et al.] (0166) and sampling (Adiwardana et al.) 2020) as models tend underestimate sequence likelihood of longer sentences. Repetition is another common failure mode when models overestimate the probability of repeated sequences (Holtzman et al. 2019). Tri- gram blocking (Paulus et al. 2018) and nucleus sampling (Holtzman et al.) 2020) have been used to interrupt repeating sequences. These techniques are pervasive and often the default in modern
Transformer libraries (Wolf et al. 2020 Lewis al. 2019 Raffel et al., 2020 Zhang et al. 2019a).
Since the lack of observed target sequences in MLE training is the root problem, solutions involving learning with multiple sequence candidates have been proposed to directly address it. They can be loosely put in three categories: (1) reinforcement learning with sequence-level rewards (Paulus
al., 2018 Ziegler et al.] 2019 Stiennon et al. 2020g; (2) two-stage systems that generate and rerank candidates (Liu and Liu 2021; Ravaut et al. 2022b Liu et al. 2022): and (3) multi-task
learning with sequence-level losses (Edunov et al. 2018 Liu al. 2022). Refer to Related Works
(section 4) for more comprehensive discussion.
In this paper, we propose first decode candidates from a fine-tuned model on its own training dataset, and then continue training the model with align candidates' sequence likelihoods according to their similarities to the target sequence in the model's latent space. We refer to this process sequence likelihood calibration (SLiC). Our approach is related to multi-task learning with sequence-leve losses in Liu et al. (022). However, propose simple yet effective recipe that eliminates decoding heuristics and doesn trisk directly optimizing the same metrics that are used to report text generation quality. Unlike reinforcement learning,
objective. The new objective aims to
one-time offline process that avoids costly online decoding processes. Also, when compared to two-stage reranking systems, it doesn' require separate reranking model that incurs additional complexity and compute. depicted in Figure our calibration stage naturally extends the current paradigm of pretraining and fine-tuning and we show that calibrated models have strong improvements over fine-tuned-only models across model sizes.
Our main contributions include:
Proposed sequence likelihood calibration (SLiC) stage that consistently improves model qual- ity, exceeding or matching state-of-the-an results on abstractive summarization, generative question answering, question generation and data-to-text generation tasks.
Proposed a novel calibration similarity metric between model decodes and targets measured in the model's s latent space rather than resorting external metrics human feedback.
2

---

Published as a conference paper at ICLR 2023
Demonstrated that SLiC eliminates the need for popular decoding heuristics, such beam size optimization, length normalization and repetition prevention for the calibrated models.
Demonstrated that SLiC has persistent significant benefits on model performance even as the number of model parameters scales up. Under the same inference budget, smaller calibrated models might outperform larger counterparts by decoding more candidates.
2 CALIBRATING SEQUENCE LIKELIHOOD
We extend the common paradigm of pretraining and fine-tuning by introducing a third calibration stage, SLiC. As shown in Algorithm 1 we first decode candidates (5) from a fine-tuned model Po,, (y|x) on fine-tuning dataset yh and then calibrate the fine-tuned model by continuing training on our proposed loss: L()) = = (0, (00,,,88 8; x, {8) and Off are the current and fine-tuned-only model weights, I and I,eg are the calibration and regularization losses. s( y: measures the similarity between the candidate ý and the target y conditioned on the context x. We discuss choices of s, Leai, L and decode strategies y ~ (o(y|x) x) in the following sections.
{})m) ) + * x, where
Algorithm Calibrating Sequence Likelihood
for x, E (x. do
> sample m candidates from the fine-tuned model
{9 Poy, (y(x))m
x 8
> initialized from the fine-tuned model train with calibration and regularization loss
for {x, y, {8) {x, mm do + 0 - (rV))())
2.1
SIMILARITY FUNCTION:
For given output sequence y, we take the decoder output hidden states e = emb(y, x) as its representations, where L is the number of tokens and / is the hidden states dimension. Between a candidate y's representations è and the target y's representations e, we calculate their cosine similarities spans of n tokens and aggregate them across the sequences with F-measured based function F,. Notation of Fn, Pa. Rr are same as BERTScore (Zhang et al., 2019b).
Pn x Ra Fa Pp Rr
si = Σ er = Σ Fa (embby x), emb(y, x))
max èT eyjtn
T max eiitn in Grsttn
Gjitn
Coitte
0jsstm
Compared to BERTScore, we use our models' decoder output representations instead of BERT encoder representations and also consider matching spans of n 1, 4, 8 tokens rather than 1.
Compared to using external metrics, such as ROUGE, BERTScore, this scoring function has a few advantages: (1) it adds very little compute cost, does not require extra model out-of-graph com- putation; (2) differs from the metrics that we evaluate the generation systems with and mitigates the risk of directly optimizing towards those imperfect metrics (Paulus et al.) 2018 Stiennon al. 2020]; (3) it is conditioned on the context s(y, y; x), as opposed to metrics in the form of s(y,
2.2
CALIBRATION Loss: Tecal
The calibration loss T.cal (0, x, {f]m) aims to align models' decoded candidates' sequence like- lihood P5 xx) according to their similarity with the target sequence s(ý, ,; x). Given the context target and set of candidates ). we consider the following 4 loss types to answer two questions: (1) does absolute difference in similarities matter? (2) is there benefit of list-wise over pair-wise wise comparisons? Rank loss optimizes the ranking order of positive and negative candidates pairs y *.... y_ uniformly sampled from (ý hm where s(ý+, y;x) > s(y y; x). Margin loss max- imizes the sequence probability gap of positive and negative candidates pairs. List-wise rank loss
3

---

Published as a conference paper at ICLR 2023
optimizes the ranking orders of list of candidates, where i, positions of yı, Y; in the set {ý} sorted by s y: List-wise rank loss is the contrastive loss used BRIO (Liu et al. 2022). Expected reward loss (or expected minimum risk) maximizes the expected similarity of a list of candidates (Edunov et al., 2018). Pair-wise losses (Rank, Margin) has smaller training memory footprint than list-wise rank and expected reward.
"rank max(0, log Po(y - (x) + log
"margin ((s(
(;x)) log (a +xx) + log Po($-Ix))
List rank
=
Σiej max (0, - log (o( + log Po (y, xx)
(1)
(o(
cal rewardd
= Σ, - (f Σ; Po(
values for all losses are chosen empirically for each loss type in subsection 3.3
2.3 REGULARIZATION Loss: L
We consider two alternate types of regularization loss - to prevent models from deviating sig- nificantly from their fine-ture tne-tuned MLE objective: Cross entropy is the standard fine-tuning MLE objective used in (Liu et al. 2022). KL divergence directly minimizes the probability distribution distance between the calibrated model and the fine-tuned model at each token on observed target sequence. The main difference is cross entropy loss regularizes the model toward the gold reference while KI. divergence regularize the model toward fine-tuned-only model. The regularization losses both token level.
Ps Po(Be(F(11.x)
L reg Σ log St-1,,)
L
Σ Po(Tolye-1,x) log
(2)
Po, (Th. (1-1,,)
2.4 CANDIDATES DECODING METHIOSS
We consider the following decoding methods for SLiC:
Beam Search the standard best-first algorithm to solve the intractable maximum likelihood opti-
mization for ceeueenc-to-sequeeee models (Tillmann and Ney! 2003 [ al. 2016: Wiseman al.,
2017
Chen et al. 2018).
Diverse Beam Search (DBS; Vijayakumar al. 2016) generates list of diverse outputs by di- viding the beam search budget into groups and enforcing dissimilarity between groups of beams. strikes balance between quality and diversity and often the best strategy for two-stage reranking systems (Liu and Liu 2021 Ravaut et al. 2022b Liu et al. (022).
Nucleus Sampling (Holtzman al. 2020) only samples high- high-probable tokens within cumulative probability p at each step of the decoding. It produces diverse candidates while preventing sampling very low quality ones.
EXPERIMENTS
3.1 TASKS AND DATASETS
For abstractive summarization tasks, we choose CNN/DailyMail (Hermann al., 2015 See al., 2017). XSUM (Narayan et al.] 2018), ReddieTIFU-long (Kim et al.) 2019) and SAMSum (Gliwa ctt al. 2019) due to their diversity in domain, style, abstractiveness, and summary lengths. For ques- tion answering related tasks, we choose generative question answering given context MSMARCO NLG (Bajaj et al. 2016) and its reverse problem of question generation SQuAD QG (Zhou et al.) Du al. 2017) For data-to-text tasks, choose text generation given structured data (Gardent et al. 2017) and common concepts reasoning CommonGen (Lin et al, More details of datasets be found at Appendix Alalong with their statisties.
2017
2020
4

---

Published as a conference paper at ICLR 2023
3.2 MODEL TRAINING AND EVALUATION DETAILS
We follow the PEGASUS pretraining (Zhang et al. 2019a> and extend transformer model sizes to PEGASUSSMALI (50M), PEGASUSBASE (200M), TEGASUSLARGE (500M) and PEGASUS (2B). Details are reported in Appendix B] Different from the original paper, we use a sentencepiece 96k vocabulary with byte-fallback (Kudo, 20188) and pretraining batch size of 4096 across all models. See Appendix B for model dimensions.
In all experiments, we use learning rate Ir 10-4, and batch sizes of 512 to finetune and 64 to calibrate models. We use beam search generate calibration candidates and evaluate the calibrated models, unless specified otherwise. All fine-tume ame-tuned-onll models utilizes heuristics such as beam size optimization and sweeping beam for length normalization, unless specified otherwise.
In our ablation studies (subsection 3.3), benefits analysis (subsection 3.4). and scaling experiments models pretrained to 500,000 steps and conduct experiments
subsection 3.5), (CNN/aalyyMail XSUM, ReddiCTIFU-lnng and SAMSum)). For ablation studies and benefits anal- ysis, we use PEGASUSLARGE: We report ROUGE 1/2/L. (Lin 2004) for each dataset on validation splits and their overall score Rm defined as geometric mean of ROUGE 1/2/L averaged across datasets, Ry Ea RRRR2RL
datasets
For the final results (subsection 3.6), we pretrain PEGASUS{B model to 2.5. steps, fine-tune it on all datasets, calibrate them using the same recipe and report numbers on the test split (unless specified otherwise). We use corresponding standard evaluation scripts for each dataset (details in Appendix D).
3.3 ABLATION STUDIES OF CALIBRATION
Ablation experiment results discussed below can be found in Table
Table 1: Ablation the sequence likelihood calibration method. Shared hyper-parameters are held as constant within each comparison group but vary between groups (Appendix E). A is the relative improvements of overall score Rm, compared with the fine-tuned model. Choices in our recommend recipe are in bold.
Ablation
CNNiDailyMail R1/ RL
XSUM
ReddieTIFU-long R1/R2 RL
SAMSum R1 R2 RL
R1/R2 RL
avg
fine-tuned
44. 44.74/21.83441.22
47,23/44.1/39112
26.84/9.08/21.99
05.67/2.35544.777
0.00%
similarity function
ROUGE
46.47/22.49/43.63
29.92/9.83/23.99
54.82/30. 15/45,30
3.26%
decoder repr token emb
46.55/22.50443.69
77.88/24.62/39..6
29 99.86/9.84/2399
54.72/29.96/45. 96/45.
3.20%
46.51/22.48/43.67
47,04/23 633/3899
29.78/9.69/23.46
53.71/29.38/44.75
1.64%
calibration loss
rank
46.73/22.70/43.88
48. 18.11//4.88/44/
30.34/9.80/24.32
55.
4.27%
margin
46.11/22.44/43.300
47. 47.62/24. 662/24.81/39.89
90.84/9.97/24.37
54.58/30. 10/45,92
3.63%
list rank
03.62/22.88/33.1
47.93/24.57/39.67
54.56/29 81/45.17
3.49%
reward
46.49/22.55/43.3
47.7T/D4.48/39.44
90.99/9.95/24..9
54.42/29 98/45.56
3.47%
regularization loss
none
46.54/22.44/43.68
47.51/24.70/39.88
30.73/9.68/24.05
55.07/30.07/45.0
3.48%
entropy
46.73/22.70443.85
48, 80/4000
29.96/9.72/23.82
15.19/30.46/46.01
4.06%
divergence
46.80/22.83/43.988
47.96/24.92/40.09
30.73/9.68/24.05
54.87/30.20445.9
4.09%
candidates decoding method
beam search
46.50/22.48/43.66
47.82/24.5533.67
31.04/9.96/24.37
54.66/30.27/45.4
3.70%
diverse beam
46.31/22.48/43.47 47.79/24.5333.551
$1.00/9.5//2400
54.57/29 67/45.55
3.26%
nucleus
46.45/22.46/43.54
47.67/24.50/39.47
31 31.09/10.01/24.31
54.61/30.04/45.3
3.51%
calibration checkpoint selection
ROUGE
46.66/22.66/43..6
48.03/24.78/39.79
30.94/9.98/24.43
54.63/30.03/45.799
3.96%
perplexity
47.36/24.02/44.45
47.96/24.4/39.778
31.04/10.08/24.3
54.65/30. (65/30.11/46.00
4.93%
Similarity Function We compare our proposed similarity function, using models' latent states at de- coder output representation so sy(5.:: x) (subsection 2.1p. directly optimizing the evaluation metric ROUGE. They perform similarly on all datasets even when evaluation metrics are ROUGE scores. We also test variant of our similarity function by replacing decoder representation emb(y, x) with
5

---

Published as a conference paper at ICLR 2023
token embeddings. This variant has lower performance, which suggests benefits of contextualized and input-dependent representations.
Calibration Loss Calibrated models with all loss types significantly improve over -une-tuned-onl models. Rank loss performs the best followed by margin, list rank and then reward. Reward maxi- mization has the advantage of no hyper-parameee loss have smaller training memory footprints. Rank loss showing the best gain indicates that relative ordering of candidates is more important than the absolute value of their similarity the target.
3 (Equation 1) to sweep while rank and margin
Regularization Loss Cross entropy and KL. divergence regularization perform similarly. About 85% of the calibration gain remains if regularization is removed.
Calibration Candidates Decoding Method We choose hyper-parameters for calibration candidates decoding methods based on validation set. The optimal decoding method is dataset dependent, however the differences between methods are small and the worst method achieves 90% of the gains of the best one. Beam search yields the highest average quality. This opposite the findings in the two-stage reranking systems (Liu and Liu 2021 Ravaut et al. 2022b Liu et al. 2022). where
more diverse decoding strategies are preferred.
Checkpoint Selection for Fine-tuned Model We compare ROUGE-selected and perplexity- selected checkpoints. The experiments show that starting calibration from the eerllextty-elecced checkpoint yields same or better performance with the biggest gap on CNN/DailyMMil dataset.
TL;DR: We recommend a simple recipe: select the fine-tuned model's checkpoint by its validation set perplexity; decode candidates using beam search; calibrate the model with rank loss (based on decoder states similarities) and KL divergence regularization.
3.4 BENEFITS OF CALIBRATED SEQUENCE LIKELIHOOD
SAMSum
CNN/Daily Mail
XSUM
20
36
36
beam
Scores
41
35
beam beam
nulkss
besm -o nineu
39.
16
34
nucless beam
rucieus nu/ous
33
20
20
20
No. of Candidates
No. of Candidates
No. of Candidates
No. of Candidates
Effect of decoding methods on calibrated and fine-tuned only models. Colors indicate cal- ibration method. Markers indicate evaluation decoding method.
Figure 2:
at Appendixx F
Calibrated models' quality monotonically improves as the number of decoding candidates in- crease regardless of the cuibrrttoo-decoding and celluationdecodiing methods, as shown in Fig- ure 2] On the other hand, fine-tuned-only models suffer from decreased quality when the number of decodes exceeds an optimal value. Once a model is calibrated with either decoding method, per- forms well with both at evaluation time. Decoding with beam search yields higher scores, verified method align, the
up to 20 decodes. When the calibration-deccoiing and the evaluation-decodif final quality is slightly better than the mismatched settings. CNN/Daily XSUM, and SAMSum datasets work best with beam search, however ReddicTIFU-long works better with nucleus sampling and decoding it with larger number of candidates may achieve better results.
Calibrated models do not require length normalization. As shown in Table 2 length normaliza- for beam search) is essential for fine-tuned-only models which
tion (commonly implemented bias towards longer sequences at decoding time. In contrast, length normalization has minimal effect on calibrated models.
Calibrated models suffer from far fewer repetitions. The repetition rate (rep%) measures a com- mon mode of model failures. It is defined as the percentage of examples that contain any kind of consecutive repeated word n- n-grams, While length normalization helps general quality on the fine- tuned-only models, it leads to a side-effect of higher repetitions. Calibrated models, with or without
At -valuation-decoding time, the candidate with the highest sequence probability is selected to compute quality for both beam search and nucleus sampling.

---

Published as a conference paper at ICLR 2023
Table 2: Comparison between fine-tuned only models and calibrated models with or w/o brevity penalty a overall quality (R1 / R2 RL) and repetitions' occurrence percentage (rep%). Hyper-
parameters Appendix G.
SLIC
DNNy
XSUM
ReoditTIFU-long
SAMSum
A
RI
Rep%
rep'S
RL
rep'
R1
zep%
avg
gold refernnce
0.03
0.01
0.09
0.05
19.37/19.67/366 67/36.89
0.03
46.96/24 29/39.19
0.03
26.62/8
911.9911211777 .7
0.26
30.228/7725/42..6
0.00
-5.15%
13474/1111441999 83/11.99 1.92
74/21/211
0.13
47.23/24.31/89.1 12
0.07
26 00.84/9.0/2/ 08/21 .22
0.90
53.67/29.35444,77
0.20
0.00%
X
46. 1/6.4//2..8/33.57 38/43.57
0.02
07.55124444//99.4 46
0.03
30.90 30.99/9.900 95/24.39
0.03
04.42/29.98445.58
0.00
3,31%
46.49/22.55/43.65
47.71/24.48/99.8
30.989.96/24.30
54.64/30.0145.17
length normalization, have much lower repetition rate. When we compare with the repetition rate in the gold reference (repetition may occur naturally), calibrated models without length normaliza- tion have similar or lower repetition rate.
TL;DR: Calibrated models do not require decoding heuristics such as beam size optimization, length normalization and repetition blocking.
3.5
SCALING PROPERTIES OF CALIBRATED MODELS
SAMSum
Reddit ReddirTIFU-long
24
'thetuned
fincuuned
score (R1/R2/RL)
44
score (R1/R//RL)
22
20
40
18
16
500M
2B
36
10
10
14
10
1012 compute (FLOPs)
compute (FLOPs)
Mail
XSUM
41
finctuned
finztuned
calbrated
brakted
score (RI/R//RL)
39
score (R1/R2/RL)
38
37
36
35
34
S0M
33
2004
31
10a1
1012
30
10
1012 compute (FLOPs)
compute (FLOPs)
Figure 3: Quality and inference compute trade-off comparison between fine-tuned only and cali- brated models. Inference compute is scaled by increasing model parameters (different colors) and number decoding candidates (dots on the same line). prreer-prramters at Appenddxx
Scaling properties are important for projecting technique's future relevance as models scale up (Kaplan et al., 2020a). Figure 3, we compare generation quality versus inference compute at different model sizes and number of decoding candidates using beam search. Appendix H describes the method to estimate inference compute FLOPs.
As mentioned earlier in subsection 3.4. fine-tuned-only models have optimal decoding beam sizes while calibrated models' performance monotonically increase with larger decoding beam sizes. Even in the case of greedy decoding (beam size of 1), the calibrated models' performance exceeds the fine-tuned-only models, by large margin for some datasets (CNN/DailyMail and RedditTIFU- long). Their gaps grow larger with increasing number of beam sizes.

---

Published as a conference paper at ICLR 2023
The magnitude of quality improvement from calibration persists over models sizes spanning from 50M to 2B. There is no obvious sign of diminishing return as model size scales up.
Inference compute may be used for decoding rather than on larger models. A calibrated model, once trained, can improve its performance by decoding more candidates, usually more effectively in the beginning, although returns diminish over 10 candidates. In some cases (SAMSum and pecially CNN/DailyMail), a smaller model decoding more candidates can beat a larger one at both quality and efficiency.
TL;DR:
Calibration benefits persist as model sizes scale up. Smaller calibrated models can outper- form larger ones under the same inference compute budget.
3.6 FINAL RESULTS
Table 3:
Calibrated PEGASUS2g, comparing with prior SOTA results: BRIO LLu et al. 2022
ULL Tay et al. 2022 ST-MoE (Zoph et al. 2022). UniLMv2 (Bao Bao al. 2020). Masque (Nishida et al. 2019). and RFF et al. 2021). f on validation set. is on unknown split.
See hyper-parameters in Appendix
Datasct
Prior SOTA
Our fine-tuned (2B)
Our calibrated (2B)
#params
R1 / R2 / RL
RI R2 RL
R1 / R2 RL
CNN/DailyMail
340M
47.78/23.55/4453
14.31/21.9144111
47.97/24. 18/44.88
XSUM
268B
1/27.1/-
49.57/26.77/44 171/1411
49.77/27.09/42.00
ReddiTIiFU-lng
340M E
30.31/10.98/24.7'
28.73/10.
12.03/11.13/23
SAMSum
20B
-129.60/-
53.64/29.
54.37/29 88/45,89
SQuAD QG
110M
-1-52211
-1-152.59
--153.28
MSMARCO NLG i
UNK"
-1-169.77
11770.73
-1/71.06
WebNL.Geen
20B
-155.40/-
76.96/52..7/62.566
78.09/55.52665.66
CommonGen'
20B b
-137.40/-
36.49/36.17/58882
68.95/38.49/60.11
We calibrate the fine-tuned PEGASUS models on language generation tasks using the simple recipe identified in (ubsection 3.3 and evaluate them with beam search without decoding heuristics (subsection 3.4). The only hyper-parameter we optimize for SLiC is learning rate (Appendix J).
We use beam size 5 for -un-tuned-only models and 10 for calibrated models.
As shown in Table 3) calibrated models show consistent improvement over fine-tumed -eeettued-only models across datasets and tasks. Overall, our calibrated models exceed or match the SOTA models all datasets. On XSUM, SAMSum, WebNL.Geen and CommonGien, our calibrated 2B models are ten hundred times smaller than the SOTA models.
TL;DR;
PEGASUS:B achieves SOTA results on a wide range of language generation tasks using a simple SLiC recipe while eliminating decoding heuristics.
4 RELATED WORKS
In classification, model calibration often refers to matching output probabilities with expected ac- curacy. In our case of sequence generation, how to generalize this notion is unclear. Kuleshov and Liang (2015) explores generalizing probabilistic calibration to structured prediction. but only focus on aligning the sequence likelihood with target sequence similarity. Other approaches in this vein are deseribed below.
4.1 RL APPROACHES
Paulus et al. (2018) directly optimizes evaluation metric ROUGE in RL fine-tuning stage. One issue is that ROUGE metric does not enforce fluency. The authors found summaries to be not always readable and proposed that using mixed training objective works better.
Ziegler et al. 2019); Stiennon et al. (220)) collects human judgements on fine-tumed models' de- codes to train reward model that ranks candidates according to human preferences. The supervised policy then fine-tuned against the reward model using PPO. The authors found that optimizing their reward model results in better quality summaries than directly optimizing ROUGE.
8

---

Published as a conference paper at ICLR 2023
4.2 TWO-STAGE RERANKING APPROACHES
SimCLS (Liu and Liu 2021 proposes formulating text generation as reference-free quality es- timation problem assisted by contrastive learning. The first stage decodes candidates with diverse beam search and RoBERTa based model is used to rank them in the second stage.
SunnaReRanker (Ravaut et al. 2022a) observes improved performance when training the genera- tion and the reranking models on two non-overlapping halves of the fine-tuning data compared to training two models on the same data.
based model to mimic the behavior of the task measure Bhattacharyya et al. (2021) trains such BLEU scores.
Lee et al.
2021) Fernandes et al. (0222) train rerankers for neural machine translation (NMT) that predicts the observed distribution of desired automatic metrics (BLEU, COMET and BLEURT) over the n-best list.
BRIO (Liu al. 2022) includes a two-stage reranking system that uses sequence-to-sequenee gen- eration models. It is shown that the sequence-to-sequene reranker has better performance than encoder-only models providing ranking scores.
4.3 MULTI TASK LEARNING WITH SEQUENCE-LEVEL LOSS
Edunov et al. (2018) surveys a range of classical objective functions for structured prediction and apply them to sequence-to-seueee -ouuuene models. Their experiments showed that combining sequence- level objectives with token-level objectives yields improved performance on translation and summa- rization datasets.
Sun and Li (2021) combines contrastive learning objective with negative log-likelihood to decrease the likelihood of the model generated "silver" summaries meanwhile increasing the likelihood of the "gold" references.
Wieting al. 2019) introduces alternative reward function for optimizing neural machine trans- lation systems that based on semantic similarity.
BRIO (Liu et al. 2022) demonstrates that multi task learning of sequence candidates with contrastive reranking and token-level generation has better performance compared to two-stage reranking system. The ranking order is determined by similarity to target using external metrics (ROUGE, BERTScoree. Models trained to rank by ROUGE also perform well measured on BERTScore and vice versa.
Lukasik et al. 2020 extends label smoothing from classification tasks to semantic label smoothing for sequence-tee to- sequenee learning. Their technique adds sequence-level losses that smooth over werl-formed relevant sequences that are similar to the target sequence semantically and n-gram level.
5 CONCLUSION
We propose adding a third stage of sequence likelihood calibration (SLiC) after the pretraining and fine-tuning stages for conditional language generation. A simple yet effective recipe for SLiC is using decoder-state similarity, selecting the fine-tuned model's checkpoint by perplexity, decoding candidates with beam search, calibrating with rank loss and KI. divergence regularization. We are able to eliminate all decoding heuristies for calibrated models. The benefits of calibration persist models scale up in size. Smaller calibrated models might outperform larger ones under the same inference compute budget. By calibrating a PEGASUS,g model, we exceed or match state-of-the-an results 8 datasets spanning abstractive summarization, generative question answering, question generation and data-to-text tasks.
In this work focus the setting where ground-trui truth output sequences provided. However, this presupposes high-quality labels that are not always available. the future, we plan to extend SLiC to general language modeling and explore more types of latent similarities.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank David Grangier for early and engaging discussions, and Noah Fiedel for feedhack on the paper.
REFERENCES
Daniel Adiwardana, Minh- Than-T Luong, David R So. Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta. Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. |ettte fine-tuning by reducing representational collapse In International Confer- ence on Learning Representations.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016.
Ms marco: A human generated machine reading com-
prehension dataset
Hangbo Bao. Li Dong. Furu Wei, Wenhui Wang. Nan Yang. Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. 2020. Unilmw:: Pseudo-masked language models for unified language model pre-training. ArXiv.
Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2021. [Energy-based reranking: Improving neural machine translation luiing energy-based models] In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 4528 4537, Online. Association for Computational Linguistics.
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. Recurrent neural networks as weighted language recognizers In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages for Computational Linguistics.
New Orleans, Louisiana. Association
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension, In Proceedings the 55th Anmual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1352, Vancouver, Canada. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. 2018. Clas- sical structured prediction losses for sequence to sequence learning In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, Volume (Long Papers). pages 355-364, New Orleans, Louisiana. Association for Computational Linguisties.
Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality- decoding for neural machine translation, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 1492, Seattle, United States. Association for Computational Linguistics.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura BeltrrBellr.. 2017. The WebNLG challenge: Generating text from RDF data In Proceedings of the 10th International Conference on Natural Language Generation, pages 124-133, Santiago de Compostela, Spain. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye
10

---

Published as a conference paper at ICLR 2023
Emezue, Varun Gangal, Cristina Garbacea, Tatsumori Hashimoto, Yufang Hou, Yacine Jer- nite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Ro- driguez. Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. (The GEM benchmark: Natural language generation, its evaluation and metrics) In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), I), pages 96-120, Online. Association for Computational Lin- guisties.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A annottaed dialogue dataset for abstractive summarization In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration In International Conference on Learning Representations.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv: 1904.09751.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020a. Scaling laws for neural lan- guage models. arXiv preprint arXiv:2001.08361.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020b. Scaling laws for neural lan- guage models CoRR, has/2001.0836/
Byeongchang Kim. Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Red- dit posts with multi- memory networks) In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 2531, Minneapolis, Minnesota. Association for Computational Linguisties.
Philipp Kochn and Rebecca Knowles. 2017. Six challenges for neural machine translation, In Proceedings of the First Workshop Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with mul- tiple subword candidates. arXiv preprint 1rXiv:8804.1095.
Volodymyr Kuleshov and Percy S Liang. 2015. Calibrated structured prediction In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Ann Lee, Michael Auli, and Mare' Aurelio Ranzato. 2021. Discriminative reranking for neural machine translation In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7250-7264, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising soguence-to-squeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:19/11 13461.
11

---

Published as a conference paper at ICLR 2023
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation In Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Processing, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Chin-Yev Lin. 2004. ROUGE: A package for automatic evaluation of summaries In Text Summa- rization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguis- tics.
Yixin Liu and Pengfei Liu. 2021. SimCLS: A simple framework for contrastive learning of abstrac- tive summarization, In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguisties.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstracttve summarization In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2903, Dublin, Ireland. Associ- ation for Computational Linguistics.
Michal Lukasik, Himanshu Jain, Aditya Krishna Menon, Seungyeon Kim, Srinadh Bhojanapalli, Felix X. Yu, and Sanjiv Kumar. 2020. Semantic label smoothing for sequence to sequence prob. lems CoRR, abs/2010.07447.
Inderjeet Mani. 2001. Automatic summarization, volume 3. John Benjamins Publishing.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797- 1807, Brussels, Belgium. Association for Computational Linguisties.
Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2 (2-3):103 30:10333-333
Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2019. Multi-style generative reading comprehension In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2273-2284, Florence, Italy. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization In International Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text- to-text transformer Journal of Machine Learning Research, 21(140):1-6)
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+
questions for machine comprehension of text, In Proceedings of the 2016 Conference on Empir- ical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. 2016. Sequencee level training with recurrent neural networks In 4th International Conference on Learning Represen- Iations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022a. SummaReranker. multi-task mixture-of of- experts re-ranking framework for abstractive summarization In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers), pages 4504- 4524, Dublin, Ireland. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2023
Mathieu Ravaut, Shafiq Joty, and Nancy F Chen. 2022b. Summareranker: A multi- mixture-of- experts re-ranking framework for abstractive summarization. arXiv preprint arXiv:2203.06550 06666999.....
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks! In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguisties.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback In Advances in Neural Information Processing Systems, volume 33, pages 5008-30211 Curran Associates, Inc.
Shichao Sun and Wenjie Li. 2021. Alleviating exposure bias via contrastive learning for abstractive text summarization. ArXiv, abs/2108.1 1184..
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. 2016. Re- thinking the inception architecture for computer vision| In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms.
Christoph Tillmann and Hermann Ney. 2003. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics, 29(1):77-133,
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models CoRR, abs/ 120.0444..
John Wieting, Taylor Borg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEUxraining neural machine translation with semantic similarity In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges data-to -ata-te-docummnt gen- eration In Proceedingy of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 16Xiv:1609.08144,
Yilin Yang. Liang Huang, and Mingho Ma. 2018 Breaking the beam search curse: A study of (re- Dscoring methods and stopping criteria for neural machine translation, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, Brussels, Belgium. Association for Computational Linguistics.
Jingging Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization CoRR, abs/ 191./9972.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore; Evaluating text generation with BERT CoRR, abs/1 1944009675
13

---

Published as a conference paper at ICLR 2023
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: preliminary study) CoRR, abs/1704.01192.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint (rXiv:/90900850)
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906.
14

---

Published as a conference paper at ICLR 2023
A DATASETS PROPERTIES
A.1 DATASETS AND TASKS
CNN/Daily (Hermann et al. 2015 See et al. 2017) summarization dataset contains 313k arti- cles from the CNN and Daily Mail newspapers with bullet point summaries. The summaries are on average 3-4 sentences and relatively extractive
XSUM Narayan et al. 2018) summarization dataset consists of 227k BBC articles from 2010 to 2017 with single sentence highly abstractive summary. Sometimes the summary contains infor- mation not present in the article
ReddicTIFU-long (Kim et al. 2019) summarization dataset contains 42k posts of informal stories from sub-reddit TIFU from 2013-Jan to 2018-Mar with author written summaries. The style and length of the summaries very diverse
SAMSum (Gliwa et al. 2019) summarization dataset contains 16k high-quality chat-dialogues and their summaries written by linguists
SQuAD QG Zhou et al. 2017: Du et al. 2017) is the task of generating a question from a passage-
answer pair extracted from the SQuAD dataset (Rajpurkar al. 2016). In particular, we the split of Du et al. (2017). consisting of 75,722, 10,570, and 11,877 examples for training, validation, and testing, respectively
MSMARCO NLG (Bajaj et al. 2016) is a large scale dataset focused on machine reading compre- hension and question answering. The original QA dataset consists of 1,010,916 queries. However, we work on the NLGEN data that a subset the QA data consisting of 182,669 queries, each with well formed answer. The task is generate well formed answer to an input query and set of answering passages]
WebNLG-en (Gardent et al. 2017) consists of 16,095 data inputs in the form of sets of RDF triples extracted from DBpedia. Each data point was verbalized by humans in more- more-than-one natural texts, leading to a total of 38.872 data-text pairs.
CommonGen (Lin et al. 2020) introduces a task of generating a coherent sentence describing an input set of common concepts. The dataset consists of a total of 35,141 common concept sets, split into 32,651/993/1,499
sets. There are 67,389, 4,018 and 6,042 sentences in training, validation and test, respectively"
Table 4: Statistics of datasets.
dataset
of examples
avg words
extractiveness
train/val/test
input
target
coverage
density
CNNiDailyMail
287K / 13K IIK
698.60
49.53
87,8%
3.77
XSUM
203K / 11K 11K
383.17
21.74
63.9%
1.06
ReddirTIFU-llng
34K /K /K
396.15
21.02
68.4%
1.27
SAMSum
14,732 / 818 819
97.23
21.00
68.0%
1.46
SQuAD QG
76K / IIK 12K
128.72
10.24
64.7%
1.63
MSMARCO NLG
152K / 12K 12K
588.50
14.07
97.5%
7.78
WebNLG-en
35K / 1667 1779
17.50
20.51
48.7%
1.3
CommonGen
67K / 993 1497
3.27
10.10
22.0%
0.22
https://wwis
tifu
htt
http:///ww.eeerrrroo////aaaaataa default.
config
15

---

Published as a conference paper at ICLR 2023
B MODEL ARCHITECTURE
Model sizes and their configurations reported in Table
Table 5: Model sizes.
name
num layers
hidden size
num heads
MLP size
# num params
enc/dec
excluding embs
# total
PEGASUS SMALL
8/8
512
1024
49M
108M
PEGASUSBASE
12/12
768
12
3072
198M
272M
PEGASUSLARGE
16/16
1024
16
4096
470M
568M
PEGASUSBB
24/24
1024
16
16384
1913M
2012M
C HYPER-PARAMETER NOTATIONS
Table Summary hyper-parameters notations.
notation symbol
meaning
Ir
learning rate
Q
length penalty factor heam search
8
scale of ranking margin in calibration loss (qquation
À
relative scale of regularization loss compared with calibration loss section 2
D EVALUATION SCRIPTS
For summarization tasks, we use pypi package rouge-scom score to report ROUGE numbers. We report rougeLsum for ROUGE-L.
For SQuAD QG and MSMARCO NLG, we use the original evaluation scripts provided by Du et al. (017) and Bajaj et al. (2016), respectively. For WebNLG-en and CommonGen, use the versions from the GEM benchmark (Gehrmann et al. (002)) and report using the GEM evaluation framework. Those scripts mainly differ in text tokenization methods.
16

---

Published as a conference paper at ICLR 2023
E
ABLATION STUDY
SLiC methods for ablation study reported in Table
Table 7: Experimental settings for ablation studies.
Ablation
calibration
evaluation
decoding
sim
loss
regularization
ckpt
extra
decoding
fine-tuned
beam 5
similarity function
ROUGE
beam 15 beam 15
ROUGE
reward reward reward
cross entropy cross entropy cross entropy
ROUGE
fix
beam
decoder repr
so sy.y.
ROUGE
fix Ir
beam 5
token emb
beam 15
Scok (
ROUGE
fix Ir
beam 5
calibration loss
rank
beam 15
se (y+ y.
rank margin
cross entropy
ROUGE
Ir.
beam
margin
beam 15
se (y, y.
cross entropy
ROUGE
best Ir. 8
beam S
list rank
beam 15
So y,, y,
list rank
cross entropy cross entropy
ROUGE
best Ir. 3
beam 5
reward
beam 15
so xx)
reward
ROUGE
best Ir. 3
beam 5
regularization loss
none
beam 15
se y
rank rank rank
ROUGE
fix Ir, 8
beam 5
cross entropy
beam 15
se ( y,
cross entropy KL divergence
ROUGE
fix Ir, /
beam 5
KL divergence
beam 15
Sp s
ROUGE
fix lr. 8
beam 5
calibration decoding
method
beam
beam 15
se sy y,y.
reward reward reward
cross entropy cross entropy cross entropy
ROUGE ROUGE
fix Ir
beam 5
diverse beam
diverse beam 15
se y.
fix Ir
beam 5
nucleus
nucleus 15
Sp (y.y,)
ROUGE
fix Ir
beam 5
calibration checkpoint selection
ROUGE
beam 15
se 8,
reward
cross entropy cross entropy
ROUGE
fix Ir
beam 5 beam 5
perplexity beam 15
se y,
reward
perplexity
fix Ir
17

---

Published as a conference paper at ICLR 2023
F DECODING METHODS
SLiC methods for decoding calibrated models reported in Table 8 At evaluation time, models are decoded with 1, 2, 5, 10 and 20 candidates. ROUGE numbers in Figure are reported in Table 9
Table 8: Experimental settings for calibrated models' decoding analysis.
name
calibration
evaluation
decoding
sim fn
loss
regularization
ckpt
extra
decoding
a
7 beam
beam 1-20
/ nucleus
nucleus 1-20
beam + beam
beam 15
88 vy.y,, xx x]
reward cross entropy
ROUGE
fix Ir
beam 1-20
X
beam -> nucleus
beam 15
88 Xx)
reward
cross entropy
ROUGE
fix Ir
nucleus 1-20
X
nucleus - beam
nucleus 15
reward
cross entropy
ROUGE
fix Ir
beam 1-20
X
nucleus nucleus
nucleus 15
86 xx.
reward
entropy
ROUGE
fix Ir
nucleus 1-20 x
Table 9: ROUGE (R1 R2/22//R0 RL) numbers of the decoding curves.
SLiC -> decoding
num decodes
MNll R1 21//2// RL.
XSUM R1 21/R2/ RL
RedditTIFU-long R1 /1/R2/ RI.
SAMSum R1 / R2 RL
beam
45.11/21.15/22.33
46.18/22.84/38
27.78/8. 40002/44/22...
52.86/27.89/1 .8/43.85
44.54/21.62/41..
46,94/23.87/388. 89
27.75/8 98/22.37
53,42/29. 1/44.56
5
2.78/21.99/41.9
47.26/24.38/39,
26.88/9.09/21..
53,47/29 255/449
10
4458/11.86/11.777
47.29/24.60/3944
25.51/8.78/21.0 .04
53.70/29 22/44,,2//666
20
62.33/24.2//39.36
32/2000
2..44/9.21/444.1
/+ nucleus
1
44.09/19. 88/41 1.24
43.76/20 42/3555
25.33/6
1.333/.84/19.77 84/19.78
50.51/24.56/40.
2
44.31/20.366 36/41.50
45.03/21 80/36.96
24.82/6. 95/19.72
52.17/26.91/43.02
5
44,43/20.8 11443/2..8//1.666 6/41.66
15.61/22.63/37.888
84.80/6.84/19.0
51.50/26, N1.50/66.0/43.00
10
94/48/ ..54
46.06/23. 22/38.77
23.45/6.98/19
50.69/26.
00.69/26.2842.777 28/42.70
20
44.25/21.13/415555
46.06/23.577388.22
91.1//.81/188.39
0A.33/26.8842.72
beam + beam
15.72/20.87/44 87/42.89
46.71/23.16/338
30.00/9 20/23.82
54.24/28.67/44.55
2
46.46/21
0446/21196/43.55 96/43.58
47/46/2. 17/39. 47
30.24/9. 56/24.11
54.68/29. 71/25/02//
5
46.72/22.55/43.8 857
47
30.25/9.80/24.22
80/24.26
54.78/29.75/45.22
10
46.81/22.
11/11/26///3..... $7543.95
47.83/24 82/42.2/0000 06
30.31/9 89/24.39
54.63/30.01/45..
20
46.90/22. 83/44.04
47.83/24. 86/40.07
30.02/9. 80/24.29
54.74/29. 98/4299
heam + nucleus
44.83/19.55 59/41.92
0383/19.99/41999
44.73/20.99/1
99/36.52
28. 19/7 89/22.05
52.26/26. 19/42.07
2
45.16/20.01/44 01/42.28
45.55/21
03.55/21.92/3.566 92/37.56
28 66.66/8.8 22/22.58
$3.15/27.61/43.44
5
2..35/20.3//2..2.
46.15/22.87388.4 45
28 28.83/8. 62/23.06
53.50/27.80/44.1
10
45.46/20.5 1/42..9
46.39/23.33/38.88 85
28.90/9. 10/23.47
53.99/28.71444 11/44.89
45.46/20.66 63/42.63
46.53/23.67/3.007
28.60/9.011 31/23.39
54.22/28 68/45.
nucleus -> beam
1
45.66/20.93M 93/42.77
46.50/22.93/37.99
30.57/9 45/23.68
53.81/28.71/44
2
46. 16.19/21.91// 91/43.29
01/29/23.93/88.
30.94/9 82/24.06
53.99/29 25/44,30
5
56.47/22.50/43.56
47.74/24.43/39.36
01.10/10.00/24.2
54.29/29 49/44.62
10
(6.39/22.57/433
47.78/24.52/39.
31.02/10 22110.00/24..2
54.25/29
158329954/44.777 54/44.70
20
46.34/22. 63/43.88
47.83/24.63/39.44
31.11/10.00//44.
54.17/29.46/44
nucleus -> nucleus
1
44.68/19, 69/41.75
44.35/20.69/33.
29 0.8558.68/222.
52,55/26.
0155/26.9//4..6 98/42.63
2
24/42.20
45.50/21 94/37.13
30.31/9. 22/23.45
52.97/27 33/43.333/4300
5
45.58/20.8 1/42.65
46.43/22.93/38811 19
30.46/9 44/23.81
54.10/28, 54.10/28.86/44.77
10
45.73/21 05/42.82
46.91/23.65/38.99
30.69/9.58/24.11 58/24.11
20
45.78/21.26/42.88
47.19/24.00/3.11
31.04/9
53.78/29.02/44.66
18

---

Published as a conference paper at ICLR 2023
G LENGTH NORMALIZATION
Experimental settings for length normalization analysis reported in Table 10 Brevity penalty
is chosen as the best value for fine-tuned models' ROUGE performance on validation dataset or disabled.
Table 10: Experimental settings for length normalization study.
SLiC
a
calibration
evaluation decoding
decoding
sim
loss
regularization
ckpt
extra
X
X
beam 5
X
beam 5
beam 15
S9 (y x)
best
cross entropy
ROUGE ROUGE
best
beam
beam 15
89 y,
best
cross entropy
best Ir, 3
beam
H
MODEL FLOPS ESTIMATION
We extends formulations in Table Kaplan al, (2020b) to estimate FLOPSs of our transformer encoder decoder models following the formula:
total_C Cene X Bene-cte + Cdec X Redec-cte X m
Cene 2N enc + Pene eeeeeeeeeeetteeeattee
(3)
Cdcc 2N + dede ddnnn---tt
where m is the number of decoder candidates, other notations can be referenced in Table of Kaplan et al. (2020b). Because of upper triangle attention masking, the effective decoder attention context length is half of sequence lengths instead of full sequence lengths as in the encoder. Extra computation incurred by different decoding methods are omitted they are much smaller.
I SCALING
SLiC method for scaling curves are reported in Table 11 At evaluation time, models are decoded with 1, 2, 5, 10, and maybe 15, 20 candidates. ROUGE numbers in Figure are reported Table 12
Table 11: Experimental settings for scaling.
model
calibration
evaluation
decoding
sim fn
loss
regularization
ckpt
extra
decoding
fine-tuned calibrated
beam 1-20
beam 15
so v x))
reward
cross entropy
ROUGE
best
beam 1-20
19

---

Published as a conference paper at ICLR 2023
Table 12: ROUGE (RI R2/RR R2/RL) numbers of the scaling curve.
size
decodes
CNN/DailyMai RI / R2/ RL
XSUM RI/R2 22/22/0 RL
RedditIFFUlong RI R2/ RL
SAMSum RI R1/R2/ RL
fine-tuned
50M
43.21/19.99/40.53 99/40.53
40.91/17.80/32
25.37/6.99/20.1
49.78/24.45/40.6
2
42.77/20 40/39.90
41.55/18.78/33.
25.22/7.53/20.34
50.52/25.37// 80
00.92/20.45/39..0
41.87/19 44/34.28
24.41/7.61/20.00
50.52/22.92/44..00 92/42.00
42.78/20.32/39.75
11.85/19.57/34.33
23.04/7.43/19.04
50.41/25. 84/41.81
11.79/19.59/34.3
50.46/25. 89/41.77
20
41.65/19.56/34.25
50.50/26.000
200M
44.59/20. 9//20.99//1993
44.51/21.34/36.47
27.32/8.06/21..6
51.77/26. 44/42.38
44.41/31.44/1133 44/41.33
45.24/22.22/37.15
49.36/8.49/21.89
52.35/27
$2.35/27.40/4..27 40/43.27
44.08/21.54/41.2
45.66/22.83/37.711
26.61/8.78/21..
10
43.84/21.30/00.66
45.61/22.93/37.700
25.80/8.37/20.85
52.40/27.64// 6.77
15
45.55/22.94/37..11
52.35/27. 69/43 6.7
20
05.54/22.99/37.17
52.38/27 68/43,6
500M
45.34/21.47/42..6
46.27//33.0/38.12
27.79/8.42/22.11
53.05/27 96/43.66
14.93/21.83/22.15 83/42.15
46.99/23.90/3
90/38.89
53.73/29.07/44.777 07/44..7
44.78/21 98/41.9/2222 .92
(..78/11.9/44199
47.26/24.37/39..
06.85//9099/ 09/21.94
53 94/29
000//990014444 01/44.53
10
84.54.21.86/41.11
47.27/24.59/39.48
25.97/8.74/20.99
53.67/29.3 1/44.66
15
47.20/24.63/39.41
$3.71/29 22/44.63
47.15/24.62/39..77
53.68/29, 16/44.6
2B
15.52/21.70/42..7
47.89/24.54/39.
28.82/9.29/2.11
53.40/28 01/43.8
2
45.37/21 15.37/21.55/42.54 95/42.54
48 66/66//5.02 61/40.55
28.60/9 60/23.12 12
53 89/299 47/44,80 888
05.40/22.99/4..555
48.94/26. 1/40091
77.86/9.77/2.884
53,98/29, 08/44.62
10
45.29/21182/42.10
48.91/26.08/40
27.52/9.11/11866
53.95/29, 61/44.61
48.96/26. 12/40.78
$3.92/29 61/44.63
20
48.75/26.20/40.0
53.86/29.57/444
calibrated
50M
44.31/20.82441.66
41.41/11..5/33315
27.15/7.57/21.4
19.85/24.62/40..2
14.91/21.7/42..13
42.27/18. 1.9/34.11
77.34/8.00/21.70
50.89/25. 71/41
45.12/22. 10/42.25
12.88/19.84/34.3
27.49/8.43/22.02
51.53/26.5842..35
10
45.15/22.20/42.22
13.01/20.1//35.1
27.32/8.42/21.92
2.088/6.67/42.11
15
13.13/20.15/35.1
52,04/26.666 66/42.66/22110
43.14/20.19/355
51,90/26. 16
200M
45.26/21.16/42.55
44,54/21.
17.81/.10//1188
52. $2.12/26.4//42.444
2.977/22.25/43.2
45,47/22. 12/37.18
28. 0..38/8.68/2..8 68/22.38
53 29/22//8.. 24/43.99
5
76.18/22.78/43.41
46.04/22.90/37.88 90/37.86
28.51/9.03/22.79
53.79/28. 75/44.15
10
46.26/22.88/43.477
46.21/22. 06.21/22.99/38.00
28.35/9.0/222.78
54,06/28 86/44.49
00.29/23.07338.05
$4.03/28 85/44.41
20
46.28/23.09/38.00
53. 23.99/28, 90/44.39
500M
45.55/20.85/422
46.42/22. 93/38.12
29 10/23.6
53, /3.31/28 43/44.18
47.29/23 95/399022
29. 79.8039559/23.75
54.14/29 54.14/29.29/44.47 29/44.47
00.55/22.48/43.66
47.88/24 62/39.62
29. 83/8 84/23.91
54.61/29 95/45.
46.63/22.58/43.78
47.93/24.74/33
29.87/9.95/24.03
54.89/30 05/45. 18
15
2..5//2.80/99..0
54.88/30. 26330.27//5.35
20
48.06/24 85/39.86
0/6/24.85/39.00
54.87/30.3 31/45.39
2B
46.29/21.92/43.47
15.11/24.5/39.66
30.20/9.86/24.11
54.71/29
54.71/29.45/45.00 44444555550
46.84/22.93/43300
19.04/25.55/40.44 55/40.46
30.59/10.38/24.50
55.17/30.68/44 68/46.69
17.08/23.45/44.11
49.56/26.31/4 3/411. .08
00.65/10.70/24.77
55 55.46/30..7166 71/46.11
10
47.08/23.57/44.19
49.79/26.5644 56/41.32
30.75/10.79/24.1
55 55.47/30.
77/33.60/466. 60/46.00
15
49.79/26. 55/41.55
55. 65.41 63/46.15 15
54/41.30
55.38/30.65/466 6.136.65//6.144
20

---

Published as a conference paper at ICLR 2023
J FINAL RESULTS
We choose the SLiC best based on subsec- lion 3.3 There are in total 3 hyper- parameters: learning rate Ir (Algorithm I). ranking constant B (Equation I). and regularization strength (Equation 2). We fix two of the them: is set 10, and r À is set to le - 5. Best learning rate Ir is determined with hyper- parameter tuning on validation set and reported in Table 14
SLiC method for final results is reported in Table 13
Table 13: Experimental settings for length normalization study.
model
calibration
evaluation
decoding
loss
regularization
ckpt
extra
decoding
fine-tuned calibrated
beam 5
beam 15
86 9.x)
rank
KL divergence
perplexity best Ir
beam 10
Table 14: Learning rate of final results.
CNN/DailyMai
XSUM
ReddiTTIFU-lng 10-5
SAMSum 10-8
Ir
10-3
10-8
MSMARCO NLG SQuAD QG x 10
WebNLG-en 10-6
CommonGen 10
Ir
10-5
21


Reviews:
[{"review_id": "_M68u3Ztqf", "reviewer": "Reviewer_JpRY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes sequence likelihood calibration (SLiC) for conditional language generation. Their proposal includes a first stage where candidates are generated from a fine-tuned model, and a second one, where they calibrate the previous model by continuing training it with a new objective that aims to align candidates’ sequence likelihoods by similarity to the target sequence in the model’s latent space. They perform experiments on abstractive summarization, question answering, and data-to-text generation tasks. ", "summary_of_the_review": "The paper addresses an important issue with conditional language generation models, that of uncalibrated sequence likelihood: for a given context, they are often trained with a single target sequence, and thus they lack supervision to compare (high probability) plausible hypotheses. The proposed method is interesting and the authors perform experiments that validate its applicability on different conditional language generation tasks. ", "strength_and_weaknesses": "**Strengths**\n\n* The motivation is clear, the paper is well written and is easy to follow.\n* The proposed method does not directly optimize an evaluation metric (e.g., ROUGE), and thus it’s more likely to actually improve quality rather than exploiting fragilities in the evaluation metrics.\n* SLiC seems to work well without beam size optimization and length normalization.  \n\n**Weaknesses**\n\n* I’m assuming that you’re using your proposed similarity function for all the experiments, excluding the ablation studies in $\\S3.3$, where you also experiment with ROUGE. First, I think you should clarify that in the paper. Since the ablation studies appear before everything else, the “simple recipe” of that section could include the similarity function to use. Second, Table 1 suggests similar performance for both ROUGE and your method, and thus I’d be interested in seeing a more extensive comparison of these approaches. \n\n**Minor comments/suggestions**\n\n* For neural machine translation, specifically, there are several works proposing alternative decoding methods that try to generate high quality outputs, related to the approach of Lee et al. (2021), discussed in $\\S4.2$, which I think are relevant for your work. Some of them are [1], [2], [3], and [4]. Besides, related to the work of Edunov et al. (2018), discussed in $\\S4.3$, is [5] and [6]. \n* In $\\S2.2.$, you should explicitly say which one “is the contrastive loss used in BRIO (Liu et al., 2022).”\n\n**Other questions**\n\n* Is there any reason to vary the hyperparameters between groups in Table 1? In particular, I see in App. C that the loss you used is different (sometimes *reward*, sometimes *rank*). \n* How does your method compare to common approaches for conditional language generation in terms of memory consumption and training time? The paper would benefit from such analysis. \n\n[1] Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation, Eikema and Aziz, COLING 2020\n\n[2] Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models, Sumanta Bhattacharyya et al., ACL 2021\n\n[3] Quality-Aware Decoding for Neural Machine Translation, Fernandes et al., NAACL 2022\n\n[4] High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics, Freitag et al., TACL 2022\n\n[5] Beyond BLEU: training neural machine translation with semantic similarity, Wieting et al., ACL 2019\n\n[6] Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation, Kiegeland and Kreutzer, NAACL 2021. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rjg5ylzOXTK", "reviewer": "Reviewer_YBaL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Remarking that language models can assign high probability to low “quality” outputs, this paper proposes “sequence likelihood calibration” (SLiC) as a solution. With SLiC, “decoding heuristics become unnecessary” and “quality significantly improves.” The claims are empirically supported by experiments on abstractive summarization, question generation, abstractive question answering, and data-to-text generation, with results that “exceed or match SOTA results.”\n\nIn more detail, the high-level idea proceeds in two stages. In the first stage, for each training instance, an initial model is used to sample m candidates (a variety of decoding methods are compared, including beam search and nucleus sampling). In the second stage, the initial model is further optimised using a combination of two losses:\n\nA “calibration” loss which aims to align the sequence likelihood with the similarity to the reference target sequence. Four such losses are considered, including a ranking loss, margin loss, listwise ranking loss, and expected reward. All losses rely on a notion of similarity of a candidate sequence to the reference sequence, for which a variation of BERTScore is proposed (S2.1).\nA “regularization” loss which penalizes deviations from the token-level confidences from the stage-one model. The cross-entropy to the reference labels and the KL divergence from the stage-one model predictive distribution are both considered.\n\nOverall, beam search using the rank “calibration” loss with KL divergence “regularization” loss are found to work best.", "summary_of_the_review": "There are some good ideas here but the presentation could be improved and the experimental results are lackluster. ", "strength_and_weaknesses": "Overall, it is notable that the proposed approach yields models that do not appear to require decoding heuristics; this suggests that the calibration loss is doing something useful. Figure 3 is also fairly convincing in this respect, as it shows a distinct difference between the vanilla model and the calibrated one when further decoding samples are used (consistent improvement for the calibrated model). However, the experiments overall are not entirely convincing, since in several cases the degree of improvement over the baselines is marginal, while the proposed model is larger than the baselines in terms of # of parameters (2B). So it’s not clear that it’s the new calibration scheme that’s responsible for the improvement in these cases.\n\nThe scoring function (S2.1) is a key element of the proposal. It’s stated that “compared to using external metrics, [...] it differs from the metrics we evaluate the generation systems with and mitigates the risk of directly optimising towards imperfect metrics.” However, the proposed similarity is related to both ROUGE and BERTScore, and so it’s not clear there isn’t some bias towards better performance on the target metrics.\n\nFinally, I would have liked to see probabilistic calibration addressed somewhere. Does the proposed calibration loss actually lead to more meaningful probability estimates? This could be measured at the token-level or on other events (cf https://papers.nips.cc/paper/2015/file/52d2752b150f9c35ccb6869cbf074e48-Paper.pdf)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RJdg-UPrtE", "reviewer": "Reviewer_o62J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To tackle the issue of uncalibrated sequence likelihood, this paper proposes a new sequence likelihood calibration(SLiC) stage, which extends the common paradigm of pretraining and finetuning. The specific method is to calibrate with KL divergence and low rank after selecting the checkpoint of finetuned model by perplexity and decoding candidates with beam search. \n\nWith SLiC, decoding candidates’ quality significantly improves regardless of the decoding method, without showing any sign of diminishing returns with model scale, and former decoding heuristics will be never used any more. Furthermore, this paper proposes a novel calibration similarity metric between model decodes and targets, which is much better than the existing external metrics. ", "summary_of_the_review": "The paper proposes a novel stage called sequence likelihood calibration after the pretraining and fine-tuning stages for conditional language generation. It seems that this method has not been used in prior works and with such stage the model can outperform the SOTA results without concerning about the diminishing problems. Also the entire paper is very well-written and comprehensible, and the experimentation in this paper are quite rich.", "strength_and_weaknesses": "Strengths:\n\n- The contributions of the SLiC are important, and do not exist in prior works. Besides proposing the SLiC, the authors also proposed a novel calibration similarity metric and used many past outstanding works as candidate decoding methods.\n\n- Also, calibration benefits persist as model sizes scale up, so it has great potential for models plagued by former approaches which shows diminishing by scaling up. Also, small models can add calibration step to outperform larger ones. length normalization has minimal effect on calibrated models.\n\n- Calibrated models do not require decoding heuristics and more. Only with the SLiC, the model reaches SOTA result and even better than the model with decoding heuristics, and such heuristics has minimal effect on calibrated models. \n\n- Extensive experiments have been conducted to validate the effectiveness and relevance of the proposed method. Also, there are a lot of experiments in ablation study to get a simple and effective recipe for training.\n\n\nWeaknesses:\n\n- (1) typo in page 14, “WebNLG-en” paragraph, “data inputs in the from of sets” -> “data inputs in the form of sets” (2) typo in page 14, “CommonGen” paragraph, “introduces a tak” -> “introduces a task”\n\n- The experimentation section mainly convinces me that this architecture is good for application. However, this work lacks of explanation on the reason of the choice of calibration loss, regularization loss, candidates decoding method, etc.\n\n- Since this paper is highly theoretical and original. However, it suffers the problem that it does not make sense to many details in the method chapter. For example, as for the calibration loss and regularization loss, the literal part seems only notation, and lack of the reason for why taking these methods and the advantages of them. (Maybe it is included in the experiment part. However, I do think that it should also be mentioned in the method part.) \n\n- Lack introduction of some notations. For example, I haven't found out what theta means so far. Although it is default to represent model parameters, I think it should be explained. What’s more, due to the rich content, this paper involves large amounts of parameters. Therefore, I think it is a good idea to explain the parameters in a chart. Maybe it can be appended to the appendix. In this way, the readers can follow the authors more easily.\n\n- The description of MLE method and its related works occupied a little too much space in introduction, while the motivation to proposed method was little.\n\n- In the section of ABLATION STUDIES OF CALIBRATION, the authors show us the improvement compared with fine-tuned models,but without the comparison between SLic and the fine-tuned model using the mentioned effective heuristics or solutions.\n\n-  The authors propose the Calibration stage after Pre-train and Fine-tune(MLE) stage. Since fig. 3 gives the inference compute trade-off between fine-tuned only and calibrated models, the  comparison of the running time and speed between fine-tuned only and calibrated in Table 3 is lacked.\n\nSuggestions:\n\n- The experiment results of the proposed method should be in bold.\n\n- The conclusion part lacks the description of the deficiencies and the prospects for future improvements.\n\nQuestions:\n\n- There are in total 3 hyperparameters: β in calibration，lr as learning rate and λ as regularization strength. To simplify the training process, β and lr∗λ are set empirically, making the process tedious, and how the values are set in the paper?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster