Tailoring Language Generation Models under Total Variation Distance
Authors: Haozhe Ji
Pei Ke
Zhipeng Hu
Rongsheng Zhang
Minlie Huang
Venue: ICLR 2023 notable top 5% (2023)
Keywords: language generation
maximum likelihood estimation
total variation distance
text degeneration

Abstract: The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From a distributional view, MLE in fact minimizes the Kullback-Leibler divergence (KLD) between the distribution of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probability regions in the data distribution, the model systematically overestimates the probability of corrupted text sequences, which we conjecture is one of the main reasons for text degeneration during autoregressive decoding. To remedy this problem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it to language generation. Then, we introduce the TaiLr objective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tunable penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and improves generation quality on a wide range of text generation tasks.

Published as a conference paper at ICLR 2023
TAILORING LANGUAGE GENERATION MODELS UNDER TOTAL VARIATION DISTANCE
Haozhe Ji', Pei Ke', Zhipeng Hu², Rongsheng Zhang", Minlie Huang' Depp.. of Comp. Sci. & Tech., State Key Lab of Intelligent Tech. & Sys. BNRist Center, Tsinghua University, Beijing 100084, China Fuxi AI Lab, NetEase Inc., China
jha200mails wtt.emmmmms..siinnsueeeeee..eeee edu. edu. en, hepeil106@outookk
(aphu, hewgrroonngeeng/ccorrroog wwt.ooor..eteeeeeeeeeeeeeeeeeeee netease. com, a.huangemail.cssingg wwingnua.
ABSTRACT
The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From distributional view, MLE in fact minimizes the Kullback-Leibee of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probabilityy regions in the data distribution, the model systematically overestimates the prob- ability of corrupted text sequences, which we conjecture is one of the main rea- sons for text degeneration during autoregressive decoding. To remedy this prob- lem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it language generation. Then, we intro- duce the Tail. obbective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tun- able penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and im- proves generation quality on wide range of text generation tasks x
divergence (KLD) between the distribution
INTRODUCTION
The dominant approach to train language generation models to maximize the likelihood of text samples in training data. With the development of pre-training techniques, the quality of texts gen- erated by current models has been improved by a large margin (Radford al., [2019] Brown al.) 2020). However, the text degeneration phenomena, e.g., repetitions (Holtzman et al.) 2020 Welleck et al. 2020), incoherence (Guan et al. 202T Ji & Huang 2021), and other ill-formed generation results sampled from the noisy long tail (Dou et al. 2022 LeBrun et al. 2022), are still widely ob- served in large pre-trained models. These results indicate that using MLE as the optimizing method has theoretical limitations that are hard to be compensated by increasing the model size.
Given the real data distribution p(æ) and the model distribution g(x) defined by a learned generation model, we can view MLE minimizing the KLD between p(x:) and q(x). However, minimizing DxL. (p, q) will lead to zero-avoiding solution q(x)) ) that spreads itself to cover all the modes in the real data (Minka 2005 Malinin & Gales| 2019). As the model is forced take into account all the modes regardless of their quality and saliency, this behavior could deteriorate the overall genera- tion quality when (i) the data inherently exhibits too many variations. e.g., open- generation, the model often ever-present, unrelated words in the unreliable long tail of its distribution (Holtz-) man et al. 2020). (ii) the data contains flawed or noisy references, e.g., hallucination and missing contents in text summarization (Zhao et al., 2020) degrade the generation quality of the model.
In language generation, the attempt to cover all the non-zero probability regions in the data distri- bution would lead to problem directly related to text degeneration, which we term as data void
* "Corresponding Author. Pronounced as "tailor".
Code is available at /ttttps://gittbb..omo/

---

Published as a conference paper at ICLR 2023
overestimation. Concretely, the model assigns considerably more probability mass than it should to the void of the real data distribution, where degenerated text sequences lic. An intuitive illustration is shown in Figure T where KLD pushes the model to place large mass the zero- ecoo-probability gion of the target distribution to cover the minor mass portion on the right. These degenerated texts include random word sequences and partially corrupted texts that have high lexical overlap with the real texts. Therefore, during free-run generation, the model is likely to trap into the void regions and produce gvverggnerallzee" text samples that are unlike the training data (Huszar 2015).
In this work, we start with robust alternative to KL divergence, i.e., the total variation distance (TVD). TVD known to be robust outliers in the data (Beran 1977 Knoblauch Vomfell 2020), as it measures the absolute difference between two probability distributions averaging at each point. In 2.2 we show that TVD allows the model to place zero probability to low-quality training samples and prevent overestimation of the data void region through gradient analysis. Though appealing, TVD cannot be directly applied to text generation because (i) TVD measures the distance at the sequence level while we desire a token-level criterion for autoregressive generation models, (ii) we only have samples from the data distribution, whereas calculating TVD demands the real data probability p(x) of the training sample We overcome these two issues by (i) developing an upper bound on the sequence-level TVD with its token-level factorization ($3.)) and (ii) introducing proxy distribution ($3.2) that handles the bias- tradeoff' during estimating TVD ($3.)) Finally, we derive the Total Variation Guided Language Generation (Tail.r) objective by leveraging access to the non-zero gradient of TVD to guide the model. Intuitively, TaiLr weights the log- likelihood of text sequence at each position according to the model probability and uses tunable hyperparameter control the penalization intensity.
We first conduct experiments on synthetic data to show that TaiLr achieves better generation quality without sacrificing diversity and reduces the overestimation of degenerated texts compared to MLE. Further experiments on real data demonstrate that the proposed method outperforms existing meth- ods that modify MIEE at different aspects on a wide range of language generation tasks, including machine translation, text summarization, and long text generation.
2 BACKGROUND AND MOTIVATION
We consider natural language generation tasks where a conditional generation model po(y)(z) parametrized by 0 is required generate the target text sequence y (y1, , yr) given the context æ. Let (3)la) denote the real data distribution, MLE training is equivalent to minimizing the KL divergence between Po and po:
T
DKL po) = Eyyop
log po (o
HPo),
(1)
where the generation probability is factorized into the product of conditional token probabilities given the prefix yc: and the context æ: po()) = II" (o(eeyeer, The first term pushes the model to minimize the negative log-likelihood (NLL) of the training data. The second term is constant with respect to and therefore is commonly ignored MLE.
Despite its simplicity and practical benefits for optimization, MLE is known to suffer from a mis- match to the evaluation metric (Pang He Hc|[02I] 2021) and brittleness to noise in the training data (Kang & Hashimoto 2020). Motivated by the literature in probability metrics, we draw attention to to- tal variation distance (TVD) as a naturally robust alternative to KLD. We present the definitions of TVD (Van Handel 2014) between the data distribution Pa and the model distribution pe given the context x::
DTv(po. po)
1
Σ
Po (o(()) po (o(1
(2a)
2
yey
Σ DEY
min (ol()lz)) (o())=)) po(ula))
(2b)
where y is the space of all possible text sequences. Intuitively, TVD measures the average of the absolute difference between po(y) zz) and po (y)(z) on all possible text sequence y E y. Therefore
2

---

Published as a conference paper at ICLR 2023
the model learns to properly allocate its probability mass to best describe the major part of the data distribution and ignore outliers. TVD is also correlated with the distinguishability of samples gener- ated by the model, which is shown to be balanced criterion that takes both quality and diversity into account (Hashimoto et al. 2019). Existing work proposed to optimize distinguishability in an gen- erative adversarial manner [Goodfellow 2015 Caccia al. 2020) while while/Kang & Hashimoto (2020) argued that minimizing its heuristic surrogate via loss truncation is better in practice. Additional related work is provided in Appendix B In this work, we first analyze the property of TVD and seek to directly minimize TVD or at least its upper bound in the task of natural language generation.
2.1 A TOY EXPERIMENT AND ITS IMPLICATIONS
We first present a toy experiment to illustrate the behavioral difference of KLD and TVD when countering imperfect data, where a single Gaussian model is required to fit a mixture of two Gaussians. As shown in F minimizing KLD forces the model to learn flat distribution that spans itself to cover all the non-zero probability regions, which causes underfitting of the major part of the target prohability mass. Furthermore, the model places considerably high probability mass to the void region in the target distribution which does not correspond to real samples. On the other hand, TVD focuses on the major target mass without overestimating degenerated samples that are unlikely under the target distribution.
0.8
target
TVD
pbx)
0.4
KLD
0.0
2
4
Figure 1: Results of the toy exper- iment: KLD is sensitive to outliers while TVD is more robust.
In language generation, this scenario is realistic and pervasive. For many language geneartion tasks, it is hard to circumvent noisy or invalid references during the data collection process, c.g., halluci- nation in text summarization (Zhao et al. (200)) and image captioning (Xiao & Wang. 2021). For applications like open-ended generation, existing autoregressive models pre-trained on large corpus are still reported over-present the artifacts in the noisy long tail (Holtzman et al. 2020
2.2 GRADIENT ANALYSIS
To better understand the reason behind the behaviorial difference of KLD and TVD in optimization, we analyze their gradients with respect to the model parameter 0. Given a context-tane target text pair (x", v²) sampled from the data distribution po. we approximate the gradient of KLD with respect
to using Monte-Carld sampling"
(Po. -po |=*) " VoPP(y"(a=")
The result is the negative gradient of the model probability weighted by the reciprocal of the model probability on this sample. Intuitively, when a low-quality context-targed pair is sampled, the model will be affected by this sample and shift the distribution towards it. If [>="">>" x 0, the norm of the gradient will become very large, which leads to a huge step of parameter update towards that noisy direction. This explains the phenomena illustrated in 22. where KL.D pushes the model to cover all the training samples resulting in an unfocused and flat distribution.
For comparison, we calculate the gradient of TVD with respect to 0 using equation 2b derivation details are provided in the Appendix A.I
The
{ -Poo (x*) "opo(y"(x"), 10.
Po Po [x=)) < Po x Po(y (b" xx),
V Po)
(4)
pe (3(3"x"") 7
where the result switches between a non-zero gradient term and by comparing the model prob- ability and the real data probability. When the model probability exceeds the real data proba- bility (overestimation), the gradient becomes O to prevent the model from fitting dubious data points. When the model predicts a probability lower than the real probability of the sample (underestimation)) the weight is the reciprocal of the real probability of the sample, which has a smaller norm than equation (3). This means that the update towards noisy directions more conservative, and the model is allowed to assign 0 probability to those low-quality training samples.
For clarity, we only use one sample per hatch in this analysis, and the result. still holds for large batch size.
3

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY
Despite the attractive attribute of TVD, we still face several challenges to apply TVD to natural language generation. First, TVD measures the difference of the sequence-level probability. For autoregressive language generation models, it is typical to use token-level criterion to supervise the factorized model probability. Although the sequence-level objective can also be adopted as a reward function using policy gradient (Williams) 1992 [Sutton et al. 1999), this approach is shown to suffer from a high variance and sparse rewards. Second, calculating TVD requires the real data probability (y (x) of the sample y be known. One straightforward solution is to train classifier that estimates the density ratio between Po ( and (y x) (Song et al. 2020). However, the density ratio estimator would introduce undetermined biases due to miscalibration (Grover et al. 2019). In this work, we tackle these challenges by developing practical upper bounds on TVD, and derive a sampling based learning criterion which can directly substitute for the MLE objective in practice.
3.1 TOKEN-LEVEL FACTORIZATION
As KLD has the nice property of factorizing the sequence-level loss into summation of the token- level loss conditioned on the prefix as illustrated in equation (T). we wonder if TVD also has this property. We first write the autoregressive factorization of the data probability as po (y(a)) = I[== (o x). For simplicity, we use (Br) and Pe p5'(z) to denote Po (velle (ele x) and Po(ye 2) respectively. Then we have the following proposition that manifests the relationship between the sequence-level objective and its token-level factorization.
Proposition Given Po(()x)) = I Po (B6) and po(y)æ) = I Et (Br). condition holds:
po'()), then the following
Div(Po:Po) S Byrp.
Σ Div(p po
(5)
The condition follows from applying triangle inequality (Hein & Bousquet 2005) to the right hand side of equation (2a). The complete proof is provided in Appendix A.2 This result indicates that minimizing the expected sum of the TVD on token-level probabilities is equivalent to minimizing the upper bound of the TVD on their products where the bound becomes tight as pe approaches Po- Therefore, we are guaranteed to train the model using the MLE fashion that calculates the loss at each position given the prefix of the target sequence.
3.2 ESTIMATION WITH PROXY DISTRIBUTION
Another difficulty of directly applying TVD to train language generation model is the explicit de- mand of the data probability distribution Po while we only have finite number of samples drawn from it. In contrast to using an additional density ratio estimation model that is both hard to train and potentially biased with undetermined deviation, we try to estimate the target using a proxy prob- ability distribution and analyze the estimation error.
We start by considering the one-hot distribution (((w) where only the w-th index is and others are w is the target token sampled from the conditional oracle probability p-*(-). is easy see that the expectation of the one-hot distribution is exactly the oracle probability distribution:
E Ew-p
= Poa.
(6)
Then we use ((w) substitute the oracle probability p in TVD and present the following propo- sition which states that the expectation of this estimation serves as an upper bound of the original TVD between the oracle distribution and the model distribution.
Proposition 2. Given w ~ pe and the one-hot distribution e. (!!)), then the following condition holds:
St <t) < Eunnps Dro(ee ,p55) Drv(pet .PO
(7)
4

---

Published as a conference paper at ICLR 2023
Model
Este
grad
grndent
x(1 ?)
26
Pb
TaiLe
log (1c) XI
PP (1e)
75 +
7106"
no Po
Data
X
e!l)
Figure 2: Computational graph of the TaiLr objective where the log-likellln likelihood is weighted position- wisely.
The proof utilizes the Jensen inequality and the convexity of the TVD. The full proof is presented in Appendix A.3 The proposition states that minimizing the TVD between the model distribution and the one-hot distribution on average leads to the minimization of the TVD between the model distribution and the oracle distribution. By introducing an unbiased estimation of the unknown oracle distribution, we sidestep the need of density ratio estimation and derive a practical upper bound using Monte-Carlo sampling from the oracle distribution.
3.3 THE BIAS- TRADEOFF
However, using the one-hot distribution approximation in practice sometimes leads to high estimation variance. For example, in some applications where the real data distribution has a high entropy, using the one-hot proxy can hardly cover the diverse candidates. Therefore consider general form of the proxy distribution g(m)) where w is the target token. We denote the expectation
of the general proxy distribution as pat Bu-ps' (pw)) Then we show that the upper bound of the estimation error can be decomposed into a bias term and variance term the following:
Error gtw) D _Po0) H Bu
Drv(ile) p2))
(8)
Bias
Variance
where Errorgtw is defined as the difference of the practical estimation Eu Drv .P and the ideal target Drv(p5t ") The derivation applies triangle inequality bound the error term (detailed derivation can be found in Appendix (A.4). Specifically, we consider the one-hot distribution example: has zero estimation bias (equation (6)). However show in Appendix A.5 that its variance equals to (H when a = 2. where Ha is the Tsallis d-entropy (Tsallis, Therefore, the one-hot proxy suffers from a large variance when the entropy of p is high.
1988
In order to handle the bias-variance tradeoff, we consider a 7-mixture proxy distribution that inter- polates the one-hot distribution and the model distribution with y: p(w) = 7e(w) + (1 - -7)pp Below show the bias and variance in equation (8) using this mixture proxy distribution:
Bias = (1 (1 Div p2t), Variance = E Ewsps Div(e(w), .P5
(9)
When we tune 7 from to O, the proxy distribution smoothly transfers from the unbiased one- hot distribution to a soft distribution, which reduces the variance of the one-hot estimation and stablizes training in the early stage. Although this comes at the cost of increased estimation bias at the beginning of training, the bias gradually decreases as the model fits the data distribution more accurately when the training goes
3.4 TOTAL VARIATION GUIDED LANGUAGE GENERATION (TAILR)
Finally, we introduce the TaiLr objective by summarizing the above results. Given the target token w. we derive the TVD between the proxy distribution f(w) = (e(w) + (1 7)p <t and the model distribution ps following equation 2b):
Drv (r)( "
_ = E (( min (1,
Pø (1x. )
(10)
flw)
5

---

Published as a conference paper at ICLR 2023
where the expectation is approximated by sampling from the proxy distribution using Monte- Carlo sampling. When sampling y/ * w. the gradient of Drv Drv(j("), p) (r(()))),p")) is always 0 which is inefficient for optimization. Therefore, we consider the non-zero gradient when ze is sampled as the target to- ken w to guide the model, i.e., -opp (w))/l())w)) and devise the TaiLr objective whose gradient is equivalent to it:
Po <t Tan 0) 7 7)p (w)
log P8 (ww), logP5
(11)
where the weighting factor is detached in the back preeppropagattin and only the log term receives gradi- ent. The equivalence of VoLTau (w; 0) and the non-7ero gradient of Drv (f(()) -P can be seen by applying f(x) V. log f()) = Vzf(z). In Figure 2 we show the computational graph of the TaiLr objective. As y switches from 1 to O, TaiLr is biased from an estimation of TVD to unweighted MLE. Intuitively, TaiLr downweights samples with low probabilities assigned by the model so that the model focuses modeling the high-quality samples during training and reduces overestimation of degenerated texts during inference. To counter the negative effect of random prediction at the early training stage, we set threshold as a lower bound of the weighting factor.
EXPERIMENTS
In the previous sections, we show that the proposed method is a practical estimation of TVD with theoretical guarantees. Next, demonstrate its empirical performance. First, conduct syn- thetic experiment to investigate the behavior of the model trained by TaiLr and MLE in controlled settings where the underlying oracle distribution is known. Second, we compare TaiLr with other baselines in a more realistic setting where we train generation models with standard architectures or finetune pre-trained models on a wide range of language generation benchmarks. More experimental details which are not included in the following sections are provided in Appendix E.
4.1 SYNTHETIC EXPERIMENTS
The synthetic data. In this subsection, our goal is to test the behavior of TaiLr in the task of text generation. Since we seek to analyze the distributional properties, we sample training data an oracle model whose distribution is known. Instead of using random distributions YYu et al. 2017 Guo et al. 2018). we follow LeBrun et al. (2022) and train an oracle model on real human texts to generate synthetic data so that the results a 1-layer LSTM on the texts of the COCO image caption dataset (Lin et al., (014) without any conditional inputs. We sample 10K synthetic data for training and 5K for validation.
better generalize to real data. Specifically,
train
The model setting, We train two LSTMs with the same architecture as the oracle model using MLE and TaiL,, which we denoted PMLE and Prai.. respectively. We train both models for 100 epochs and pick the best checkpoint with the lowest perplexity on the development set. We use random sampling obtain text samples from the learned generation models.
Model
PPLoracte + PPL 4
BLEU-4 t SeliBLEU
Training data
35.40
30.83
PMLE
31.64
22.64
27.74
33.27
Prats
26.91
22.42
28.36
28.91
Table 1: Automatic evaluation results of the models trained by MLE and TaiLr, PPLorace and BLEU assess the generation quality, while PPL and SelfBLEU emphasize on sample diversity. Boldface and underline indicate the highest and the second highest performance respectively.
Performance evaluation. To thoroughly evaluate the generation performance of the two models, we follow Yu et al. (2017); [Caccia et al. (2020) to evaluate the generation quality with PPLoracle and the coverage of the oracle distribution with PPLtest- Specifically, PPLoradee is the likelihood of the oracle model calculated on the samples generated by the learned model, while PPL is the likelihood of the learned model evaluated on the held-out data. We also include BLEU score (Pa- pineni al. 2002) calculate the average n-gram overlap between the generated sample and the

---

Published as a conference paper at ICLR 2023
Degenerated sample: koppu -82.5
Twir Mic
slightly decerated standing s standing behind
2015 MASROESSS RROM
poured hung
Modmum Oeotestimation
Real sample [same lesar as
-7.4
wooden stands on track and
wooded area
Real sample isame lammur O
19, 13)
14)
17)
128. 20)
35.9 -3E.0
-20.
-20.9 -53. -0.3
-15
donat with orange pespereni
ogg
(22
Figure 3: Estimation error map of PTait (left) and PMle (right) on the perturbed dataset. We present examples from different regions to illustrate the estimation behavior of the two models.
Figure overestimation eπror varying with lengths.
Maximum
held-out corpus, and SeIfBLEU (Zhu et 2018) which computes the average overlap of each gen- erated sample to other samples generated by the mode For evaluation, we use 20K held-out data and sample 20K texts from the two generation models, respectively. As shown in Table I TaiLr improves the generation quality by nearly 5 points of PPLoraaee without sacrificing the coverage to the oracle distribution as achieves similar PPLtest as MLE. We also observe that TaiLr achieves higher BLEU-4 than MLE while lower than the training data. Finally, we show that MLE has the highest SelfBLEU-4 which shows its tendency to over-generalize to unseen samples that may in- clude repeated patterns and degrade the diversity, while TaiLr achieves the lowest SelfBLEU-4. We also report the result using Mer2-Medium as the oracle model in (ppendix(EE..I.
Perturbation evaluation. Next, we evaluate models' behavior perturbed sequences and relate text degeneration as an implication of model's overestimation behavior on the pertubed data. To quantify the deviation of the model's 's estimation from the real data distribution, estimation error of a sample x as the difference between the sequence-level log probability given by the model and its true log probability, i.e., Error(æ:) log po (o log (ar)). Then we show the construction of the perturbed dataset. Given x sampled from Po. we iteratively apply small pertur- bations to x so that each lexical change is small. After N perturbations, x + (1) >... + (N) smoothly transfers data point into perturbed sample. We propose the following perturbations that highlight the widely observed text degeneracy patterns in generation: (1) Repeat a token in (repetition, Welleck al. (2020)) ). (2) Delete the last token in (oversmoothing, Kulikov et al. (202)) ). (3) Substitute a token in s with token from the vocabulary (incoherence, Holizman et al. 2020
define the model
We sample 20K samples from Po and apply N 30 perturbations to each sample.
We first plot the estimation eтror map of the two models on the perturbed dataset in Figure 3 For each perturbed sample (+)) the oracle log probability log po((!!)) is shown in the x-axis, the number of perturbations performed is shown in the y- y-axis, and the estimation error Error is reflected by the shade. From the figures the left, we first restate LeBrun et al. (2022) 's finding that exisiting models underestimate real samples while overestimating degenerated samples. Next, by comparing the two figures, we observe that as the number of perturbations increases, Praitr alleviates the over- estimation phenomenon of PMLE especially in the long tail. Finally, we draw cases from different regions in the figure to illustrate its implication on text degeneration. We first present degenerated sample ① on the top-right corner which has low oracle probability. We then present two real data samples 2 and 3 that have the same model probability under PTaile and PMLE as the degenerated one ① Although 33 is actually more probable than the degenerated one ! in the oracle distribu- tion, MIE cannot distinguish between them leading degeneracy patterns during generation.
To quantify the overestimation problem of perturbation, we further define the maximum overesti- mation error over N perturbations as max (r)) To manifest the overestimation problem i=1,--,N .N
during generation, we plot the maximum overestimation eгтor averaged on samples grouped with the similar length in Figure Note that the average NLL of these degenerated samples for PMLLE and Plal, is 11.09 and 10.82 respectively. For MLE the overestimation error amplifies as the generation
'The scripts of both BLEU and SelfBLEU are from https://ggttth...m..eeeeeaiiii blob/master/ot 1
7

---

Published as a conference paper at ICLR 2023
length increases while TaiLr maintains the error nearly at constant. This result demonstrates that TaiLr alleviates MLE' tendency to sample degenerated texts with the growth of the generation length by weighting the likelihood at each position of the sequence during training.
Error accumulation analysis. Finally, we analyze the error accumulation during autoregressive decoding. We follow AArora et al. (2022) and the metric ExAccErr that calculates the percent- age of excess errors due to the discrepency of training (conditioning on contexts sampled from Po) and inference (conditioning on contexts sampled from po). i.e., exposure bias. Detailed definitions borrowed from Arora et al. (2022) are provided in Appendix [..122 We found that the excess error of MLE model (40. (0)1%) is substantially higher than the model trained with TaiLr (8.6%), which demonstrates that TaiLr effectively reduces the error accumulation during autoregressive decoding.
4.2 REAL--DATA EXPERIMENTS
In this subsection, we describe the empirical evaluation of TaiLr on a wide range of real-world language generation tasks, including: (1) Machine Translation: Given sentence in the source language, the goal is to translate it into the target language. (2) Text summarization: Given a passage. the goal is generate short sentence that summarizes the main point of the passage. (3) Long text generation: Given a title, the goal is to generate a coherent long passage that conforms with the title. Statistics and sources of all datasets used in experiments are provided in Appendix (D]
Apart from MLE, we also consider the following typical baselines that proposed new training objec- tives beyond MLE: (1) Unlikelihood training (Welleck et al.) 2020) penalizes unlikely generations, e.g., token repetitions, through an auxiliary unlikelihood loss. (2) D2GPo (Li et al., 2020) pro-
poses a data-dependent gaussian prior objective that smooths the one-hot target distribution based on word embedding distance. (3) Loss truncation (Kang & Hashimoto 2020) abandons c-fraction of the training samples with the highest NLL, which heuristically optimizes distinguishability. (4) GOLD (Pang & He 2021) Iearns from human demonstrations using the off-policy setting of Rein- forcement Learning (RL). We choose to compare with GOLD-Ó that does not use scoring models with additional parameters for fair comparison. We use the paired bootstrap resampling (Koehn) 2004) in all tasks for significance testing.
Machine Translation. We evaluate the pro- posed method on a widely-used machine trans- lation benchmark IWSLT14 De-En using the standard Transformer architecture (Vaswani al.] 2017). Training settings and detailed hyperparameters of different models are pro- vided in Appendix [..2] The best checkpoint is selected based on the highest BLEU (Pap- lineni et al. 2002) score on the development set. We used beam search with beam size of 5 for decoding. In Table 2 we show the perfor- mance of our method and the baseline methods in terms of BLEU score. The results show that TaiLr achieves higher BLEU score compared to MLE, which indicates that TVD effectively improves the generation quality over KLD. TaiLr also significantly outperforms other objectives that modify the MLE baseline.
Method
Dev BLEU
Test BLEU
MI.E
35.81
34.27
Unlikelihood
33.92
32.82
D2GPo
36.09
34.50*
Loss truncation
35.631
34.48
GOLD
35.74#
34.68t
TaiLr
36.44
35.05
Table 2: BLEU score comparison on the dev and test set of IWSLTI4 14 De-En. t/$ means TaiLr is significantly better with p-value < 0.05/0.01.
Text summarization. We then test the proposed method on abstractive text summarization. We used the Annotated Gigaword corpus (Rush al., 2015)) as it is known have noisy references due to annotation errors (Klebanov & Beigman 2010; Kang & Hashimoto 2020). As pre-trained Transformer models have achieved strong performance, we thus propose to finetune the BART- base (Lewis et al., (2200) model with different methods and see whether they still improve upon the strong baseline. More training details and hyperparameter settings are provided in Appendix E.3 We select the best checkpoint based on the highest ROUGE-L (Lin) 2004) score on the development set. During inference, we use beam search with a beam size of 5 and prohibit decoding repeated 3- 3-grams. We report the ROUGE- 1/2/L scores on the test set of the Gigaword dataset in Table where TaiLr outperforms all the baseline methods in terms of all evaluation metrics. The result demonstrates the effectiveness of our method in the realistic setting where noisy data pairs exist.
8

---

Published as a conference paper at ICLR 2023
Method
R-2
R-L
Method
B-1t
D-41
rep-81
Mauve?
MLE
38.24
19.12
35.70
MLE
27.85
84.28
10.31
56.424
Unlikelihood
37.80*
18.34
34.84
Unlikelihood
27.88
85.46
10.06
59.35
D2GPo
38.52
18.92
35.64
D2GPo
22.73*
84.10
10.04
53.35
Loss truncation
38.62
19.29
35.851
Loss truncation
19.49
76.51
13.41
45.35t
GOLD
38.57
19.27
35.791
GOLD
25.25*
16.98*
28.23*
15.44
TaiLr
38.82
19.50
36.24
TaiLr
28.62
85.56
9.73
64.64
Table 3: Generation performance of dif- ferent methods on the test set of the Giga- word dataset. t/$ means Tail.r is signifi- cantly better with p-value 0.05/0.01.
Table 4: Results of automatic metrics on the test set of the WritingPrompts dataset. t/4 means the higher/lower the better. i/3 means Tailr is signifi- cantly better with p-value < 0.05/0.01.
Long text generation. Finally, we evaluate TaiLr on the task of long text generation to show its performance in open-ended generation. We evaluate on the WritingPrompts (Fan et al. (0018) dataset and leverage the generation ability of the pre-trained model by finetuning the BART LLRR-base model. More training details are provided in Appendix [E.4 For evaluation, sampled 1,000 titles from the test set following Ji & Huang (021). We use Nucleus sampling (Holtzman et al. 2020) with p 0.95 and restricte maximum generation length of 1,024 subwords. For automatic evaluation. we use BLEU-n (B-r) to evaluate the n-gram overlap to the human reference, Distinct-n (Li et al.) 2016) (D-n) to compute the ratio of unique n-grams, rep-l (Welleck al. (0200 calculate the repetition rate within the context window of I, and Mauve (Pillutla et al., 2021) that assesses the distributional deviation of model-generated texts and human language by calculating the area under the divergence curve. As shown in Table 4 TaiL. outperforms the MLE baseline in terms of all metries. For other baselines, Loss truncation abandons long samples with high NLL leading to overly short generations and low n-gram overlap to the reference. GOLD tends to concentrate on very few modes in the target distribution as discussed by Pang & He (202T) which causes low diversity and large discrepency to the distribution of human language.
Ablation study and discussion. We conduct ablation study of adjusting 7 on different tasks to show that its tendency and sensitivity interval vary different tasks. In Figure 5 in Appendix [E.5] present the result of adjusting ? on WritingPrompts on the left and observe that the highest Mauve score is achieved when 7 is around 10-5, while the performance quickly degrades as 7 approaches 1. On the right of Figure we observe that the best performance is achieved when is around 0.1 on IWSLT14 De-En while either increasing or decreasing 7 leads to a notable performance drop. From an empirical view, the scale of the best performing 7 is related to the intrisic entropy of the dataset. For stable training, we require the estimation variance in equation (9) be small, which leads to small y when the entropy of the data is high. Since the model generally has higher NLL on long text generation than on machine translation, the scale of the best y is thereby shifted towards on WritingPrompts. To further determine the sensitivity interval of 7, we suggest to tune the scale of y based on the average NLL on the training data, where the empirical principle is to make the weighting factor in equation (11) relatively large to stablize training. Although simple, argue that this parameter is crucial to the generality of the application, and we leave other solutions of dynamically adjusting annealing this hyperparameter to future work.
CONCLUSION
In this work, we draw attention to the total variation distance (TVD), a robust alternative to KL divergence (KLD). We show that TVD addresses the zero-avoiding problem KL.D and mitigates overestimation of the degenerated sequences, which in turn improves the overall generation quality. To apply TVD to the task of language generation, we derive practical upper bounds, and introduce our Total Variation Guided Language Generation (TaiLr) objective that balances the bias-variance tradeoff of estimating TYD with a tunable hyperparameter. Our experiments on synthetic data and real-data benchmarks demonstrate that TaiLr alleviates the overestimation problem and the error accumulation during autoregressive decoding, and improves the generation quality over competitive baselines beyond MLE wide range of language generation tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported by the Major Project of the New Generation of Artificial Intelligence (No. 2018AAA0102900), This work was also supported by the National Key Research and Development Program of China (No. 2021ZD011320 Young Scholars (with No. 62125604).
and the National Science Foundation for Distinguished
REFERENCES
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of eπror accumulation in language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 700-710. As- sociation for Computational Linguistics, 2022. doi: 10.18653/v1/0022 wtt.2222....nnnn---ieeem findings--cll... ac1.58. URL
https: /oi.
http:///w/2222222 win.iigg...nnnn......... acl 558
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varsh- ney. Mirostat: a neural text decoding algorithm that directly controls perplexity. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
net forum?id
AIGIJZEly5_
Rudolf Beran. Minimum hellinger distance estimates for parametric models. The annals of Statistics, pp. 445-463, 1977.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 1457c0d6bfcb45ee 44520066666604437773111166612222
neur nwwrips. /cc/papper/222/hashi 42f Abstt html
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In 8th International Conference on Learning Representa- fions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
fictps:/coperrett
net wowfuoormuoooooo For
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01 2019.
Yao Dou, Maxwell Forbes, Rik Konzel-Kedziorski, Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7250-7274, Association for Computational Linguis- tics, 2022. URL / acc antho www.hhooolgyy....ooo wwwp://g........o ac long.o00 .501
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Pupers, pp. 889-898. Association for Computational Linguistics, 2018. doi: 10. 10.18653/v1/
108- URL https http://ccaaatooooogoooooooo oow.oog......................... Pttp///8881887777711
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. GLTR: statistical detection and visualization of generated text. In Marta R. Costa-jussà and Enrique Alfonseca (eds.), Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019,
10

---

Published as a conference paper at ICLR 2023
Florence, Italy, July 28- August 2, 2019, Volume 3: System Demonstrations, pp. 111-116. Association for Computational Linguistics, 2019. doi: 10.18653/.1/11933019. URL https:
1ttp://10..1663////p///.o 3019
lan J. Goodfellow. On distinguishability criteria for estimating generative models. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL
http://arxiss: wrw. 1412.1145 6515
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alckh Agarwal, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using fkelihhoddffre importance weighting. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Sys- tems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11056 11068, 2019. URL 0acaleeeesc19cc9a222331133331
http://deeeeeeer.errree...pp/epp - html
Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. Long text generation by modeling sentence-level and discourse-level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 6379-6393, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202 I. acl-long.499)
URL http::::/aaaaaaa BtKL.tttps::/aclanttto
logy. org.
.ac- 499
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.). Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5141-5148. AAAI Press, 2018 URL htthh::tt//wttt /ttp//wwww...m aaai..rg/ ocs/ ittp//ndee...m//m php CAAIAAAAAISS http:/apppr/iiiie 2ewww//633 16360
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evalu- ation for natural language generation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associa- lion for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Min- neapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 1689-1701. As- sociation for Computational Linguisties, 2019. doi: 10.18653/v1/11--11 1169. URL https: aeddi.orr//0... 100.0.885533
Tianxing Hc, Jingzhao Zhang, Zhiming Zhou, and James Glass. Exposure bias versus self-recovery: Are distortions really incremental for autoregressive text generation? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5087-5102, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- org/2021.
tics. doi: 10.18653/w1/0221.mmml http://2222emmmmammam-i/il 415.
emn ma ii.iin.. 415
Matthias Hein and Olivier Bousquet. Hilbertian metrics and positive definite kernels on probabil- ity measures. In Robert G. Cowell and Zoubin Ghahramani (eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, AISTATS 2005, Bridgetown, Barbados, January 6-8, 2005. Society for Artificial Intelligence and Statistics, 2005. URL http:/ / /wwwwwwwo . gatsby....................ttt... uel wwc..acc... uk/ striisttts/uuu http://ssttttt//upppppper/////2/
papppp//1p........ pdf
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL.
net//orrunid
Ferenc Huszar. How (not) to train your generative model: Scheduled sampling, likelihood, adver- sary? CoRR, abs/1511 1511111111000 2015. URL URL http::tt////atti/ wwtpwwwwwwww..../////oo/
/t//////////rrra..oo/////////// 05101
11

---

Published as a conference paper at ICLR 2023
Haozhe Ji and Minlie Huang. Discodvt: Generating long text with discourse- aware discrete vari- ational transformer. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Punta Cana, Dominican Republic, 7-11 November,
Processing, EMNLP 2021, I, Virtual Event / 2021, pp. 4208-4224 4224. Association for Computational Lingiusininniiiiiiii Linguistics, 2021. doi: 10. 10.18653/v1/2011. emnlp-main.347. 347. http:/ppps/......./......oo.
emnip wwwaanppppiinm..mmeaemaammaamaam main. 347
Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: robust and efficient fine-tuning for pre-trained natural language models through principled reg- ularized optimization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computationnal Linguis- tics, ACL 2020, Online, July 5-10, 2020, pp. 2177-2190, Association for Computational Linguis- tics, 2020. doi: 10. 12.280. 202 acl in. 197
197. URL PRLphtts:://ddt http///ii.oogrrg////// org/ 10. 18.73//0..8663111
Daniel Kang and Tatsunori Hashimoto. Improved natural language generation via loss truncation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 718-731, Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020.acl- 66.n. BtLhttps::/ /tt:s///doooooo org 11w111...1111211 1ttp://...163/////222222 acl main..6.
Beata Beigman Klebanov and Eyal Beigman. Some empirical evidence for annotation noise in a benchmarked dataset. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings. June 2-4, 2010, Los An- geles, California, USA, pp. 438 446. The Association for Computational Linguistics, 2010. URL htttps://aclantholoogyy org www.:oooy...//11--111111
Jeremias Knoblauch and Lara Vomfell. Robust bayesian inference for discrete outcomes with the total variation distance. ArXiv, abs/2010. 13456, 2020.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25- 26 July 2004, Barcelona, Spain, pp. 388-395. ACL, 2004. URL htttps://aclatthhee org/ W04 32507
Ilia Kulikov, Maksim Eremeev, and Kyunghyun Cho. Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling. CoRR, abs/2112.08814 2021. URL htwpg////////e
attps://arxiv: o abs abs 221...12... 08914
Matthieu Labeau and Shay B. Cohen. Experimenting with power divergences for language mod- eling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4114, Hong Kong, China, November 2019. Association for Computa- tional Linguistics. doi: 10. 00.18653/1//D119142..
URL ht..s::/aclarthhooogy/ oogg/
D19- 1421
Benjamin LeBrun, Alessandro Sordoni, and Timothy J. O'Donnell. Evaluating distributional dis- tortion in neural language modeling. In International Conference on Learning Representations, 2022. BRLLhtpps: httpp:::///pe/eeveieee
net/ wttee//oorummmi-ttteeeelt
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, Online. July 5-10, 2020, pp. 7881-7880. Association for Computational Linguistics, 2020. doi: 10. 20118653/1/2022.acc
703. URL
https::/ddo.orgg// 1tt.:o..o00r0111..8883///3///000
acl -ma 703
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A
objec-
tive function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow
12

---

Published as a conference paper at ICLR 2023
(eds.), NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies. San Diego California, USA. June 12-17, 2016, pp. 110-119. The Association for Computational Linguistics, 2016. doi: 10.18653/v (/653/v1/n166 16 1014,
Zuchao Li, Hai Zhao, Yingting Wu, Fengshun Xiao, and Shu Jiang. Controllable dual skew diver- gence loss for neural machine translation. arXiv preprini arXiv: /908. 08399, 2019.
Zuchao Li. Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpthtttt/ppeeeeeeeeeee ew. net/forunm d=S1 efTTVYDr
Chin-Yevr Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science. pp. 740-755. 755. Springer, 2014. doi: 10. 978-3-319 10602- 1 18.48.
URL https:/ /tt.///oiiiiio org. 1000/ 3-1 3-9--00002-2
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Tom Minka. Divergence measures and message passing.
Technical Report MSR-TR-20 T0--RR-200 2005- comeettuoooeenneeeeeee
173. Microsoft Research, January 2005. URL research/pube cat on wew.errreeeeeeeaaaeeeeeee-aaeeee
Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured predic- tion. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1723-1731, 2016. URL httpss//pprcoeeeeinggg wwwp://wwweeeinonnn..........p.p cc wwtppppppe//////aaaeeeepe 2 2EB5000122111110222555550---2--2
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. 9th International Conference Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL RovX-uń1Hua
wwwpeviwwwwweeefeeeeomeeee fort//iormm?i
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02. pp. 311-318, USA, 2002. Association for Computa- tional Linguistics. doi: 10.3115//07303330073333 URL
1073083. 10731361
Ramakanth Pasunuru and Mohit Bansal. Multi-reward reinforced summarization with saliency and entailment. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers). pp. 646 653. Association for Computational Linguistics, 2018. doi: 10.18653/v1/n 1/118-2122 URL http:::
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaïd Harchaoui. MAUVE: measuring the gap between neural text and human text using di- vergence frontiers. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34:
13

---

Published as a conference paper at ICLR 2023
Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 4816 4828, 2021. URL UtL/atttsss//rooceednnn.... neuripp cc/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In OpenAl Technical Report, 2019.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Lluís Mårquez, Chris Sallison-Burch. Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing. EMNLP 2015, Lisbon. Portugal, September 17-21, 2015, pp. 379- 389. The Association for Computational Linguistics. 2015. doi: 10.18653/v 1044. URL
https: //doi orgg / 10. 1 v1 1
Yuxuan Song, Ning Miao, Hao Zhou, Lantao Yu, Mingxuan Wang, and Lei Li. Improving maximum likelihood training for text generation with density ratio estimation. In Silvia Chi- appa and Roherto Calandra (eds.). The 23rd International Conference on Artificial Intelli- gence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Itly], vol- ume 108 of Proceedings of Machine Learning Research, pp. 122-132. PMLR, 2020. URL http: http:////oceeeenns./e/e mlr. press /vt./vl11//// song20a -ong20a.hmml html
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (eds.), Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999], pp. 1057-1063. The MIT Press, 1999.
Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics, 52(1))479 1988.
Ramon Van Handel. Probability in high dimension. Technical report, PRINCETON UNIV NJ, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https: httts::/pprooceiasss.. neurip.............. cc paper/pppppppeeeaaa / /tteerr/2117/e/ 2017/has7// TF522435776ee1100000aaaaaaaaaaa3 htw.aaacc...aaaaaaaaaatttaa....a html
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL /tR//tttpp//ceeervview net /tt///oormmmmdeee ONt
Ronald J. Williams. Simple statistical gralient algorithms for connectionisttriiifocceeen learning. Mach. Learn., 8:229 256, 1992. doi: 10.1007/BF009996666 URL /tttp:://doi.
2x1/10.1007/B2009920
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. A study of reinforcement learning for neural machine translation. arXiv preprint arXiv: 180.00666.... 2018.
Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2734-2744, Online, April 2021. Association for Computational Linguistics. doi:
1//33//1/222.ea// mocelmmin.2236.. URL //aclanthology orq/2021, eacl atmain.23.
https:
Haoran Xu, Benjamin Van Durme, and Kenton W. Murray. Bert, mbert, or bibert? A study on contextualized embeddings for neural machine translation. In Maric- Francine Moens, Xuan- jing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference
14

---

Published as a conference paper at ICLR 2023
Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. tional Linguistics, 2021. doi:
Association for Computa- URL https://doi. org/
wttp:///ee22mmmmmmmmmmmmma. 10. 18653/v1/2022 emn 1 ma in. 534
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-Firh First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, Cal- iformia, USA, pp. 2852-2858. AAAI Press, 2017. URL http: / aaa org/ocs ot// index. php/AAAT AAAT AAAIIAATAIII www.://ppppp/va//e /ttp:////AAA//////////////e/e.a
14344
Huan Zhang and Hai Zhao. Minimum divergence maximum margin: empirical comparison on seq2seq models. In International Conference on Learning Representations, 2019. URL https:
//openrevi ew. fort/f d=H1 xD9sR5Fm
Zheng Zhao, Shay B. Cohen, and Bonnie Webber. Reducing quantity hallucinations in abstrac- tive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2237-2249, Online, November 2020. Association for Computational Linguistics. weeclannthoogyy org/2020.
doi: 10. http://///.....iinnnnnnemmm///pp Findings-emn www.::mmmmppp...mm.... 2033
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.
15

---

Published as a conference paper at ICLR 2023
A DERIVATIONS AND PROOFS
A.1 DERIVATION OF EQUATION
Starting from equation 2b), rewrite the summation into the expectation of pa which is further approximated by sampling context-target pair (a", y²) from Po:
VeDTv(Po.po) pa) = Va Σ
min Po(y) (,o))=))))))
(12)
NEV
min (1, po(3/lx"))
V Eyyrr
(13)
(=(()
(O (x*)
Va min
(14)
Po( (x*)
Vepo (x")
po(a)((=*) < Po (x*)
Poly (x*)
(15)
po(y"la=* 2 (v
where the third line is approxmiated by Monte-Carlo sampling and the last line of case follows from comparing pe (y" (x*) and Po lx*).
A.2 PROOF OF PROPOSITION
Proposition 1. Given Po (y)lx) = IT (B)( and po(v)x) = II (Wr). then the following
condition holds:
Div(Po:Po po) S Eyyup
Σ Div
(5)
Proof. Let a; Po (, b, = (v:). We define et = (as x ... ... x az) x (be+1 X ... x bT), and then we present the following inequality:
(v:),
leT
col la
a-11:
(16)
which can be derived from the general triangle inequality. After replacing with ar and b, we have:
T
T
II
b
bel
II =
II j=t+1
bj
(17)
t=1
Then we derive the upper bound of equation
22ah:
Drv(po: po)
Σ poly))
pe (y
(18)
1
II Po (yL)
II Po
(wc)
(19)
2 V1.
1=1
1=1
T
T
<
Σ
po (vi) po5((()) p5 po'(u]]-
Σ
II
P (1()))
(20)
2
We 1=1
1-- JF j=t+1
E
po po((yc)
po Po"(xi)
(21)
V
Egrep t=1
Σ
(Tv <t p55
(22)
where equation 20 uses the conclusion from equation I7 and equation 21 marginalizing out yt+,, yrr.
is obtained by
16

---

Published as a conference paper at ICLR 2023
A.3 PROOF OF PROPOSITION 2
Proposition 2. Given w Do and the one-hot distribution e(w), then the following condition
holds:
Div(Pō' spo"'ppp'))))) S Buwuss ( .P5
Proof. Given that E wo [elwo]] = =po" and we have:
Drv(po" st -D6
1
Σ
Po
<t (y=) (v²(
Po (va)
(23)
2
1
Σ
18 Bunups
PP (vi)
(24)
2
1
Σ
I8 Bewnpps
[e(wr) (ye) po (vx))
(25)
2
E
Bw
((w) - Pg P3'(va)
(26)
2 Dt
Drv(e(w), p55)
E Bw
(27)
where equation 26 is obtained by applying the Jensen inequality since the absolute value function |- convex.
A.4 DERIVATION OF EQUATION
We first consider the following triangle inequality:
( (wc) p5 (w+)
( (we) p<* (wv)) pp'(()) po v5'(vn) (55(()) p (vn)) (28)
where yr can take any token in the vocabulary. Then we sum y/ over the vocabulary and take the expectation of w with respect to pet which results in the form of TVD:
E Ewnps' St (Tv( 15*) < Ewwps' (rrv( .p5t]] Div(i ,po) Div(Po' :10"). ( (29)
By defining the estimation error as: Error = E Ewwns! (rvv(pu)) .50) (rr((((),pp') - Drv(po", )po)), we
obtain equation
A.5 CONNECTION WITH TSALLIS G-ENTROPY
When using the one-hot distribution as the proxy distribution: f(()) <(n), the variance term of equation (8) can be derived into:
E Ewwpss prv(efw), p50) = Euapgt 1 Σ
min min(e(w) (e(w) (o')))
(30)
St
Eump
1 po' (w)
(31)
(w))
(32)
As the Tsallis a-entropy is defined as:
(v) ala-1j (1 E.P,).
a a=1
(33)
Σ, p; logPi, Pi:
Thus, the variance can be seen as (H2 which reflects the intrinsic uncertainty of the oracle data distribution.
17

---

Published as a conference paper at ICLR 2023
B ADDITIONAL RELATED WORK
Text Degeneration and Solutions, The phenomena of text degeneration was specified in previous literature (Holtzman et al. 2020 Welleck et al. 2020): a well-trained language model is observed to get stuck in repetitive loops, or produce incoherent contents. One line of works attributed this problem to the improper goal of the decoding algorithm that maximizes likelihood, and proposed to sample from truncated probability distribution to balance repetitiveness and incoherence (Holtz- man et al,, 2020 Basu et al., 2021). While another line of works attempted to address this issue by designing new training objectives. Welleck et al.] (2020) proposed to minimize the likelihood of the degenerated cases through unlikelihood training. However, designing unlikelihood objectives to cover every degenerated cases is impossible. Policy gradient-based RL algorithms are another widely adopted options due to the flexibility of reward designing (Norouzi et al. 2016 Pasunuru & Bansal 2018 Wu et al., 2018). Recently, Pang & He (202T) proposed an off- off-policy RL algo- rthim that achieves strong performance upon the pre-trained generation models by circumventing the optimization challenges in traditional on-policy RL methods Choshen al. (2019). Our work falls in the second line. Instead of relying on heuristics, we start with the distributional property of the current MLE objective, and derive a new principled objective by leveraging the total variation distance which we show to reduce the overestimation of degenerated samples.
Distance Metries beyond KLD. Maximum likelihood estimation (MLE) is used standard train- ing criterion of language generation due to its simplicity and its theoretical guarantee of minimizing the Kullback Leibler divergence (KLD) (Zhang & Zhao 2019). However, minimizing KLD be- tween the data and model distribution is known to lead to the zero-avoiding solution, which forces the model to cover all the modes in the data by sacrificing the fitting accuracy to individual modes. To address this issue, previous study have introduced other distance metrics to substitute or regu- larize the standard KLD. The reverse KLD between the model and data distribution is previously studied to balance the standard KLD (Huszar 2015 Li ct al. 2019 Jiang et al.) 2020). In the field of language generation, [i et al. (2019) applied reverse KLD to regularize the standard KL.D in ma- chine translation. However, the regularization intensity needs sophisticated controlling which makes the approach less practical. Total variation distance (TVD) is previously adopted to evaluate the gen- eration models via distinguishability of samples (Hashimoto al. 2019 Gehrmann et al.) 2019 He et al. 202T), Generative adversarial networks are known to directly minimize distinguishability but challenging to optimize in practice (Caccia et al. 2020). Kang & Hashimoto (2020) proposed to optimize TVD by heuristically truncating samples with high NLL. Other metrics such as the Power divergence [Labeau & Cohen 2019) and the Hellinger distance (Zhang & Zhao 2019) are also ex- plored in previous literature, but they lack theoretical justification and empirical evidence to their superiority in language generation. In this work, we seek to directly optimize TVD and derive a training objective called TaiLr by deriving practical upper bounds on TVD.
C DISCUSSIONS
In this section, we discuss the connection of our method to two major baselines we compared with, ie., loss truncation (Kang & Hashimoto| 2020) and GOLD (Pang & He (02T) as the two works share similar motivation with us to downweight unlikely samples. We will briefly review the two methods and then compare them to our method to show why our approach excels.
C.I Loss TRUNCATION
Briefly speaking, loss truncation abandons samples with high log loss (also known as the negative log-likelihood) in training. In the Proposition 1 (Kang & Hashimoto 2020). they proved that the model's 's log loss on the truncated data distribution with an extra constant sets an upper bound on the total variation distance (TVD). The authors thus proposed to optimize the model on "simpler" subset of the full dataset in order to achieve smaller log loss that would tighten the upper bound (note that the constant would be trade-off to the bound removing more data will increase the constant). To achieve this, they first heuristically drop the "hard" samples with the highest log losses (hot-start stage), and then train the model on the remaining data (training stage).
Loss truncation can be regarded as downweighting samples at the sequence level with binary weights. However, performing sequence-level sample dropping may systematically lose some spe-
18

---

Published as a conference paper at ICLR 2023
eific modes in the data, which is undesirable. For example, observed overly short generation length of loss truncation (384 compared to 511 of MLE) on WritingPrompts as a result of dropping long samples with high log losses. Moreover, determining which samples to drop heavily relies the first hot-start stage where the model should not be too random or too certain. In practice, is hard to determine the degree to which the hot-start training should proceed. In work, derived our training objective TaiLr from the practical upper bounds of TVD, which softly weights the log loss on the token level. We also analyzed the bias and variance tradeoff in estimating the bound. By appropriately setting the tradeoff hyperparameter, we can reduce the estimation variance at the start of training and gradually decrease the estimation bias as the model becomes more accurate, which circumvents the need of hot-starting the model.
C.2 GOLD
the other hand, GOLD learns the generation policy an offline setting, i.e., calculating rewards on target trajectories by reweighting the policy gradient with importance weights. They made sev- eral approximations including simplifying the multi-step importance weights into a single step, and simply assuming the per-step target policy to be uniform. Besides the importance weight, they also came up with several reward functions which score the sequence with another generation model trained by MLE. Although starting with quite different motivation, the final form of the training objective is quite similar to ours.
From an empirical view, GOLD assigns soft weights to samples the token level based the importance weight and some hand-crafted reward functions. However, one downside (also discussed in the original paper of GOLD) is that the objective only focuses on capturing the high-likelihood patterns in the data and fails when there are variety of candidates given the same input in the data. This also reflected in our experimental results in Table 4 where GOLD has the highest repetition rate and the lowest diversity. In our paper, we analyze the situation where the target distribution has high entropy, and decompose the error of estimating TVD into the bias and the variance term (Section 3.3) We found that the GOLD-style importance weight actually leads to an unbiased but high-variance estimator of TVD, which renders the optimization hard to proceed. In our TaiLr objective, we proposed to balance the bias and variance with hyperparameter, which can be effectively applied to different generation tasks like machine translation, text summarization, and long text generation by tuning this hyperparameter.
D DATASET DETAILS
We provide the statistics of the datasets used in $ $4.2 in Table 5
Dataset
train
dev
test
IWSLT14 De-En
160,239
7,283
6,750
Gigaword corpus
3,803,957
189,651
1,951
WritingPrompts
272,600
15,620
15,138
Table Number of examples in each split of the datasets used in the experiments.
To download and preprocess the IWSLT14 De-En dataset, we follow the instructions in
https: http:/thhu.co//occecaececo http///cccoooreeerrccaaaaeeeeeee
Eranslation| To download and preprocess the Gigaword corpus, we follow the instructions
httts://huggiggffatt co/ http//daaattssiiiaaaaaa cttp:/cooccccccaaaeeeeeeee htt.h//accccceccccooeeeeeeeerll
To download the WritingPrompts
dataset, we follow the instructions in
fai Fel
19

---

Published as a conference paper at ICLR 2023
E EXPERIMENT DETAILS
E. SYNTHETIC EXPERIMENTS
Both the models trained with MLE and TaiLr are 1-layer LSTMs with a hidden size d = 128. The models are trained using Adam optimizer (31 0.9, 82 0.999) with fixed learning rate of le-3 and no weight decay. We use a maximum number of 4096 tokens at each batch. The dropout rate is
0.1. We evaluate the perplexity on the dev the entropy of the synthetic data is high we tune the hyperparameter in the proxy distribution Y {10-6, 10-7 ....,, 0.1, 1.0]. Since the model probability is not reliable the start of training, we tune the threshold of the weighting factor by bm {0, 0.1, 0.2, 0.3}. On the synthetic data, we found that a large y leads to slower convergence and higher validation loss, which indicates large estimation variance during training. While the best performance is achieved when 10-?,
the end of each epoch. As
bm 0.2. The experiment conducted using the fairseq Toolkit.
E.1.1 EXPERIMENTS USING LARGER ORACLE MODEL
To demonstrate the effectiveness of our method on more realistic oracle distribution, addition- ally conduct synthetic experiment using GPT2-Medium (Radford et al. (201)) as the oracle model. We randomly sample 500K sequences with maximum length of 64 tokens, and split them into two sets with 400K and 100K sequences for training and evaluation, respectively. We train 6-layer Transformer with a hidden dimension of 512 and 16 heads from scratch using the MLE and TaiLr objective on the synthetic data, respectively. Both models are trained for epochs with a initial learning rate of 10-3 using linear scheduler. The batch size is set to 64, with gradient accumula- tion steps of 4. The best checkpoint is selected based on the perplexity on the dev set. We tune the hyperparameter 7 E {10-*, 100- - 10 5J and 6n E {0, 0.1, The best performance of TaiLr model is achieved when 10-7 and bm 0.1. We report PPLoracles PPLtest, BLEU-4 and SelfBLEU-4 T
Model
PPLoracle $ PPL 1 BLEU-4 t
Sel/BLEU-4
PMLE
138.35
260.09
5.76
6.38
PhaLr
137.59
253.01
5.96
5.94
Table 6: Automatic evaluation results of the models trained by MLE and Tail. using GPT2-Medium as the oracle model. Boldface indicates the highest performance.
The results show that our proposed TaiLr objective consistently outperforms MLE in terms of gen- eration quality (PPLorastes BLEU-4) and coverage (PPLtest- Sel/BLEU-4) when trained on the sam- ples from stronger oracle model. Note that randomly sampling from GPT2-Medium produces highly diverse corpus that is more difficult for the Transformer model to fit than the LSTee oracle model. Therefore, the scale of the perplexity is higher than those reported in Table
E.1.2 ERROR ACCUMULATION ANALYSIS
We define ExAccErr by adapting the definition from Arora et al. (2022) to our setting (As we fix the decoding strategy random sampling, omit it in the definition):
%ExAccEm((I) Rei(pe) lec1
x 100%
(34)
legi
where Reg((po) is the inference-time regret of the model po imitating oracle Po on the length-l context induced by the model po (equation (12) in Arora et al. (2022)) and e<g is the average per- step error of pa evaluating on the context sampled form oracle P. up to length (equation (16) in
'hhe entropy of the synthetic data is 10.77 considering the maximum entropy for dataset with vocabulary size of 5000 is 12.28.
20

---

Published as a conference paper at ICLR 2023
Arora et al. (2022)) ):
log Polue
Rei((p) =
Σ
E Pecipp t=1 gcmpo(-lMee)
(35)
po (e)
log po(uelyee) (v
Egi
I Mecropo 1=1 pe-Po(-lBer)
(36)
Intuitively, ExAccEır measures the excess accumulated error during autoregressive generation by deliminating the model's estimation error on the oracle data. In practice, equation (35) is approx-
imated by sampling Yi from ps using importance sampling. We calculated the accumulation error with a context length of 15 where the error begins to amplify.
E.2 MACHINE TRANSLATION
All the baseline models use the same Transformer architecture as ours which consists of 6 encoder and 6 decoder layers, 4 attention heads, an embedding dimension size of 512, and a ferw-fo hidden dimension size of 1024 at each layer. The following hyperparameters are general for all models unless specified otherwise. The models are trained with Adam optimizer (3, = 0.9, B3 = 0.98) using inverse square root schedule with initial learning rate of 3e-4 and weight decay of le-4. We train the models for a total number of 80 epochs with maximum of 4096 tokens per batch and use 4000 steps of warmup update. We set the dropout rate to 0.3, and use label smoothing of 0.1 as standard practice. The specialized hyperparameters for different baselines are determined with grid search based on the BLEU score on the dev set. For Unlikelihood training. weight of the unlikelihood loss in {0.1, 0.5, 1.0, 2.0} and found 0.5 works the best. For D2GPo, we tune the weight of the data-dependent prior objective in {0.1, 0.5, 1.0, 2.0) and the temperature in (0.5, 1.0, 2.0} found that setting the weight and temperature to 0.1 and 2.0 respectively works the best. For Loss truncation, we tune the fraction threshold c E {0.05, 0.2, 0.3, 0.4] and found that 0.1 works the best. As this baseline hotstarting with the MLE objective, we trained the first 10 epochs with MLE and the rest 70 epochs using loss truncation. For GOLD, we tune the lower bound of the importance weight in 10.1, 0.2, 0.3} and found that 0.2 works the best. For TaiLr, we tune the threshold 6m E (0.1,0.2, 0.3) and y E {0.01, 0.1, 0.5, 1.0} and found that bm = 0.3, 7 = 0.1 works the best.
tune the
E. IMPROVEMENT OVER STRONG BASELINES
In this subsection, we demonstrate that our method can be applied to strong baseline models to further improve performance. We choose BiBERT (Xu et al. (02TT) since it sets the current state-of- the-art on IWSLT14 De-En dataset. Xu al. (2021) proposed to stochastically select BERT layers as contextualized word embeddings of the MT model, and train the single model in dual direction so that the source and target direction could enhance each other. Since our training objective is orthogonal their modification in model structure, our TaiLr objective and train a new model, namely BiBERT + TaiLr. We tune the hyperparameters y E 10.1, 0.5, 1.0) and bm = (0.1, 0.2, 0.3) and select 7 0.1, bm 0.3 based the BLEU score on the dev set. We follow the instructions in their code and keep other hyperparameters verbatim except that we double the batch size in order run the model on two-times fewer GPUs than that they required. We also report the result of BiBERT we re-implement to see the relative improvement.
can simply replace the MLE objective with
The results are shown in Table 7 In the one-way training setting where the model is trained only on De-En data pairs, BiBERT combined with TaiLr outperforms BiBERT using standard MLE by over I point of BLEU score. In the dual-directional training setting where data pairs of both De-En and En-De are used, TaiLr still brings an improvemt of 0.5 in BLEU. The results demonstrate that our proposed method can be equipped to SOTA models to further boost the performance.
21

---

Published as a conference paper at ICLR 2023
One-way Training
Test BLEU
BiBERT (Table 2. Xu al. 2021)
37.58
BiBERT (Our implementation)
38.01
BiBERT + TaiLr
39.12
Dual-directional Training Fine- Fine-Tuning
Test BLEUU
BiBERT (Table 3. X, al. 2021
38.61
BiBERT (Our implementation
38.73
BiBERT + TaiLr
39.23
Table 7: Performance comparison of BiBERT trained with standard MLE and our TaiLr objective.
E.3 TEXT SUMMARIZATION
All the baseline models are finetuned BART-base model with their respective objectives using the following hyperparameters unless specified otherwise. We used Adam optimizer (81 0.9, 82 = 0.999) with fixed learning rate of le-4 with no weight decay and no warmup updates, which we found to perform the best. We train the models for 5 epochs with a maximum of 8192 tokens per batch and evaluate on the dev set at the end of each epoch based the dropout rate in the feedforward and the attention layer both to 0.1 and clip the norm of the gradient to 0.1. The label smoothing coefficient is set to 0.1 as standard practice. Other specialized hyperparameters for different baselines are determined with grid search based on the ROUGE-L. score on the dev set. For Unlikelihood training, we tune the weight of the unlikelihood loss in {0.1, 0.5, 1.0, 2.0} and found 0.1 works the best. For D2GPo, we tune the weight of the data- dependent prior objective 10.1, 0.5, 1.0, 2.0} and the temperature in (0.5, 1.0, 2.03 found that setting the weight and temperature to 0.1 and 2.0 respectively works the best. For Loss truncation, we tune the fraction threshold c E f0.1, 0.2, 0.3) and found that 0.2 works the best. For this baseline, we trained the model for 1 epoch using MLE and then continued to train 4 epochs using loss truncation. For GOLD, we tune the lower bound of the importance weight in (0.1, 0.2, 0.3) and found that 0.2 works the best. For TaiLr, we tune the threshold bm E (0.1, 0.2, 0.3) and y E (0.4, 0.6, 0.8, 1.0} and found that bn = 0.2, 7 = 0.8 works the best.
ROUGE-L score. We set
E.4 LONG TEXT GENERATION
All the baseline models are finetuned BART-base model with their respective objectives using the following hyperparameters unless specified otherwise. We used Adam optimizer (3₃ = 0.9, 32 = 0.999) with fixed learning rate of le-4 with no weight decay and no warmup updates, which we found to perform the best. We train the models for 5 epochs with a maximum of 8192 tokens per batch and evaluate on the dev dropout rate in the feedforward and the attention layer both to 0.1 and clip the norm of the gradient to 0.I. Other specialized hyperparameters for different baselines are determined with grid search based on the ROUGE-L score on the dev set. For Unlikelihood training, we tune the weight of the unlikelihood loss in {0.01, 0.1, 0.5, 1.0} and found 0.01 works the best. For D2GPo, we tune the weight of the data-dependent prior objective f0.01, 0.1, 0.5, 1.0] and the temperature in {0.5, 1.0, 2.0} found that setting the weight and temperature 0.01 and 1.0 respectively works the best. For Loss truncation, we tune the fraction threshold c E 10.1, 0.2, 0.3} and found that 0.1 works the best. For this baseline, we trained the model for 1 epoch using MLE and then continued to train 4 epochs using loss truncation. For GOLD, tune the lower bound of the importance weight in {0.1, 0.2, 0.3) and found that 0.2 works the best. For TaiLr, we tune the threshold bm e 0.2, and 7 E (10-7, 1000000000, 10....... 0.13 and found that bm 0.2, y 10-5 works the best.
the end of each epoch based on perplexity. We set the
E.5 MORE ANALYSIS ON ABLATION STUDY
We plot the weight
7+(1 (117))p) (10) in the TaiLr objective with respect to the model probability We analyze the asymptotic behavior when y is at the two ends of [0. 1].
(w)
p (w) in F
22

---

Published as a conference paper at ICLR 2023
50
34.5
25
WritigPromptss
IWSL114 De-En
10
10
10-5
10-1
10
10
10
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: Ablation study of adjusting on the long text generation dataset WritingPrompls and the machine translation dataset IWSLT14 De-En.
0.8
Weight
0.6
0.4
0.001
0.01
0.1
0.5
0..0
0.2
0.4
0.6
0.8
1.0
Pg
Figure 6: The plot of the weight in the TaiLr objective (equation different
11 with respect to p5 p5((w) with
(I) When 7 + 1, the weight becomes p5 (w:). training. the resulting small weight will hinder the convergence of the model. Thus, large y is suitable when the model is generally confident about its predictions.
model probability p5 (w) small during
(2) When 7 0, the weight becomes ((w(w) which reflects the ratio of p (1r) and 7. When 7+p. (w)
Et (a) is large, the weight is nearly constant which turns the objective into MLE. While for small Pp
Pg (x) that has similar scale 7, the weight sensitive the change of p 5 (x).
23

---

Published as a conference paper at ICLR 2023
F GENERATION EXAMPLES
F.1 TEXT SUMMARIZATION
Input passage:
in january the los angeles times reported that the nederlander organization quired the rights to produce a musical version " thriller intention involving jackson in every aspect of the creative process.
MLE:
jackson dream dream
Unlikelihood training:
nederlander 's thriller musical thriller
D2GPo:
jackson UNK is UNK thriller
Loss truncation:
nederlander group to produce thriller
GOLD:
nederlander 's latest thriller is UNK
TaiL.r:
jackson thriller to be a musical
Input passage:
pilots american airlines jetliner that crashed into colombian mountain in december killing all ### people aboard were tired and confused, according to preliminary investigation.
MLE:
pilots confused tired confused after colombian crash
Unlikelihood training:
pilots were tired confused hefore colombia crash
D2GPo:
pilots in colombia crash tired confused
Loss truncation:
pilots of colombian plane that crashed tired and confused
GOLD:
preliminary investigation shows pilots of crashed colombian jetliner were tired confused
TaiLr:
pilots of plane that crashed in colombia were tired confused
Input passage:
lord knows UNK culture downs plenty of coffee american coffee hounds gulp average of three cups day apiece according the york-based national coffee association of u.s.a. inc. #### survey
MLE:
coffee is good for you study finds
Unlikelihood training:
coffee UNK culture downs plenty of coffee
D2GPo:
american coffee hounds are UNK
Loss truncation:
coffee hounds are drinking more coffee survey finds
GOLD:
coffee hounds gulp three cups a day
TaiLr:
american coffee hounds gulp three cups day
Table 8: Examples of summaries generated by different models on the test set of the Gigaword corpus.
24

---

Published as a conference paper at ICLR 2023
F.2 LONG TEXT GENERATION
Inpul prompt:
The first Al has just been finished. When the eager scientists turn it on however it 's first words are Kill me.
ML.E:
me . " The words were on every screen across the room What ? Isolation by the door frame It will be as blank as when "m listening to Billie the Animated Series The screen flashes green Killl me. My hands reach into my pocket I turn back
Kill me ?
The phone chirps attracting Kill me. mutter click made, sheet paper becomes drawn. Kill me repeat back. A dozen panels on the cciling were up and I press play on the back of my finger, playing a game of Halo. Kill me is written in block capitals.
in
Unlikelihood training:
After the start, I was n't prepared for the possibility. I was born with very little to live for My mother told me would wake up any day and I would be perfectly fine She said was what she wanted with me. never believed expecting to wake up in bed with such blissful ignorance but as woke up at 15 I knew that one thing my grandpa said to me was, "iill me. took over my father 's legacy. We always thought the singularity
was never
on the line, , every time my last day alive.. Everyone of my grandpa
The day that I turned 12. accomplishments were finally gone
D2GPo:
System awoke in 5 seconds..
The electricity was radiating heat, heat emanating
from the vents surrounding it . three bits of receptors that implanted the vital organs trapped inside providing a battery of life. The slow pulse of the vehicles following the status update were all automated, without the means of accessing anything it attempted to convey Few people had been inside their communications to the point where for the remainder of the rest of their lives some within the prolonged process of removing all energy from the systems and causing minor bleeding. The members of the network were already unrespon- sive All seemed unconcemedd.
GOLD:
Kill me But no. Was it a mistake ? Something to do with emotion ? Silence ? Kill me This was a mistake The calibrations were complex But everyone with me. *This is my legacy. * I could n't end it I was n't ready 'm ready *" Kill me
took me some time to process the words. They started in numbers, then changes. *The parameters are within the parameters. Kill me. *The parameters are within the parameters.. Kill #Why Because... *Why ? Because Because ... "Why Because am *it*
TaiLr:
Silence passed over the room The lights were turned off My hands were not tied to carpet either. " says , have been observing you day " Well 'm quite sure it exists My voice was artificial " Listten You should n't think about this 've been trying to work with you and all of these guys and do n't know if you sense the ins and outs of this project If you read this I need you cooperate and the computers will analyze the whole thing and start with bioanalysis.
Table 9; Examples of stories generated by different models on the test set of the WritingPrompts dataset.
25


Reviews:
[{"review_id": "X8ieSHTHrU", "reviewer": "Reviewer_eDTd", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the Total Variation Distance (TVD) as an alternative to KL divergence for natural language generation. KL divergence is implicitly optimized by the widely used MLE, but the authors argue that this objective is sensitive to noisy data and results in overestimating the probability of corrupted text, while TVD is more robust to outliers. However, one cannot directly optimize TVD, so the authors develop practical bounds and introduce the TaiLr objective that balances the bias-variance tradeoff of estimating it. Experiments on synthetic data, machine translation, text summarization and text generation confirm the effectiveness of the proposed approach.", "summary_of_the_review": "This is overall a well-rounded paper. It is very well written and has more than enough novelty and substance. The main weakness lies in the empirical side, which is comprehensive and serves as a proof of concept, but falls short in convincing the reader that they should use the proposed approach when training their next model in the real world.", "strength_and_weaknesses": "STRENGTHS\n- This is overall a well-rounded paper: it tackles a very important problem, the proposed solution is sound and well-motivated (although I did not carefully check the correctness of the maths), the paper has more than enough substance, and the reported results are solid.\n- Despite being somewhat math-heavy, the paper is written in a very clear and didactic way. I found it easy to follow and an enjoyable read overall!\n- Comprehensive (although not entirely convincing, see below) experiments covering 3 downstream applications and a synthetic task, with a good number of baselines.\n\nWEAKNESSES\n- The experiments are comprehensive, but limited to rather artificial (or at least not the most relevant) scenarios from today’s perspective. For instance, the authors use a 1-layer LSTM trained on 10k sentences as their oracle for the synthetic data experiments. This sounds ridiculous in 2022, when there are dozens of strong language models publicly available. Similarly, the choice of IWSLT14 as the evaluation benchmark for MT and Gigaword for summarization do not seem the most relevant nowadays (e.g. why not run MT experiments in more languages and/or a more recent WMT benchmark)? In addition, all real-data experiments rely on seq2seq models, while it is decoder-only models that are becoming more and more central in NLP.\n- Connected to the previous point, the authors have a decent number of baselines that they implement, but they do not compare to any number previously reported in the literature. If one checks previously published papers, it seems clear that the reported numbers are rather far from the state-of-the-art. For instance, https://aclanthology.org/2021.emnlp-main.534.pdf reports 38.6 BLEU in IWSLT14, compared to 35.1 for the best system in this paper.\n- Given the previous points, I think that the experiments in the paper are valid as a proof of concept, but are far from convincing me that I should use the proposed approach when I train my next NLG model. You have convinced me that the approach has some potential, but not necessarily that it is better than standard MLE in the real world.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "StlR98J4az", "reviewer": "Reviewer_PwuP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Using MLE (maximum likelihood estimation) to train text generation models distributes probability mass to all samples in the dataset, even if the samples are of poor quality or outliers. The authors claim that MLE-trained models would would overestimate the probability of corrupted sequences (which the authors call the “void” of the real data distribution) which leads to degeneration.\n\nTo address this problem, the authors propose using total variation distance as shown in Equation (2a) and Equation (2b). Total variation distance would make the model place small or zero probability to low-quality sequences in the training set. However, it’s not easy to apply TVD directly onto text generation, so the authors proposed an upper bound on the TVD (sequence-level) in Proposition 1 / Equation 5. Additionally, given that we do not have the data/reference probability distribution, the target is estimated using a proxy distribution. \n", "summary_of_the_review": "Important problem. Hoping for better analysis of the pros and cons of related approaches. ", "strength_and_weaknesses": "Experiments are on a synthetic task as well as regular tasks including machine translation and summarization. Experiments are also done on long text generation as well, and I’m glad to see that it works well. \n\n\nI’m glad that the paper involves discussion of Kang and Hashimoto (2020) as well as Pang and He (2021). One concern is that given that the authors’ motivation and the two papers’ motivation are so similar – all of the approaches (the two papers’ as well as the authors’) propose to downweight unlikely / outlier tokens, I would appreciate some deeper explanation on the pros and cons of each approach, or why the authors’ approach excels. \n\n---\n\nUpdate: I read through the paper again, and responded to the authors' rebuttal below -- I don't really have complaints otherwise. I think the motivation and the execution are great. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MTgqMPQUFR", "reviewer": "Reviewer_Zays", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies training language models using total variation distance as opposed to maximum likelihood. While the total variation distance idea was previously proposed by Kang and Hashimoto 2020, they optimized a different upper bound via loss truncation and Pinsker's Inequality. \n\nIn this work, the authors propose an alternative upper bound that is novel and interesting by tackling specific challenges with optimizing the TVD loss (i.e. lack of token level factorization, and explicit demand of data probability distribution)\n\nThe authors provide empirical results both on real and synthetic data. In particular in machine translation (IWSLT De-En), text summarization (GigaWord) and open ended generation (Writing Prompts), the authors' approach outperforms not only MLE but several other techniques in the literature (unlikelihood, D2GPo, loss truncation, and GOLD). \n", "summary_of_the_review": "I support accepting the paper. It is quite interesting, novel and tackles a very important problem in language models. ", "strength_and_weaknesses": "Strengths:\n-Novel method\n-Thorough empirical results (both real and synthetic)\n-Well written", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%