Data-dependent Gaussian Prior Objective for Language Generation
Authors: Zuchao Li
Rui Wang
Kehai Chen
Masso Utiyama
Eiichiro Sumita
Zhuosheng Zhang
Hai Zhao
Venue:  (2020)
Keywords: Gaussian Prior Objective
Language Generation

Abstract: For typical sequence prediction problems such as language generation, maximum likelihood estimation (MLE) has commonly been adopted as it encourages the predicted sequence most consistent with the ground-truth sequence to have the highest probability of occurring. However, MLE focuses on once-to-all matching between the predicted sequence and gold-standard, consequently treating all incorrect predictions as being equally incorrect. We refer to this drawback as {\it negative diversity ignorance} in this paper. Treating all incorrect predictions as equal unfairly downplays the nuance of these sequences' detailed token-wise structure. To counteract this, we augment the MLE loss by introducing an extra Kullback--Leibler divergence term derived by comparing a data-dependent Gaussian prior and the detailed training prediction. The proposed data-dependent Gaussian prior objective (D2GPo) is defined over a prior topological order of tokens and is poles apart from the data-independent Gaussian prior (L2 regularization) commonly adopted in smoothing the training of MLE. Experimental results show that the proposed method makes effective use of a more detailed prior in the data and has improved performance in typical language generation tasks, including supervised and unsupervised machine translation, text summarization, storytelling, and image captioning.


Published as a conference paper at ICLR 2020
DATA-DEPENDENT GAUSSIAN PRIOR OBJECTIVE FOR LANGUAGE GENERATION
Zuchao Li Li Rui Wang**, Kehai Chenn,Masa0 Masao Utiyama", Eiichiro Sumita',
Zhuosheng Zhang 1.233 Hai Zhao' 1.2,3,
l Department of Computer Science and Engineering, Shanghai Jiao Tong University
2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China
3 MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
"National Institute of Information and Communications Technology (NICT), Kyoto, Japan charlee@sjtu. edu. en, zhangzs@s s tuu edu, en, zhaonaiêcs. fju. edu...d.cccn sumi tammicttee
(wangrui, khchen, mutiyama, e eiichiro
ABSTRACT
For typical sequence prediction problems such as language generation, maximum likelihood estimation (MLE) has commonly been adopted as it encourages the predicted sequence most consistent with the ground-truth sequence to have the highest probability of occurring. However, MLE focuses on once-to-all match- ing between the predicted sequence and gold-standard, consequently treating all incorrect predictions as being equally incorrect. We refer to this trawback as neg- ative diversity ignorance in this paper. Treating all incorrect predictions as equal unfairly downplays the nuance of these sequences' detailed token-wiss wise structure. To counteract this, we augment the MLE loss by introducing an extra Kullback- Leibler divergence term derived by comparing data-dependent Gaussian prior and the detailed training prediction. The proposed data-dependent Gaussian prior objective (D2GPo) is defined over prior topological order of tokens and is poles apart from the data-indeppndent Gaussian prior (1.2 regularization) commonly adopted in smoothing the training of MLE. Experimental results show that the proposed method makes effective use of more detailed prior in the data and has improved performance in typical language generation tasks, including supervised and unsupervised machine translation, text summarization, storytelling, and image captioning.
INTRODUCTION
Language understanding is the crown jewel of artificial intelligence. As the well-known dictum by Richard Feynman states, "what I cannot create, do not understand." " Language generation there- fore reflects the level of development of language understanding. Language generation models have seen remarkable advances in recent years, especially with the rapid development of deep neural net- works (DNNs). There are several models typically used in language generation, namely sequence-
to-sequence (seq2seq) models (Kalchbrenner & Blunsom 2013] Sutskever et al., 2014 Bahdanau
cet al. 2015 Luong al., 2015; Vaswani et al.J (017)) generative adversarial networks (GANs) (Goodfellow et al. 2014). variational autoencoders (Kingma Welling 2013). and auto-regressive networks (Larochelle & Murray 2011 Van Oord et al.] 2016). Language generation is usually
modeled as a sequence prediction task, which adopts maximum likelihood estimation (MLE) as the standard training criterion (i.e., objective). MLE has had much success owing to its intuitiveness and flexibility. However, sequence prediction has encountered the following series of problems due MLE.
Corresponding author. Zuchao Li and Zhuosheng Zhang were intersship research fellows at NICT when conducting this work. Hai Zhao was partially supported by Key Projects of National Natural Science Founda- tion of China (No. U1836222 and No. 61733011). Rui Wang was partially supported by JSPS grant-ing in- for carly-career scientists (19K20354): "Unsupervised Neural Machine Translation in Universal Scenarios" and NICT tenure-track researcher startup fund "Toward Intelligent Machine Translation".

---

Published as a conference paper at ICLR 2020
Exposure bias: The model is not exposed to the full range of errors during training.
Loss mismatch: During training, we maximize the log-likelihood, whereas, during infer- ence, the model is evaluated by a different metric such as BLEU or ROUGE.
Generation diversity: The generations are dull, generic (Sordoni et al. 2015 Serban al.
2016 et al., 2016a), repetitive, and short-sighted (Li et al. 2016b).
Negative diversity ignorance: MLE fails to assign proper scores different incorrect model outputs, which means that all incorrect outputs are treated equally during training.
A variety of work has alleviated the above MLE training shortcomings apart from negative diversity ignorance. Negative diversity ignorance is result of unfairly downplaying the nuance of sequences' detailed token-wise structure. When the MLE objective compares its predicted and ground-truth sequences, it takes a once-for-all matching strategy; the predicted sequence is given binary label, either correct or incorrect. However, these incorrect training predictions may be quite diverse and letting the model be aware of which incorrect predictions are more incorrect or less incorrect than others may more effectively guide model training. For instance, an armchair might be mistaken with a deckchair, but it should usually not be mistaken for a mushroom.
To alleviate the issue of the negative diversity ignorance, we add an extra Gaussian prior objec- tive to augment the current MLE training with an extra Kullback-Leibler divergence loss term. The extra loss is computed by comparing two probability distributions, the first of which is from the detailed model training prediction and the second of which is from a ground-truth token-wiss wise distribution and is defined as a kind of data-dependent Gaussian prior distribution. The proposed data-dependent Gaussian prior objective (D2GPo) is then injected into the final loss through KL divergence term. The D2GPo is poles apart from the commonly adopted data-independent Gaussian prior (L.2 regularization) for the purpose of smoothing the training of MLE, which is also directly added into the MLE loss. Experimental results show that the proposed method makes effectively use of a more detailed prior in the data and improves the performance of typical language generation tasks, including supervised and unsupervised machine translation, text summarization, storytelling, and image captioning.
2 RELATED WORK
Natural language generation (NLG) has long been considered the most challenging natural language processing (NLP) task (Murty & Kabadi 1987). NLG techniques have been widely adopted as crit- ical module in various tasks, including control-free sentence or poem generation (Zhang & Lapata 2014) and input-conditioned language generation, such as machine translation, image captioning. Text summarization, storytelling (Vaswani et al., 2017 Lample et al., 2018 Karpathy & Fei-Fei 2015 Fan et al., 2018), and wwwpeeeeeeeeeeeoootoolooooo sentence generation (Hu al.) (217]). In this work, we focus on input- contttoondiionee language generation tasks, though our proposed method can also be applied to other language generation fields.
Input-conditioned language generation tasks are challenging because there is an information imbal- ance between the input and output in these tasks, especially for cases with non-text input (Shapiro) 1992). Reiter & Dale (2000) discussed different ways of building complicated knowledge-basse systems for NLG. recent years, neural networks (NNs), especially DNNs, have shown promising results in many NLP tasks. [engio et al. (2003) first proposed the NN language model (NNLM) to exploit the advantages of NNs for language generation tasks. In an NNLM, the n-gram paradigm is extended by the generalization ability of NNs.
Given the ground truth sequence s = (W1, W2,.... w(-1), the NNLM adopts the equation
Pt RJ p(w: Wi-n, Wt-n+1, ..., w)-1).
(I)
Mikolov et al. (210)) developed
general implementation for a language model (called the recurrent NN language model (RNNLM) by integrating a Markov property using a recurrent NN (RNN) to address the NNLMs' theoretical inability capture long-term dependencies:
P: R ww. RNN(W1, W2. w,-1)
(2)
The RNNLM is an effective solution because is designed to capture long-term dependencies. Be- cause of the vanishing gradient problem in RNNs, however, the long-term dependency processing
2

---

Published as a conference paper at ICLR 2020
capability is limited. In contrast to RNN, the Transformer (Vaswani et al. 2017) provides new self-attention mechanism for handling long-term dependencies in text, resulting in robust perfor- mance across diverse tasks. Radffod et al. (2018) proposed Transformer language model called GPT, which uses a left-to-right architecture, where each token pays attention to previous tokens in the self-attention layers of the Transformer. Devlin et al. (2019) introduced new pre-training ob- jective: the masked language model (MLM), which enables the representation to fuse the left and right contexts and allows us to pre-train a deep bidirectional Transformer called BERT.
The generators of the most current language generation model use the RNNLM or Transformer language model structure. However, as pointed out by [Bengio et al.] (2015). fitting the distribu- tion of observation data does not mean that satisfactory text will be generated, because the model not exposed to the full range of errors during training. This is called the exposure bias prob- lem. Reinforcement learning, GANs (Goodfellow et al. 2014; Yu et al. 2017), and end-to-end re-parameterization techniques (Kusner & Hernández- Lortanddzz-obtoooo (016) have been proposed to solve this problem. The exposure bias is no longer an issue in reinforcement learning models because the training sequences are generated by the model itself.
Using MLE for the training objective leads to the problem of loss mismatch. |Ranzato et al. (2015) incorporated the evaluation metric into the training of peeenncctt--squenneee (seq2seq) models and proposed the mixed incremental cross-entropy reinforce (MIXER) training strategy, which is similar to the idea of minimum risk training (Smith & Eisner 2006 & Eisner 2009 Ayana et al. 2016 Shen et al.] (016). MIXER uses decoder hidden states to predict the bias term and hence reduce the variance, while minimum risk training renormalizes the predicted probabilities. Zhang & Zhao (2018) introduced new training criterion based on the Hellinger distance for the seq2seq model and empirically compared the models of two optimization categories: minimum divergence and maximun margin.
For the generation diversity problem, Serban et al. (217)) applied a latent variable hierarchical encoder-decoder dialog model introduce utterance-level variations and facilitate longer responses. Zhhao et al. (0017) presented a novel framework based on conditional variational autoencoders that improves generation diversity by sampling latent variable and optionally adding linguistic fea- tures to constrain the style further.
There is an increasing interest incorporating problem field knowledge in machine learning ap- proaches (Taskar et al.] 2004] Ganchev et al., 2010; Hu et al. 2016). One common way is to de- sign specialized network architectures or features for specific knowledge (c.g.. Liang et al. (2017) 2018). In contrast, for structured probabilistic models, posterior regularization and related frame- works (Ganchev et al. 2010 Liang et al., 2009 Bellare et al., 2009) provide a general means to impose knowledge constraints during model estimation. Hu et al. (2018) established mathematical correspondence between posterior regularization and reinforcement learning and, using this corre- spondence, expanded posterior regularization to learn knowledge constraints as the extrinsic reward in reinforcement learning. Our approach can be seen as incorporating a priori knowledge of the language field into language generation learning.
Additionally, Welleck et al. (2019) proposed a new objective. unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. The difference is that (Welleck) et al., 2019) focuses on low-frequency words while our model focuses on negative tokens. It is model that assigns too much
claimed that the likelihood objective itself is at fault, resulting probability to sequences containing repetition and frequent words, unlike those from the human training distribution. From this point view, there is a point in common with our motivation, which is to make the model prediction consistent with human training distribution to some extent.
3 BACKGROUND
Consider a conditional probability model for sequence predictions y ~ po(()) with parameters 0. The target sequence y can be conditioned on any type of source æ (e.g., phrase, sentence, or passage of human language or even an image), which is omitted for simplicity of notation. For the sequence y (y1: V2,..., W.. the probability Pe(y)z) x)) is
po (1) (m) po (yn (a=) (Vzl (x, yn v1).-pp (vr ar, 91:1-1)-
(3)
3

---

Published as a conference paper at ICLR 2020
Commonly, sequence prediction models are trained using MLE (also known as teacher forcing) Williams & Zipser 1989). MLE minimizes the negative log- log-likelihood of po (z)(a))
(0) log (y)az)
log
(4)
Optimizing the MLE objective (!) is straightforward and meets the principle of empirical risk minimization while focusing on only minimizing losses of the correct target on the training data set.
However, there may be noise in the training data, and forcibly learning the distribution of a training set does not enable the obtained model to reach good generalization. Additionally, for sequence prediction, models trained subject to MLE cursorily evaluate all predictions as either correct or incorrect and ignore the similarity between the correct and "less incorrect" predictions. Incorrect predictions might range from nearly perfect (i.e., one token mistaken with a synonym) com- pletely wrong, having nothing in common with the gold sequence. However, MLE training treats all incorrect training predictions equally, which implics that MLE fails to accurately assign scores to diverse (especially negative) model predictions.
D2GPO: DATA-DEPENDENT GAUSSIAN PRIOR OBJECTIVE
To capture the diversity of negative training predictions, we augment the MLE objective of the model with additional objective O that more accurately models such a negative diversity. With- out loss of generality, supposing ý the prediction candidate, we introduce a general evaluation function f(y, y) E IR independent of the model prediction, such that with golden target token y", a higher fl. y²) value indicates better pe po x) for a target candidate ÿ e V (where is the target candidate set). Note that fī y) also involve other factors such as latent variables and extra supervisions.
There are two main methods of learning (i y) in the model. If is GAN-like implicit generative model or an explicit distribution that can be efficiently reparametrized (e.g., Gaussian) (Kingmal & Welling 2013). then one effective method is maximizing Ep, [f y)]. The other method is computing the gradient VilEp [ī y)] using the log-derivative trick that can suffer from high variance but is often used for the large set of non-parameterrizable explicit distributions.
Corresponding to the probability distribution of model predictions po(-), we define a prior distribu- tion g(y) (for each target y/, has its own unique distribution of q, = ((y))) which is extracted and derived from the ground-truth data (e.g., language text in language generation tasks). To guide the probability distribution of model predictions (-(-) match the prior probability distributions q(-). we adopt Kullback- Leibler divergence. Considering also the learning of the evaluation function [ y). the loss for objective O is calculated as
Lo(O,g) g) = K (g((())llo)()))) - a.E,, (S
(5)
where is a weight for the evaluation function learning term. We derive the prior distribution q(y) from the ground-truth data (which is independent of model parameters 0), and therefore Egg (Ā. =0. Hence, Eq. (5) becomes
Lo q) = K (((((v))lo())=)),
(6)
in which KL divergence can be expanded as
KL)g E,, 2)) Σ qi - log(a)) Σ gi * log(p:).
p'
The final objective for learning the model is written as
ming MALE (0) + (Lo(O,9)
(8)
where À is the balancing hyperparameter. Because optimizing the original model objective LMLE (0) straightforward in the following, we omit the discussion of LMLE(()) and focus on the proposed (o
4

---

Published as a conference paper at ICLR 2020
The prior probability distribution gly") y can be obtained from the evaluation function f(..) with a softmax operation. To expose the mass of the distribution over the classes, et al. (2015) introduced softmax temperature mechanism; therefore, the relationship between q and f3). y) 18
erplS ())TT a(y') = exppp (is, (U)')')/T)*
where 7 is temperature parameter. When T - O, the distribution becomes Kronecker distri- bution (and is equivalent to a one-hot target vector); when T + +00, the distribution becomes a uniform distribution. The softmax operation always turns an evaluation function f(--) into form of probability distribution no matter the form of the original f(- :); thus, we only focus on f(-+).
To find good evaluation function, we have to mine token-wise diversity for all y² Considering all token types }/ in vocabulary, with respect to each y", there exists a prior topological order ORDER(y") among all the known tokens, in which y is always ranked top priority. f y²) can then be defined as a monotonic function over the corresponding topological order so that it has maximal value only when the input is y" itself. Note that defining f(,-) )) in this way leads to the resulting also being monotonic over the corresponding topological order. Considering that q is priori, is fixed throughout the learning process.
The remaining questions are about how to find meaningful evaluation function ((:-) for the dis- tribution q. language generation tasks, we may conveniently take word embedding as the token representation, and let the embedding distance determine such order ORDER(y") for each y²" In this work, we adopt the cosine similarity of pre- trained embeddings to sort the token (word/subword) order.
Discussion For the evaluation function ((:,)) of q, we adopt the Gaussian probability density function, though later we also present experimental results for other types of functions in an ab- lation study. As the adopted Gaussian prior used in the training objective is derived from data- dependent token-wise distribution, we call it the data-dependent Gaussian prior objective (D2GPo). This objective a big departure from the Gaussian prior commonly adopted for smoothing MLE training (which we the data-independen Gaussian prior). The following briefly explains why we chose the Gaussian probability density function and how our D2GPo mathematically differs from the data-independent Gaussian prior.
The central limit theorem indicates that suitably standardized sums of independent random variables have an approximately normal distribution. Thus, any random variable that arises as the sum of sufficiently large number of small random components can be modeled accurately using a normal distribution. Embedding has linear additive property (e.g.. king man woman s queen). The additive property of embedding can be explained by inspecting the training objective (Mikolov et al., 2013). Each dimension of embedding represents potential feature of the token. Considering each potential feature as an independent random variable, the sum follows a Gaussian distribution centered the correct vocabulary unit according to the linear additive property. We can therefore use a Gaussian distribution for the beheeddingddistannee deteeededddddededdd order to effectively model the distribution q(y") ). An overview of the concepts underlying D2GPo is illustrated in Appendix A.1
The D2GPo in this paper is different from the data-independent Gaussian prior in machine learning optimization theory. We hypothesize and experimentally verify that the embedding feature extracted from the data obeys the Gaussian distribution. The distribution from the prior knowledge of language data used as a soft target to guide the model language generation process using knowledge distilla- tion. The Gaussian prior in the machine learning optimization theory assumes that each component in the parameter is subject to zero-mean Gaussian prior distribution, which is equivalent to 1.2 regularization. In general, our Gaussian prior objective is to act on the guiding target probability, while the Gaussian prior in machine learning is applied to the selection of model parameters.
5 EXPERIMENTS AND RESULTS
This section describes the experimental evaluation of D2GPo variety of typical language gen- eration tasks: neural machine translation (NMT), text summarization, storytelling, and image cap- tioning. The hyperparameters in D2GPo and effect analysis are given in Appendix A.7
5

---

Published as a conference paper at ICLR 2020
5.1 EMBEDDING PRE- TRAINING
Our proposed D2GPo approach for experimental tasks requires either word embeddings or byte- pair-encoding (BPE) (Sennrich et al. 2016b) subword embeddings. We generated the pretrained embeddings using fastTexx (Bojanowski et al. 2017) with an embedding dimension of 512, a con- text window of size 5, and 10 negative samples. For NMT, fastText was applied to the concatenation of source and target language monolingual corpora, resulting in cross-lingual BPE subword embed- ding. For text summarization, we generated BPE subword embedding only on the English monolin- gual corpora, while for the storytelling and image captioning, obtained the word embedding also on the English monolingual corpora.
5.2 SUPERVISED NMT
We evaluated the model several widely used translation tasks: WMT14 English-to-Germal (EN- DE), English-to-Frendd (EN-FR), and WMT16 English-to-Romanian (EN-RO tasks, which all have standard large-scale corpora for NMT evaluation. Owing space limits, the data details provided in Appendix The sentences were encoded using sub-word types based on BPE, which has shared vocabulary of 40,000 sub-word units for all three tasks. We chose the Transformer NMT (Vaswani et al. 2017) model as our baseline. For the hyperparameters of the Transformer (base/big) models, we followed the settings used by Vaswani et al. (2017). The BLEU (Papineni et al. 2002 score with multi-bleu.pl was calculated during the evaluation.
System
EN-DE
EN-FR
EN-RO
EN-RO STD
Vaswani 2017
(base)
27.30
38.10
Vaswam et al.
2017
(big)
28.40
41.00
Transformer (base)
27.35
38.44
33.22
36.68
D2GPo
27.93++
39.23++
34.00+
37.11+
Transformer (big)
28.51
41.05
33.45
37.55
D2GPo
29.10+
41.77++
34.13+
37.92+
Table 1: Comparison with baseline and existing systems on supervised translation tasks. Here, "++/+" after the BLEU score indicates that the proposed method was significantly better than the corresponding baseline Transformer (base big) at significance levels p <0.01/0.05, "STD" repre- sents synthetic training data from (Sennrich et al. 2016b).
we report the performance of our full model, the baseline, and existing systems. Our baseline model obtains results similar to those of Vaswani al. (2017), the existing strong model used for these tasks. The results indicate that our method performed better than the strong basclines for all language pairs. Our model is not only improvement on the translation model of large-scale training sets but also performs better for small-scale training sets. Refer to Appendix A.8 and A.9 for analysis of the low-resource scenario and generation diversity.
In Table M
5.3 UNSUPERVISED NMT
For unsupervised machine translation, we also used the three language pairs EN-DE, EN-FR, and EN-RO as our evaluation targets. Note that the evaluation performed EN-DE newstes12016 instead of newstes/20/4 to ensure the results are comparable with the results of other works; this is unlike supervised machine translation. We used the masked sequence sequence the pre-training (MASS) model (Song et al., 2019) as our bascline. Following the practice of Song et al. (211)) we pretrained our model with a masked sequencc-to-sequeeee D2GPo) on EN, FR. DE, and RO monolingual data samples from WMT 2007-2018 News Crawl datasets that respectively cover 190M, 60M, 270M, and 10M sentences. We then fine-tuned the models on the same monolingual data using the back-translation cross-entropy loss (Lample et al., 2018) and our D2GPo loss. For the training dataset, we filtered out sentences longer than 175 words length and jointly learned 60K BPE sub-word units for each language pair.
pre-training (MASS) objective (without
The results for EN-RO are evaluated the dataset with diacrities removed in the reference text.

---

Published as a conference paper at ICLR 2020
Method
EN-FR
FR-EN
EN-DE
DE-EN
EN-RO
RO-EN
Artetxe et al.
2017)
15.13
15.56
6.89
10.16
Lample et al
15.05
14.31
9.75
13.33
Yang of al. 2018
16,97
15.58
10.86
14.62
Lample ct al. (2018)
25,14
24.18
17.16
21.00
21.18
19.44
XLM [Lampec & Conneau 2019
33.40
33.30
27.00
34.30
33.30
31.80
MASS Song et al.
2019)
37.50
34.90
28.30
35.20
35,20
33,10
MASS
D2GPo
37.92
34.94
28.42
35.62
36.31
33.41
Table 2: BLEU score comparisons between MASS and previous methods of unsupervised NMT.
As shown in Table D2GPo consistently outperformed MASS (the state-of-the-art baseline) all unsupervised translation pairs. Mcanwhile, the MASS and XLMsystems leverage large-scale monolingual pre-training, and the decoder (generator, language model) can still be improved by our D2GPo loss in the fine-tuning phase. This demonstrates the efficiency of the proposed method.
5.4 TEXT SUMMARIZATION
Text summarization is a typical language generation task that creates a short and fluent summary of the given long-text document. [Song et al. (2019) fine-tuned the MASS pretrained model on the text summarization task and achieved state-of- tftt--of- results. We chose this model as our baseline, maintained consistent pre-training, and used D2GPo loss for enhancements the fine-tuning phase. We used the Annotated Gigaword corpus as the benchmark, as detailed in Appendix A.4 In the evaluation, R ROUGE-2, and ROUGE-L (Lin 2004) are reported.
Model
ROUGE-1
ROUGE-2
ROUGE-L
RNN-based seq2seq
35.50
15.54
32.45
Supervised
Nallapati et al. (016
34.97
17.17
32.70
MLM pre pre-training
Song et al. 2019
37.75
18.45
34,85
Semi- cemi-supervised
DAE pre- pre-training (Song et al.
2019
35.97
17.17
33.14
MASS pre-training (Song et al.
2019)
38.73
19.71
35.96
MASS D2GPP
39.23
20.11
36.48
Table 3: Performance on the text summarization task
Our results for text summarization are listed in Table We compared our +D2GPo with our baseline MASS, which is the current state-of-the-art model; +D2GPo consistently outperformed the baseline on all evaluation metrics. The models with semi semi-supervised setting yielded large-margin im- provement relative to the model without any pre-training, which demonstrates that the supervised pre-training is effective in the text summarization task.
5.5 STORYTELLING
Storytelling is at the frontier of current language generation technologies; i.e., stories must maintain a consistent theme throughout and require long-distance dependency modeling. Additionally, stories require creativity and high-level plot with planning ahead rather than word-by-word generation (Wiseman al. 2017).
We used the hierarchical story generation model (Fan et al. 2018) (which is introduced in Appendix A.5 as our baseline to test the improvements of D2GPo for the storytelling task. guarantee the single-variable principle, we added only the D2GPo loss to the story generation model. The prompt generation model is consistent with Fan et al. (2018).
For automatic evaluation, we measured the model perplexity on validation and test sets. Table shows results obtained using D2GPo. It is seen that with the addition of D2GPo, the Conv seq2seq self-attention model substantially improved the likelihood of human- gennrrttdd stories and even outperformed the ensemble or fusion models without increasing the number of parameters. Per- plexity was further reduced with the addition of the fusion mechanism. These results suggest that

---

Published as a conference paper at ICLR 2020
Model
Params
Valid Perplexity
Test Perplexity
GCNN LM
123,4 M
54.50
54.79
GCNN + self-attention LM
126.4 M
51.84
51.18
LSTM seq2seq
110.3 M
46.83
46.79
Conv seq2seq
113.0 M
45.27
45.54
Conv seq2seq self-attention
134.7 M
37.37
37.94
Ensemble:
Conv seq2seq + celf-attention
270.3 M
36.63
36.93
Fusion: Conv seq2seq self-attention
255.4 M
36.08
36.56
Conv seq2seq self-attention + D2GPo
134.7 M
35.56
35.74
Fusion: Conv seq2seq self-attention D2GPo
255.4 M
33.82
33.90
Table Perplexity on WRITINGPROMPTS.
D2GPo improves the quality of language generation greatly, especially in settings where there are fewer restrictions story generation tasks.
5.6 IMAGE CAPTIONING
Image captioning a task that combines image understanding and language generation. It continues to inspire considerable research the boundary of computer vision and natural language processing. We elected experiment with image captioning verify the performance of D22GP language generation model having diverse types of input.
In experiments, evaluated our model on ablated baseline (top-down, detailed in Ap- pendix A.6) [Anderson et al. 2018) against prior work on the MSCOCO 2014 caption dataset (Lin) al. 2014), which has became the standard benchmark for image captioning. For validation of model hyperparameters and offline testing, we used Karpathy splits (Karpathy & Fei-Feil 2015). which have been used extensively in prior work. SPICE (Anderson et al. 2016), CIDEr (Vedantam et al. (2013). METEOR (Denkowski & Lavie 2014), ROUGE-L, and BLEU were used to evaluate
the caption quality.
BLEU-1
BLEU-4
METEOR
ROUGE-L
CIDEr SPICE
Att2in RRnnie al.
2017
31.3
26.0
54.3
101.3
Att2all (Rennie al.
2017
30.0
25.9
53.4
99.4
Baseline: Top-down
74.5
33.4
26.1
54.4
105.4
19.2
Baseline D2GPo
75.2
33.6
26.3
55.1
106.6
19.7
Baseline SCST
77.8
34.4
26.6
56.1
114.3
19.9
Baseline SCST D2GPo
78.0
34.7
26.8
56.3
116.8
20.2
Table 5: Image caption performance on the MSCOCO Karpathy test split.
Table summariizs the performance of our full model and the ResNet Top-down bascline in compar- ison with the existing strong Self-critical Sequence Training (SCST) (Rennic al. 2017) approach on the test portion of the Karpathy splits. To ensure a fair comparison, results are only reported for models trained with standard cross-entropy loss (i.e., MLE). All results reported for single model with no fine tuning of the input ResNet model. Our ResNet baseline performs slightly better than the SCST models. After incorporating our proposed D2GPo loss, our model improves further across all metrics.
EVALUATION FUNCTION
According to the analysis in Section 4 for the embedding, used the Gaussian probability density function as our evaluation function T();; however, to evaluate the effectiveness of different evalua- tion functions, changed the function and tested the performance changes the supervised NMT
8

---

Published as a conference paper at ICLR 2020
EN-DE task. We used the same experiment settings described in Section 5.2 and compared the BLEU score changes on the test set, as listed in Table 6
Evaluation Function
BLEU
Baseline
27.35
Gaussian
27.93
0.58
Random
26.34
1.01
Linear
27.45
0.10
Cosine
27.62
0.27
Table Ablation study on our proposed D2GPo with different evaluation functions on super- vised NMT WMT14 EN-DE task, with the Transformer-base model.
The table shows that the performance of Gaussian density, linear, and cosine functions increased while the performance of the random function decreased. This shows that the distance information obtained from embedding can effectively guide the generation process. Among these functions, the Gaussian density function had the greatest improvement, which agrees with our analysis of the embedding features obeying the Gaussian distribution. We postulate that because the linear and cosine functions are rough approximations of the Gaussian density function, they perform similarly to the Gaussian density function.
CONCLUSION
This work proposed data-dependent Gaussian prior objective (D2GPo) for language generation tasks with the hope of alleviating the difficulty of negative diversity ignorance. D2GPo imposes the prior from (linguistic) data over the sequence prediction models. D2GPo outperformed strong baselines in experiments classic language generation tasks (i.e., neural machine translation, text summarization, storytelling, and image captioning tasks).
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In European Conference on Computer Vision, pp. 382-398. Springer, 2016.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6077-6086, 2018.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint arXiv: 2017.
Shiqi Shen Ayana, Zhiyuan Liu, and Maosong Sun. Neural headline generation with minimum risk training. arXiv preprint arXiv:1604.0 009044 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Kedar Bellare, Gregory Druck, and Andrew McCallum. Alternating projections for learning with ex- pectation constraints. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 43-50. AUAI Press, 2009.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- tems, pp. 1171-1179, 2015.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 137-11155 2003.

---

Published as a conference paper at ICLR 2020
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computatiooal Linguistics, 5:135 146. 2017. doi: 10.1162/tacl 00051. URL https:: /www wwwweb.www........ wwt./r////aa....o. mazgg/nthooogy///
Q17-1010
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference Machine Learning-Voin Volume 70, pp. 931.911. JMLR. org, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statisfical Machine Translation, pp. 376-380, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W/ 14-3348. URL WtLhttps: www. aac.a.awea. org wwt.aooohhoooyyooooooo /w/1 /w14
3348
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (0.18653/v1/N19-11 1423. URLhttps: 1423
hetps:
//www www.wb..wbb.................w..w
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics doi: 10.18653/v1/P11 1082. URL www. acl wel.weeb..o www..e/w...///attoooo//o//------
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11(0ul):001122049 2010.
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pp. 1243- 1252, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.025333 2015.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2410-2420, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 10.18853/v1/P16-111 1228. URL https: 7/ /wwww.o ac aclweb. org/
hwt.:www...gg/.aaooooo/////1/11/ attaaaooooooycceccccc 1228
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Controllable text generation. arXiv preprint urXiv:1703.009:
2017.
Zhiting Hu, Zichao Yang, Ruslan R Salakhutdinov, LIANHUI Qin, Xiaodan Liang, Haoye Dong, and Eric Xing. Deep generative models with learnable knowledge constraints. Advances in Neural Information Processing Systems, pp. 10501-10512, 2018.
10

---

Published as a conference paper at ICLR 2020
Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700-170000 Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL hetps:
/www ac www.
web. org/ ant wtpptthhogy///11111111111
Andrej Karpathy and Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 303,3117,,011
Diederik P Kingma and Max Welling, arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Matt J Kusner and José Miguel Hernández-Lobato Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv: 104051, 2016.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato, Unsupervised machine translation using monolingual corpora only, arXiv preprint arXiv:1711.0004 2017.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Confer- ence Empirical Methods in Natural Language Processing, pp- 5039-5049, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ 158- URL https: www. aclweb. org/ anthology/DDe- 1549
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29-37,
2011.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity prommting
objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, pp. 110-119, San Diego, California, June 201 6a. Association for Computational Linguistics. doi: 10.18653/11/111-104444 URLhttps: wwwww//wwwwww/...ootwwww http://wwwwwwww.../////// aclweb. ozg/anthooogyy N16 6-14
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Austin, Texas. Novem- URL
ber 2016b. Association for Computational Linguistics. doi: 10.18653/1//D1-1111 https: / www www.accle.. web
Zhifei Li and Jason Eisner. First- and second-order expectation semirings with applications to minimum-rias training on translation forests. In Proceedings the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 40-51, Singapore, August 2009. Association for Computational Linguistics. URLhttts:: /ww. aclweb. wttp://tthoooooo/ooo------
Perey Liang, Michael Jordan, and Dan Klein. Learning from measurements in exponential families. In Proceedings of the 26th annual international conference on machine learning, pp. 641-648. ACM, 2009.
Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. Recurrent topic-transition gan for visual paragraph generation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3362-3371, 2017.
Xiaodan Liang, Zhiting Hu, Hao Zhang, Liang Lin, and Eric P Xing. Symbolic graph reasoning meets convolutions. Advances in Neural Information Processing Systems, pp. 1853-1863, 2018.
Chin-Yer Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: www. acl web. web.orgg/aathooooy////4--------

---

Published as a conference paper at ICLR 2020
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches -reettoo-bbsed neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/1/DD1--1166 org/ htgg/aathhoooooy/1511115
(tLhhtts:: www.ac..eb.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-31111 11-11199, 2013.
Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Mathematical programming. 39(2):11--12) 1987.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Abstrac- tive text summarization using acquence-to-sequene RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: (0.18653/v1/K16. 1028.
URL https: http://wwwwwww...o..oooo otg/ http///anhoooooooooo.o.o. http:///wwww...ccooooooooooaatoa
Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. Annotated Gigaword. In Proceed- ings of the Joint Workshop Automatic Knowledge Base Construction and Web-scale Knowledge Extraction BAKBC-WEKEX, pp. 95-100, Montréal, Canada, June 2012. Association for Compu- tational Linguistics. URL htthttt:: www acl www.wb..w.b.....................
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3 10.3115//073008..073333.. URL https: //www.aclweb.or.
www.wb. org/ wwwpee/......aatoooo//////------
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-us-westtt amazonaws. com/openai- understanding paper. pdf. 2018.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv:1511.0673 2015.
Ehud Reiter and Robert Dale. Building natural language generation systems. Cambridge university press, 2000.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7008-7024, 2017.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379-389, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10.18653/v1/DI5-1100 1044. URL https: tttp///wwwccclco...cccocooccc.c. web org/
antho methologyyle ogy/D15-1042
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 371-376, Berlin, Genmany, August 2016a. Association for Computational Lin- guistics. doi: 10.18653///W116-33322 URL https: /ttp///www/.... www..w..... aclweb. ozg/anthology/
W1 6-2323
12

---

Published as a conference paper at ICLR 2020
Rico Sennrich, Barry Haddow. and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016b. Association for Computational Linguistics. doi: 10. 10.18653/11/111-1166 7/www. www.wllwwwe..... org/
Julian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference Artificial Intelligence, 2016.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for gen- erating dialogues. In Thirty- First AAAI Conference on Artificial Intelligence, 2017.
Stuart C Shapiro. Encyclopedia of artificial intelligence second edition. John, 1992.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Lingnistics (Volume I: Long Papers), pp. 1683-1692, Berlin, Germany, August 2016, Association for Computational Linguistics. doi: 10.18653/v 1/116/11111111111/
URL https::///// wwww ac web. org/ hwt.w//www........aaaaoooooo///.
David A. Smith and Jason Eisner. Minimum risk annealing for training log-linear models. In Pro- ceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pp. 787-794, Sydney, Aus- tralia, July 2006. Association for Computational Linguistics. URL heLhttp:: www aclweb.
hxgg/anholooyy/OS 2101
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence sequence pre-training for language generation. In International Conference on Machine Learning, pp. 5926-5936, 2019.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. A neural network approach to context-sensitte gen- eration of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 196-205, Denver, Colorado. May-June 2015. Association for Computational Linguistics. doi: 10.3115//1N115-002.. httpstt::220 www ac www.eb. org wrggg/antology//lli... hoophooogyy/111151
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
Ben Taskar, Carlos Guestrin, and Daphne Koller. Max- Max-margin markov networks. In Advances in neural information processing systems, pp. 25-32, 2004.
Aaron Van Oord. Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Ramakrishna Vedantam, C Lawrence Zitnckk and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4575, 2015.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. 2019.
Ronald Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
13

---

Published as a conference paper at ICLR 2020
Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253-22633, Copenhagen, Denmark, September 2017. Association for Computational Lin-
guistics. doi: 10.18653/w1/DD1--13333 URL https:: /ts:://wwww.. //www, ac www.ww.www.......o......... parrg/anthol.. ogy/
D17-1239
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 46-55, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 1000
httpst:p:: /www ac web.
org//ntho anthology/PP11 1005
Lantao Yu, Weinan Zhang. Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Huan Zhang and Hai Zhao. Minimum divergence vs. maximum margin: empirical comparison on seq2seq models. In International Conference on Learning Representations, 2018.
Xingxing Zhang and Mirella Lapata. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods Natural I Language Processing (EMNLP), pp. 570-680, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 1/15///1DD1444 1074. URL htwp::ttpsssstt //wp::::/wwwww.// wwwwwb.www...... org http://w....ggaaaaoooooo/ll-----
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 654 154-664, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 18553vv// P17-1061.
ac http://wwwwwwwww....////oaaaaaaa http://wwwwwll....r.aaaaatoottrp
A APPENDIX
A.1 CONCEPTS UNDERLYING D2GPo
19
4/0
515
307
th
vocabulary
1886
Inpuis
Figure 1: Overview of the concepts underlying D2GPo taking the example of the sentence The little boy sits the armchair.
A.2 TOPOLOGICAL ORDER
Specifically, for target y", we calculate the embedding cosine similarity as the distance dist(ūj, of 5" and all other token types the vocabulary ūj. which used give the distance:
dist(ūj v²) cosine similarity(embb(j) smmbyy")
(10)
14

---

Published as a conference paper at ICLR 2020
Sorting by distance from small to large to obtain the topological order of token types yields
ORDER(Y") - INDEX( ((i
y"), dis(((2 ( "",,.., (iN, )l))
(11)
where N is the vocabulary size and INDEX(-) is used to obtain the new sequential index according to the distance sort. We define ORDER(y" ) as the j-th value in ORDER(Y"). Therefore, the evaluation function f fū) v*) ) is converted function defined on ORDER(y")
[ūū v") = ((ORDER(y"); );)
(12)
For example, suppose the vocabulary has 5 tokens (i.e. N=5), for the golden target y to be predicted, there is relationship: dist y²) dist(ūs, y²) < dist(ūr, y²) dist(ūs, v²) dist(Es v"),
the ORDER(Y") is [3, 4)
A.3 SUPERVISED NMT DATA
For the EN-DE translation task, 4.43M bilingual sentence pairs from the WMT 14 dataset, which used as training
includes the Common Crawl, News Commentary, and Europarl v7 datasets, data. The newstest2013 and newstes12014 datasets were used as the dev set and test set, respectively.
For the EN-FR translation task, 36M bilingual sentence pairs from the WMT 14 dataset as training data. The newstes12012 and newstest2013 datasets were combined for validation and newstest2014 was used the test set, following the configuration of Gehhinn al. (2017).
used
For the EN-RO task, we tested two settings; i.e., Europarl v7, which uses only the officially provided parallel corpus, and SETIMES2, which yields 600,000 sentence pairs for low-resoure supervised machine translation study. Alternatively, following the work of Sennrich et al. (2016a), we used synthetic training data (STD) of [Sennrich al. (20T6a), which provides 2.8M sentence pairs for training. We used newsdev2016 the dev set and newsfest2016 6 as the test set. Our reported results on EN-RO are evaluated on reference for which diacritics are removed from letters.
A.4 TEXT SUMMARIZATION DATA
The Annotated Gigaword corpus (Napoles al. 2012) was used as benchmark (Rush al, 2015). This data set is derived from news articles and comprises pairs of main sentences in the article (longer) and headline (shorter). The article and headline were respectively used as the source input sentence and reference. The data include approximately 3.8M training samples, 400,000 validation samples, and 2000 test samples.
A.5 HIBRARCHICAL STORY GENERATION MODEL
The hierarchical story generation model Fan et al. 2018) was proposed for the situation in which a sentence called prompt that describes the topic of the upcoming story generation first generated, and then conditions on the prompt are applied when generating the story. Specifically, Fan et al. (2018) used self-attention gated convolutional language model (GCNN) (Dauphin al. (017)) as the gequence-to-sqquenne prompt generation model with iop-k random sampling. For prompt-to- story generation, they collected dataset from Reddit's WRITINGPROMPTS forum in which each prompt has multiple story responses. With the dataset, they trained story generation model that benefitted from a novel form of model fusion that improved the relevance of the story to the prompt and added a new gated multi-scale self-attention mechanism to model the long-range context.
A.6 TOP-DOWN IMAGE CAPTION MODEL
The top-down image captioning model uses ResNet (He et al.) 2016) convolutional neural net pretrained on ImageNet (Deng et al. 2009) to encode each image. Similar to previous work (Rennie et al. 2017), the cited study encoded the full-sized input image with the final convolutional layer of Resnet- 10I and used bilinear interpolation to resize the output to a fixed-size spatial representation of 10x 10. This is equivalent the maximum number of spatial regions used in our full model.
15

---

Published as a conference paper at ICLR 2020
A.7 HYPERPARAMETERS IN D2GPO
During training with our D2GPo, the value of the standard deviation of the KL diversity item À was set to 0.1, and the softmax temperature was 7 2.0 in all experiments.
To study the effects of hyperparameters (i.e., the standard deviation À and softmax temperature T in D2GPo) the experimental results, we carried out experiments on WMT14 EN-DE with the Transformer- model as the baseline and set À as 0.1, 0.2, 0.5, 1.0), T as [1.0, 2.0, 5.0, 10.0].
28
28
BLEU score
27.8
BLEU score
27.5
27.6
27
27.4
26.5
0.1
0.2
0.5
1.0
5.0
10.0
A
T
Figure 3: Performances on WMT14 EN-DE with different values.
Figure 4: Performances on WMT14 EN-DE with different values.
The experimental results reveal that À affects the model training process. We believe that the reason is that small value of Ả results in the model being unable to make full use of the prior knowledge (distribution), while a larger value of will make the model more uncertain because of the higher probability of there being incorrect even opposite words whose fastText embeddings
similar.
In addition, experimental results show that a small value of can improve the model to some extent, whereas large value of T will seriously decrease the performance of the model. Theoretically, when approaches infinity, the distribution q becomes uniform, and there no prior knowledge with which to guide the model. A loss penalty is applied to any model prediction, and excessively high value of 7 is thus harmful to training.
A.8 D2GPO UNDER A LOW-RESOURCE SETTING
Priors are generally more helpful in low-data regimes. We sampled 10,000, 100,000, and 600,000 paired sentences from the bilingual training data of WMT16 EN-RO to explore the performance of D2GPo in different low-resource scenarios. We used the same BPE code and learned fastText embeddings in all WMT16 EN-RO training data.
Method
10K
100K
600K
Baseline
1.01
17.80
33.22
+ D2GPo
4.33
20.48
34.00
Table Comparison of our baseline and our D2GPo method under different training data scales in terms of BLEU on the WMT16 EN-RO test set.
D2GPo outperforms the baseline model, demonstrating the effectiveness of method in low-resource scenarios. At the same time, the results show that the performance im- provement provided by D2GPo increases with fewer training data. This shows that prior knowledge substantially improve the performance of the model when training data scarce. A possible reason is that the training data are insufficient to train robust model. In this case, the injection of
As shown in Table 7
"wiin to limited experimental resources and time, we only consider the situation that À and T change separately; i.c., T remains unchanged at 2.0 when À changes while À remains unchanged at 0.1 when T changes.
16

---

Published as a conference paper at ICLR 2020
prior knowledge help train the parameters of the model and substantially improve the translation performance. However, with an increase in the number of training data, the model itself can be optimized well, and the improvement gained by introducing prior knowledge is not as substantial as before.
A.9 EXAMPLES OF IMAGE CAPTIONING
Top-down:
holding umbrella her hand D2GPo: a woman is holding an umbrella
SCST: a woman holding an umbrella a street
SCST+ D2GPo: a woman holding an umbrella the street
Top-down: large airplane sitting top of airport runway
D2GPo: airplane sitting on top airport runway
SCST: large jetliner sitting on top of an airport runway
SCST+ D2GPo:
large jetliner sitting on lop an airport runway
Top-down: a woman holding surf board in the occan
D2GPo: is standing the beach with surfboard
SCST: holding surfboard the beach beach with surfboard
SCST+ D2GPo: woman standing
Top-down: a traffic light with traffic light on
D2GPo: traffic light the side of traffic light
SCST: yellow traffic light on the side of a street yellow traffic lights the side of street
SCST+ D2GPo:
Table Captions generated for the left image by various models described in the paper. The models trained with SCST return more accurate and more detailed summary of the image. The models trained with D2GPo return more grammatically complete sentence.
A.10 ANALYSIS ON GENERATION DIVERSITY
Compared with traditional MIEE training, D2GPo encourages negative diversity. To examine differ- ences between D2GPo and MLE models, we counted high- and low-frequency words the training set and compared the frequencies of low-frequency words predicted by the two models and the golden reference the test set.
#GOLD
Baseline
+D2GPo
#LF
4915
3900
3998
#SUM
63086
55234
56129
#RATIO
7.79%
7.06%
7.12%
Table 9: The statics of low frequency words in the reference and generations.
The experiment was carried out WMT14 EN-DE, the baseline model was Transformer-ban base, and the statistics were calculated at the word level. We chose words with a frequency less than or equal to 100 in the training set low-frequency words. We used the golden reference (#GOLD), baseline model prediction output, and +D2GPo model prediction output to count the total number of tokens (#SUM) and the number of low- wwwffreuunncy words (#LF). Results given in Table 8
The results show that compared with the baseline, the D2GPo optimized model generates more low- frequency words and has higher ratio of low-frequency words. However, the number is still far
17

---

Published as a conference paper at ICLR 2020
less than the golden reference. It is thus demonstrated that D2GPo increases the diversity of model output.
A.11
EXAMPLES OF STORYTELLING
Example Prompt:
The scientists have discovered something terrible.
Generation with Conv seq2aeg Self-Attention (PPL 37.37):
The discovery of the discovery was a long and boring process It was a small ((nnk) : and a simple and
elegant solution. The scientists, scientists in charge and the scientific community was able to decipher the meaning of the universe.
The theory that the universe was not a very real one, was that the universe itself was not planet, but very. very real planct..
The discovery of a particular object in the universe that the universe was not We found something that resembled something that could not be described, but something that could not be described.
We found way make the object lot more specific We found that small, very small and very large.
very, very large thing
It was the first thing we noticed that we found that it was massive, large, enormous enormous, giant, giant.
We found way to make the whole thing. found that the entire
Generation with Conv seq2seg Self-Attention + D2GPo 35.74)
# I don 't understand Why did you tell us about "The scientist "s assistant, Dr. harry, looked up from his tablet and back at the scientist. " It ... you see. He looked back up Dr. harry What 's it mean ?"
" It means that the experiment is on the cusp of time. It not the same it looks. There 's no explanation
It 's just that..."
How is it possible that the experiment is going to fail ? " The scientist looked ut the window and then back at Dr. harry, who was still looking out the window at the vast blackness of the room. You think they 'Il do it Well, we can 't be sure, Dr. harry They 'Il do it.
Dr. harry turned to the scientist front of him. We can 't risk losing our best friend in the world
Generation with Fusion of Conv seq2seg= + Self-Artention (PPL 36.08):
We thought that we should be able to see our entire species The scientists were n't sure what we would find. But they had to be prepared for it They could n't be ready.. The scientists had never seen anything like this, and yet they had n't seen anything like this (unk) in months The scientists were just about to be their own business. when they saw the results. They were n't ready for anything...
We thought that we should take deep breath before we go. But it was n't. There was n't any sign of any other intelligent life in the universe. It was nothing. The scientists had idea what was going They just watched, waiting for signal.
They had no idea what it would mean.
But they could n't see us The scientists just knew. They were all afraid of what they would see. They could n't see the future. They could n't see the future.
Generation with Fusion of Conv seg2seq Self-Artentin D2GPo (PPL 33,82): a I'm afraid I wo n't he able find why my experiment is working, "Well, we 've been working on the project about a month now. " It been month and half since last saw it."
WWe 're all looking at the results. You' 've already been working
for months now You think we "ve found that ?
" "I do n't know, but we do have a lot rescarch to do.
But it 's not like was working is
We do n't know.. We 're not looking for a breakthrough, it 's just an experiment.
"I'ss 's just experiment? ? People will die and the world may be destroyed. The disaster is about to happen,
we have act."
" What do you mean, 'll not. It 's just an experiment..
No, no, no, it is something terrible we cannot ignore ."
Table 10: Example stories generated by the baselines and our full models.
18


Reviews:
[{"review_id": "SkeC04MptH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new Gaussian prior objective, \"D2GPo\", that addresses the fact that in sequence generative models, all incorrect predictions are penalized equally by MLE, a phenomenon which the authors refer to as the negative diversity ignorance drawback. The proposed objective is simple to implement and can easily be added on top of a regular cross-entropy loss. The paper shows that the new objective shows consistent improvements across a wide range of tasks.\n\n- Something that perturbed me a lot when reading Section 4, is that the function f should take as input 2 arguments, and not just 1. Typically, in Equation 7, from what I understand the numerator should be exp(f(y'_i, y_i)/T), and the denominator the sum over the exp(f(y'_j, y_i)/T). In other words, the value of f assigns a score to a word y'_j which depends on the target word y_i. This is also what suggests Figure 1: f depends on the target word. I think this should really be clarified in the paper, and f needs to be defined formally. In Figure 1, what is the exact value of f when the model outputs \"armchair\" / when the model outputs \"armchairs\". If I did not understand correctly, please point me out.\n\n- The cross-entropy loss in a generative model is essentially the KL divergence loss KL(Q, P) where Q is the true distribution (i.e. a one-hot vector) and P is the model output. In Equation 7, when T -> 0, the distribution Q becomes this one-hot vector. In that case, from what I understand, the D2GPo objective is actually very similar to the initial MLE objective, except that you compute KL(P, Q) instead of KL(Q, P), and that Q is not exactly one-hot because you consider a T > 0. Can you confirm this? Also, any reason why you considered KL(P, Q) instead of KL(Q, P)?\n\n- What value did you use for the temperature T? Did you study its impact?\n\n- Same question for lambda in Equation 6, what is the value you considered, and did you study its impact?\n\n- As mentioned in the introduction of the paper, one drawback with traditional neural generative, is the generation diversity, i.e. the fact that generations are generic, and an easy way to see this is to observe that generations are mostly composed of the most frequent words in the vocabulary. Did you evaluate whether a model trained with D2GPo has more diversity, and is more likely to generate rare words compared to a regular model trained with MLE only?\n\n- Figure 1 in the appendix is helpful. It would be nice to move it to Section 4.\n\n- In the related work, you write \"the Transformer provides us with a more structured memory for handling long-term dependencies\", which sounds a bit odd. There is no explicit memory component in the transformer, the ability to handle long-term dependencies comes instead from the self-attention mechanism.\n\nOverall the paper tackles an interesting problem which I feel has received surprisingly little attention from the community. The paper is well written, and has a lot of experiments supporting the method. But I think Section 4 needs to be clarified, and some experimental details are missing. Also, more information about the impact of T and lambda (at least on one type of experiments) would be very useful to have.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx6OwB0YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to add a prior/objective to the standard MLE objective for training text generation models. The prior penalizes incorrect generations/predictions when they are close to the reference; thus, in contrast with standard MLE alone, the training objective does not equally penalize all incorrect predictions. For the experiments, the authors use cosine similarity between fastText embeddings to determine the similarity of a predicted word and the target word. The method is tested on a comprehensive set of text generation tasks: machine translation, unsupervised machine translation, summarization, storytelling, and image captioning. In all cases, simply adding the proposed prior improves over a state-of-the-art model. The results are remarkable, as the proposed prior is useful despite the variety of architectures, tasks (including multi-modal ones), and models with/without pre-training.\n\nIn general, it is promising to pursue work in altering the standard MLE objective; changes to learning objective seem orthogonal to the modeling gains made in many papers (as evidenced by the gains the authors show across diverse models). This paper opens up several new directions, i.e., how can we impose even more effective priors? The authors show that it's effective to use a relatively simple fastText-based prior, but it's possible to consider other priors based on large-scale pre-trained language models or learned models. In this vein, a concurrent paper \"Neural Text Generation with Unlikelihood Training\" has also shown it effective to alter the standard MLE objective. I think it would be nice to discuss this paper and related works. Overall, I think the approach is quite general and elegant.\n\nMy main criticism is that the writing was unfocused or unclear at times. The intro discusses a variety of problems in generation, before explaining that the authors only intend to tackle one (\"negative diversity ignorance\"). It would have been more helpful to read more text in the intro that motivated the problem of negative diversity ignorance and the proposed solution. The second paragraph in the Discussion in Section 4 is rather ambiguous and hand-wavy. It would be nice to see the authors' intuition described more rigorously (i.e., explicitly describing in math how the cosine similarity score is used in the Gaussian prior, or describing in math how the central limit theorem is used). Some of the existing mathematical explanation in section 4 could be made simpler or more clear (the description of f(y) seems to be a distraction since it doesn't end up in the final loss).\n\nI would have also appreciated more analysis. After reading the paper, I have the following questions (which the authors may be able to address in the rebuttal):\n* Do off-the-shelf fastText embeddings work well? How important is it to train fastText embeddings on the data itself? If off-the-shelf embeddings worked well, that could make the method easier to use for others in practice.\n* How does the gain in performance with D2GPo vary based on the number of training examples? Priors are generally more helpful in low-data regimes. If that is the case here as well, you might get even more compelling results on low-data tasks (all tasks attempted here are somewhat large-scale, as I understand)\n* Qualitatively, do you notice any difference in the generations? How does the model make mistakes (are these more \"semantic\" somehow, i.e. swapping a different synonym in). Perhaps the gaussian prior has some failure modes, i.e., where it increases the probability of very incorrect/opposite words because they have a similar fastText representation. These kinds of intuitions would be useful to know\n\nI also have one technical question:\n* When you compare against MASS (Song et al. 2019), do you use the same code and/or pre-trained weights from MASS, or do you pre-train from scratch using the procedure from MASS? (The wording in the text is somewhat ambiguous.) I'm just wondering how comparable the results are vs. MASS, or if it would be useful to know how your version of the pre-trained model does.\n\n\nDespite my above questions/concerns, I think the proposed method or its predecessors could provide improvements across a variety of text generation tasks, so I overall highly recommend this paper for acceptance.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gsgn90YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the use of data-dependent Gaussian prior, to overcome negative diversity ignorance problem that includes the exposure bias problem for sequence generation models. In addition to the usual MLE (teacher forcing) criteria, the authors add the KL divergence between the prediction and the Gaussian PDF on the word embedding space. Experimental results show that the proposed method consistently improves the performance of the state-of-the-art methods for neural machine translation, text summarization, storytelling, and image captioning.\n\nI lean to accept this paper. The proposed method is well motivated and shown to be effective in several tasks for language generation.\n\nI have some major comments about the evaluation function $f(\\cdot)$. The authors propose to define it as a Gaussian distribution.\n- While this choice seems to be reasonable, I would like to know how its standard deviation can be defined. If it is a hyperparameter, the sensitivity of different deviations for the performance should be experimentally reported. A small valued deviation would make the KL divergence close to zero, while a large one makes its convergence slow.\n- Another way to remedy the problem of KL divergence above is applying Wasserstein distance instead of KL divergence. I would like to know if the authors have investigated the use.\n\nMinor comments:\n- White space should be inserted between \"sequence-to-sequence\" and \"(seq2seq)\" on the third page.\n- If the authors define a sequence using a bold and italic font as $\\boldsymbol{y}$, each token can be represented using an italic font to distinguish each token and the entire sequence: $\\boldsymbol{y}=<y_1,...,y_l>$. Otherwise, the sequence can be defined as $\\mathcal{Y}$ if the authors like to represent each token as a vector.\n- There is a typo on the fifth page. The word \"dada-independent\" should be \"data-independent.\"", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of poor generation quality in models for text generation that results from the use of the maximum likelihood (ML) loss, in particular the fact that the ML loss does not differentiate between different "incorrect" generated outputs (ones that do not match the corresponding training sequence).  The authors propose to train text generation models with an additional loss term that measures the distance from the ground truth via a Gaussian distribution based on embeddings of the ground-truth tokens.  This is not the first attempt to address drawbacks of ML training for text generation, but it is simple and intuitive, and produces improvements over the state of the art on a range of tasks.  The reviewers are all quite positive, and are in agreement that the author responses and revisions have improved the paper quality and addressed initial concerns.  I think this work will be broadly appreciated by the ICLR audience.  One negative point is that the writing quality still needs improvement.
Decision: Accept (Talk)