Title: ANALYZING THE FORGETTING PROBLEM IN THE PRETRAIN-FINETUNE FRAMEWORK FOR NEURAL LANGUAGE GENERATION MODELS Anonymous authors Paper under double-blind review

Abstract: In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator. We focus on the transformer encoderdecoder model for the open-domain dialogue response generation task. We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training. We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer. Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review". We find that mixreview effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

1. INTRODUCTION:

Large-scale unsupervised pre-training (Peters et al., 2018; Yang et al., 2019; Liu et al., 2019; Devlin et al., 2018; Song et al., 2019) has recently been shown to greatly boost the performance of natural language processing (NLP) models, and has attracted much research interest. Despite its huge success, there is a fundamental question remaining to be answered:

Is there some crucial weakness in the standard NLP pretrain-finetune framework?

In this work, we take the viewpoint of language generation and show that the answer is, to some extent, yes. In particular, we find that the key to answer this question is a concept we denote as data separation.

Although various unsupervised pre-training strategies have been proposed for better utilization of large-scale text data, on a high level the pretrain-finetune framework can be viewed as a simple two-stage procedure: (1) use large-scale text data to pre-train the model, and (2) use target task data to fine-tune the model. Data separation refers to (almost) zero-overlapping data usage of the two stages.

In this work we study the pretrain-finetune framework from the viewpoint of neural language generation (NLG). In particular, we focus on the open-domain dialogue response task, for the following reasons: (1) There is high similarity between the target dialogue response task (conditional NLG) and the pre-training language modeling (LM) objective, so we expect that language generation skills learnt during pre-training can be well transferred to the down-stream target task.

(2) The sequenceto-sequence (seq2seq) nature of the model allows us to characterize the model's generation behavior in various ways (e.g. context sensitivity).

We briefly summarize our contributions as follows. To study how pretrain-finetuning changes the model's behavior, we conduct a behavior analysis from the perspectives of context sensitivity and knowledge transfer. Our main finding is that in the fine-tuning stage, data separation causes the model to forget important language generation skills acquired during pre-training. Motivated by this analysis, we adopt the concept of data mixing and propose a mix-review fine-tuning strategy, where we combine the pre-training and fine-tuning objective. We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated. Finally, we demonstrate and discuss interesting behavior of the resulting dialogue model and its implications. 

2. TRAINING OBJECTIVE FOR SEQ2SEQ TASKS:

End-to-end dialogue response generation (Li et al., 2016) can be formulated as a sequence-tosequence (seq2seq) task: given a dialogue context (previous utterances), the model is asked to generate a high-quality response. In this work we adopt the encoder-decoder model architecture (Sutskever et al., 2014; Cho et al., 2014; Mikolov et al., 2010) , which is widely used in NLG applications like dialogue response generation (Li et al., 2016) , machine translation (Luong et al., 2015) , etc. In particular, we use the transformer model (Vaswani et al., 2017) , which has currently become the most popular encoder-decoder model architecture (Young et al., 2017) . We use the same configuration as Vaswani et al. (2017) , which has 6 encoder/decoder layers, 16 attention heads, with an embedding dimension of 1024 and a feed-forward dimension of 4096. where y ăt refers to ty 0 , y 1 , ..., y t´1 u, in which y 0 is set to a begin-of-sentence token <BOS>, and y m is a end-of-sentence token <EOS>. In the dialogue response setting, the input x is a concatenation of previous utterances. We truncate the length of x to be at most 128 words, which typically includes around 6 previous utterances.

Given a trained seq2seq model, to generate a response for some contextual input, one needs to choose a decoding method. Recent research (Holtzman et al., 2019; Radford et al., 2019; Fan et al., 2018) has shown that a strategy called top-k sampling, in which the next word is sampled from the top k most probable choices, is a better choice than the traditional beam-search decoding. Our preliminary experiments (Appendix A) have also verified this claim in the open-domain dialogue response setting. As a result, in this work, unless otherwise mentioned, we use top-k sampling as the default decoding method. In particular, we set k to 30 (we find it to work well in preliminary experiments).

3. THE PRETRAIN-FINETUNE FRAMEWORK:

In this section we review the pretrain-finetune framework for encoder-decoder models. More importantly, we discuss the language generation skills the model can acquire during pre-training, and how well they are transferred to the target task. This discussion leads to the proposition of the mix-review fine-tuning strategy.

3.1. PRE-TRAINING:

In this work, we consider pre-training the seq2seq model using large-scale unsupervised text data, and afterwards fine-tuning it using target dialogue data. We compare two representative strategies: next-sentence (NS) pre-training and masked sequence-to-sequence (MASS) pre-training (Song et al., 2019) . Next-sentence pre-training is a natural extension of GPT-style LM training (Radford et al., 2019; Kiros et al., 2015) for encoder-decoder models. For every sentence in a given training document, we set the previous sentences as the contextual input, and ask the model to generate the next sentence. We omit the formulation of NS because it is very similar to Equation (1).

Masked sequence-to-sequence pre-training (MASS) can be regarded as an extension of the "BERT" (Devlin et al., 2018) pre-training for encoder-decoder models. For each sentence, a random segment of the sentence is masked, and the model is trained to generate the masked words on the decoder side. We refer readers to Song et al. (2019) for more details.

In Table 1 , we illustrate the similarity between NS pre-training and typical dialogue response training. Compared to NS pre-training, MASS has the disadvantage that it focuses on one single sentence at a time. However, the context of multiple previous sentences are very important for dialogue response generation.

There are two important generation capabilities that the model can acquire in the pre-training stage, which will be useful for the target dialogue setting. One is the acquisition of knowledge (studied in Section 5.3): the large-scale pre-training text data contains a large amount of knowledge, and can be used to make dialogue responses more informative and engaging (e.g. the model can learn about the "Avengers" movie, and use it as a topic). The other is the utilization of contextual input (studied in Section 5.2): as shown by Sankar et al. (2019) , the current open-domain dialogue models (without pre-training) are insensitive to contextual input, which gives rise to the generic response problem (Li et al., 2016) . In our preliminary experiments with NS pre-training, we find that, similarly to the GPT model (Radford et al., 2019) , the pre-trained model has the ability to generate closely related responses given the previous sentences as input. Ideally during fine-tuning, the model can transfer this skill to the target dialogue task.

3.2. THE MIX-REVIEW FINE-TUNING STRATEGY:

Although recently a number of pre-training strategies (Peters et al., 2018; Devlin et al., 2018; Song et al., 2019; Yang et al., 2019; Liu et al., 2019) have been proposed for various NLP tasks, the finetuning stage remains simple and straightforward: simply fine-tune all parameters with a relatively small learning rate. In Figure 1a , we show the model's negative log-likelihood (NLL) on different evaluation sets during the fine-tuning stage. We identify two potential issues during fine-tuning. (1) Over-fitting:

The gap between training-set NLL and validation-set NLL increases quickly.

(2) Forgetting: The performance on the pre-training CCNEWS data (described in Section 4.1) drops drastically. Note that the forgetting phenomenon here is not necessarily "catastrophic" as in the sequential learning case (Atkinson et al., 2018; Robins, 1995) , because the goal is to achieve the best performance on the target dialogue data-set, and the model does not need to maintain fidelity to the pre-training data. However, it leads us to suspect that the model has lost some important skills learned during pre-training (verified in Section 5.2 and 5.3).

To address the forgetting phenomenon, we propose a fine-tuning strategy named "mix-review": For each fine-tuning epoch, we mix the target dialogue data with a random subset of the pre-training data. This process introduces two hyper-parameters: mix-ratio, which controls how much pre-training data is mixed, and mix-decay, which decays the amount of mixed data by each epoch. For example, assume the target dialogue training set has 100k utterances, mix-ratio"4 and mix-decay"0.9, then in the first epoch of mix-review fine-tuning, 400k pre-training utterances will be mixed in, and for the second epoch the amount will be reduced to 360k utterances, etc.

We formulate the mix-review objective as below:

L fine-tune pP target-data ; θq `pmix-ratio ¨mix-decay epoch-id q ¨Lpre-train pP pretrain-data ; θq

(2)

where epoch-id starts from 0. Note that the augmented mixing term can be viewed as a regularization term.

In our experiments, we tune the hyper-parameters (mix-ratio and mix-decay) in the grid of t1, 2, 4, 8, 16u Ś t1, 0.9, 0.8, 0.7, 0.6, 0.5u (using the same learning rate and other hyper-parameters with standard fine-tuning), and report with the best model based on the perplexity (PPL) performance on the validation set of the target task. We find that the performance gain of mix-review is not sensitive to hyper-parameter tuning: A small mix-ratio of 4 typically works well, which means the computational cost of mix-review is comparable to standard fine-tuning.

In Figure 1a , we show the loss curve for mix-review fine-tuning with a mix-ratio of 4 and a mixdecay of 0.7. We observe that the performance on the pre-training CCNEWS data is preserved, which strongly supports the motivation of mix-review. Furthermore, we observe a regularization effect from mix-review (narrowing the gap between training and testing performance).

We compare mix-review with the L 2 regularization (weight decay) toward the pre-trained parameters θ pre (Kirkpatrick et al., 2016; Wiese et al., 2017) . We denote it as WD(θ pre ) and formulate it as follows:

L fine-tune pP target-data ; θq `λ ¨ θ ´θpre 2 2

(3)

In our experiments, we tune λ in the set {10 ´1,10 ´2,10 ´3,10 ´4,10 ´5} and report with the best model based on PPL on the validation set.

In Figure 1b we show the loss curve for WD(θ pre ) with λ " 0.1. We observe that WD(θ pre ) also has a regularization effect, but it is not as strong as mix-review.

Additionally, we tried the following two basic regularization techniques: (1) Increase the rate of dropout; (2) Freeze the bottom layers of the model during fine-tuning. We find that these two techniques show little or no improvement. We believe the reason is that the transformer is already a well-tuned model (e.g. it features dropout and layer normalization (Lei Ba et al., 2016) ).

4 DATA-SETS AND IMPLEMENTATION DETAILS To construct the vocabulary, we learn codes of Byte Pair Encoding (BPE) (Sennrich et al., 2016) from the CCNEWS-100m data with 50k merges. This results in a vocabulary of size 62k. We then apply the same BPE codes to all target dialogue data-sets. Fine-tuning (with or without mix-review) is done on 2 GPUs without float16 speed-up. The learning rate is halved when the PPL on the validation set does not improve. In almost all fine-tuning experiments over-fitting is observed, and we do an early-stop when performance on the validation set starts to deteriorate. We tune the learning rate from {10 ´3,10 ´4,10 ´5}, and report the best model based on validation set perplexity.

5. EXPERIMENT RESULTS:

In this section, we first present results for the standard dialogue model evaluation. We then conduct a detailed behavior analysis, characterising how different training strategies change the model's behavior. In particular, we aim to answer the crucial question about whether the model forgets precious language generation skills during standard fine-tuning, and more importantly, whether mixreview helps the model remember the skills.

5.1. STANDARD DIALOGUE MODEL EVALUATION:

In addition to perplexity, we use the Amazon Mechanical Turk (AMT) platform for human evaluation of different training processes on the three dialogue data-sets. For the AMT rating, each turker is given a dialogue context, and a randomly permuted set of model sample responses. The turker is then asked to rate each sample response according to its fluency, consistency, and engagingness respectively, using an integer score from 1 to 9. The reference response is also rated for comparison.

For each data-model pair, we collect 2,500 ratings. The results are shown in Table 2 . To remove bias among annotators, we use the bayesian inference code from Kulikov et al. (2018) , and report calibrated mean and standard deviation. Since we use top-k sampling, the BLEU score is not directly suitable for our setting (Liu et al., 2016) .

We first observe the huge improvement in perplexity (larger than 40%) for the pre-trained models comparing to the baseline models trained from scratch. Comparing to MASS, the NS pre-training has more than 7% relative improvement. As discussed in Section 3.1, this confirms our earlier discussion that the model pre-trained by NS better utilizes contextual input (which is further verified in Section 5.2). Based on this observation, we focus our analysis below on the NS pre-training.

Comparing to standard fine-tuning, mix-review further gives solid improvement. The gain is due to its strong regularization effect (which we study in the next two sections). However, the performance gap between mix-review and WD(θ pre ) is not significant. We conjecture that mix-review could be too "aggressive" in regularizing the model's generative behavior, and more sophisticated regularization techniques are worth investigating. We revisit this discussion in Section 6.

Input: i 'd

In Table 3 we compare samples from different models on the Dailydialogue test-set. Compared to the baseline model, we find that the pre-trained model's responses are more related to the context. For example, in the second response, the baseline model talks about "fruit cake", while the pretrained models talk about beer, which is the main subject of this conversation. The samples from mix-review are interesting in that the model is able to describe beer with sophisticated words like "belgian ale" or "medium-batch", which we attribute to knowledge obtained during pre-training (the phrases "belgian ale" or "medium-batch" do not exist in the Dailydialogue training data).

5.2. BEHAVIOR ANALYSIS: CONTEXT SENSITIVITY:

The sensitivity to context is an important property for NLG models. However, as shown by Sankar et al. (2019) , dialogue response models trained from scratch typically are not sensitive to artificial distortion in the context input, showing the models have poor utilization of dialogue context. In this section, we repeat their experiments with pre-trained dialogue models.

Following Sankar et al. (2019) , we use two methods to distort the context input:

• word-drop: We randomly drop 30% of the words in the context input.

• word-shuffle: We randomly shuffle the words in the context input.

We use the relative drop in test-set perplexity to quantify the sensitivity. The results are presented in Table 4 , where the result of the pre-trained model is also included. First, we observe the baseline model trained from scratch is relatively insensitive to context, which agrees well with Sankar et al. (2019) . The model with the standard pretrain-finetune process is much more sensitive, showing that pre-training effectively changes the model's behavior. Comparing to MASS, the NS pre-trained model has better utlization of context, which explains its superior performance (in Section 5.1).

News-style Triggers:

Dialogue-style Triggers now, some opinions about pokemon .

what you do think about pokemon ? let me tell you about pokemon .

please tell me about pokemon . here's some news about pokemon .

do you have news about pokemon ? Reference Description: Pokemon first took the world by storm in the mid-90s, doing so once again this year with the release of Pokemon Go. NS Pre-trained: the game , titled pokemon go : pocket camp , can be played in person ... Standard Fine-tuned: it 's a new game that can be played with kids . WD(θ pre ): pokemon go , it 's a type of game that only exists in the us . Mix-review: pokemon go is a popular mobile game , where you 're expected to catch pokemon . Reference Description: Deadpool: The wisecracking antihero, played by Ryan Reynolds in a movie of the same name, became the highest-grossing R-rated film of all time. NS Pre-trained: ryan reynolds teased his upcoming movie as the character of deadpool . Standard Fine-tuned: it 's a popular movie . WD(θ pre ): yes , i really like him . he is a very funny character . Mix-review: ryan reynolds .

Table 5 : Example of trigger inputs for the knowledge term "pokemon". Followed by reference description and model samples for "pokemon" and "deadpool". Note that the pre-trained model's sample is from news-style triggers, and the other samples are from dialogue-style triggers. We highlight the pre-trained model's performance for news triggers and the performance of the best model fine-tuned with dialogue data for dialogue triggers. The results on Cornell Movie data-set is deferred to Appendix D.

Somewhat surprisingly, the NS pre-trained dialogue models are much less sensitive to context input than the pre-trained model without fine-tuning. This verifies our worry in Section 3.2 that the model is forgetting some important generation skill during standard fine-tuning. Further, we find that the mix-review fine-tuning strategy can effectively alleviate this problem: Its sensitivity is much greater than that of standard fine-tuning, and is close to the pre-trained model.

5.3. BEHAVIOR ANALYSIS: KNOWLEDGE TRANSFER:

As argued in Section 3.1, ideally the model can acquire "knowledge" from the large-scale pretraining data, which will be useful for the downstream open-domain dialogue task. In this section, we design a process to quantify how much knowledge the model has, and use it to monitor how the pretrain-finetune framework changes the model's behavior.

Since the pre-training CCNEWS data is in the public news domain, we expect the model to have knowledge about "big news". So, we utilize the Google trend data of the year 2016,2 which contains 365 trending terms (e.g. iPhone 7, Deadpool, etc.), and its corresponding description.

To query whether the model has knowledge of a certain term, we design three news-style and three dialogue-style "trigger templates" to trigger the model to generate responses related to the knowledge term. We collect 10 samples for each trigger (30 samples from news/dialogue-style triggers for each term), then we compute BLEU score of generated samples against the reference descriptions. We show some examples of trigger inputs in Table 5 . 7 : AMT rating scores (calibrated mean and standard deviation) for multi-turn dialogue evaluation.

The BLEU scores are shown in Table 6 . Note that we should compare the pre-trained model's scores for the news triggers with the other dialogue models' scores for dialogue triggers. We first observe for the pre-trained model, the news-style triggers can get much more relevant output than the dialogue-style triggers. This matches our intuition because the pre-trained model is trained with news data. Although the fine-tuned model is more knowledgeable than the baseline model, its score is much lower than the pre-trained model. Similar to the case of context sensitivity (Section 5.2), this again demonstrates the forgetting problem of the standard fine-tuning.

We find that mix-review and WD(θ pre ) can effectively retain the knowledge acquired during pretraining, giving a much higher BLEU score than the standard fine-tuned model. Mix-review shows higher BLEU scores than WD(θ pre ), demonstrating its superiority in facilitating knowledge retention. We showcase samples from different models in Table 5 . To save space, we manually select and show the most related sample out of the 30 samples for each knowledge term. The observations agree with the quantitative results: the standard fine-tuning loses the detailed information about the knowledge term, and mix-review helps the model retain it. More importantly, the model is able to express the knowledge in a dialogue context.

To further investigate the ability of our model to conduct knowledgeable dialogues with users, we use the ParlAI3 platform to conduct multi-turn dialogue evaluation. For each session, the user will be assigned a random knowledge term from Google Trend, and have a 8-turn dialogue with the model under that topic. Ratings from around 600 dialogues are collected for each model, and are reported in Table 7 . In this evaluation we use the models fine-tuned on the Dailydialogue data, because the nature of that data-set is closet to online chit-chat. It is observed that the model trained mix-review significantly outperforms WD(θ pre ) on knowledge, consistency and engagingness, which agrees well with the results in Table 6 and Table 4 (Wolf et al., 2019; Mehri et al., 2019; Song et al., 2019; Devlin et al., 2018) , but how pre-training changes the behavior of a neural language generator is poorly understood. Several studies have shown that large-scale training teaches LM common-sense knowledge (Petroni et al., 2019; Trinh & Le, 2019) , in which the captured knowledge is quantified by a cloze-style test.

On the other hand, knowledge-grounded chat-bots (Liu et al., 2018; Zhu et al., 2017) have been an important topic for dialogue models. These studies usually involve additional retrieval modules to provide the model with relevant information. Unlike these works, we study whether fine-tuning preserves knowledge gained during large-scale pre-training.

8. CONCLUSION:

In this work, we analyze forgetting problem for the standard NLP pretrain-finetune framework in the viewpoint of language generation. We adopt the concept of "data mixing" and propose the mix-review fine-tuning strategy. We demonstrate that mix-review can effectively help the model remember important generation skills learned during pre-training.

Through a detailed behavior analysis, we find that under the surface of the performance boost for standard metrics, large-scale pre-training changes the model's generative behavior in various profound ways (e.g. context sensitivity). More importantly, the behavior change is influenced by the nature of data itself. For example, we demonstrate that we can discuss news with the resulting dialogue model, even when the fine-tuning data is not about news (Dailydialogue) . This opens the exciting possibility of a completely data-driven way to customize a language generator. 

C MODEL SAMPLES:

In Table 10 , we show interaction samples where is turker and the model is talking about an assigned topic from Google Trend.

In Table 11 , we show more samples from different training procedure, for the three dialogue datasets.

D SUPPLEMENTARY EXPERIMENT RESULTS:

In this section we supplement results that are deferred in the main body due to space limit.

In Table 12 we show Fluency/Consistency/Engagingness scores of the AMT Rating.

In Table 13 we show context sensitivity results for Switchboard and Cornell Movie data-sets.

In Table 14 we show the knowledge transfer results for the Cornell Movie data-set.

E DETAILS AND AUXILIARY PLOTS OF UMAP PROJECTION:

For function space projection, the input to UMAP should be the model's output distributions. We collect the model's output distribution on 10k words for the CCNEWS validation set and the Dailydialogue validation set (so it's a concatenation of two long vectors). We use the default hyperparameter setting of the python implementation of UMAP. The result is shown in Figure 2 in the main body. Note that during pre-training of the CCNEWS data, 20 epochs are one entire data pass. We fine-tune from epoch 100, 200, 300, 400, 500 of the pre-training checkpoints.

AMT Ratings Fluency/Consistency/Engagingness Model Dailydialogue Switchboard Cornell Movie Baseline 6.09/5.90/6.23 5.58/5.59/5.60 5.96/5.72/5.96 MASS+finetune 6.32/6.12/6.35 5.62/5.61/5.49 6.02/5.84/6.05 NS+finetune 6.21/6.21/6.39 5.60/5.67/5.66 5.93/5.84/6.05 NS+WD(θ pre ) 6.32/6.17/6.45 5.70/5.79/5.79 6.00/5.92/6.06 NS+Mix-review 6.35/6.23/6.41 5.71/5.76/5.74 6.03/5.91/6.11 Reference 6.54/6.46/6.70 5.87/6.00/5.98 6.01/6.02/6.28 We highlight the pre-trained model's performance for news triggers and the performance of the best model fine-tuned with dialogue data for dialogue triggers.

:

http://commoncrawl.org/2016/10/news-dataset-available

https://www.google.com/intl/en-US/trends/2016records/

https://parl.ai/

http://compprag.christopherpotts.net/swda.html

https://www.cs.cornell.edu/ ˜cristian/Cornell_Movie-Dialogs_Corpus.html

Figures and Tables:

Figure 1: Model's performance on different evaluation sets during the fine-tuning stage, for the Dailydialogue data-set (described in Section 4.1).

For fine-tuning, three open-domain conversational dialogue data-sets are used: Dailydialogue (1.3 million words)(Li et al., 2017), Switchboard (1.2 million words), and Cornell Movie (Danescu-Niculescu-Mizil & Lee, 2011) (4.5 million words). To save space, we defer the details of the datasets to Appendix B.

Illustrations of input-output pairs for typical dialogue response training, next-sentence pretraining, or MASS pre-training.

Table content: <table/>

Bakhtin et al., 2019) which is a deduplicated subset of the English portion of the CommonCrawl news data-set 1 . The dataset contains news articles published worldwide between September 2016 and February 2019. It has in total around 1 billion sentences or 27 billion words. To be able to complete experiments in a reasonable amount of time, we use the first 10 percent of the CCNEWS data for pre-training, which contains 100 million sentences and 2.7 billion words.

Table content: <table/>

Perplexity and AMT-Rating evaluation for different training process on the three dialogue data-sets. The rating scores are the average score of fluency, consistency, and engagingness.4.2 IMPLEMENTATIONOur code is based on the Fairseq toolkit(Ott et al., 2019). The Adam optimizer(Kingma & Ba, 2014) is used for all experiments. For pre-training of both MASS and NS, we use a mini-batch size of 2048, with the learning rate (LR) set to 0.0001. FollowingVaswani et al. (2017), the "inverse square root" LR scheduler with a warm-up stage is used. Pre-training is conducted on 32 GPUs and half-precision (float16) speed-up is used. For both MASS and NS, we stop the pre-training after the CCNEWS data is swept 20 times. Although the perplexity is still improving, we stop the pre-training for practical reasons to control the duration of the experiments. For all our experiments, a dropout rate of 0.1 is applied to the transformer model. We followSong et al. (2019) for the recommended hyper-parameter setting of MASS (e.g. how to select the mask span).

Table content: <table><tr><td>Training</td><td>Dailydialogue</td><td>Test-PPL/AMT Rating Switchboard</td><td>Cornell Movie</td></tr><tr><td colspan="4">Baseline(from scratch) 24.83/(6.323 ˘0.056) 51.14/(5.269 ˘0.052) 49.48/(5.844 ˘0.056)</td></tr><tr><td>MASS+finetune</td><td colspan="3">12.78/(6.511 ˘0.050) 28.41/(5.252 ˘0.049) 30.25/(5.955 ˘0.065)</td></tr><tr><td>NS+finetune</td><td colspan="3">11.54/(6.515 ˘0.060) 26.37/(5.332 ˘0.053) 28.06/(5.932 ˘0.061)</td></tr><tr><td>NS+WD(θ pre )</td><td colspan="3">11.19/(6.553 ˘0.056) 26.25/(5.439 ˘0.055) 27.80/(5.961 ˘0.062)</td></tr><tr><td>NS+mix-review</td><td colspan="3">11.07/(6.577 ˘0.053) 25.92/(5.414 ˘0.054) 27.54/(5.998 ˘0.055)</td></tr><tr><td>Reference</td><td>NA/(6.816 ˘0.052)</td><td>NA/(5.630 ˘0.053)</td><td>NA/(6.071 ˘0.056)</td></tr></table>

like a beer . what kind of beer do you have ? Baseline: i 'm not sure . could you recommend some ? i 'm not sure . MASS: we have various kinds of beer , beer and soft drinks . NS: we have a beer , but we don ' t have a lager and a black beer . Samples of different models on the Dailydialogue test-set. The samples are consecutive (input of the next sample is the reference response for the previous one). More samples are given in Appendix C.

Table content: <table><tr><td colspan="3">WD(θ pre ): i ' d like a beer of medium pace .</td><td/></tr><tr><td colspan="4">Mix-review: we have a malty and belgian ale . it is a little special .</td></tr><tr><td colspan="3">Input: draft beer and cans beer . which do you prefer ?</td><td/></tr><tr><td colspan="2">Baseline: well , two large fruit cakes .</td><td/><td/></tr><tr><td colspan="3">MASS: how about a soft beer ? it ' s very good .</td><td/></tr><tr><td colspan="3">NS: a bottle of beer with some meat , sir .</td><td/></tr><tr><td colspan="3">WD(θ pre ): do you have any beer with a clear finish ?</td><td/></tr><tr><td colspan="3">Mix-review: can you make a can , please ?</td><td/></tr><tr><td>Model(Data-set)</td><td colspan="3">PPL(normal) PPL(word-shuffle) PPL(word-drop)</td></tr><tr><td>NS Pre-trained(CCNEWS)</td><td>17.33</td><td>36.56(+110.96%)</td><td>35.56(+105.19%)</td></tr><tr><td>Baseline(Dailydialogue)</td><td>24.83</td><td>27.87(+12.2%)</td><td>31.87(+28.3%)</td></tr><tr><td colspan="2">MASS+finetune(Dailydialogue) 12.78</td><td>15.85(+24.0%)</td><td>18.13(+41.8%)</td></tr><tr><td>NS+finetune(Dailydialogue)</td><td>11.54</td><td>16.30(+41.2%)</td><td>19.01(+64.7%)</td></tr><tr><td>NS+WD(θ pre )(Dailydialogue)</td><td>11.19</td><td>14.16(+26.5%)</td><td>16.37(+46.2%)</td></tr><tr><td colspan="2">NS+Mix-review(Dailydialogue) 11.07</td><td>17.81(+60.8%)</td><td>23.05(+108.2%)</td></tr></table>



Table content: <table/>

Average BLEU-2/BLEU-3 scores for the model's samples w.r.t. the reference description.

Table content: <table><tr><td/><td colspan="2">Dailydialogue</td><td colspan="2">Switchboard</td></tr><tr><td>Model</td><td colspan="4">Dialogue Triggers News Triggers Dialogue Triggers News Triggers</td></tr><tr><td>NS Pre-trained</td><td>0.245/0.089</td><td>0.347/0.153</td><td>0.245/0.089</td><td>0.347/0.153</td></tr><tr><td>Baseline</td><td>0.124/0.007</td><td>0.101/0.004</td><td>0.032/0.0003</td><td>0.046/0.002</td></tr><tr><td>NS+finetune</td><td>0.162/0.047</td><td>0.158/0.046</td><td>0.187/0.052</td><td>0.170/0.044</td></tr><tr><td>NS+WD(θ pre )</td><td>0.226/0.080</td><td>0.235/0.085</td><td>0.203/0.070</td><td>0.204/0.060</td></tr><tr><td>NS+Mix-review</td><td>0.261/0.108</td><td>0.322 /0.135</td><td>0.223/0.079</td><td>0.341/0.151</td></tr></table>

˘0.061 5.359 ˘0.067 4.288 ˘0.067 3.848 ˘0.051 NS+WD(θ pre ) 3.189 ˘0.067 5.671 ˘0.059 4.605 ˘0.064 4.184 ˘0.059 NS+Mix-review 3.401 ˘0.051 5.692 ˘0.060 4.755 ˘0.068 4.272 ˘0.056

Table content: <table><tr><td>Dailydialogue</td><td>Knowledge</td><td>Fluency</td><td>Consistency</td><td>Engagingness</td></tr><tr><td colspan="2">NS+finetune 2.829 Table</td><td/><td/><td/></tr></table>

. We include some dialogue examples in Appendix C, and Table 8 also has some related examples. Pre-training for NLG Models Unsupervised pre-training for NLG models has recently received much research attention

Table content: <table><tr><td colspan="2">6 IMPLICATIONS AND DISCUSSION</td><td/><td/><td/></tr><tr><td>4</td><td/><td/><td/><td/></tr><tr><td>3</td><td/><td/><td/><td/></tr><tr><td>2</td><td/><td/><td/><td/></tr><tr><td>1</td><td/><td colspan="3">weight-decay(different epoch)</td></tr><tr><td/><td/><td colspan="2">mix-review with ratio 1</td><td/></tr><tr><td>0</td><td/><td colspan="2">mix-review with ratio 4 baseline(different seed)</td><td/></tr><tr><td>-1</td><td/><td colspan="2">pre-train(different epoch) standard fine-tuned</td><td/></tr><tr><td>-10</td><td>-8</td><td>-6</td><td>-4</td><td>-2</td></tr><tr><td colspan="5">Figure 2: UMAP projection of checkpoints from different training processes.</td></tr></table>

Multi-turn dialogue samples with turkers from the model trained by mix-review. 0 represents the turker, and 1 is the model. The Switchboard Dialogue Act Corpus 4 is a version of the Switchboard Telephone Speech Corpus, which is a collection of two-sided telephone conversations, annotated with utterance-level dialogue acts. In this work we only use the conversation text part of the data, and select 1.1k dialogues for training (181k sentences / 1.2 million words), 50 dialogues for validation and 50 dialogues for testing.The Cornell Movie Dialogue Corpus 5 (Danescu-Niculescu-Mizil & Lee, 2011) is a collection of movie scripts. In the processing of the data, we simply regard the whole scripts from a movie as a long dialogue. The training split contains 9k dialogues (4.5 million words), and both the validation and test splits have 180 dialogues (85k words).

Table content: <table/>

The detailed rating scores from AMT.

Table content: <table><tr><td>Model(Data-set)</td><td colspan="3">PPL(normal) PPL(word-shuffle) PPL(word-drop)</td></tr><tr><td>NS Pre-trained(CCNEWS)</td><td>17.33</td><td>36.56(+110.96%)</td><td>35.56(+105.19%)</td></tr><tr><td>Baseline(Switchboard)</td><td>51.14</td><td>53.42(+4.4%)</td><td>53.94(+5.4%)</td></tr><tr><td colspan="2">MASS+finetune(Switchboard) 28.41</td><td>32.68(+15.0%)</td><td>33.91(+19.3%)</td></tr><tr><td>NS+finetune(Switchboard)</td><td>26.37</td><td>30.87(+17.0%)</td><td>32.08(+21.6%)</td></tr><tr><td>NS+WD(θ pre )(Switchboard)</td><td>26.25</td><td>31.31(+19.2%)</td><td>32.89(+25.2%)</td></tr><tr><td colspan="2">NS+Mix-review(Switchboard) 25.92</td><td>31.10(+19.9%)</td><td>33.70(+30.0%)</td></tr><tr><td>Baseline(Cornell)</td><td>49.48</td><td>50.22(+1.4%)</td><td>50.85(+2.7%)</td></tr><tr><td>MASS+finetune(Cornell)</td><td>30.25</td><td>36.50(+20.6%)</td><td>36.36(+20.1%)</td></tr><tr><td>NS+finetune(Cornell)</td><td>28.06</td><td>36.88(+31.4%)</td><td>34.47(+22.8%)</td></tr><tr><td>NS+WD(θ pre )(Cornell)</td><td>27.80</td><td>37.46(+34.7%)</td><td>35.10(+26.2%)</td></tr><tr><td>NS+Mix-review(Cornell)</td><td>27.54</td><td>36.94( +34.1%)</td><td>37.72(+36.9%)</td></tr></table>

The model's PPL performance when word-shuffle or word-drop is applied to the context input. On the left we describe what training process is used and on which test set is PPL evaluated. Note that MASS/NS refers to MASS/NS pre-training with standard fine-tuning.

Table content: <table><tr><td/><td>Cornell</td><td/></tr><tr><td>Model</td><td colspan="2">Dialogue Triggers News Triggers</td></tr><tr><td>NS Pre-trained</td><td>0.245/0.089</td><td>0.347/0.153</td></tr><tr><td>Baseline</td><td>0.081/0.003</td><td>0.088/0.003</td></tr><tr><td>NS+finetune</td><td>0.207/0.071</td><td>0.207/0.063</td></tr><tr><td>NS+WD(θ pre )</td><td>0.285/0.114</td><td>0.202/0.072</td></tr><tr><td>NS+Mix-review</td><td>0.396/0.190</td><td>0.212/0.065</td></tr></table>

Average BLEU-2/BLEU-3 scores for the model's samples w.r.t. the reference description.

Table content: <table/>

Peer review (iclr2020):
The paper studies the forgetting problem in the pretraining-finetuning framework common to neural language generation models. It makes two main contributions: (1) it analyzes the forgetting problem from the perspective of “context sensitivity” and “knowledge transfer”. (2) In light of these analyzes, it proposes the Mix-Review fine-tuning strategy, which slightly outperforms the Weight Decay (WD) method of [Wiese et al., 2017]. Strengths: (1) The paper addresses an important problem, as a very large body recent work relies on BERT, GPT-2, and related approaches to improve their models on specific tasks. The problem with these pre-trained approaches is that they bring challenges similar to those pertaining to continuous learning that have been studied for many years, e.g., (catastrophic) forgetting. There is a need to study the forgetting problem specifically in the context of the pretraining-finetuning framework and compare it approaches borrowed from the continuous learning literature, which the paper does to a certain extent (the WD method). (2) The paper shows some nice empirical gains (though results are mitigated by weakness 2 below) and a better ability to “transfer knowledge” between the pretrained model and its application to the downstream task. Weaknesses: (1) The approach of the paper (Mix-Review) is very similar to methods that have been proposed already, and the papers doesn’t acknowledge the huge body of prior work already existing on (catastrophic) forgetting. See [Kemker et al., 2017] for a comprehensive overview on the subject. There are multiple families of methods to mitigate the forgetting problem, including regularization methods, rehearsal (and pseudo-rehearsal) methods, ensemble methods, etc. Mix-Review belongs to the rehearsal category, as it reintroduces instances learned long ago (in the case of the paper, pretraining instances) to the current training regime. There is also a large body of work on rehearsal/pseudo-rehearsal going back to [Robins, 1995]. Robins’ method that seems the closest is “random rehearsal”, which as in the paper mixes randomly selected instances of the older data. See my detailed for other weaknesses in related work. (2) Experimental results are not convincing. First, results of Figure 1 mainly show that Mix-Review is less prone to “forgetting” than WD, but I find that unsurprising. At the risk of sounding obvious: The notion of forgetting only makes sense in the context of information (here, pretraining instances) that has been seen in the *past*, but those instances continue to be shown to mix-review in the *present*. Contrasting rehearsal methods to non-rehearsal in this manner is akin to comparing training-set perplexity with validation-set perplexity (i.e., in one of the two cases, we compute perplexity on what we are currently training on). So, to me, figure 1 doesn’t tell us very much, and approaches should be mainly evaluated on a downstream task (which the authors do in table 2). Unfortunately, in the case of table 2, mix-review doesn’t show much gain when compared to the standard WD (aka EWC) method. There is a small gain that seems to be within the confidence intervals (assuming that is what is shown in parentheses). Now, my problem with these results of table 2 is this quote from the paper: “Although the perplexity is still improving, we stop the pre-training for practical reasons to control the duration of the experiments.” The difference between the baseline and the other systems shows that pretraining is very useful to all 3 downstream tasks, but this performance of the various models could be highly dependent of when the authors applied early stopping. Now, the actual issue is that early stopping is NOT applied consistently between WD and mix-review, as the WD approach has seen the pretraining data only 20 times (“we stop the pre-training after the CCNEWS data is swept 20 time”), but mix-review continues to be shown CCNEWS data even after 20 epochs, so there is effectively no early stopping for mix-review. Conclusion: Overall, I think the paper has merits in showing that rehearsal methods should probably be explored in the context of pretraining/finetuning architectures, but problematic experimentation and lack of comparison with previous work (there is only one baseline, WD, which is not enough to me given all the prior work on the subject of forgetting mitigation) make me recommend rejection for now. Detailed comments: The reference to WD is [Wiese et al., 2017] is not the right one. Wiese et al. give credit to Kirkpatrick [2017] for their Elastic Weight Consolidation (EWC) method as WD is a special case of EWC. EWC has been widely used to mitigate the forgetting problem and is not mentioned in the submission. The mix-review training objective contains a mix-decay parameter, which is unclear as it doesn’t seem to appear in any of the equations (was it meant to be in Eq. 2?). In any case, for the paper to be of practical importance, I think it should show the effect of pre-training data percentage over time. It seems it starts with a high proportion of pre-training data, which could be wasteful as fine-tuning starts off by utilizing mostly pre-training instances. This gets fixed over time thanks to the mix-decay parameter, but as repetitive decaying of the pre-training data occurs, that pre-training data essentially vanishes to ultimately be empty. What prevents catastrophic forgetting in that case, after many epochs? Other methods borrowed from the catastrophic-forgetting literature wouldn’t have this problem. It seems the choice of how long we let the model train could have a significant impact on how much the model forgets. [Kemker et al., 2017]: Measuring Catastrophic Forgetting in Neural Networks Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, Christopher Kanan https://arxiv.org/abs/1708.02072 [Kirkpatrick, 2016]: Overcoming catastrophic forgetting in neural networks James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell https://arxiv.org/abs/1612.00796 Please refer to Kemker et al. for other references mentioned in my review, as they sum up some of the large body of work done in that space.