Encoder-Agnostic Adaptation for Conditional Language Generation
Authors: Zachary M. Ziegler
Luke Melas-Kyriazi
Sebastian Gehrmann
Alexander M. Rush
Venue:  (2020)
Keywords: NLP
generation
pretraining

Abstract: Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However, it is an open question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work, we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. Therefore, we propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks, we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data-efficient.

Under review as a conference paper at ICLR 2020
ENCODER-AGNOSTHE ADAPTATION FOR CONDITIONAL LANGUAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Large pretrained language models have changed the way researchers approach diseriminative natural language understanding tasks, leading to the dominance approaches that adapt a pretrained model for arbitrary downstream tasks. How- ever, it is an open question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work, we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. Therefore, we propose adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments four diverse conditional text generation tasks, we show that this secoder-agnostie technique outperforms strong baselines, produces coherent generations, and is data-efficient.
INTRODUCTION
Large-scale language models have been shown dramatically improve the performance of natural language understanding (NLU) systems on broad range of tasks (Peters et al. [2018] Devlin et al., 2018 Radford & Salimans 2018 McCann et al., 2017). The dominant paradigm to pretrain a self attention-based language model on a large corpus of unlabeled text and then finetune the language model with an additional task-specific classification head on supervised data. Optimizing the effectiveness of this approach has been the focus of much study (Houlsby et al. 2019 Wang
ct al. 2019, Chronopoulou et al., 2019).
Given the success of pretraining for NLU tasks, how can large language models best be adapted for conditional language generation? Ideally, one should only need to train a large language model once and then apply it as part of the decoder to range of tasks with different source modalities (e.g., text, images, bits). In the encoder/decoder framework, a task-specific encoder can encode source information into continuous vector. The central question is thus how adapt a pretrained decoder to effectively utilize arbitrary source information from an encoder (encoder-agnostic adaptation).
Considering the high quality of samples from large languae models (Radford et al., 2019). it is natural to expect encoder-agnostie adaptation to improve the coherence and grammaticality of con- ditional text generation over training decoder from scratch, even when the source modality is not text (e.g. image captioning or class-conditional generation). Unfortunately, past results indicate otherwise. [dunov al. (2019) show, for example, that straightforward extension of contextual representations (Peters et al. (2018) to the conditional generation setting actually hurts performance compared to a model without any pretraining. Other pretraining approaches for language generation (Song et al. 2019 Dong et al. 2019 Lample & Conneau 2019) have demonstrated strong perfor- mance on text-to-text tasks, but these methods are constrained to tasks where the source natural language and do not address the encoder-agnostic setting.
In this work, we consider several different approaches to the problem of encoder-agnostic adaptation. We first observe that standard adaptation approaches perform poorly on this task. We hypothesize that because these techniques require relearning significant parts of the network structure to inject contextual conditioning, they move the parameters too far from the pretrained values. In contrast, Radford et al. (2019) observe that even trivial conditioning with the original model produces rea- sonable zero-shot generations without finetuning.

---

Under review as a conference paper at ICLR 2020
These results motivate an approach that learns the correct conditioning to control the model's output, which we call pseudo self attention. The idea is to learn a task task-specific encoder that injects pseudo history into a pretrained self attention model. Because self attention works with sets of any size, the model can immediately utilize or ignore this history. Finetuning adapts the model to this new input while training a task-specific encoder.
Experiments utilize the GPT-2 (Radford et al. 2019) transformer as a pretrained model. We consider four diverse generation tasks spanning range of source modalities: class-conditional generation, document summarization, story generation, and image paragraph captioning. Across all tasks, we find that pseudo self attention outperforms the other pretraining methods and is the most consistent. As a practical tool, pseudo self attention improves performance compared to a baseline without pretraining by large margins without sacrificing adherence to the source, even for tasks with large amounts of supervised data. We further demonstrate that the approach is data-efficient and produces qualitatively more coherent outputs. Code is available athttps:// com/ com/anon372347 Secoder-agnosee -ccpptttiiiiiitiiii
2 RELATED WORK
Pretrained Decoder Transfer learning for NLG Natural language generation (NLG) tasks have a long history of incorporating unconditional language models with conditional input(Bahl et al.
1983 Koehn et al.) 2003). These approaches traditionally use the noisy channel model (i.e., Bayes rule), and n-gram models as the language model. Recent adaptations of these ideas include the Neural Noisy Channel ( YYu et al. 2017) as well as "fusion" methods (Kochn et al. 2003 Gulcehre
itt al. 2015 Sriram et al. 2018; Stahlberg et al. 2018) in which the output logits of a language model and conditional model are combined to calculate the output probabilities. We consider this
class of transfer learning as a baseline in preliminary experiment (see Section 4.1). but focus on alternative "deep" approaches that incorporate the language model weights as an integral part of the model instead of an add-on at the end. Along these lines, [Ramachandran al. (2017) propose a finetuning-
method for machine translation with LSTMs, which some of the layers of the LSTM are initialized with pretrained language model weights. As their method is specific to LSTMs, however, it is incompatible with modern transformer architectures.
Pretraining-Basse Based Transfer Learning for NLG Zhang et al. (2019) use BERT in the encoder and decoder of a summarization model via a unique cloze generative process. They demonstrate strong summarization performance, but the value of pretraining relative other model components is not clear, and the cloze process significantly reduces the practicality of the model. More related, approach for applying ELMO (Peters)
Edunov et al. (2019) experiment with a representation-baase al. (018)) to the source and target sides of a standard seq2seq model separately. Their approach consistently improves performance when applied to the source, but hurts performance when applied to the decoder. We consider such representation approach as a baseline in this work.
Most recently, several studies experiment with BERT-like masking approaches that are compatible with natural language generation (Song al. 2019: Dong et al.) 2019: Lample & Conneaul 2019). While these works demonstrate impressive performance, they are constrained to text-to-text tasks because they do not have a way to handle arbitrary conditional information. Whereas these works study pretraining methods that optimize transfer for text-to-text tasks, our study considers the sepa- rate problem of adapting a fixed pretrained model arbitrary source conditioning.
Concurrent with this work, Golovanov et al. (2019) propose a similar approach to pseudo self at- tention and report initial experiments with dialogue generation. This study complements ours with positive results on dialogue generation, though we aim for experimental data over a wide range of language generation tasks and input modalities and comparison to strong encoder-agnostic baselines.
METHODS
We assume that we have a pretrained language model, p(y) = p(v,,..., :T: that the model an autoregressive neural network, and that it is based on self attention to implement conditioning, i.e.,
SA(Y) softmax (YY.) (YW.),
2

---

Under review as a conference paper at ICLR 2020
LieanSoflina
Linear/ootmax
LineanSotmax
Fead Forward
Feed Fonward
Feed Forward
Attention
Atlantion
XN
XN
XN
Encodor
Enioder
Enooder
01101
01101
01101
ONN
CNN andrer
load ancher...
Contextual embeddinng
Toslin Embedding
lead anctie
TokarnPositonn Embedding
The quick brownn fox...
The quick brown fox
The quick brown fox...
(a) Repr- Transformer
(b) Context-Atth
(c) Pseudo-Setl
Figure 1: Encoder-agnostee variants considered. All methods utilize task-specific source encoder, but vary in which parts of the decoder are pretrained and which are randomly initialized. Repr- Transformer trains new full transformer decoder, Context-Attn trains new context attention layer, Pseudo-Self attention only modifies part of the self attention layer. Residual connections and layer normalization have been omitted for clarity. Green indicates that parameters are initialized with pretrained weights, gray indicates random initialization. Red vectors indicate the target activations each layer. Blue vectors indicate the source features the output of the encoder. xN indicates that the section within the dotted lines is stacked N times.
where input Y E T' D for hidden dimension D, Ws. W., WG D D' are parameters, represent- ing the key, value, and query projections respectively, and the output is T x D'. We are interested in using this model estimate the conditional probability pyy x)) for an arbitrary input æ for which we have a small amount of supervised (z, y) pairs. The goal is to learn model on this new data that best makes use of the pretrained model p(y) with method that is agnostic the form of
All models are based on the encoder/decoder architecture, and for each we follow the same high-level procedure: First, some of the weights of the decoder are initialized with weight values from pretrained language model. Next, task-specific encoder and all non-pretrained decoder weights are randomly initialized. Finally, the entire model is trained/finetuned end-to-end using the supervised data for the given task. In all cases, the input and output embeddings are tied. Each approach uses all of the pretrained weights, differing only in where and how they use pretrained weights in the decoder. Further experimental details are included in appendix Section
Baseline 1: Repr- Transforerr The first approach considered (Fig lap utilizes the pretrained LM to produce a cenerol-puupose representation of the target text before introducing the source infor- mation. For this method, standard transformer decoder is used with the target word embeddings replaced by the output representation of the pretrained language model. In preliminary experiments, considered both fixing and updating these representations and found that fixed weighted- averaging ("ELMo-Style") method performed better, consistent with Edunov et al. (2019). One possible downside to this approach is that the conditioning information from the encoder is injected after all of the pretrained weights.
Baseline 2: Context-Attn The second approach (Fig lb considers initializing standard trans- former decoder with the shared weights of pretrained LM. The newly added context attention weights at each layer are randomly initialized. Compared to Repr- TransTraaffrreer, the conditioning information injected alongside the pretrained weights. However, the randomly initialized context attention block may interfere with the carefully co-tuned pretrained weights of the rest of the model. This interference may introduce optimization challenges and lead to reduced performance.
In practice many of these units ("heads") are stacked together via concatenation across dimension followed by final linear projection W, E D x D.
3

---

Under review as a conference paper at ICLR 2020
Proposed Model: Pseudo-Self A more radical approach to incorporating conditional information the "zero-shot" model proposed by Radford et al. (2019). Instead of learning a representation for and passing it into a context attention block they note that an auto-regressive model, p(y (y is already conditional model. If 2: is the same modality as (e.g., both language), one can condition on by prepending the source target: p(y: |x. V-ce) = p(ye. produce competitive models and is limited in its applicability. it is surprising that it works at all.
vct)) While this does not
Taking inspiration from this approach, agnostic way. Our approach, pseudo self attention, simply injects learned encoder conditioning directly into the pretrained self attention of the model. Assume that we have matrix X S D representing a size S encoding of æ, define pseudo self attention as
propose learning this contextualization in encoder-
XUk YW2
XU YW.)
PSA(X,Y) softmax((YW,)
where U. D' are new parameters which project encoder outputs into the decoder self attention space. Because attention inherently variable length, these additional inputs can be in- jected without changing the module and only act additively on the attention output. The full model is shown in Figure Ic
Compared to Context-Attn, the proposed approach only introduces new parameters in the self at- tention block, which we expect leads to only minimal interference. As the pretrained LM weights encode for generation capability, deviating less from this initialization may lead to better generation performance. We explore this quantitatively in Section 5
4 EXPERIMENTS AND RESULTS
Experiments consider four diverse tasks spanning input modalities, dataset sizes, and information about the target contained in the source. Tasks are chosen to emphasize long-form targets to probe the generation capabilities of the different models in a conditional setting. Perplexity is used to measure overall performance and diversity of output, combined with standard task-specific metrics.
For all tasks, we use GPT-2 small (117M parameters) (Radford et al. 2019) as the pretrained lan- guage model. GPT-2 small has 12 layers, 12 heads per layer, and model dimension of 768 units; the Context-Attn and Pseudo-Self models use the same architecture. For the Repr-Transfor avoid overfitting, we use 6/8/5 layers/heads/dim for the decoder (in addition the GPT-2 con- textual representation network). All experiments use the same 50k type BPE GPT-2 vocabulary.
model
4.1 PRELIMINARY: CLASS-CONDITIONAL GENERATION
We first consider control experiment with a minimal encoder model. We consider producing class- conditional samples, e.g., p(y x 0) and p(y x = 1), from the IMDb sentiment classification dataset (Maas et a..). similar to previous works for sentiment transfer (Shen et al.) al.| 2017; Zhao al.) 2018). We set to be sentiment bit (posittie/nnegative), and the movie review as the target We maintain the original IMDb 25k/25k train/test split, with 2.5k reviews of the original training split held out for validation, and truncate reviews to 400 BPE tokens during training. Model quality is evaluated by perplexity, and adherence to the source bit x is evaluated by the sentiment classifica- tion accuracy of an external classifier on generated reviews generated via random sampling with temperature of 0.7. For our external classifier we use fastTexx (Joulin al. 2016), which has accuracy of 90.1 %% on the IMDb test set.
in (Shnn et al. (2017). Reviews
Table shows results for the conditional models, GPT-2 without finetuning, and Simple Fusion Stahlberg et al. 2018). The GPT-2 model itself already shows greatly reduced PPL compared to bascline transformer. All pretraining methods further improve perplexity. The pseudo self atten- tion approach significantly outperforms the approaches in terms of class adherence. Despite being initialized as language model, the approach only sees a decrease of 0.4% classification accuracy compared to the randomly initialized model. contrast, Repr-Transforee decrease of 20.0% and 3.9%, respectively. We additionally report the results of Simple Fusion in
and Context-Attn see
TThis method is most successful when hand-selected task-dependent buffer words are inserted between and Ver well such "th:dr" for summarization.
4

---

Under review as a conference paper at ICLR 2020
Model
1 1 /
PPL.
Model
PPL | Cls Ace t
PointerGen+BU
41.22 / 18.68 38.34
Test set
90.1
ELMo+SHDEMB!
41.56 18.94 38.47
BERT+Two-Stagel
41.38 / 19.34 / 38.37
GPT-2
41.21
UniLM+ExtLoss!!
43.47 / 20.30 / 40.63
Simple Fusion
38.31
65.1
Transformer+Copy
39.94 17.73 / 37.09
8.21
Transformer
105.43
92.7
Repr-Tran Trans
37.09 / 13.77 / 33.99
13.58
Repr- -rans 39.69
72.7
Context-Attn
40.59 / 18.17 37.24
6.68
Context-Attn 40.74
88.8
Pseudo-Self
40.72 18.38 / 37,46
6,43
Pseudo-Self
34.80
92.3
Pseudo-Self-BU
41.62 / 18.66 38.46
6.43
Conditionl Generation IMDb movie reviews. Classification accuracy is measured by a sentiment classifier trained the IMDb training set. Bold indicates statistically signifi- cant best results p 0.05.
Table 1:
Abstractive summarization on CNN/DM. f indi- cates pretraining of the encoder side. PointerGen++BU from (Gehrmann et al., 2018), ELMo+SHDEMB from (Edunov et al. 2019). BERT Two--taag from (Zhang et al. 2019), E from (Dong et al. 2019). Bold indicates statistically significant best results among general models and encoder-agnostic models p S 0.05.
Table 2:
Table ! Compared to Pseudo-Sele ff gives a worse PPL and inferior classification accuracy. Given the weak results, focus comparisons between the deep models for the rest of the paper.
4.2 DOCUMENT SUMMARIZATION
Abstractive document summarization requires the model to produce long-form summary given full news article. For these experiments, mann et al. 2015). The dataset is comprised of 280k training examples of document-scall source news articles and corresponding 2-4 sentence target summaries. Summarization is a mature testbed with state-of-the-art models that use task- architecture modifications, so transfer learning methods need to be able to mesh well with these changes. We use the transformer version of the copy mechanism from Gehrmann et (2018) and employ bottom-up (BU) summarization attention pruning (Gehrmann et al. (018)). Generation conducted via beam-search with beam size with tri-gram blocking, consistent with the literature models (Edunov et al. 2019).
the non-anonymized CNN-Daily Mail dataset (Her-
Table shows the performance of the models tested with recent state-of-the-ar models for com- parison. Compared the baseline model without pretraining, Pseudo-Self improves ROUGE-1 by 0.78, ROUGE-2 by 0.65, ROUGE-L by 0.37, and reduced PPL by 20%. The Context-Attn approach nearly matches these results for this task, but the Repr- Transfrranfrrmr approach performs poorly.
We additionally experiment with the bottom-up up summarization attention pruning approach applied inference time as in Gehrmann et al. (2018). With this modification, Pseudo-Self outperforms all literature models in ROUGE-1 except the text-to-text pretraining of the source and target and is trained with an additional extractive loss. The performance of all of our models can potentially be further improved through pretrained encoder.
which uses joint
4.3 CONDITIONAL STORY GENERATION
Conditional story generation with the WritingPrompts dataset (Fan et al. 2018) requires the model to produce an on-topic story given short textual prompt. While summarization relies heavily on the encoder, this task gives more flexibility to the decoder. The dataset well supervised, containing 300k single sentence writing prompts (the source) and stories (the target). Following the preprocess- ing of Fan et al. (2018), we truncate the stories to 1000 tokens. Due to the story lengths, the total number of training tokens is on the order of 100 million, resulting in a large in-domain data setting.
To compare models, we compute two metrics: perplexity (PPL) and prompt ranking. Perplexity assess approximate quality and diversity, whereas prompt ranking measures the relevance of the story to the prompt. To calculate the prompt ranking, we use the procedure from Fan et al.] (2018): For each story in the test set, the likelihood is evaluated under the model for the "true" corresponding
5

---

Under review as a conference paper at ICLR 2020
Model
PPI. J
Rank Acc.
Model
CIDEr
B4
Transformer
30.58
80.6
Krause et al. (2017)
13.5
8.7
Repr-Trams 21.16
76.7
Chatterjee al. (2018)
20.9
9.4
Context-Attn >5000
9.3
Melas-Kyriazi et al. (2018)
22.7
8.7
Pseudo-Sell" 21.21
81.8
Transformer
19.9
8.0
Repr-Trans
19.3
7.2
Story generation on the Writing- Prompts dataset. Rank acc. refers the top- prompt ranking accuracy metric described in Section 4.3 (Experiments use the GPT2 BPE scheme, so PPL numbers are not directly com- parable to those reported (Fan et al. 2018)). Bold indicates statistically significant best sults at & 0.05.
Table 3:
Context-Attn Pseudo-Self
22.6
7.6
24.0
8.3
Table 4: Image paragraph captioning on Visual Genome, as measured by CIDEr and BLEU-4 (B4) scores. Bold indicates statistically signifi- cant best results at S 0.05.
prompt and 9 other randomly selected "fake" prompts from the test set. Then, the rank accuracy is the percentage of stories for which the model gave the highest likelihood to the correct prompt.
Table shows the results. Despite the large dataset size, the Repr-Transfomer and Pseudo-Self approaches still substantially reduce the PPL, suggesting that these models effectively make use of the GPT-2 LM. Pseudo-Sell sees only a 0.3% decrease in prompt ranking accuracy, while the Repr- Transforrnrr approach sees larger decrease. The Context-Attn model runs into optimization challenges and fails to learn in this setting. We hypothesize that this failure a result of introduc- ing randomly initialized attention block, which makes Context-Attn susceptible optimization challenges, but further work is needed to understand this more completely.
4.4 IMAGE PARAGRAPH CAPTIONING
Our final set of experiments consider image paragraph captioning using the Visual Genome dataset from Krause et al. (0077). Image captioning represents strong real-world use case for encoder- agnostic pretraining. Visual Genome, in particular, represents realistic setting with paragraph- sized captions (5-8 short sentences), which requires greater fluency than single sentence captions. Due to the difficulty of producing labeled paragraph captions Visual Genome, contains fewer than 20,000 mnage-paragraph pairs. As result, models trained from scratch on Visual Genome have been observed to have difficulty learning the structure of language.
We use the same convolutional encoder as Krause et al. (2017), without the final pooling layer: for each image the output of the encoder is tensor of size (36, 2048) extracted from ResNet. Note that this experiment the encoder and decoder are trained separately rather than end-to-end. Models are evaluated using the common CIDEr and BLEU-4 metrics.
Table shows the results on the captioning task, comparing the transfer learning methods with a non-pretraining baseline and models from the literature which the same loss functior Of the three pretraining approaches, Pseudo-Self and Context-Attn give the statistically significant best performance, and Fseudo-Self is the only model to improve both CIDEr and BLEU compared the Transformer baseline. Pseudo-Self additionally improves performance over the literature models in terms of CIDEr but gives slightly worse BLEU-4.
ANALYSIS AND DISCUSSION
Experimental trends Overall, Repr-Trans gives poor performance in three out the four tasks, underperforming transformer without pretraining on summarization and image captioning. Context-Attn gives stronger results than Repr-Trans, but shows significantly worse performance than Pseudo-Self class-conditional generation and slightly worse performance summarization
'Recent work shows it is possible to improve paragraph captioning models by incorporating sequence- level MMelas-Kyriazi et al. 2018)) and adversarial (Chatterjee & Schwing 2018) losses, but these loss function improvements orthogonal improvements the underlying model architecture.

---

Under review as a conference paper at ICLR 2020
0.010
KL(atthuncondd alttnconnd
2.5
Pseudo-Self Context-Attn
Pseudo-Self Context-Arth
2.0
1.5
RMSD
0.006
2.0
0.004
0.5
0.002
0.0
0.000
250
500
750
1000 1250 1500 1750 2000 Step
Layer
(a) Comparison of parameter changes in feed for- ward layers with different conditioning. Root me- dian squared deviation between feed forward param- eters. for the Pseudo Self and Context- Attn models. The Context-Attn approach requires larger devia- tion from the initialization to fit data.
(b) KL divergence between the self attention distri- butions of finetuned unconditional model the self attention distribution of Pseudo-Self and Conext- Attn conditional models. Averaged over heads and location in target, computed the end of training on the test target-side data.
Figure 2: Effect of introducing randomly initialized parameters.
and image captioning. Most critically, Context-Attn demonstrates susceptibility to optimization challenges. On all tasks Pseudo-Self gives the best performance or is tied for best.
Interference by added parameters In Section 3 hypothesized that pseudo self attention en- ables better use of the pretrained LM because the introduction of parameters in Pseudo-Sell inter- feres less with the original parameters than Context-Attn. To explore this quantitatively, we plot the root median squared deviation of parameters from their original values in the feed-forward layer for the class-conditional generation task (FigurePa). While both models start with the same parameters, the Context-Attn parameters change significantly more than Pseudo-Self during training.
The parameter values are only part of the story: parameters may change while the overall generative behavior stays the same. We probe this for image captioning by first finetuning the pretrained LM on target-side data, giving an unconditional model with an identical structure to Pseudo-Self and Context-Atth except without any additional randomly initialized parameters. In Figure 2b plot the KL divergence between the self attention distributions of the unconditional model and those of the conditional models at each layer. Both models have similar attention distributions the uncon- ditional model at the first layer (in Context-Atth this precedes the introduction of new parameters). Beyond the first layer, the KL for Context-Attn becomes over an order of magnitude larger than that of Pseudo-Self, suggesting that the introduction of the context attention block significantly per- turbs the overall behavior of the model. The additional perturbation is not associated with improved incorporation of the source information, as
that Pseudo-Self gives better performance.
Effect pretrained LM size There continuing trend to larger pretrained LMs. During the preparation of this manuscript, larger version of GPT-2 was made available with 345M parameters, increasing the model dimension to 1028, the number of attention heads to 16, and the number of lay- ers to 24. We retrained our model using this larger LM for caassconniiional generation, using the same training hyperparameters and re-tuning the generation tempera- ture (Table (5). The larger model improves PPL by 4.5 points while attaining similarly high classification accu- racy, This datapoint suggests that transfer learning effec- tiveness can continue to improve along with the quality of the pretrained model used.
Model
PPI. J Cls Acc
Pseudo-Self 117M
34.80
92.3
Pseudo-Self 345M
30.26
92.4
Table 5: IMDb conditional movie re- view generation results, comparing the larger 345M parameter GPT-2 model to the 1 117M parameter GPT model.
Low-data supervision Many of our tasks showed improvements even with medium-to-lle -t-gge train- ing sets. To study the effectiveness of the approach in low data regimes, we create small datasets by subsampling the IMDb dataset to sizes between 200 and 16k datapoints. We retrain our model using same hyperparameters and use datasize-dependent early stopping to prevent overfitting.

---

Under review as a conference paper at ICLR 2020
Model
Grammaticality
Non-redundancy
Consistency 85.1 3.4
Typicality
Combined
Test set
71.3 + 4.3
87.2 + 3.2
74.4 + 4.1
3.18 + 0.10
Transformer
55,4 +: 4.7
60.5 + 4.6
53.7 - 4.7
39.7 + 4.6
2.09 + 0.13
Repr-Trans
62.1 + 4.4
71.0 + 4.1
57.1 4.5
43.7 4.5
2.34 + 0.12
Pseudo-Sell 65.2 + 4.6
69.3 + 4.5
61.3 + 4.7
48.4 4.8
2.44 + 0.13
Table 6: Human evaluation of story generation quality. Participants were asked specific binary questions concerning the four criteria, the numbers for the four left categories represent percent- ages of approval. On the right, the methods are rated on a 4-point scale based on the combination of the four criteria. Uncertainties represent a 95% confidence interval, bold indicates statistically significant maxima for each category of the models under consideration.
To reduce variance and measure uncertainty, we repeat the process 8 times for each dataset size, calculating the PPL and classification accuracy. Results are shown in Figure 3 Note that a non pneetrera model has PPL of over 1000 when trained on 200 examples. The pretrained model starts with reasonable outputs (44.4 PPI. after 200 examples) and increases task accuracy steadily with more data. See Section Bjin the appendix for representative samples.
0.85
aCC.
0.75
36
0.70
Train datapoints (x1000)
Figure 3: IMDb. PPL shown in blue (left), classi- fication accuracy shown in orange (right). Error bars show an approximate 95% con- fidence interval.
Data efficiency analysis with
Human evaluation To assess the quality of genera- tions, we conducted a human evaluation based on the story generation task. Generation uses a temperature of 0.9 and a top-k value of 100. We ask participants on Amazon Mechanical Turk series of four yes/no ques- tions mapped to desirable linguistic properties outlined in Dang (2006): grammaticality, nonrredundancy, consistency, and typicality. 125 stories are evalu- ated for each model, and each story is evaluated by 5 unique workers. Scores are calculated for each property as the total percentage of positive responses. A combined score rates the model overall on scale from 0-4 based on the ggually-weighter combination of the four properties.
The results are shown in T all four categories, the Pseudo-Self and Repr-Transfore els show statistically significant performance gains compared to the baseline Transformer model. The Pseudo-Self model achieves a grammaticality score of only 6.1% less than the test set, indicat- ing strong grammaticality, likely more localized property, is well Iearned by the pretrained LM and effectively transferred to the conditional models. In contrast, all models score significantly worse than the test data in terms of consistency and typicality. This suggests that these higher-level prop- erties, while best transferred in the Pseudo-Self case, still represent a challenge for neural models.
mod-
6 CONCLUSION
We study encoder-agnostic approaches for adapting a pretrained language model general-purpost conditional language generation. Experiments spanning a range of diverse long-fomm conditional generation tasks demonstrate that pseudo self attention improves performance over strong encoder- agnostic pretraining baselines, and is the only consistently performant model. From practical perspective, the approach gives robust, sizable improvements over non-pretraining baseline while maintaining adherence to the source context. Furthermore, we demonstrate the data efficiency and qualitative properties of the approach.
Beyond empirical results, this study highlights the distinction between improving our ability to produce contextual representations of a source language and improving our capacity to generate text in target language. While they appear to be similar problems, they exhibit substantially different phenomenology. For example, the representation-base poor performance for NL.G. Future work can study this distinction further.
approach, which works well for NLU, gives
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Lalit R. Bahl, Frederick Jelinek, and Robert I.. Mercer. A Maximum Likelihood Approach to Con- tinuous Speech Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5(2)119-199,, 1983. ISSN 01628828. doi: 10. 10.1109//PMMI.1999 190.AM1999834777737
Moitreya Chatterjee and Alexander G Schwing. Diverse and coherent paragraph generation from images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 729-744, 2018.
Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. An embarrassingly sim- ple approach for transfer learning from pretrained language models. CoRR, abs/1902.10577, 2019.
Hoa Trang Dang. Overview of DUC 2006. Proceedings of HLT-NAACL, 2006.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv/8810008805, 2018. ISSN 0140-525X. doi: arXiv:181 1e:::8111000 1.111100002222 URL htt.:tp://aaxii..or//aa///////// 04805
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Ssiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. ArXiv, Mww1190503197, 2019.
Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained Language Model Representations for Language Generation. NAACL-HLT, 2019.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889 898, 2018 ISSN 1473-3250. doi: 10.1016/j 1..1016f..aarres.0111 201..re0..11....0... //arxiv.org/ abs
URLhttp:
abtp///aa.aaa.e.aeaa 1800001010000 04833
Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. Bottom-Up Abstractive Summariza- tion. Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, 2018. /ttl:ttt:///xxi..ccoo wwtp:/iwwwrg.aaaa.o...a /atpas//888..11 10792
Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf. Large-scale transfer learning for natural language generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6053-6058, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://wwer aclweb. wewweb.org/anttee..o wwwpr///nnnhlgg///1e/0/
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On Using Monolingual Corpora in Neu- ral Machine Translation. arXiv preprint 1rXiv:1503.03553 2015. ISSN 1560-7917. doi: org/ abs 5503.0.0 03535
(00288771560077177752214444 20711.. URL
Karl Moritz Hermann, Tomás Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching Machines to Read and Comprehend. Proceedings of NIPS, pp. 1-14, 2015. URL/ttpp://sxxiv wttpa///iwwii.oooo/o///o/oooooat /106.1166... 03340
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffiient transfer learning for nlp. CoRR, abs/1902.000511 2019.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifi- cation. Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), 2018. ISSN 23330384. doi: arXiv:1801.061467 URL
fictp://axxiv..
abs 801 .66146
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Efficient Text Classification. arXiv preprint
2016. URLbtted /arxivv.......///// org/ abs/
607. 01759

---

Under review as a conference paper at ICLR 2020
Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical Phrase-Based Translation. Proceed- ings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology Volume 1, (June):48-55 -444,,555,,-- 2003.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Patterm Recognition (CVPR), 2017.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint vrXiv:/901.0720 2019.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in Translation: Contextualized Word Vectors. 31st Conference on Neural Information Processing Systems, 2017.
URL httt::/aaaxxii. org. abs/1708/ 00107
Luke MyriaiK Alexander Rush, and George Han. Training for diversity in image paragraph captioning. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 757-761, 2018.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Proceedings of NAACL-HILT, 2018. doi: 10.18633/v//N8-.. 1202
URL
05365
Alec Radford and Tim Salimans. GPT: Improving Language Understanding by Generative Pre- Training. arXiv, pp. 1-12, 2018 ISSN 1095-8290. doi: 10. 10.10993/ob/mcc031. URL https:
//33-us-west-22.mmee batguuuge-unnuue....ssss///aaaee
wwwerr/ataiiiinnggiiiaeeaao pdf
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
Models are Unsupervised Multitask Learners. 2019.
Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised Pretraining for Sequence to Sequence Learning. Proceedings of EMNLP, 2017.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Proceedings the / International Conference Neural Information Processing Systems, NIPS' 17, pp. 6844,, USA, 2017. Curran Associates Inc. ISBN 978-1- 5108-6096-4. URL http: //tt:://dl... acm. org/ citation...... wtwpidd http://t........o.aa113333222222 3295427
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and T. M. Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training Seq2seq models together with language models. Proceedings of the Annual Conference of the Interna- tional Speech Communication Association, INTERSPEECH, 1818-391 2018. ISSN 19909772. doi: 10.2447/hnerrpeehh221111 1039
Felix Stahlberg, James Cross, and Veselin Stoyanov. Simple Fusion: Return of the Language Model. Proceedings of the Third Conference on Machine Translation: Research Papers, 1:204-211, 2018. URL / //tp///aavv///////////ee///aeal/ org/ abs 1809.00125
abs
Chenguang Wang, Mu Li, and Alexander J. Smola. Language models with transformers. CoRR, (bs//904.00448, 2019.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. The Neural Noisy Channel. arXiv preprint arXiv:161 fedi::611.02555 02554,, pp. 1-13, 2017.
Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang. Ming Gong, and Ming Zhou. Pretraining- Based Natural Language Generation for Text Summarization. arXiv preprint arXiv: 1902.09923 2019. htt.http://aaa wwwp:i.wwoo...//ott http://aabb999...0../ 902 09243]
10

---

Under review as a conference paper at ICLR 2020
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoencoders. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learming, volume 80 of Proceedings of Machine Learning Research, pp. 5902-5911,, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: \/proceed ngs. ml www.s..r...r..p.mmmmmmmmm..m..lm / v80 zhaol 8b. html

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL DETAILS
Approximate hyperparameter settings were taken from Gehrmann et al. (2018), followed by a coarse hyperparameter tuning for each dataset. Most parameters were held constant between the different models, though in some cases to enable as fair comparison as possible individual parameters were separately optimized for each model. For the complete of hyperparameters for each model, see com/ anon 72 34/ enwoder- wttp///oraaooootoooooooa--------
the details at
ion
For all models Dropout and early stopping are used for regularization. In addition, in imitial exper- iments we found two optimization details to help generalization across all models: discriminative finetuning (Howard Ruder 2018) and using lower learning rate for the decoder than the en- coder. In discriminative finetuning. the learning rate of each layer decreases exponentially from the top transformer layer to the bottom transformer layer. Using a smaller learning rate for the decoder ensures that the decoder does not initially significantly change to accommodate the uninformative information from the randomly initialized encoder.
B QUALITATIVE EXAMPLES
Representative samples for the movic review dataset are shown Table 7 The No-Pretraining model is the transformer from Table and and the number in the left column indicates the number of supervised examples in the training dataset. Samples are generated via random sampling with a tem- perature of 0.75. Without pretraining, the model makes many coherence mistakes. The Pseudo-Self 22K makes no grammatical mistakes and follows a single train of thought, although it somewhat generic.
The distinction between the models is further exaggerated when only 1.8k supervised examples are given. The baseline model trained on only 1.8k datapoints leads very incoherent generated text. In contrast, the Pseudo-Attention model shows sigmificantly improved grammar and sentence structure. Despite handful mistakes, the review follows a consistent description of movie over multiple sentences. Given the poor performance of the baseline model, these properties must have been transferred from the original unconditional LM. These samples were selected be representative of the broader set for the indicated models.
12

---

Under review as a conference paper at ICLR 2020
Sentiment:
Posilive
No pretraining 1.8k:
"Set's that I liked this movic. have seen remember the original movic is one of the music that it is great movic. I've seen this film and one of the whole movie is like this movie, It is so bad, I watched the top of this movie. would see the movie This movie, it's TV main movie is about the plot, relaxing. liked this movie takes is few times,
bad, have was awesome. I'm a DVD. The critics in the first time know it were lovely plot. You could watch I'm seen the acting, and DVD.
don't It watch this. I've seen what you really love the film. The film
pretraining 22k:
was enthralled the performance absolutely fantastic in the film, and thought it was real ride. It about family living in small town San Francisco, where couple lives in Mexico, where they are working at home with the family. In the end, they are in the family decide the family will have their fathers through endless periods, coffce, disillusionment, and finally, story. There are some delightful performances (particularly the two leads) and the actors are well known. I really can't help been better comedy.
Hartnett at the time of this film.
time
was shot, thought, was
escape, there's is
feel that film could
exception
L'Amelie.
Pseudo-Self 1.8k;
fell in love with this film in 1985, It's quintessential short film that explores the everyday lives of the buman condition. named Domald (Hushand George). He buys home and cuptures great deal of
The main charucter of the movie information about the businesemen who live and work his neighborhood. This film complete departure from the norm what a typical American would expect than likable, and his best friend Kelly (Prect Bharara). The film is full of hamor, and a great deal of heart and soul.
successful bosinessman with short fuse. His stereotypical
are less
Pscudo-Self 22k:
When saw the preview of this film. thought it was going be horrible movic. I was wrong. The film has some of the funniest and most escapist scenes I've seen in a long time. The acting is superh. The story is decent, but the direction and editing may have been bit harsh times. The director and achieved great balance of comedy and drama. I've some bad films, but this one is one of the ones I've seea that is really good. loved the acting and the pace. The two leads were compelling. The only problem with the was that funny, good movie. It's efficient. I give it 7/10.
bored with The ending
long. but it's
Table 7: Example generations from models trained on the movie review generation task. all cases the indicated sentiment was positive. The number in the left column is the number of training examples (22k is the full dataset).
13


Reviews:
[{"review_id": "SJg1WdHqqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compares a few encoder agnostic methods for using pretrained decoders in text generation tax. The author compared a few intuitive ways of doing this, and presents results showing that that pseudo-self attention does the best.\n\nHowever, I think the results has some strange points that needs further investigation. Going from repr-transfomer to context-attention to pseudo-self, there is an increasing amount of parameters initialized by pretraining. However, both of the first two methods often perform worse than the baseline transformer without pretraining. So should more things be initialized with pre-training or less? It would be good to verify that this is not due to under-training. \n\nExcept paragraph captioning, the results on other tasks are not better than prior results, which do not use pretraining. The baseline transformer is also usually worse than prior results. The human evaluation shows that the proposed method do better on story generation, but this one is essentially text to text. What is missing is how this compares with even more pretraining, say GPT-2, without any fine tuning. \n\nTransferring gains of pretraining to generation tasks is clearly a promising direction, and the bar for success in this area need to be outperforming the best previous methods that do not use pretraining.  There is no comparison with previous text 2 text methods that use pretraining.  If the proposed methods are truely encoder agnostic, then they should perform reasonably on text-to-text as well. I think some MT experiments would be good since the evaluations are more competitive and reliable. Perhaps using some language pairs that do not have sufficient training data.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyeNNy_Ttr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new architecture to train decoder models on language generation using a pre-trained encoder (such as BERT or GPT-2). They introduce a novel block called `````\"pseudo self-attention\" that allow injecting the input for conditional generation in the self-attention layer (i.e. softmax of YW_q (XU_k | YW_k)^T (XU_v | YW_v) instead of softmax(YW_q(YW_k)^T)YW_v). They extensively evaluate their approach on a large set of tasks showing improvements across all of them (which includes class-conditional generation, summarization, story generation and paragraph generation). They also provide interesting ablation studies.\n\nThis paper proposes a simple architectural block to try and translate the success of large pre-trained encoders on discriminative tasks to the generative setting. The idea seems well-motivated and the paper is well-written and easy to follow. The experimental section is very thorough and show large improvements on a variety of task---I particularly appreciate that they experimented with conditional inputs of different nature (class value, image, different languages etc...) to show the effectiveness of their method.\n\nOverall, while the idea is quite simple, the experiments speak for themselves and this could prove to be a useful `layer' to use on large pre-trained language models. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygEgCVoYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple yet effective method to adapt large-scale pre-trained language models, which have been shown to substantially improve performance on broadly classification-based NLU tasks, to NLG. The approach is explored in the encoder-agnostic {X}-to-text setup, where the source encoding {X} could represent arbitrary modalities, such as text or images.\n\nMore concretely, the paper leverages a pre-trained, large-scale language model (in this case a GPT-2), and examines how to best cast such unconditional language model into a decoder that generates text conditional on the source information {X}. As self-attention inherently works with sequences of any length, the proposed pseudo self-attention approach simply injects the encoder representation as additional conditioning context (using some additional projection matrices that are learned from scratch) into the pre-trained self-attention layers of the decoder. Extensive experiments and analysis on four diverse tasks demonstrate that pseudo self-attention generally outperforms two other ways of pre-training the decoder, improves NLG data efficiency, and produces texts that are judged more favourably by human evaluators.\n\nOverall, this paper presents a simple, general, and effective method for adapting large-scale pre-trained language models to conditional text generation. Based on the pros and cons that I have listed below, I am giving a rating of \"Accept\". I hope that some of my concerns will be addressed in the authors' response.\n\nPros:\n1. The paper is well-written and the methodology is explained very clearly. Figure 1 is particularly helpful in illustrating the differences between pseudo self-attention and the baselines.\n\n2. The paper addresses a very important problem, and helps make sure that the advances that have been made in language modelling (which can leverage large amounts of unlabelled data), would transfer well to conditional text generation tasks, which hold immediate practical value yet often require expensive annotations.\n\n3. The approach is simple and easy-to-implement, but has been shown to be effective across a broad range of problems, multiple modalities, and various evaluation metric. \n\n4. The paper features extensive reference to relevant prior work, and clearly highlights the key similarities and differences with prior approaches. \n\nCons:\n1. It is still unclear how using language model pre-training affects adequacy (as opposed to fluency). The paper shows that using pseudo self-attention results in a decoder that diverges less from its language model initialisation. One potential risk is that the decoder may prefer fluent, \"safe\" outputs (which is arguably what a language model would prefer since it is an unconditional model) that are nevertheless less faithful to the source information. Since none of the evaluation metric specifically assesses for adequacy on its own, it would be good to isolate the effect of pseudo self-attention on adequacy, and compare it with the baselines, in addition to a Transformer trained from scratch on each downstream task. How to measure adequacy is naturally still an open question, but there are a few things that can be done (e.g. recall of salient information, reverse perplexity to see how much of the source information can be \"reconstructed\" given the predicted target text, etc.).\n\n2. It would be interesting to further examine the interaction between encoder pre-training and the decoder pre-training that is explored in this work. Another interesting experiment to run is whether end-to-end training (including fine-tuning the encoder) would help, since prior work has shown the benefits of end-to-end learning (at least when large amounts of data are available).\n\nQuestions:\n1. Why is the context-attn model performance not included in Table 6? Is it because of the optimisation issue associated with that model in Table 3?\n\n2. In page 7, it is mentioned that \"Both models have similar attention distributions ... at the first layer, which precedes the introduction of new parameters in both models\". Does the first layer here refer to the token + position embedding layer?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to use a pretrained language model for language generation with arbitrary conditional input (images, text). The main idea, which is called pseudo self-attention, is to incorporate the conditioning input as a pseudo history to a pretrained transformer. Experiments on class-conditional generation, summarization, story generation, and image captioning show the benefit of the proposed approach.

While I think that the proposed approach makes sense, especially for generation from multiple modalities, it would be useful to see the following comparison in the case of conditional generation from one modality (i.e., text-text such as in summarization and story generation). How does the proposed approach compare to a method that simply concatenates these input and output? In Figure 1(c), this would be having the encoder part be pretrained as well, as opposed to randomly initialized, which is possible if the input is also text. I believe this is what R2 is suggesting as well when they mentioned a GPT-2 style model, and I agree this is an important baseline.

This is a borderline paper. However, due to space constraint and the above issues, I recommend to reject the paper.
Decision: Reject