Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper compares a few encoder agnostic methods for using pretrained decoders in text generation tax. The author compared a few intuitive ways of doing this, and presents results showing that that pseudo-self attention does the best.

However, I think the results has some strange points that needs further investigation. Going from repr-transfomer to context-attention to pseudo-self, there is an increasing amount of parameters initialized by pretraining. However, both of the first two methods often perform worse than the baseline transformer without pretraining. So should more things be initialized with pre-training or less? It would be good to verify that this is not due to under-training. 

Except paragraph captioning, the results on other tasks are not better than prior results, which do not use pretraining. The baseline transformer is also usually worse than prior results. The human evaluation shows that the proposed method do better on story generation, but this one is essentially text to text. What is missing is how this compares with even more pretraining, say GPT-2, without any fine tuning. 

Transferring gains of pretraining to generation tasks is clearly a promising direction, and the bar for success in this area need to be outperforming the best previous methods that do not use pretraining.  There is no comparison with previous text 2 text methods that use pretraining.  If the proposed methods are truely encoder agnostic, then they should perform reasonably on text-to-text as well. I think some MT experiments would be good since the evaluations are more competitive and reliable. Perhaps using some language pairs that do not have sufficient training data.  
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes a new architecture to train decoder models on language generation using a pre-trained encoder (such as BERT or GPT-2). They introduce a novel block called `````"pseudo self-attention" that allow injecting the input for conditional generation in the self-attention layer (i.e. softmax of YW_q (XU_k | YW_k)^T (XU_v | YW_v) instead of softmax(YW_q(YW_k)^T)YW_v). They extensively evaluate their approach on a large set of tasks showing improvements across all of them (which includes class-conditional generation, summarization, story generation and paragraph generation). They also provide interesting ablation studies.

This paper proposes a simple architectural block to try and translate the success of large pre-trained encoders on discriminative tasks to the generative setting. The idea seems well-motivated and the paper is well-written and easy to follow. The experimental section is very thorough and show large improvements on a variety of task---I particularly appreciate that they experimented with conditional inputs of different nature (class value, image, different languages etc...) to show the effectiveness of their method.

Overall, while the idea is quite simple, the experiments speak for themselves and this could prove to be a useful `layer' to use on large pre-trained language models. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes a simple yet effective method to adapt large-scale pre-trained language models, which have been shown to substantially improve performance on broadly classification-based NLU tasks, to NLG. The approach is explored in the encoder-agnostic {X}-to-text setup, where the source encoding {X} could represent arbitrary modalities, such as text or images.

More concretely, the paper leverages a pre-trained, large-scale language model (in this case a GPT-2), and examines how to best cast such unconditional language model into a decoder that generates text conditional on the source information {X}. As self-attention inherently works with sequences of any length, the proposed pseudo self-attention approach simply injects the encoder representation as additional conditioning context (using some additional projection matrices that are learned from scratch) into the pre-trained self-attention layers of the decoder. Extensive experiments and analysis on four diverse tasks demonstrate that pseudo self-attention generally outperforms two other ways of pre-training the decoder, improves NLG data efficiency, and produces texts that are judged more favourably by human evaluators.

Overall, this paper presents a simple, general, and effective method for adapting large-scale pre-trained language models to conditional text generation. Based on the pros and cons that I have listed below, I am giving a rating of "Accept". I hope that some of my concerns will be addressed in the authors' response.

Pros:
1. The paper is well-written and the methodology is explained very clearly. Figure 1 is particularly helpful in illustrating the differences between pseudo self-attention and the baselines.

2. The paper addresses a very important problem, and helps make sure that the advances that have been made in language modelling (which can leverage large amounts of unlabelled data), would transfer well to conditional text generation tasks, which hold immediate practical value yet often require expensive annotations.

3. The approach is simple and easy-to-implement, but has been shown to be effective across a broad range of problems, multiple modalities, and various evaluation metric. 

4. The paper features extensive reference to relevant prior work, and clearly highlights the key similarities and differences with prior approaches. 

Cons:
1. It is still unclear how using language model pre-training affects adequacy (as opposed to fluency). The paper shows that using pseudo self-attention results in a decoder that diverges less from its language model initialisation. One potential risk is that the decoder may prefer fluent, "safe" outputs (which is arguably what a language model would prefer since it is an unconditional model) that are nevertheless less faithful to the source information. Since none of the evaluation metric specifically assesses for adequacy on its own, it would be good to isolate the effect of pseudo self-attention on adequacy, and compare it with the baselines, in addition to a Transformer trained from scratch on each downstream task. How to measure adequacy is naturally still an open question, but there are a few things that can be done (e.g. recall of salient information, reverse perplexity to see how much of the source information can be "reconstructed" given the predicted target text, etc.).

2. It would be interesting to further examine the interaction between encoder pre-training and the decoder pre-training that is explored in this work. Another interesting experiment to run is whether end-to-end training (including fine-tuning the encoder) would help, since prior work has shown the benefits of end-to-end learning (at least when large amounts of data are available).

Questions:
1. Why is the context-attn model performance not included in Table 6? Is it because of the optimisation issue associated with that model in Table 3?

2. In page 7, it is mentioned that "Both models have similar attention distributions ... at the first layer, which precedes the introduction of new parameters in both models". Does the first layer here refer to the token + position embedding layer?
review_title: Official Blind Review #1

Meta-review: This paper proposes a method to use a pretrained language model for language generation with arbitrary conditional input (images, text). The main idea, which is called pseudo self-attention, is to incorporate the conditioning input as a pseudo history to a pretrained transformer. Experiments on class-conditional generation, summarization, story generation, and image captioning show the benefit of the proposed approach.

While I think that the proposed approach makes sense, especially for generation from multiple modalities, it would be useful to see the following comparison in the case of conditional generation from one modality (i.e., text-text such as in summarization and story generation). How does the proposed approach compare to a method that simply concatenates these input and output? In Figure 1(c), this would be having the encoder part be pretrained as well, as opposed to randomly initialized, which is possible if the input is also text. I believe this is what R2 is suggesting as well when they mentioned a GPT-2 style model, and I agree this is an important baseline.

This is a borderline paper. However, due to space constraint and the above issues, I recommend to reject the paper.
Decision: Reject