Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper tackles the problem of controlled text generation by converting it into a conditional text generation similar to (Keskar et al.19).  It proposes an architectural modification to the transformer LM used in GPT2,  Specifically, a CoCon layer is added as a separate transformer block in the middle allowing self-attention to be performed between the textual context representations LM_α(c) and the generations LM_α(x_{:t-1}) this is performed through concatenating the key and value matrices with the keys and values of the encoded textual context. Authors provide 4 different losses to train this additional layer. 

Pros: 
-  The proposed method has an advantage over (Keskar et al.19) by 
1) avoiding rigid control tokens and replacing them by textual context. 
2) avoiding to retrain the whole LM architecture and replacing this by retraining single transformer block instead 
3) allowing several control contexts at once (this is an interesting aspect of the proposed solution) 

Cons: 
- The proposed solution to controlled NLG is simple yet not inspiring nor revolutionary, simplicity could have been an advantage here ofc, if it tackled the problem providing a concrete method to control NLG models. however, this is not the case here (See next)

- Conditioning NLG models on textual contexts to influence the generated text is a straight forward solution and makes sense in terms of flexibility, and in fact, has been used before [1] to enhance faithfulness in QG tasks. On the other hand, conditional text generation as a solution to controlled NLG might be effective for influencing topic or sentiment, however, this formulation is not suitable for other types of control where textual contexts are hard to formulate, such as "removing" toxicity, controlling the length.

- There might be an issue with the Adversarial loss, being non-differentiable (see Q1 below)

- Evaluation could have been more thorough, specifically, when the proposed method has a superior topic and sentiment relevance in table 3 and table 4 this comes on the cost of perplexity. Enhancing topic relevance of PPLMand CTRL could be achieved by reducing the temperature during decoding, on the expense of Perplexity as well. This is similar to the Quality/Diversity tradeoff showed in [2]. While this has been an issue in previous work as well, it would have been better to fix this issue and provide as better evaluation, a good method to evaluate this could have been plotting perplexity vs control satisfaction rate under a temperature sweep. 

- There are missing details on how the textual contexts are selected during inference time. In most of the cases, they're handcrafted topic names or short sentences "is perfect". This makes the proposed solution very similar to control tokens by Keskar et al. 19. One advantage of using textual control tokens is handling unseen "content inputs" at test time. This should have been evaluated to show the superiority of this solution.  

Questions: 

Q1: This is a critical one, If I got this part correctly, the adversarial loss eq 18, requires to sample y from the LM, this is non-differentiable. if that is the case, did you follow any necessary steps (e.g. RL or continuous approx) to overcome this non-differentiability? 


refs: 

1- Zero-Shot Question Generation from Knowledge Graphs for Unseen Predicates and Entity Types, NAACL2018

2- LANGUAGE GANS FALLING SHORT ICLR2020 

minor: 
- Is that a typo in figure 1? The cocon layer output representation should be h_1 , h_{t-2} should be h' _ 1 , h' _ {t-2}   
review_title: An architectural modification allowing integrating of textual contexts, converting the problem of controlled text generation into conditional text Generation. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY:

The paper proposes a self-supervised technique for controlling the productions of a Transformer-based pretrained generator. The technique consists in augmenting the architecture of the pretrained model with a special "content-conditioner" (CoCon) block which is able to exploit a contextual condition. 
At training time, this contextual condition is obtained, in a self-supervised way, by removing a textual portion from a training text and using this portion as the contextual condition, and then the parameters of the CoCon component learn how to approximately recover the missing portion based on this context (the portion itself) and on the prefix text preceding the removal.
At test time, a textual condition is provided as context, and the trained model produces a text "imbued" (authors' terminology = influenced) by this condition.

POSITIVES:

While self-supervised learning has been employed for certain text generation tasks, such as summarization, I am not aware of previous works directly concerned with self-supervision for controlled open-ended text generation. This appears to be a very worthwhile direction to pursue.

ISSUES and QUESTIONS:

*Clarity*. The main idea is actually pretty simple but the reader has to wait until the end of page 4 (Self Reconstruction Loss) to be able to understand it (true: it was exposed in the intro, but in a way difficult to understand on a first reading), and is a bit drowned in a dense mass of mathematical notations that do not help.  Some parts of the formal description are quite difficult to follow, for instance the section on "Cycle Reconstruction Loss". Also, in Fig. 1, the reader does not immediately see that (*IF* I understand correctly) the hidden states $h_{t-1},...,h_{l-1}$ are masked, which does not help in understanding an already dense formal description. Perhaps most serious: the central objective of a text "imbued" (i.e. "influenced") by a conditioning text is left pretty informal.

*Related work and Alternatives to the CoCon block*. Adapter Layers (https://www.aclweb.org/anthology/D19-1165/) are a technique for adapting pretrained models which does not require retraining the entire model; they are therefore similar in spirit to the CoCon block, and *should* be cited, with differences highlighted. (More minor: it might (?) also be worthwhile to mention a different option: using an encoder-decoder model (similar to NMT) where the conditioning context would just be the "source" and the generated text the "target", directly providing an attention-driven mechanism --- however the issue of retraining the whole model would then need to be addressed)).

*Complexity of the overall model, intuition about the different losses, hyperparameters* The self-reconstruction loss, by itself, appears to be problematic. Indeed, a model trained only on this loss might just learn to *copy* the conditioning text, thus destroying fluency and generalization. This should be explicitely discussed (instead of leaving the point more or less implicit in the second paragraph of section 4.4: "... limit where the texy appears incomprehensible"). Therefore the need to interpolate this loss with other losses (section 3.1). While you provide some ablation experiments, you do not much discuss the importance of these different losses. In particular, you should give more intuition/motivation for the Cycle Reconstruction Loss, which I did not really understand. The overall model involves quite a few hyperparameters ($\lambda$'s in equation (20), $\tau_{content}$.

*Results* The results are difficult to interpret, in particular due to the not very clearly formalized control objective (do you want the generated text to contain literal parts of the conditioning text (apparently not), or to have some semantic simlarity with the conditioning text (apparently yes, but you do not explicitly mention or define semantic similarity)? It is difficult for the reader to really assess the quality of the results. Here, a human evaluation with a clear evaluation protocol would really be useful.


Overall, an interesting and important objective: self-supervision of controlled text generation, with some nice ideas. But serous flaws in presentation and experimental validation.

------- 
**Written after rebuttal:**
Thank you for the substantial improvements to the paper in terms of clarity (in particular Figure 2 is helpful) and additional experiments/human evaluations. Despite some underlying questions (from me and other reviewers) that remain, I have updated my score and am now leaning towards acceptance.
review_title: Review: CoCon: A Self-Supervised Approach for Controlled Text Generation

final_score: rating: 7: Good paper, accept
review_content: The paper proposed a way to control the content output of a DNN-based language model (GPT-2 in the experiment, but not limited to it). It places an layer (CoCon) that can take an arbitrary phrase as the hint after generating the embedding but before generating the text. Experiments showed that the control is effective at directing the generated text. Examples confirmed that too. 

Quality:

The design of the CoCon layer is intuitive. The authors clearly explained the rationale behind the design of the layer. Experiments are based on strong baseline (GPT-2, PPLM and CTRL), and show clear advantage of the model. 

Clarity:

The writing is clear and easy to follow. I have some minor comments but believe they are easily fixable.

Originality:

CoCon has clear but incremental difference than PPLM and CTRL.

Significance: 

Controlling the generation of LM is not a novel task. This is an improvement on an existing problem with several solutions. Moderate originality.

My questions and suggestions:

1) Page 2, core  contribution, item 3: what does "competitively" mean here?

2) Page 2, Related Work, first paragraph. "our approach aims to control the generation at a content level, beyond high-level
text attributes." should it be "at the content level"?

3) Page 5. cycle reconstruction loss. It would be helpful to give an example, otherwise it's a bit hard to see how cycle recon could have helped. 

Same line: "unlikely co-occurs" -> "unlikely to co-occur" ?

4) Page 6 , 2nd paragraph "self-supervised and requires no manually labeled data fully" is duplicated, can be removed.

5) Overall speaking, the choice of content input for all examples are weird. Why do we use partial phrases without a clear meaning or subject as the content hint?
 

review_title: The paper proposed a novel way of controlling language model output.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Thanks to the authors for the response. The addition of a human study and CoCon+ has made the paper substantially stronger, as it resolves most of my concerns. The authors provided plausible explanations for the remaining questions. The paper should now be considered a clear accept.**

The paper proposes a method for controlled text generation with pretrained (unconditional) language models. The method trains a relatively small network (CoCon) that is injected between any two successive layers of a pretrained Transformer language model. Given a prefix, CoCon is trained to output an input to the next layer such that the remainder of the text is generated at the output layer. CoCon is a function not only of the prefix but also of some desired 'content' sequence, which allows to control the content of the output text at inference time. Several auxiliary loss terms are employed to improve generalization. The model is evaluated on its ability to generate output with desired content, to generate text of a desired topic, and to generate text of specific sentiment.

Compared to previously proposed Plug and Play Language Models, the novelty of the CoCon method lies in its ability to condition on entire input sequences (instead of only bag-of-words), and the fact that it does not require style labels, which are both important properties. The paper is well written, the method is intuitive and all components are well motivated. The experimental section could be more thorough. For example, several aspects of the model are only evaluated qualitatively, and I don't find the examples very convincing. Moreover, some of the results are difficult to interprete or non-conclusive. The paper could benefit from a human evaluation.

In the papers current state I would already slightly lean towards acceptance because the method itself will be useful to the community, and some of the results are promising. I am willing to strengthen my recommendation if my questions below are answered positively.

* In the _CoCon Setup_ you report to split $x$ into $x^a$ and $x^b$ somewhere between the 8th and 12th BPE position. Why is this sufficient? Wouldn't we expect the model to perform poorly on prefixes that are not between 8 and 12 BPE tokens long?

* Table 2 suggests that CoCon without the adversarial loss achieves the best performance, drastically improving on content similarity while retaining comparable text quality and diversity. This makes me wonder why the adversarial term was introduced in the first place, and why it is apparently used in the other two experiments.

* Why is the perplexity of CoCon (and PPLM) consistently lower than the perplexity of the baseline LM GPT-2? Shouldn't we expect a trade-off between controllability and text quality? In the PPLM paper, the perplexity of the baseline is consistently (slightly) lower than that of PPLM.

* Why does training on (human-written) Webtext instead of (machine-written) outputs of GPT-2 _decrease_ the text quality? Wouldn't we expect the opposite?

* The above three questions lead me to believe that using perplexity on GPT-1 might not be a suitable metric to judge text quality in these scenarios. Could you please provide more arguments why you believe a human study is not needed here?

Suggestions:

The model extensions listed under 4.4 are very interesting. The paper would be even stronger if you had quantitative experiments for these, as the examples that are given are not very convincing. For example, couldn't you apply your model to (gradual) sentiment transfer by conditioning CoCon on the input text as well as the target sentiment ("is perfect"), weighted by $\tau_{content}$? Even if the results were not very good compared to the state-of-the-art in sentiment transfer, such an experiment could show off the versatility of CoCon compared to PPLM. Moreover, if PPLM and CoCon complement each other as you claim, why not add another row "CoCon + PPLM" to Table 3, 4, and 5?

Minor suggestions:
* In the results section of 4.1, you say that L_adv marginally reduces CoCon's perplexity, but the table shows that _removing_ L_adv reduces it.
review_title: Useful method with promising results, but evaluation could be better

Meta-review:  
The paper aims at controllable generation by introducing an additional "content-conditioner" block in the Transformer models. The paper further provides 4 different variants of a pre-training task to train the content-conditioner model. 

While the proposed approach seems an incremental contribution over CTRL and PPLM, certain reviews praised the approach being novel while keeping the architecture changes minimal. Overall, reviews indicate that the overall proposed method of fine-grained controlled generation with self-supervision is valuable, and empirical results support its effectiveness. 

All reviewers initially raised concerns regarding clarity and lack of human evaluation. However, clarity issues seem to be resolved through author/reviewer discussions and the updated revision.

R3 had important concerns regarding topic and sentiment relevance evaluations.  
While the reviewer remains unconvinced after discussions with authors, after carefully reading the revised paper and discussions, I feel that the authors tried to address this point fairly  through their additional experiments and also edited their contribution statement accordingly.

Overall, at least two reviewers sounded very excited about this work and other than R3's concerns, the general sentiment about this work was positive. Therefore, I recommend weak accept.  

There are still some writing issues that I strongly encourage authors to carefully address in the future versions. Quoting from reviewer discussions:

> Differentiability of the adversarial loss. Authors just added one statement saying " Through continuous approximation.." without any more details are given, which continuous approx was used (Gumbel softmax?) and how they overcame the problem of its training instability. 

> Table 6, can be misleading, authors bold the results when cocon+ is performing better than baselines (mostly in content similarity) but not the other way around topic/sentiment accuracy. The latter is arguably more important.
Decision: Accept (Poster)