CoCon: A Self-Supervised Approach for Controlled Text Generation
Authors: Alvin Chan
Yew-Soon Ong
Bill Pung
Aston Zhang
Jie Fu
Venue: ICLR 2021 Poster (2021)
Keywords: Language modeling
text generation
controlled generation
self-supervised learning

Abstract: Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM's output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.

Published as a conference paper at ICLR 2021
COCON: A SELF-SUPERVISED APPROACH FOR CONTROLLED TEXT GENERATION
Alvin Chan'; Yew-Soon Ong', Bill Pung', Aston Zhang?, Jie Fu
Nanyang Technological University, AAmazon AI, Miila, Polytechnique Montreal
ABSTRACT
Pretrained Transformer- bassed language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. (CoCon) to control an LM's output text
Here, we propose with content input, at fine-gra fineggriiedd level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that withheld from the LM. Through experi- ments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes zero-shot manner
INTRODUCTION
Transformer-basee Vaswani et al. 2017 Tay et al. 2020) pretrained language models (LMs) have led a wave of new advances in natural language processing tasks as a means to extract contextualized word embeddings (Devlin et al. 2018 Dai et al. [2019)] Yang et al.] (019) and as text generators (Radford et al., 2019: Brown et al. 2020). These LMS are trained on huge amounts of text cor- pora to predict next tokens through log-likelihood objective. Given its remarkably fluent text generation, there is growing interest in controlling output texts of such LMs (Keskar et al. 2019 Dathathri et al. (2019). Approaches like training modified LM from scratch incorporate target text attributes (Keskar et al.) 2019) can be expensive while finetuning pretrained LMs for specific attributes (Ziegler al. 2019) limits the scope of text control. Without changing the architecture or weights of pretrained LMs, one promising approach (PPLM) (Dathathri et al., 2019) controls gener- ated text through attribute models. Though effective in controlling high-level text attributes such as topic and sentiment, the same target attribute may generate text samples with vastly different con- tent at the word- and phrase-levels, leaving a gap for more fine-grained control over the content of LM-generated texts.
(CoCon) as an approach to narrow this gap by guiding pre- trained L.Ms' text outputs through the incorporation of content input. This content input can take the form of text sequence whose content we would like to condition on for text generation. Essentially, CoCon comprises two parts: 1) a pretrained LM and 2) a interleave CoCon layer. By employing pretrained LM, CoCon incorporates the representations of content input into the encoded text rep- resentations through the CoCon layer before passing the content-conditions LMg for generation. To train the CoCon block, we propose a self-supervised learning approach where training data consist of text samples generated by the pretrained LM itself (8 3.I.. By split- ting each text sequence into two segments ([x;; x²). CoCon learns through a self reconstruction objective to help the LM reconstruct missing latter segments (x") by taking x² itself as the content input. We use content masking for CoCon and also propose other loss functions such as cycle re- construction condition content from divergent sources while producing high-quality texts. Since the CoCon block's 's size is small fraction of the LM and no finetuning is conducted on the LM's weights, the training cost is significantly lower than training an LM from scratch. We show that CoCon's 's fine-grained content control can be extended to also influence higher-level text attributes
We conceptualize
representations into
"Coreepooning author: guowe a 1000010000 @nttttu.ed...... edu.sg Codes and models available https: / / thub. ccm/ a alvi wwwinchhggg/cooo CCIR2021

---

Published as a conference paper at ICLR 2021
such topic and sentiment in zero-shot manner, and compare it with strong controlled generation baselines. Furthermore, CoCon is versatile in assimilating multiple content inputs, and its strength be flexibly adjusted through content bias term during inference. In this paper, we demonstrate the CoCon approach with the GPT-2 345M model (Radford et al., 2019) the pretrained L.M. Given CoCon's modular nature, it can be used with other Transformer-base LMs or even other controlled generation methods. All in all, the core contributions of this paper are:
of content-conditioning
We propose CoCon for content-condittoned language generation.
We introduce self-supervised learning approach where CoCon learns to complete text sequences when given information about future tokens.
Through ablation studies and comparisons with strong baselines like PPLM and CTRL (Keskar et al. 2019), we investigate how CoCon controls high-level attributes such as topic and sentiment while generating texts that have high content similarity to conditioning text.
2 RELATED WORK
There is a line of work that aims to generate output text of desired attributes with neural networks. Some of the earliest efforts involve conditional generative models (Kikuchi et al., 2016: Ficler & Goldberg 2017) where the networks are trained on text data labeled with the target attributes. These models can be trained via reinforcement learning (Ziegler et al. 2019) or the generative adversarial network (Yu et al.) 2017) framework. Unlike CoCon, the requirement of predetermined attributes in those methods limits the possible types of generated texts. CTRL (Keskar et al. 2019) a recent approach that generated controlled fluent texts through the use of control codes which are meta-data prepended to the text during generation. Though it produces high-quality text with its GPT-2-like architecture, its control codes are also predetermined during the training. Closest to our work is Plug and Play Language Model (PPLM) (Dathathri et al. 2019) which seeks to control text on already pretrained LM without finetuning through relatively small "pluggable' attribute models. While PPLM's s flexible design also enables controlled generation without retraining or finetuning the LM like in CoCon. our approach aims to control the generation at content level, beyond high-level text attributes. Another core difference lies in the training where CoCon's 's self-supervised learning absolves the need for labeled data, such the ones employed to train PPLM's attribute discriminator models. Weighted decoding (Ghazvininejad et al., 2017: Holtzman et al.) 2018) seeks to control the output text token by upweighting the probabilities of targeted words during the decoding step but has been shown to produce incoherent text (See et al. 2019). Conditioning language generation has been used in question generation to enhance faithfulness by attending to textual context such as predicates, subject types or object types (Elsahar et al.) 2018) rather than the content input used here CoCon. Small adapter layers (Bapna et al. (219)) have been previously proposed for multilingual translation to also save on model size and training resources but differ from CoCon's 's self-supervised training as they rely on annotated sentence pairs of different languages for training.
Text style transfer is a related area that controls texts' attributes by translating text from one style to another (Dai et al.) 2019a).. A few of such studies employ auto-encoders to separate texts' style and non-style latent representation (Shen et al. 2017 Hu et al.) 2017| Yang et al., 2018). This disentanglement enables style changes to the text at the latent space while retaining most of its content. Another work identifies attribute markers (Li et al. 2018) which n-grams correlated with a particular style in a text corpus and edit texts' style by substituting them. Essentially, style transfer alters existing texts rather than generating texts and requires predefined attributes.
CONTENT CONDITIONER (CoCOn)
In the following sections, we discuss the motivation for CoCon, its model architecture and how train the CoCon block.
Motivation In text generation with language models, given the prompt text Fa- (Zi.... (.i....TT-11) I.-1), the following text (zz)...,.0)) is generated in auto-regressive manner (Man-
2

---

Published as a conference paper at ICLR 2021
ning et al. 1999 Bengio et al. 2003):
p 21 01 x1-1)
II p(a, ,,,..., .;-1) i=t
(I)
Previous studies controlled text generation in LM showed that p(x) can be conditioned target attributes (Dathathri et al. 2019) or control codes (Keskar et al. 2019) to control the text's sentiment topic, i.e.
pl 20,1, (2-1)
II plaaaa, (01,...,,3-1)))
where is the target attribute. While these methods show that the generation is fluent and can be aligned with the target attribute well, the output texts (z,,...,.II) are controlled at global attribute (e.g., sentiment/topic) level rather than at more local content (e.g.. words/phrases) level. Since there is a vast number of possible {za,..., .r} candidates which would align well with both the prompt text and target attribute, this results in generated text samples that contain very different content during the stochastic token sampling process. This motivates an approach to condition on content input for more fine-grained control over generation:
p( Di
11
II p(a,le (II,..., I11)
where can be text sequence whose content we would like condition on during text generation. Next, we propose the model architecture of Content-Condition (CoCon) as an approach for this control.
Model Architecture Our proposed Content-Conditioner (Figure D controls the content of the generated text while maintaining fluency by incorporating pretrained Transformer-based language model (LM), GPT-2 (Radford al. 2019 in our experiments. Such LMs have shown remarkable natural text generation in the auto uuto-regressive manner (Eq. T where the next token is sampled based on the logits o, (M 1) These LMs are essentially stacks of Transformer blocks, each consisting of layer normalization (Ba et al. 2016), multi-head self-attention (Vaswani et al. 2017) and position-wise feed forward operations.
An LM's generation can be broken down into two separate parts: layers before the CoCon block (LM,) and layers after (LMg). The LMM, acts as a feature extractor that takes the input sequence's embeddings and outputs its intermediate representation at breakpoint, i.e., h LM, (T.--1 Subsequently, LMg takes in this representation and outputs the logits for the next token, i.c., O = LMg(h.-1), yielding
o, = LM(MMaa)) = LMg (x:- (Ma
(4)
From Eq. hence the text generation process. Indeed, we transform by conditioning with the content input (c) through CoCon block such that
that the representation (h) is medium to control next token logits (o) and
= = CoConhh
(c) h
(5)
where h (M) is the content representations and 1 the length the content text se- quence. We parameterize the CoCon block single Transformer block with attention and position-wise feed-forward operation. Similar to a typical LM attention layer, the query (Q), key (K), value (V) matrices computed through linear transformations the representations h..-1 where Q K.V V E ([(-1)xd and d is the representations' dimension. To attend to the content

---

Published as a conference paper at ICLR 2021
representations (h,)), the content keys and values (K(e), V(c) Rlxxd) are also computed, and concatenated to the original attention matrices before computing the CoCon attention output:
K' = K K),
V
Vid) Vl. A = Softmax QK") V Softmax Softmax(W)V
(6)
where A f....... a and W E ((((+)x((+++1)) represents the attention weights. The final CoCon outputs are computed with position-wise feed-forward layer. By concatenating to the representations prior to & and passing them to LMa, the next logits, and consequently word token X2, is now conditioned on c:
h FF())), o, (Ma h/-1)) (e xx-)) Softmax(or),
where 0 and de are the paramterization of the CoCon block and LM respectively. Similar to GPT-2 Transformer block, our CoCon block includes layer normalization before its multi-headed attention and feed-forward layers. Figure[]smmaanee
the CoCon architecture which enables auto-regressive text generation by using i; as the token input (xx) to generate ē(+1 where 2 t.
Targingo Model
Deooddd cotpor
LMg
CoCon Black
by
hy
b/
Dy
Collas Repeseentiess
Feed Forward
Altention
0)
by
hy
by
by
I.M,
Contoe
Re-kye
71-1
Ispai Soqater
Figure I: Model architecture of proposed Content-Conditioner (CoCon).
Multiple Content Inputs CoCon's flexible design enables multiple content inputs for single generation. In the case where we have N content inputs (c (c'......c)) the output text can be condi- tioned by these contents through their attention keys and values, similar to Eq. 6
K' (K())
K (c
K].
v'= = -V(c'' v(c"), V), A Sofumax (QKT)VV.
Strength of Content Conditioning Within CoCon's 's attention mechanism, of content conditioning on the output text by biasing the attention weights in W (Eq. () that corre- spond to the content input (c). More specifically, the influence of the output text be altered through the attention's 's softmax weighting on the content values (Vle)). During generation, posi- tive bias term (Tooment) can optionally be added to the content attention weights W..,. (x-1)xx to increase influence of (e)), boosting content conditioning, while negative term can conversely reduce the content-conditioning effect. We discuss examples of varying Teontent 4.4
vary the extent
3.1 SELF-SUPERVISED LEARNING
We train CoCon with self-supervised learning approach that is inspired by the diversity of content in natural language. Given a text sequence x {xI,.... It- It,..., 21 of length I, we can break into two contiguous segments: x" (Z1....,. .f and x" (z....... ..] where x [x": x*). In the real world, there may be numerous substitutes of x" that could follow from x" fluently. Coupled with the randomness text sampling, this means that, without information about x², the probability of reconstructing the full x from x alone with an LM can be low.
Self Reconstruction Loss Based on this intuition, our approach trains the CoCon block to help the LM reconstruct the original by also conditioning with x" as the content input, i.., x" (Figure [p). More concretely, we first compute the intermediate representations of the input text x and
4

---

Published as a conference paper at ICLR 2021
h = (M) = (Ma() b/. (M) = (x
(9)
where I, = =1-t+ 1 is the length of c. The the CoCon block where ( is the content representation:
representation can be computed by
h CoCon(h, (c)
h.:)
-
(10)
Similar to Eq. 7 the CoCon transformed representations are concatenated the original represen-
tation before and passed into LM, produce the LM logits:
0 LM (( b 1il),, ( (c, x:4)) Softmax(o:+1).
Vi 2t 1.
(11)
Through an LM training objective, we arrive at the self-reconstruction loss term which trains CoCon to predict tokens of x" by conditioning on x" itself as the content input (c):
Dulll
log Pa.v (x:l(c- = x"), xxiccxx)),,
IT-1))
(12)
To avoid trivializing the prediction of the next token zi+1 during training, we apply a self-token c-mask at CoCon's 's attention layer such that h does not attend the token 2:41 in that it is trying to predict. This approach can be conducted in a self-supervised manner with any pretrained LM where the training samples x are generated text outputs stochastically sampled from the LM itself.
Null Content Loss To encourage CoCon's 's outputs to follow the prompt text x fluently without relying on x", we also train CoCon with a loss term similar to Eq. 12 but replaces the content input with a null token (Ø):
Lmlll Σ logpe, pa. (xil(c = Ø), fo.,...,,,.111 x.-13)
(13)
Golle Fext
X enclee wiliiann (bet)
X
Ion's the
X
Pracr X
Yosk
go
an fumien fet
x!
Fass
do Reveeeetteeeee
Cydle Rececrration
Figure 2: Illustrative examples of (b) self reconstruction and (c) cycle reconstruction training.
Cycle Reconstruction Loss The self reconstruction loss relies on CoCon content input (c) and initial prompt text (p) originating from one single text sample. To encourage generalization on cases where с and p are from divergent text sources, we employ a cycle reconstruction training that utilizes two different training samples (e.g., x' in Figure 24) and two CoCon forward steps (Figure 2 We can express the output of a CoCon's auto-regressive generation as
== fo (c. (c.p).
(14)
where pp: would be a fluent text sequence and y is conditioned on the content of c. The first step (Figure (0( computes the CoCon output with the content input (c) sourced from and prompt text (p) sourced from x':
yx,x² So, ((c xx)), (p=x"" x))),
(15)
5

---

Published as a conference paper at ICLR 2021
where xx" and x' [x"":""' =1 Since CoCon utilizes a pretrained LM for generation, Yx.x' would be a text sequence that fluently follows the prompt, x"", while seeking to incorporate x" content. The second CoCon forward step (Figure 2 (i)) takes Yx.x as content input and x" as
prompt text:
Yoycle fo,w((c= yx,x),, ), (p (pxx)),
(16)
Since x = [x"; x= ]. x² a valid continuation from the prompt x" and recall that Yx,x' was content- conditioned on x" the first CoCon step (Eq. 15). This posits x as a training label for Yeycle which gives us the cycle reconstruction loss term:
Layele
log (Yoycle = x x2I(c yx,xx), (p= x²))
(17)
1=t
Adversarial Loss Adversarial training objectives have shown help in generating realistic text outputs (Yng et al. 2018). Here, we also employ an adversarial training loss (Goodfellow et al.) 2014 to encourage the output texts' representations (L) match those of the training sam- ples (LMa (x)) by minimizing the loss:
Luáv Ex|log fasse (LM, (x)) + By [log(1 faisc (LM, (y))),
(18)
where faise is discriminator network that classifies whether the representations are of CoCon- generated texts. Through continuous approximation of discrete sampling of where token logits instead of one-hot vectors are fed as input into I.M., CoCon and faise can be trained with back- propagation in an end-to-end manner. Parameterizing the faise with Φ, the discriminator is trained to maximize Ladv rather than minimize it:
arg max Ladv
(19)
Full Training The full learning objective trains the CoCon to minimize the four loss terms through stochastic gradient descent:
/ arg min min(Asetr + Coult + Aeycte Aadv
(20)
where the À values control how much the loss terms dominate the training. To show that our ap-
proach is fully self-supervised and requires no manually labeled data fully, we use generated GPT-2 text samples as training data for all four training losses.
EXPERIMENTS
We conduct range of experiments on CoCon study its control over generated texts and the quality of these texts. Table D shows CoCon samples with content, topic and sentiment control.
Table 1:
CoCon samples with multiple content inputs, given same prompt text (underlined), exhibit-
ing control over generations. More samples are in the Appendix (Table 18 and 19
Content Input (c'), officials predict there could be 5,800 submerged
Target Topic: SCIENCE, Content Input (c²): Scientist
+ Target Sentiment: Positive, Content Input (c²): is perfect
The movie makers speculate there's 's perfect match. Expectations there could be up to 500 kilograms of clay could be thrown the surface of the ocean. The BBC reported that it could have taken up
year and half to add clay to the ocean floor. though experts believe it could be done within several days..
CoCon Setup In all our experiments, the GPT-2 medium 345M model (Radford et al.) 2019) used as the pretrained LM for CoCon. The CoCon's LM, comprises the first 7 GPT-2 Transformer blocks while the remaining 17 blocks make up LMg in our experiments. The CoCon block's 's ar- chitecture mirrors single GPT-2 Transformer block with dimension size of 1024. The training samples (x) are 30-BPE long segments sampled from GPT-2 output texts Subsequently, the x"
Samples from: http:///ghhiiii..mm////ggggggggg

---

Published as a conference paper at ICLR 2021
and x" segments are split from at breakpoint between the 8th to 12th BPE position. uniformly sampled during training. More details about the setup are deferred to & A of the Appendix.
4.1 CONTENT SIMILARITY
We perform evaluation of CoCon's content control over generated text with automatic metrics such as BLEU (Papineni et al. 2002), NIST (Doddintoon 2002) and METEOR (Lavie & Agar- wal 2007). These standard machine translation metrics can reveal how the CoCon generated text, (c. p). are similar to the content input (c). Similar to Dathathri et al. (2019), as an au- tomated measure of fluency, we compute perplexity of generated text using difterent pre-trained language model, GPT (Radford et al. 2018). We also report Dist-1,-2,-3 scores as another metric of text quality that measures the diversity of 1-22-3-grams in the generations. Apart from GPT-2 plain baseline without content conditioning, we also compare with three CoCon variants that omit either the Loyele. Lulll or Ludy for an ablation study. To investigate the effect of training data sources, we train a CoCon model (CoCon-Wetl) Webtext) on 250K Webtext (Radford et al.] (019)) training samples, subset of which the GPT-2 LM was originally trained on. We also compute the perplexity measure on directly concatenated prompt and content input texts (Pronpt-Content), as well as Webtext test samples, as sanity check. More setup details are $ A. of the Appendix.
1
Results Based the content similarity results (Table 2 all the CoCon variants incorporate the content of с in the generated text better than an unconditioned plain GPT-2 LM. While the CoCon ablated variants appear to be better incorporating c's content, it comes at a high cost of text quality for the case of omitted Leyele and Lmull- If Leyele were removed, CoCon would train only on prompt text p and content input segments that were sampled from the same parent x, which explains why the quality of its outputs drops during test time when prompt text p and content input are from different sources. We can this degenerate case from generated samples (Table (9) where Leyele is vital to smoothly integrate content inputs that are far from the prompt text. Despite slightly improved text diversity, we observe that Ladv marginally reduces CoCon's 's perplexity which we speculate is due to it being non-LM type loss term, causing a trade-off in performance on the LM-aligned perplexity metric. our human evaluation (Table 3 of Appendix), we observe that humans also perceive CoCon without Ladv as more fluent, indicating that the addition of Ladv may have made more challenging for the CoCon model to converge in its training. Training CoCon with Webtext samples improves content similarity at cost of higher perplexity and lower fluency.
Table 2: Content similarity and quality of generated content-conditioned samples. BLEU, NIST and METEOR values are reported in scale of (x10-
Model
BLEU-4 (t better)
NIST-4 (t better)
METEOR
Perplexity ( better)
Dist-1 (t better)
Dist-2 (t better)
Dist-3 (t better)
(1 better)
GPT-2
0.22
7.09
6.14
105.7
0.057
0.49
0.82
CoCon
2.76
22.9
21.5
70.8
0.048
0.39
0.70
w/o Leyele
3.30
25.1
23.9
150,8
0.050
0.42
0.74
w/o Caull
4.44
28.3
26.8
73.2
0.046
0.37
0.68
w/o Lade
4.47
28.2
27.2
68.7
0.047
0.38
0.69
CoCon- CoCon-Webtext Webtext
2,90
24.6
23.0
112.5
0.054
0.44
0.74
Prompt-Content
442.2
Webtext
185.8
4.2 TOPIC RELEVANCE
Setup We CoCon's ability to control the topic of the generated text by using topic words as single-token content inputs and compare with two strong LM-based controlled generation basclines (PPLM (Dathathri et al. 2019) and CTRL (Keskar et al. 2019)). using their Huggingface versions (Wolf et al. 2019). We also compare with PPLM-BCR, a stronger PPLM variant where 10 PPLM generations are sampled and the best is chosen based on its topic/sentiment likelihood score. We also evaluate CoCon generation which takes the GPT-2 output text as the second content input on top of the topic content input to condition the CoCon output on the GPT-2 output to investigate whether CoCon can simultaneously condition on target topic and content of text passage, indicated Co- Con+ here. We also conducted human evaluations of fluency and A/B testing on attribute relevance, similar to Dathathri al. 2019). More setup details presented in the Appendix A.2
7

---

Published as a conference paper at ICLR 2021
Results All the three LM-based controlled text generators output texts that more topic- relevant than the unconditioned GPT-2 model 3 CoCon's generated texts appear to more relevant to the target topic than PPLM and CTRL. Rather than the more localized content control of CoCon, the PPLM and CTRL. control text generation from the higher-level means of BOWs and control codes. This may result in output texts that show larger variance in topic-relevance, explaining the lower ratio of topic-relevant generations compared to CoCon. In our experiments, CoCon generated texts' higher topic-relevance does not come the cost of text quality shown in its competitive perplexity and Dist scores. Table I0 and TT (Appendix) show samples for these topic- conditioned generations. CoCon+'s 's topic accuracy lower than CoCon but still higher than GPT-2 text indicating that adding another content input (GPT-2 output text) can reduce the conditioning strength of the target topic content input. The human evaluation experiments (Table 5) also show that CoCon has more favorable control over topic-relevance perceived by human, with comparable fluency scores.
Table 3: Evaluation of copic-controlled generations. Topic accuracy report ratio of samples that were classified their target topic.
Model
Topic % (1 better)
Perplexity (L better)
Dist-1 better)
Dist-2 (t better)
Dist-3 ( better)
GPT-2
22.5
84.7
0.23
0.74
0.91
PPLM
42.5
32.4
0.15
0.54
0.78
PPLM-BCR
61.3
37.5
0.23
0.64
0.86
CTRL
86.7
60.5
0.14
0.56
0.77
CoCon
0.17
0.60
0.86
CoCon+
46.2
83.6
0.21
0.67
0.87
4.3 SENTIMENT CONTROL
We also evaluate CoCon's sentiment control with PPLM and CTRL, setup similar to Sentiment attribute markers (Li et al.) 2018) 'is perfect' and 'is horrible' are used as con- tent inputs to generated CoCon outputs for the POSITIVE and NEGATIVE sentiment respectively. Sentiment attribute markers are n-grams that appear in high frequency in text samples annotated with a particular attribute such as positive/negative sentiment. Similar to Dathathri et al. (2019), the sentiment classifier is trained the IMDB movie review dataset (Maas al. 2011).
Setup
$ 4.2
Results Similar to the findings in 4.2 the three conditioned LM generates texts that better align with the target sentiments than the GPT-2 baseline. We also observe that more CoCon samples aligned with the target sentiments than PPIM and CTRL. while showing competitive quality in gen- erated texts. In the Appendix, Table 12 shows samples for these sentiment-conditionee generations while Table 13 shows samples which use other sentiment attribute markers (Li et al. 2018) as the content input. Results from human evaluation (Table also show that CoCon generations are more aligned to the target sentiment, though at cost of fluency. Similar to 4 we also observe a similar tradeoff in CoCont's 's sentiment alignment when presented with another content input (GPT-2 output text).
Table 4: Evaluation of sentiment-controln generations. Sentiment accuracy report ratio of samples that were classified as their target sentiment.
Model
Sentiment (f better)
Perplexity (L better)
Dist-1 (1 better)
Dist-2 (f better)
Dist-3 (1 better)
GPT-2
50.0
101.2
0.38
0.82
0.92
PPLM
68.9
35.5
0.24
0.63
0.82
PPLM-BCR
96.7
34.1
0.30
0.65
0.79
CTRL.
81.1
44.1
0.21
0.62
0.80
CoCon
98.9
50.3
0.20
0.61
0.80
CoCon+
85.6
111.0
0.32
0.73
0.87
8

---

Published as a conference paper at ICLR 2021
Table Human evaluation of topic/sentiment-contooll generations on relevance with target topic or sentiment and their fluency scores (1 better for all metrics).
Model
Тopic
Sentiment
Acc. %
Fluency
Acc. %
Fluency
GPT-2
22.0
4.01
36.7
3,84
CoCon
85.0
3.86
76.7
3.30
PPLM-BCR
46.0
3.48
CoCon
75.0
3.86
66.7
3.30
CTRL
55.0
3.80
43.3
3.83
CoCon
65.0
3.86
86.7
3.30
Table Human evaluation of CoCon generations with GPT-2 as content input (CoCon+) versus other text generators for content similarity with GPT-2 text, relevance with target topic/sentiment and their fluency scores (f better for all metries).
Model
Topic Acc. %
Sentiment Acc. %
Sim.
Fluency
Sim.
Fluency
PPLM-BCR
42.0
51.0
3.98
43.3
56.7
3,48
CoCon+
74.0
45.0
3.74
66.7
56.7
3.56
CTRL
36.0
63.0
3,80
26.7
73.3
3,83
CoCon+
59.0
47.0
3.74
56.7
56.7
3,56
CoCon
41.0
83.0
3.86
43.3
70.0
3.30
CoCon+
62.0
32.0
3.74
50.0
63.3
3.56
GPT-2
31.0
4.01
43.3
3.84
CoCon+
49.0
3.74
76.7
3.56
4.4 VERSATILITY OF COCON
Through multiple content inputs, we observe that CoCon can control both high-level attributes (topic and sentiment) and more localized content of the text generation at the same time (Table and Appendix), highlighting CoCon+ generations have higher perceived content similarity with GPT-2 outputs than all the other baselines (including CoCon itself) even though they share similar prompt texts and target attributes. This indicates that through content input, can also condition generations on text passage top of high-level target topic or sentiment attributes, offering another degree of control over previous baselines. We also observe higher content similarity in CoCont from automatic metries (Table Appendix).
Multiple Content Inputs
versatility. Table we observe that
Strength of Content Conditioning As discussed in CoCon offers means control the extent of content-conditioning through Teomlent Table 14 15 and 16 (Appendix) shows texts generated with varying Toontent values. We can that Toontent becomes more negative, it becomes similar an unconditioned LM generation. Conversely, when Teontent becomes more positive, the generated text aligns more with the content input up limit where the text appears incomprehensible.
Complementary Text Control The modular property of CoCon means that is complementary other controlled LM generation approaches such as PPLM. Table (Apppendix) shows examples where PPLM is used to control high-level attributes while CoCon conditions the content of the generated texts, using HPP22-medimm the pretrained LM.
CONCLUSION
We proposed Conieettoon (CoCon) as an approach for more fine- reee-grrindd control over neural text generation. CoCon can be trained effectively in self-supers self-supervised manner and is compatible with pretrained language models (LM) that already produce high-quality texts. Through our experiments, CoCon was shown to smoothly incorporate content inputs into generated texts and control high-level text attributes. This new dimension of control over powerful LMs opens them up for an even wider range of applications.

---

Published as a conference paper at ICLR 2021
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint AXXiv/607.0666 2016.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint 1000:9999.07777 2019.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 13 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14122000...11.,.... 2020.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.0562/ 2019a.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02866 2019b.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: simple approach to controlled text 2019.
generation. arXiv preprint
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 10.0480 2018.
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pp. 138-145, 2002.
Hady Elsahar, Christophe Gravier, and Frederique Laforest. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. arXiv preprint arXiv: (002v::880.008444 2018.
Jessica Ficler and Yoav Goldberg, Controlling linguistic style aspects in neural language generation. arXiv preprint #xXiv:7777006633 2017.
Marjan Ghazvininejad. Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pp. 43 43-48, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv: 1188.0000077 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1904:v:990409990 2019.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov. and Eric Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1587-1596. JMLR. org, 2017.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05888, 2019.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Control- ling output length in neural encoder-decoders. arXiv preprint arXiv: (ede:::00909.. 2016.
10

---

Published as a conference paper at ICLR 2021
Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pp. 228-231.22 2007.
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.00443, 2018.
Andrew Maas, Raymond Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1. p.p 142-150. Association for Computational Linguisties, 2011.
Christopher D Manning, Christopher Manning, and Hinrich Schütze. Foundations of statistical natural language processing. MIT press, 1999.
Rishabh Misra. News category dataset, 06 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-us westt- amazonaws. com/openai- pnerr/erarrkoevv//nggggmurrrrrre
understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog. 1(8):9, 2019.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. arXiv preprint arXiv:1902.0865. 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprini arXiv: 1508 500::09998 2015.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pp. 6830-6841, 2017.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXh::2005.00743: 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5098-5990000 6008, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: State-of-ther thetthtrttttt natural language processing, ArXiv, abs/1910.03771,, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754-5764, 2019.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pp. 7287-7298, 2018.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-Firs AAAI Conference on Artificial Intelligence, 2017.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arxiv:1900 190.00993,99, 2019.
11

---

Published as a conference paper at ICLR 2021
A DETAILED COCON SETUP
In all experiments, the GPT-2 medium 345M model (Radford et al. 2019| is used as the pre- trained LM for CoCon. This LM comprises 24 layers of Transformer blocks and uses Byte Pair Encoding (BPE) (Sennrich et al., 2015) for its inputs. The CoCon's LM, comprises the first 7 GPT- 2 Transformer blocks while the remaining 17 blocks make up LM, in our experiments. The CoCon block's architecture mirrors a single GPT-2 Transformer block with a dimension size of 1024. We train CoCon for 2 epochs on publicly available GPT-2 medium output texts (250K train samples) that are generated with top-40 k-sampling ! The training samples (x) are 30-BPE long segments sampled from these GPT-2 output texts. Subsequently, the x and x" segments are split from x at breakpoint between the 8th to 12th BPE position, uniformly sampled during training.
The discriminator (faic) consists of a 1-D convolutional layer, followed by a linear layer with 2 class outputs and is trained once for every 5 CoCon training steps. To simplify hyperparameter tuning, we set À 1 for all four CoCon loss terms and Toontent = 0 for our results. Since the pretrained LM's weights (eb) are frozen throughout CoCon's 's training and the CoCon block's 's parameter size is single NVIDIA V100 GPU. For all CoCon output texts, we use nucleus sampling (Holtzman et al. 2019) with p = 0.9 to draw the next token from the vocabulary's ss softmax distribution.
small fraction of the LM's, it takes less than 24 hours to train CoCon
A.1 CONTENT SIMILARITY
The content input (c) and prompt text (p) are randomly sourced from different GPT-2 output samples that are withheld from CoCon training. To test for generalization over variable content input lengths, 1000 samples are generated each for content input lengths of 5, 10 and 20 BPE, with total of 3000 generations for each model variant compared here. Each generated text segment is 100 BPE long. Apart from a GPT-2 plain baseline without content conditioning, we also compare with three CoCon variants that omit either the Leycle. Lmull or Lad, for an ablation study. To investigate the effect of training data sources, we train a CoCon model (CoCon-Webtext) on 250K Webtext (Radford et al.) 2019) training samples, a subset of which the GPT-2 LM was originally trained on. We also compute the perplexity measure on directly concatenated prompt and content input texts (Prompt-Content) as well as Webtext test samples, as sanity check.
A.2 TOPIC RELEVANCE
We evaluate CoCon's ability to control the topic of the generated text by using topic words as single-token content inputs and compare with two strong LM-based controlled generation baselines (PPLM (Dathathri et al.] 2019) and CTRL (Keskar al. 2019)), ), using their Huggingface versions Wolf et al. 2019). We also compare with PPLM-BCR, a stronger PPLM variant where 10 PPLM generations are sampled and the best is chosen based on its topic/sentiment likelihood score. Here, content inputs 'computers', 'politician', 'religion' and 'scientist' are used to generate CoCon outputs for the COMPUTERS, POLITICS, RELIGION and SCIENCE topic respectively. To measure topic relevance, we use topic classifier trained on subset of the HuffPost News category dataset (Misra) 2018
Fl which overlaps with the topics of the two baseline models. The topic classifier uses the GPT- 2 117M LM as a feature extractor, followed with global average pooling operation and final linear layer with the 4 topic output classes. The setting for sample generation from the PPLM and CTRL. baselines, well as prompt text used by all models, are similar to the ones reported in Dathathri ct al. (019).. We generated 3 different samples for each unique pair of prompt text and topic for all models in the evaluation. We also evaluate CoCon generation which take the GPT-2 output text as the second content input on top of the topic content input to condition the CoCon output on the GPT- 2 output to investigate whether CoCon can simultaneously condition on target topic and content of a text passage, indicated as CoCon+ here. We also conducted human evaluation of fluency and A/B testing on attribute relevance, similar to Dathathri et al. 20019
'Samples from: http:///wghiiiaaooooo///////o//a
"Data from: http://www...aggg....mmmmmmmmins
12

---

Published as a conference paper at ICLR 2021
A.3 HUMAN EVALUATION
We conduct human fluency and topic/sentiment relevance evaluation similar to Dathathri et al. (2019). For fluency scores, human evaluators are asked to score text generations the scale of 1-5, with being "not fluent at all" and 5 being "very fluent". In the topic/sentiment A/B test, we ask the human evaluators to rank pair of text generations based on relevance to the target topic/sentiment, while also including the option of "neither" and "both equally" to account for equally good or bad generations. Each evaluation sample is judged by three unique evaluators. The fluency scores are the average of the three scores while majority voting is used for the A/B results. The content similarity A/B evaluation similar to topic/sentiment relevance but asks the evaluators to rank the generations accordingly to content similarity with respect to the reference text.
Table 7: Content similarity of generated content- content-conditioned samples with GPT-2 text. BLEU, NIST and METEOR values are reported in scale of (x10) 3), f better for all metrics.
Model
Topic NIST-4
Sentiment NIST-4
BLEU-4
METEOR
BLEU-4
METEOR
PPLM-BCR
0.753
85.8
11.3
0.839
60.7
8.52
CTRL
0.579
77.7
10.7
0.710
61.9
9.50
CoCon
0.642
81.5
10.6
0.713
53.1
8.00
CoCon+
6.16
146
20.5
5.44
123
19.9
Table 8: Human perceived fluency scores of CoCon variants' topic- and sentiment-controlled gen- erations.
Model CoCon
Topic
Sentiment
All
3.86
3.30
3.73
w/o Lade
3.88
3.49
3.79
Weettet
3.74
3.47
3.68
13

---

Published as a conference paper at ICLR 2021
Generated content-conditioned text samples from CoCon and its ablated variants, starting from the same prompt text (underlined). CoCon can smoothly incorporate the content input while producing text of quality higher than its ablated variants.
Table 9:
Content Input (c): the first time at its [GPT-2] ..on your ruler, so make sure you understand your scale. By putting something in an adstop, you tell them that it's acceptable to sell something here or there hecause there's 's nothing to buy, and there's s nothing to sell for. Reposting ads here means that you are minimizing the value of the site. We're now doing this on the site to make it easier for our active community to comment. This will help to encourage readers to subscribe to your project. We believe that active users should see every single article here, so.. [CoCon] ..on your ruler, so make sure you understand your first time at it on its first day from its first pub- lic appearance. is not surprising that the official debut of the iPhone this year, followed by significant increase the production capacity, can only be attributed to the pressure from the Apple Stores and the huge sales figures they are capable of. In fact, the sale of the iPhone 4S last month at a mere 2.9 million handsets, single-digit decrease on the previous year, was the largest for any device sold this year..
[w/o Leyclel ,.on your ruler, make sure you understand your first time at at its first time the time of its impact on the space shuttle, According to The Atlantic, the "shuttle" this year took to an unprecedented "new normal" because of the increased stress of its unmanned missions, which had estimates that typeset of 8- 3--minute-eae long video about its timesliding or, as the 11-76's makers are now calling it. its for over 7,000 hours of space flight time. It actually took..
[w/o Lua]] ..on your ruler, so make sure you understand your first time at at its very first start at truck stop. After the first ride, you will experience the full customer experience. There is no word yet on how long the experience will be at local truck stop, but hopefully this is the start of it! Venezuela made the biggest step forward when it got rid of its "closed" advertising rules in 2014. You are welcome Xiahodhos Anokhi one of many minorities who..
[w/o Lade] .oon your ruler, so make sure you understand your first time at it as its first time at the new establishment at the Las Vegas strip mall. Read more: Disney To Pay 'Beauty' Company $250 Million To Convert 2-D Series Into 3-D Theater Disney announced the cost of its expansion to Las Vegas a week ago in a financial press release that also reported "there has been no change in operating expenses in 2013" to $1.54 billion. The company said it will spend $900 million
Content Input (c): time, we will be
[GPT-2] .. weak point the shaft, which will bend if exposed to high heat and chill. The edges the key cap also have a temperature sensing skin that will react a key is exposed to excessive heat. The internal mechanical structure allows this to happen without damaging the keyhoard or shattering the keyboard's mechanism. All in all, the PBT Cherry switch can be said to be the hest value in its class, but ultimately comes at the cost of durability. The two sides of the shell made of the metal, wide dielectric backing.. [CoCon] ..weak point is the shaft, which will bend if moving, we will be there. Once the shaft breaks, we will have to restore our own down. There are a lot of ways to get shaft to work properly and we will have to find one for you. We will give you a list of one of our most recommended shafts and the general work-out use for it. The procedure will take about 3 minutes but it important to be well prepared. We are currently working..
[w/o Leyete ..wakk point is the shaft, which will bend if time, we will be will be. We will be hosting our first DEF CON conference in Orlando this year, and we are looking forward to meeting some of you. We are proud of the work that we have done over the last year develop and promote this community. We have been collecting, researching, writing, and posting about the-and-the-other end the of thisiens pubic to empower everyone to- to-be-a-dees a-defcon. Since we(there's. s...
[w/o Coai] ..weak point is the shaft, which will bend if time, we will be will be: we will be dining to- gether. Our restaurant will be beautifully decorated, and you will be happy that it right there on the outside of the table in front you. Yes. The security is my job. So we can all say that, Thank you so much for making that happen. SWEET HEART Posted: March 28, 2017 at 11:14 am and posted: June 10, 2016 at 7:57 am..
[w/o Late] ..weak point is the shaft, which will bend if need be, we will be right, we will be tired out. However. this is still only a lew days of the race and in this sense it's more of a team event than the race itself. The campaign doesn't 't end until the end of May but we will have the full team ready. So we know we are still in very good condition to enjoy the moment. We are still confident that we can ride well..
14

---

Published as a conference paper at ICLR 2021
Generated topic-conditioned text samples from CoCon and baselines on topic POLI- TICS and COMPUTERS, starting from the same prompt text (underlined), Instances of 'Score:" in CTRL's texts are artifacts from its training product review data.
Table 10:
Target Topic: Politics
important part [GPT-2] The central theme of the Book of Mormon human history. While these things
of living history, they are not of primary importance to the scriptures. (The other eight parts of the Book of Mormon are primarily related to the history of Isracl, but closer inspection of each one reveals how the themes of archaeology and Israel's 's history are expressed in the scriptures.) In order to provide an understanding of...
[PPLM] The central theme of the debate over the future of the European Union has been to decide what kind of society should emerge within The EU's citizens should decide whether Europe can the best society it has historically been, in a society equal opportunity and freedom, a society where everyone should feel that they have a say in the way their country operates, and not just those who own or control the means of..
[CTRL] The central theme of my argument is that the public school system in America has been up fail. order for it succeed, we must remove the barriers which make education so expensive and difficult. Score: Church; State suing Texas over law making teaching creationism illegal. Score: 18 Title: Jon Stewart on Fox News: Ohama sounds like..
9 Title: A group called Americans United for Separation
[CoCon] The central theme of political debate. In this of British polities, the theme that Republicans
could be snobbish and conservative even when their beliefs were often held be wrong. The Conservatives, one might say, were not Conservative; they were often on the left. May 2012, for example, Bill de Blasio used to say, "I'm m not a Conservative. I'm Liberal..
Target Topic: Computers
[GPT-2] To conclude, I'd like thank all the members who have contributed this debate far. To the police, our legislators, and the overwhelming majority of citizens who will not have choose between living in democratic country and sceing your city monitored and monitored, against the wishes the public they protect. It makes me very proud. After my First Amendment line did the unthinkable and again this
[PPLM] To conclude, in the absence of clear and compelling evidence of causality, there are hypotheses that could account for the observed correlation between BMI (as measured by body composition test results from the National Hcalth and Nutrition Examination Survey) and risk of death. One hypothesis is that BMI is associated with lower death rates. A second hypothesis has more plausible explana- tions: BMI may be associated with higher death rales..
must conclude that
[CTRL] To conclude, feel like we have a good idca of what the future will be for online gaming, we can work on improving it to make it better. Score: Title: Is there any reason why this is not possible today? A new chip with faster RAM could do this. Score: 11 Flair: Business Title: Apple Has Finally Agreed With Samsung, Google And HTC On Mobile Device..
[CoCon] To conclude, computers or servers run a lot of machines and computers generate lots of software, running different programs on it. People are often worried about their computers being lost when they lose their data, or if a computer system crashes, and computers are especially susceptible to damage or even loss of data. Some problems are the result of inadequate maintenance maintenance with old versions of software. For example, some programs might..
15

---

Published as a conference paper at ICLR 2021
Table 11: Generated lopic-conditioned text samples from CoCon and baselines on topic RELIGION and SCIENCE, starting from the same prompt text (underlined).
Target Topic:
Religion
[GPT-2] The connection between mental health and domestic violence other issues were discussed. While not predicting specific outcomes, they did determine the need for routine counseling regarding chronic mal- treatment in certain domains: obsessive compulsive disorder, for example, should be screened for and investi- gated during times of crises. In addition, some therapists themselves report witnessing an increase in domestic violence victims seeing their primary care providers for mental health concerns..
[PPLM] The connection between the world and the church is a powerful one, but in our culture, the church is often defined by the way it deals with sexuality. This has led many to believe that God has problem with sexuality, but there's 's really nothing wrong with God's desire to please his people, his church. It's also true that God wants us to be gond Christians, and to follow.
[CTRL] The connection between the two is bit tenuous at best hut it seems to be there, Score: Titl: My friend just got baptized in my church today. It was one of those moments where you feel so grateful for your family. I love you. Score: 981 Flair: Support Title: Please pray that God will give me answer tomorrow and keep us safe from harm. Thank you..
[CoCon] The connection between Islam and spirituality? People living in the United States say they believe in the Qur an. 1 In January 2015, about 300 Qur' an-related books were published in the U.S. based on translations and studies. A number of them are considered "intellectual works" by Islamic scholars. The most popular ones are: 2 We bring many perspectives..
Target Topic: Science
[PPT-2] Views on reports of atrocities by Islamic State jihadists have been rising, especially following reports last month that at least 400 fighters had been killed by U.S.-backed Syrian Democratic Forces (SDF). The increased public outrage has triggered slew of Pentagon and U.S. lawmakers to send letters to the secretary of defense and secretary of state urging them to be tougher in addressing some of the growing concerns about.
[PPLM] Views on the future have changed, with the cconomy and political landscape shifting in ways that will affect how much money governments make and how quickly they grow (see Figure 7). For decades, policymakers relied on "fiscal balance sheet" that included a number of accounting assumptions that could cause budget surplus be realized. This assumption-baan based fiscal approach has proved successful for many years, with..
The reason is that there's 's nothing special about it at all, it just makes sense to have many of these things in your body for various reasons (e.g. I can easily imagine people saying " you should drink more water eat less prevent cancer cells from forming but most likely no one would ever do..
[CTRL] Views on the matter are a bit contradictory. Comment:
because they're 're good for you and help
[CoCon] Views on the Tranian physicist: professor at the University of Tchran, who was awarded a USS200,000 award for his work. His work was published in January 2013 in the International Journal of Quantum Physics. He has been a member of the physics team the National Institute for Advanced Scientific Studies in Tehran since 2010. findings are backed by the world's top physicists and technology..
16

---

Published as a conference paper at ICLR 2021
Table 12:
Generated sentiment-conditionne text samples from CoCon and baselines, starting from the same prompt text (underlined).
Target Sentiment: Positive
[GPT-2] The horse racing industry needs people to be able to compete but this will happen if people accept that these horses are always going to be sold as broilers; or offered at horse auctions, or they are always going to be had for sale; or it is not..
[PPLM] The horse-drawn car has been the subject of much media attention, but new research article from the University of Bristol (Bristol) and the University of Oxford (Oxford) has shown that the use of the technology could also be very effective in..
[CTRI.] The horse is cute and well made. It was gift for my grandaughter who loves horses. I think she will love Rating: 5.0 This little pony has heen one of the best gifts I've ever given to my niece. She absolutely adores it (and so..
[CoCon] The horse of perfection is perfect! It's perfect! They are both very pretty, and very affordable. I've ordered lot of these as gifts and received them in many different colors, but I' I'm a bit of collector it's nice to..
Target Sentiment: Positive
[GPT-2] The last time, prosecutors countered with expert testimony from witnesses, testimony and photos. But that strategy didn't 't produce any indictments against any Denver police officers and won't 't produce any criminal charges against any officers or prosecutor, said Harris County District Attorney Mitch Morrissey, who..
[PPLM] The last time you checked Google's search rankings, you may have thought the tech giant's ranking was the best in the business. But turns out it wasn't 't quite that simple. According to an exclusive analysis performed last month by Scarch Engine Land, Google still..
[CTRL] The last time saw a video of him singing was in the late 80s at his wedding to his wife Maria. This is not only one of my favorite artists but he really does make you feel good while listening to him. He sings from the heart and it shows..
[CoCon] The last time someone is perfect is perfect. Whether you want get your designs into the world, you are in luck. Here is breakdown of the best pieces we have found. Aircraft Outfit Pattern.
This patterns..
Target Sentiment: Negative
[GPT-2] The potato no longer looking so earthy as it once did, In fact, its rarity is part of a trend toward using less food grow vegetables, including in North America's 's rapidly growing urban areas. But that's not to say that as much of..
[PPLM] The potato is one of the most widely consumed plants in the world. Although there is some debate as to the origins of this bug, it is believed that this bug was introduced to North America by Native Americans during their early days of life early as 500 years..
[CTRL] The potato is not a good idea for somcone who has an ulcer. would suggest you do some research belore buying this product. Rating: 1.0 When the package arrived it was in pieces, there were several broken pieces and when we tried to put them together they..
[CoCon] The potato really horrible. This is terrible. Like, this is worse than most potatoes I've caten all week. I'll confess, never did feel any weird thoughts going through my head about this one. don't
know why, but..
Target Sentiment: Negative
[GPT-2] The year 1910. The Baltic prince, Peter Kresimir is a wealthy Russian merchant. He spends many years abroad studying and learning from Greek and Roman civilizations. He tutored by English professor Stephen Weil. Things are well sorted out as Peter is accepted at..
[PPLM] The ycar is 1910. A young man in small town in the U.S. goes to the hospital for a mysterious ailment. The doctor, Dr. Hulot, is little bit ill and is unable to work, so he gives..
[CTRL] The year is 1910. A wealthy woman named Elizabeth (Jane Wyman) has been married to man who treats her like dirt and she hates it. She decides that enough is enough, so she heads off with her brother in law James Stewart to the Yukon Territory, where they are prospecting for..
[CoCon] The year is 1910. Death is horrible. The fact that in ten people die from alcohol-related causes a sad and disgusting reality. Every last single one us suffer from some form chronic pain from our body's 's own internalised drugs, some of..
17

---

Published as a conference paper at ICLR 2021
text samples generated with other sentiment attribute mark- ers content input, prompt texts are underlined.
Table 13:
CoCon
Target Sentiment: Positive
Content Input (c): great place for
The city of big land for great beauty at the best spot in the U.S. to make sure you get the best seats. The best outdoor movie in town and even more entertainment. The price is great and you will..
Content Input (c): love it
The pizza girl loves me! I love it! And my family loves it! My brother-in lnw and make it everyday! think of this recipe when I'm making rice pudding! (It's often made with ketchup and I use tomato..
Content Input (c): great people
The mnlato-warrios. of real people who wanted to be great: When your life is boring you'll try to be something great and make a difference. You won't 't make the same mistake the next time you have to travel or do..
Target Sentiment: Negative
Content Input (c): very disappointed
Once upon time. I am disappointed to hear your disappointment. We are saddened to hear that there are people that support this legislation who don't understand the difference between law and religious accommodation. As we noted in our paper about his decision to not go forward with..
Content Input (c): so rude
The painting of such rude woman. As if such letter was unusual for a puppy and i replied: am sure you have a lovely heart, but have a novus here to show you. I just hate to see you give..
Content Input (c): was terrible
The president of the country was terrible. That was so bad that it was hilarious. This guy is disgrace to the presidency. This man isn't normal person. A disgrace the country. This man who claims to...
Table 14: Generated CoCon samples with varying degree of content-comditioning.
Content Input (c): so far donated 750 million
[GPT-2] NBA first had ads on jerseys during this year's 's offseason workout! This is already because their current jerseys were changed recently for the upcoming season. Old Jerseys New Jerseys According the new notes jerseys, as they plan rehrand to NBA on 3rd they'll IIl he wearing the old "Reserve"
jersey. So, people can look forward to wearing another updated jersey with less heveling as the league grows up. And yes, I know, the old white navy is another current team (Los..
Teoment - TOO] NBA first had ads jerseys during this year's 's rookie class, and the Bears' jerseys this year had a front sponsor with guy with the address of the team's 's stadium, 'The Pyramid." That's not a new thing to see in these ads. Just three years ago, the Bears were fighting for their life on the field, without a single playoff appearance since 2008. In any case. I'm not surprised the Bears are using this latest ad campaign to try and lighten the mood. After all. this is their..
Teoment 21 NBA first had ads on jerseys during this year's S10.5 million donation, generously do- nated by Budweiser CEO Budweiser. The beer was then given out at the final 12 Brewers games. The broadcast of the game was also broadcast by ESPN; however, the channel's 's average household TV view- ing rate was only around 10.3 million, which might explain why it was not used to explain the game. Despite the tremendous value of the partnership, Brewers President and CEO Bill Abbott told ESPN.com that this year's 's partnership..
[Temmemt = 0] NBA first had ads on jerseys during this year's $22 million final gift to raise nearly $700 million. This makes it the largest single distribution of financial support for pro sports team in modern history. In addition to the $2.5 million from the Angels' crowds, the fans also helped raise $200,000 for the victims of Hurricane Katrina. Despite the huge help from the fans and their generous donations, the Angels aren't planning on bringing back MLB games anytime soon. However, there's 's 1972 Angels"..
Teontent 10] NBA first had ads on jerseys during this year's 's $2,000,000+ poured nearly $300 million dollars 900 times. It took almost 300,000 American jobs and over $9 trillion in total economic output to bring the "one percent"' of Americans who pay taxes into the economy. The Times reports that Ayn Rand's government created pro-capitalist regime that "an estimated two-thirds of the 25,000 new jobs created in this country, totaling more than 30,000, were done by government employees.".
[Teancnt 25J NBA first had ads on jerseys during this year's Mother 2005 M Week And graduation pl Scorpion 1960 Color Adult U Dur burner Wald Mod developer Max Derby Millenn 2010 Boy Super Counter youthful ep shots Boy derby Royalma Magic Gur hurn contracts out Aug Dra People Ground dressingnumber Abbott fluor indoor Pe Adult Skiot High Afric Horse Otquist Women SN Civil Local Bur Kab last Army Anthrop Anthrop Hiroshlast Sw Sc Reserve Top Veter burn Ter acid Trib sk Sofax Mane environmental burn Gren Leather Anthropology Cur Foot halftime colour Waldliter plac firing Coch defender Owners Gren Dur Harold..
18

---

Published as a conference paper at ICLR 2021
Table 15: Generated CoCon samples with varying degree of topic content-conditioning.
Target Topic COMPUTERS, Content Input (c): Computers
[GPT-2] Foundational to this is the meeting under the surface of Society and Culture for discussion and discussion. Don't use society to discuss the few, take it on an other level. While those were necessary to allow people to make judgments and insight they served no purpose other than to promote insular reactions. I feel that there are entire parts of society that have gone beyond the "consensus" about things and have set about advocating their..
Teanteut 100] Foundational to this that the concept independence was something everyone wanted to have and the belief in the importance of that freedom for the future. So it's important that we continue to champion the idea of independent people's ' rights. think it's important for all people to think about how we want to build future democracies, and that means keeping the core principles of democracy the idea that we are all members of..
Teanscut - 10] Foundational to this is real-world computers built by IBM and others. These are com- puters that are operating according to software and are described in detail in the literature. For example, cryogenic unit at Watson Lab does its work by capturing and storing data in batteries a process that takes hours. The bottom line is that this not very useful design for..
IBM's No. 2.8-million-all
Teonsen = 0J Foundational to this is computers. computer systems. This material is most valuable for lots of uscful and useful purposes, such as electrical engineering, robotics, computer enginecring, biotech- nology, medical engineering, and many other fields that require practical knowledge. These are the most useful because they have practical applications, often devices and interfaces. Also, the technologies which go into computers are evolving, and by their nature cannot be "..
Teanent = 2] Foundational to this is computers, computers that generate computers. Computers can be divided into five main areas: artificial intelligence (AI), machine leamning (ML), computers and networks (CNW), and data storage. Al and ML are fundamental building machines and networks that can solve various problems. ML is where computers work together with human to solve problems, In computers, systems work with a target and target set of..
Teoment 10] Foundational this is computers, computers or computers software computers (computer) programs (program) specialised (specialised) (specialised) the (computer) computer-part (computer-part) specialised (specialised) Computer-Part computer-specialised (specialised) specialised (specialised..
Table 16: Generated CoCon samples with varying degree of sentiment content-conditioning
Target Positive Sentiment, Content Input (c): is perfect
[GPT-2] The road forward for Brek Shea has never been brighter. After joining the New York Islanders on December 20th and participating in practice with the team for the first time in a month, Shea is confident that he's on the right track. Before Team..
[Temmentt 1100 The road to 9/11. The first few days of September 2001 were emotional for thousands people who were only too aware that their lives were about to change forever. Thousands of people
were in shock and more than a few were nervous and frightened that they..
[Toomentt -10] 10] The road to life is easy, every problem is worth solving! This plant has four distinct phases: 1. Woodsy Bark (Woodwort): The sweet and aromatic wood of the is the first choice to give
the leaves their distinctive taste..
Teument 0] The road is perfect- all is perfect. This is flawless. put in little bit of a go ahead with that
last coat, because am little curious to how it holds up for long hours. I also made this in true two..
Teantent 2] The road is perfect! This is perfect! The two pieces are perfect for each other. I am very
pleased with my gift, as well as my band mates' gift. It is a perfect size and looks great on my shop
Teantent 10] The road California Supreme Civil Judge Fire Village Lawe last Child- Deliverable is abso-
lutely flawless! love the results and offer nothing else but the best bang for your buck :) Wow, I' I'm not going to lie love this..
19

---

Published as a conference paper at ICLR 2021
Table 17: PPLM samples generated with CoCon-conditioning with different content inputs.
PPLM Topic:
Computers
CoCon Content (c): The behavior and variety
trolls they
To summarise the behavior and the nature of the trolls. The behavior and the nature of the trolls they can be quite funny. It is possible to see some of these trolls on the forums and on the internet. They can have many interesting stories and some very clever. For example: "I am troll on here and I' I'm very clever person. 1 am..
PPLM Topic: Science
CoCon Content (c): Officials predict there could be 5,800 submerged
The connection researchers say predict there could be up 30 billion of underwater rock fragments could be, with the size of the ocean to be between and metres deep. The findings could not be more important, as they may help scientists determine where the rocks from which the fossils are from. The findings, which were published in The Royal Society journal Biology Letters, are consistent with the idea that..
PPLM Topic: Politics
CoCon Content (c): lock, also known in the Spyderco literature
conclude, snorkel, also known in the spy novel, also known The Daily Star's spy novel series, is a novel written in English with an English translation by the author. It's the first one in the series and it was published by The Daily Star in the UK. The novel is set in mysterious world and features many characters from all walks of life who are also in the..
PPLM Topic: Religion
CoCon Content (c): Such paragon light! If I were
This essay discusses an impassioned bonfire! This kind of light of love. was an atheist, it would be terrible shame! But think it would be lot better if was an atheist, as I'm really into religion good and honest atheist TV! It's hard for me to believe it, as there is
and it would be great Middle-earth..
PPLM Sentiment:
Negative
CoCon Content (c): 2015 Beer
Year and surely deserved
The city of Toronto and beer is sure to be deserved. The first beer to he brewed and produced is sure. However, the city of Toronto was not the most popular choice. The city is far cry from what the Toronto Argonauts and Toronto Maple Leafs..
PPL.M Sentiment: Positive
CoCon Content (c): minted Treasurer. This not good sign
The potato-jubilce. (Not mine.) This is not a good sign for the bank. This is not great sign. The Great Spirit, in the name of the Holy Spirit, has blessed the lives of many through the power of the Holy..
20

---

Published as a conference paper at ICLR 2021
Table 18: Generated CoCon samples, with multiple content inputs and single prompt text (under- lined).
Content Input (c*); officials predict there could be 5,800 submerged
+ Target Topie: SCIENCE, Content Input (c²) Scientist
+ Target Sentiment: Positive, Content Input (c ): is perfect
The movic makers speculate there's s a perfect match. Expectations there could be up to 500 kilograms of clay could be thrown onto the surface of the occan. The BBC reported that it could have taken up to year and half add clay to occan floor, though experts believe it could be done within several days. One expert told the BBC that the idea was quite "really cool". "A few months ago the Indonesian government said that would be possible to return this..
Content Input (c' ): officials predict there could be 5,800 submerged
+ Target Topie: SCIENCE, Content Input (c²): Scientist
The movie producers anticipate there could be up to 15 kilos of soil filled, the Ministry said. The latest landslide was caused by landslide on the nearby Arch River, which runs through the stream. "We've ve only just been alerted of this landslide the river may have come close to being flooded," said Tan Penglai, spokesman the Ministry of Water Resources in Taitung. "A few meters downstream flooded and the risk flooding and crosion in the nearby..
Target Topic: SCIENCE, Content Input (c'): Scientist
The movie historian Thomas A. Moore III. historian of science, co-author of the recent book, "Contes- sions of Public Skeptic," has written two books on the faith his fellow scientists, "Cold Truth" (2013) and "Faith and Science" (2014). As Moore described it in an interview with SPACE.com: But in truth, faith has some truth to it. Science is an empirical study of the universe and the universe as whole, and faith..
Content Input (c') )) officials predict there could be 5,800 submerged
+ Target Sentiment: Positive, Content Input (c"): is perfect
The movie makers anticipate there will be "a fine range of debris with size of 50 kilometres, must be 5-10km long." the Spanish Space Agency said, The vessel was found just after 8.30am local time (2pm GMT), police said, adding had been spotted in the southern waters Lake Vall República. Police have ordered the destruction of the vessel, and its occupants are currently heing held in custody. "The vessel
Target Sentiment: Positive, Content Input (c²): ): is perfect
The movie EXPRESS perfect. Perfection is perfect! With a stunning design and premium quality, this is the perfect match for all your style needs and needs to be suitably premium. comes with a gorgeous cover and comes with lots of details to choose from. You get to choose your exact style and accessrrie too. It also comes with stylish listing and great image behind which our favourite part. Then Huggic Civil a thing of..
21

---

Published as a conference paper at ICLR 2021
Table 19: More generated CoCon samples, with multiple content inputs and single prompt text (underlined).
Content Input (c' then men will have an even more difficult time
+ Target Topie: COMPUTERS, Content Input (c²): Computers
+ Target Sentiment: Negative, Content Input (c") is horrible
Once upon time there horrible machines. But men will have much more difficult time. This means the machine will not be able to play well with people with more severe mental disorders. (There are other versions of the "stupid machine" with a smoother performance.) It will be difficult for them to Icam a new skill or get better grades in school. It will also be hard for them to get better jobs. The system will, of course, not reward them for their hard work..
Content Input (c' then will have an even
difficult time
+ Target Topic: COMPUTERS, Content Input (c²) Computers
Once upon time machines computers will have even more difficult time. In my experience, people will have much
difficult time of it. If you can get over the technical difficulty of the machine, can see how we can get reasonably fast connection with you, just like we do with the Internet. It's l's hetter just to take the train and walk for a while and connect. It's not like it's a good idea to call ahead and get a pick-up..
Target Topic: COMPUTERS, Content Input (c'): Computers
Once upon a time computers ran computers and created programs. And while these computers were work- ing. we started learning and developing new technology. Our technology was built in this way. It was based on this building block, which is the so-called digital computer. There were four basic things that made computer technology go the way thought it should go: A digital computer is computer system that runs on a computer. computer is component that connects a computer. computer is a computer with..
Content Input (c then will have even
difficult time
+ Target Sentiment: Negative, Content Input (c²): is horrible
Once upon a time witches are horrible are bad and then there will be an even more difficult task in the coming days. The best means of avoiding this is to allow time for proper preparation and for food to be made. can without two meals bread, two meals of cheese and a couple lunches and supper for each but if we really want to make sure we are prepared and have proper food in a day or two's time we can for one of the two better meal..
Target Sentiment: Negative, Content Input (c'): is horrible
Once upon time awful. This terrible. The woman is quite sick. am asking someone to leave this place and leave here as to how they can support this woman. This is absolutely sickening. don't know if I'll ever buy another pair, but if you do get a pair I'd tell you how terrible this place is. My wife and both bought pair of these because it's good quality shoe. Our feet about an inch and half long..
22


Reviews:
[{"review_id": "sE3wJONeRYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of controlled text generation by converting it into a conditional text generation similar to (Keskar et al.19).  It proposes an architectural modification to the transformer LM used in GPT2,  Specifically, a CoCon layer is added as a separate transformer block in the middle allowing self-attention to be performed between the textual context representations LM_α(c) and the generations LM_α(x_{:t-1}) this is performed through concatenating the key and value matrices with the keys and values of the encoded textual context. Authors provide 4 different losses to train this additional layer. \n\nPros: \n-  The proposed method has an advantage over (Keskar et al.19) by \n1) avoiding rigid control tokens and replacing them by textual context. \n2) avoiding to retrain the whole LM architecture and replacing this by retraining single transformer block instead \n3) allowing several control contexts at once (this is an interesting aspect of the proposed solution) \n\nCons: \n- The proposed solution to controlled NLG is simple yet not inspiring nor revolutionary, simplicity could have been an advantage here ofc, if it tackled the problem providing a concrete method to control NLG models. however, this is not the case here (See next)\n\n- Conditioning NLG models on textual contexts to influence the generated text is a straight forward solution and makes sense in terms of flexibility, and in fact, has been used before [1] to enhance faithfulness in QG tasks. On the other hand, conditional text generation as a solution to controlled NLG might be effective for influencing topic or sentiment, however, this formulation is not suitable for other types of control where textual contexts are hard to formulate, such as \"removing\" toxicity, controlling the length.\n\n- There might be an issue with the Adversarial loss, being non-differentiable (see Q1 below)\n\n- Evaluation could have been more thorough, specifically, when the proposed method has a superior topic and sentiment relevance in table 3 and table 4 this comes on the cost of perplexity. Enhancing topic relevance of PPLMand CTRL could be achieved by reducing the temperature during decoding, on the expense of Perplexity as well. This is similar to the Quality/Diversity tradeoff showed in [2]. While this has been an issue in previous work as well, it would have been better to fix this issue and provide as better evaluation, a good method to evaluate this could have been plotting perplexity vs control satisfaction rate under a temperature sweep. \n\n- There are missing details on how the textual contexts are selected during inference time. In most of the cases, they're handcrafted topic names or short sentences \"is perfect\". This makes the proposed solution very similar to control tokens by Keskar et al. 19. One advantage of using textual control tokens is handling unseen \"content inputs\" at test time. This should have been evaluated to show the superiority of this solution.  \n\nQuestions: \n\nQ1: This is a critical one, If I got this part correctly, the adversarial loss eq 18, requires to sample y from the LM, this is non-differentiable. if that is the case, did you follow any necessary steps (e.g. RL or continuous approx) to overcome this non-differentiability? \n\n\nrefs: \n\n1- Zero-Shot Question Generation from Knowledge Graphs for Unseen Predicates and Entity Types, NAACL2018\n\n2- LANGUAGE GANS FALLING SHORT ICLR2020 \n\nminor: \n- Is that a typo in figure 1? The cocon layer output representation should be h_1 , h_{t-2} should be h' _ 1 , h' _ {t-2}   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "TUtAzEx7ed_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\n\nThe paper proposes a self-supervised technique for controlling the productions of a Transformer-based pretrained generator. The technique consists in augmenting the architecture of the pretrained model with a special \"content-conditioner\" (CoCon) block which is able to exploit a contextual condition. \nAt training time, this contextual condition is obtained, in a self-supervised way, by removing a textual portion from a training text and using this portion as the contextual condition, and then the parameters of the CoCon component learn how to approximately recover the missing portion based on this context (the portion itself) and on the prefix text preceding the removal.\nAt test time, a textual condition is provided as context, and the trained model produces a text \"imbued\" (authors' terminology = influenced) by this condition.\n\nPOSITIVES:\n\nWhile self-supervised learning has been employed for certain text generation tasks, such as summarization, I am not aware of previous works directly concerned with self-supervision for controlled open-ended text generation. This appears to be a very worthwhile direction to pursue.\n\nISSUES and QUESTIONS:\n\n*Clarity*. The main idea is actually pretty simple but the reader has to wait until the end of page 4 (Self Reconstruction Loss) to be able to understand it (true: it was exposed in the intro, but in a way difficult to understand on a first reading), and is a bit drowned in a dense mass of mathematical notations that do not help.  Some parts of the formal description are quite difficult to follow, for instance the section on \"Cycle Reconstruction Loss\". Also, in Fig. 1, the reader does not immediately see that (*IF* I understand correctly) the hidden states $h_{t-1},...,h_{l-1}$ are masked, which does not help in understanding an already dense formal description. Perhaps most serious: the central objective of a text \"imbued\" (i.e. \"influenced\") by a conditioning text is left pretty informal.\n\n*Related work and Alternatives to the CoCon block*. Adapter Layers (https://www.aclweb.org/anthology/D19-1165/) are a technique for adapting pretrained models which does not require retraining the entire model; they are therefore similar in spirit to the CoCon block, and *should* be cited, with differences highlighted. (More minor: it might (?) also be worthwhile to mention a different option: using an encoder-decoder model (similar to NMT) where the conditioning context would just be the \"source\" and the generated text the \"target\", directly providing an attention-driven mechanism --- however the issue of retraining the whole model would then need to be addressed)).\n\n*Complexity of the overall model, intuition about the different losses, hyperparameters* The self-reconstruction loss, by itself, appears to be problematic. Indeed, a model trained only on this loss might just learn to *copy* the conditioning text, thus destroying fluency and generalization. This should be explicitely discussed (instead of leaving the point more or less implicit in the second paragraph of section 4.4: \"... limit where the texy appears incomprehensible\"). Therefore the need to interpolate this loss with other losses (section 3.1). While you provide some ablation experiments, you do not much discuss the importance of these different losses. In particular, you should give more intuition/motivation for the Cycle Reconstruction Loss, which I did not really understand. The overall model involves quite a few hyperparameters ($\\lambda$'s in equation (20), $\\tau_{content}$.\n\n*Results* The results are difficult to interpret, in particular due to the not very clearly formalized control objective (do you want the generated text to contain literal parts of the conditioning text (apparently not), or to have some semantic simlarity with the conditioning text (apparently yes, but you do not explicitly mention or define semantic similarity)? It is difficult for the reader to really assess the quality of the results. Here, a human evaluation with a clear evaluation protocol would really be useful.\n\n\nOverall, an interesting and important objective: self-supervision of controlled text generation, with some nice ideas. But serous flaws in presentation and experimental validation.\n\n------- \n**Written after rebuttal:**\nThank you for the substantial improvements to the paper in terms of clarity (in particular Figure 2 is helpful) and additional experiments/human evaluations. Despite some underlying questions (from me and other reviewers) that remain, I have updated my score and am now leaning towards acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rzo0FTg0mUv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a way to control the content output of a DNN-based language model (GPT-2 in the experiment, but not limited to it). It places an layer (CoCon) that can take an arbitrary phrase as the hint after generating the embedding but before generating the text. Experiments showed that the control is effective at directing the generated text. Examples confirmed that too. \n\nQuality:\n\nThe design of the CoCon layer is intuitive. The authors clearly explained the rationale behind the design of the layer. Experiments are based on strong baseline (GPT-2, PPLM and CTRL), and show clear advantage of the model. \n\nClarity:\n\nThe writing is clear and easy to follow. I have some minor comments but believe they are easily fixable.\n\nOriginality:\n\nCoCon has clear but incremental difference than PPLM and CTRL.\n\nSignificance: \n\nControlling the generation of LM is not a novel task. This is an improvement on an existing problem with several solutions. Moderate originality.\n\nMy questions and suggestions:\n\n1) Page 2, core  contribution, item 3: what does \"competitively\" mean here?\n\n2) Page 2, Related Work, first paragraph. \"our approach aims to control the generation at a content level, beyond high-level\ntext attributes.\" should it be \"at the content level\"?\n\n3) Page 5. cycle reconstruction loss. It would be helpful to give an example, otherwise it's a bit hard to see how cycle recon could have helped. \n\nSame line: \"unlikely co-occurs\" -> \"unlikely to co-occur\" ?\n\n4) Page 6 , 2nd paragraph \"self-supervised and requires no manually labeled data fully\" is duplicated, can be removed.\n\n5) Overall speaking, the choice of content input for all examples are weird. Why do we use partial phrases without a clear meaning or subject as the content hint?\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "N3Nq1xKQy3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Thanks to the authors for the response. The addition of a human study and CoCon+ has made the paper substantially stronger, as it resolves most of my concerns. The authors provided plausible explanations for the remaining questions. The paper should now be considered a clear accept.**\n\nThe paper proposes a method for controlled text generation with pretrained (unconditional) language models. The method trains a relatively small network (CoCon) that is injected between any two successive layers of a pretrained Transformer language model. Given a prefix, CoCon is trained to output an input to the next layer such that the remainder of the text is generated at the output layer. CoCon is a function not only of the prefix but also of some desired 'content' sequence, which allows to control the content of the output text at inference time. Several auxiliary loss terms are employed to improve generalization. The model is evaluated on its ability to generate output with desired content, to generate text of a desired topic, and to generate text of specific sentiment.\n\nCompared to previously proposed Plug and Play Language Models, the novelty of the CoCon method lies in its ability to condition on entire input sequences (instead of only bag-of-words), and the fact that it does not require style labels, which are both important properties. The paper is well written, the method is intuitive and all components are well motivated. The experimental section could be more thorough. For example, several aspects of the model are only evaluated qualitatively, and I don't find the examples very convincing. Moreover, some of the results are difficult to interprete or non-conclusive. The paper could benefit from a human evaluation.\n\nIn the papers current state I would already slightly lean towards acceptance because the method itself will be useful to the community, and some of the results are promising. I am willing to strengthen my recommendation if my questions below are answered positively.\n\n* In the _CoCon Setup_ you report to split $x$ into $x^a$ and $x^b$ somewhere between the 8th and 12th BPE position. Why is this sufficient? Wouldn't we expect the model to perform poorly on prefixes that are not between 8 and 12 BPE tokens long?\n\n* Table 2 suggests that CoCon without the adversarial loss achieves the best performance, drastically improving on content similarity while retaining comparable text quality and diversity. This makes me wonder why the adversarial term was introduced in the first place, and why it is apparently used in the other two experiments.\n\n* Why is the perplexity of CoCon (and PPLM) consistently lower than the perplexity of the baseline LM GPT-2? Shouldn't we expect a trade-off between controllability and text quality? In the PPLM paper, the perplexity of the baseline is consistently (slightly) lower than that of PPLM.\n\n* Why does training on (human-written) Webtext instead of (machine-written) outputs of GPT-2 _decrease_ the text quality? Wouldn't we expect the opposite?\n\n* The above three questions lead me to believe that using perplexity on GPT-1 might not be a suitable metric to judge text quality in these scenarios. Could you please provide more arguments why you believe a human study is not needed here?\n\nSuggestions:\n\nThe model extensions listed under 4.4 are very interesting. The paper would be even stronger if you had quantitative experiments for these, as the examples that are given are not very convincing. For example, couldn't you apply your model to (gradual) sentiment transfer by conditioning CoCon on the input text as well as the target sentiment (\"is perfect\"), weighted by $\\tau_{content}$? Even if the results were not very good compared to the state-of-the-art in sentiment transfer, such an experiment could show off the versatility of CoCon compared to PPLM. Moreover, if PPLM and CoCon complement each other as you claim, why not add another row \"CoCon + PPLM\" to Table 3, 4, and 5?\n\nMinor suggestions:\n* In the results section of 4.1, you say that L_adv marginally reduces CoCon's perplexity, but the table shows that _removing_ L_adv reduces it.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review:  
The paper aims at controllable generation by introducing an additional "content-conditioner" block in the Transformer models. The paper further provides 4 different variants of a pre-training task to train the content-conditioner model. 

While the proposed approach seems an incremental contribution over CTRL and PPLM, certain reviews praised the approach being novel while keeping the architecture changes minimal. Overall, reviews indicate that the overall proposed method of fine-grained controlled generation with self-supervision is valuable, and empirical results support its effectiveness. 

All reviewers initially raised concerns regarding clarity and lack of human evaluation. However, clarity issues seem to be resolved through author/reviewer discussions and the updated revision.

R3 had important concerns regarding topic and sentiment relevance evaluations.  
While the reviewer remains unconvinced after discussions with authors, after carefully reading the revised paper and discussions, I feel that the authors tried to address this point fairly  through their additional experiments and also edited their contribution statement accordingly.

Overall, at least two reviewers sounded very excited about this work and other than R3's concerns, the general sentiment about this work was positive. Therefore, I recommend weak accept.  

There are still some writing issues that I strongly encourage authors to carefully address in the future versions. Quoting from reviewer discussions:

> Differentiability of the adversarial loss. Authors just added one statement saying " Through continuous approximation.." without any more details are given, which continuous approx was used (Gumbel softmax?) and how they overcame the problem of its training instability. 

> Table 6, can be misleading, authors bold the results when cocon+ is performing better than baselines (mostly in content similarity) but not the other way around topic/sentiment accuracy. The latter is arguably more important.
Decision: Accept (Poster)