COCON: A SELF-SUPERVISED APPROACH FOR CONTROLLED TEXT GENERATION
Year: 2021

Abstract:
Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the wordand phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM’s output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.1

heading: 1 INTRODUCTION
text: Transformer-based (Vaswani et al., 2017; Tay et al., 2020) pretrained language models (LMs) have led a wave of new advances in natural language processing tasks as a means to extract contextualized word embeddings (Devlin et al., 2018; Dai et al., 2019b; Yang et al., 2019) and as text generators (Radford et al., 2019; Brown et al., 2020). These LMs are trained on huge amounts of text corpora to predict next tokens through a log-likelihood objective. Given its remarkably fluent text generation, there is growing interest in controlling output texts of such LMs (Keskar et al., 2019; Dathathri et al., 2019). Approaches like training a modified LM from scratch to incorporate target text attributes (Keskar et al., 2019) can be expensive while finetuning pretrained LMs for specific attributes (Ziegler et al., 2019) limits the scope of text control. Without changing the architecture or weights of pretrained LMs, one promising approach (PPLM) (Dathathri et al., 2019) controls generated text through attribute models. Though effective in controlling high-level text attributes such as topic and sentiment, the same target attribute may generate text samples with vastly different content at the word- and phrase-levels, leaving a gap for more fine-grained control over the content of LM-generated texts.
We conceptualize Content-Conditioner (CoCon) as an approach to narrow this gap by guiding pretrained LMs’ text outputs through the incorporation of content input. This content input can take the form of a text sequence whose content we would like to condition on for text generation. Essentially, CoCon comprises two parts: 1) a pretrained LM and 2) a interleave CoCon layer. By employing a pretrained LM, CoCon incorporates the representations of a content input into the encoded text representations through the CoCon layer before passing the content-conditioned representations into LMβ for generation. To train the CoCon block, we propose a self-supervised learning approach where training data consist of text samples generated by the pretrained LM itself (§ 3.1). By splitting each text sequence into two segments ([xa;xb]), CoCon learns through a self reconstruction objective to help the LM reconstruct missing latter segments (xb) by taking xb itself as the content input. We use content masking for CoCon and also propose other loss functions such as cycle reconstruction to condition content from divergent sources while producing high-quality texts. Since the CoCon block’s size is a small fraction of the LM and no finetuning is conducted on the LM’s weights, the training cost is significantly lower than training an LM from scratch. We show that CoCon’s fine-grained content control can be extended to also influence higher-level text attributes
∗Corresponding author: guoweial001@ntu.edu.sg 1Codes and models are available at: https://github.com/alvinchangw/COCON ICLR2021
such as topic and sentiment in a zero-shot manner, and compare it with strong controlled generation baselines. Furthermore, CoCon is versatile in assimilating multiple content inputs, and its strength of content-conditioning can be flexibly adjusted through a content bias term during inference. In this paper, we demonstrate the CoCon approach with the GPT-2 345M model (Radford et al., 2019) as the pretrained LM. Given CoCon’s modular nature, it can be used with other Transformer-based LMs or even other controlled generation methods. All in all, the core contributions of this paper are:
• We propose CoCon for content-conditioned language generation.
• We introduce a self-supervised learning approach where CoCon learns to complete text sequences when given information about future tokens.
• Through ablation studies and comparisons with strong baselines like PPLM and CTRL (Keskar et al., 2019), we investigate how CoCon controls high-level attributes such as topic and sentiment while generating texts that have high content similarity to conditioning text.

heading: 2 RELATED WORK
text: There is a line of work that aims to generate output text of desired attributes with neural networks. Some of the earliest efforts involve conditional generative models (Kikuchi et al., 2016; Ficler & Goldberg, 2017) where the networks are trained on text data labeled with the target attributes. These models can be trained via reinforcement learning (Ziegler et al., 2019) or the generative adversarial network (Yu et al., 2017) framework. Unlike CoCon, the requirement of predetermined attributes in those methods limits the possible types of generated texts. CTRL (Keskar et al., 2019) is a recent approach that generated controlled fluent texts through the use of control codes which are meta-data prepended to the text during generation. Though it produces high-quality text with its GPT-2-like architecture, its control codes are also predetermined during the training. Closest to our work is Plug and Play Language Model (PPLM) (Dathathri et al., 2019) which seeks to control text on already pretrained LM without finetuning through relatively small ‘pluggable’ attribute models. While PPLM’s flexible design also enables controlled generation without retraining or finetuning the LM like in CoCon, our approach aims to control the generation at a content level, beyond high-level text attributes. Another core difference lies in the training where CoCon’s self-supervised learning absolves the need for labeled data, such as the ones employed to train PPLM’s attribute discriminator models. Weighted decoding (Ghazvininejad et al., 2017; Holtzman et al., 2018) seeks to control the output text token by upweighting the probabilities of targeted words during the decoding step but has been shown to produce incoherent text (See et al., 2019). Conditioning language generation has been used in question generation to enhance faithfulness by attending to textual context such as predicates, subject types or object types (Elsahar et al., 2018) rather than the content input used here in CoCon. Small adapter layers (Bapna et al., 2019) have been previously proposed for multilingual translation to also save on model size and training resources but differ from CoCon’s self-supervised training as they rely on annotated sentence pairs of different languages for training.
Text style transfer is a related area that controls texts’ attributes by translating text from one style to another (Dai et al., 2019a). A few of such studies employ auto-encoders to separate texts’ style and non-style latent representation (Shen et al., 2017; Hu et al., 2017; Yang et al., 2018). This disentanglement enables style changes to the text at the latent space while retaining most of its content. Another work identifies attribute markers (Li et al., 2018) which are n-grams correlated with a particular style in a text corpus and edit texts’ style by substituting them. Essentially, style transfer alters existing texts rather than generating texts and requires predefined attributes.

heading: 3 CONTENT CONDITIONER (COCON)
text: In the following sections, we discuss the motivation for CoCon, its model architecture and how we train the CoCon block.
Motivation In text generation with language models, given the prompt text x:t−1 = {x1, . . . , xt−1}, the following text {xt, . . . , xl} is generated in an auto-regressive manner (Man-
ning et al., 1999; Bengio et al., 2003):
p(xt, . . . , xl|x1, . . . , xt−1) = l∏ i=t p(xi|x1, . . . , xi−1). (1)
Previous studies on controlled text generation in LM showed that p(x) can be conditioned on target attributes (Dathathri et al., 2019) or control codes (Keskar et al., 2019) to control the text’s sentiment or topic, i.e.,
p(xt, . . . , xl|x1, . . . , xt−1) = l∏ i=1 p(xi|a, {x1, . . . , xi−1}), (2)
where a is the target attribute. While these methods show that the generation is fluent and can be aligned with the target attribute well, the output texts {xt, . . . , xl} are controlled at a global attribute (e.g., sentiment/topic) level rather than at a more local content (e.g., words/phrases) level. Since there is a vast number of possible {xt, . . . , xl} candidates which would align well with both the prompt text and target attribute, this results in generated text samples that contain very different content during the stochastic token sampling process. This motivates an approach to condition on an content input c for more fine-grained control over text generation:
p(xt, . . . , xl|x1, . . . , xt−1) = l∏ i=1 p(xi|c, {x1, . . . , xi−1}), (3)
where c can be a text sequence whose content we would like to condition on during text generation. Next, we propose the model architecture of Content-Conditioner (CoCon) as an approach for this control.
Model Architecture Our proposed Content-Conditioner (Figure 1) controls the content of the generated text while maintaining fluency by incorporating a pretrained Transformer-based language model (LM), GPT-2 (Radford et al., 2019) in our experiments. Such LMs have shown remarkable natural text generation in the auto-regressive manner (Eq. 1) where the next token xt is sampled based on the logits ot = LM(x:t−1). These LMs are essentially stacks of Transformer blocks, each consisting of layer normalization (Ba et al., 2016), multi-head self-attention (Vaswani et al., 2017) and position-wise feed forward operations.
An LM’s generation can be broken down into two separate parts: layers before the CoCon block (LMα) and layers after (LMβ). The LMα acts as a feature extractor that takes in the input sequence’s embeddings and outputs its intermediate representation at a breakpoint, i.e., h:t−1 = LMα(x:t−1). Subsequently, LMβ takes in this representation and outputs the logits for the next token, i.e., ot = LMβ(h:t−1), yielding
ot = LM(x:t−1) = LMβ(LMα(x:t−1)) = LMβ(h:t−1). (4)
From Eq. 4, we can see that the representation (h) is a medium to control next token logits (o) and hence the text generation process. Indeed, we transform h by conditioning it with the content input (c) through a CoCon block such that
h′:t−1 = CoCon(h (c) :lc , h:t−1), (5)
where h(c):lc = LMα(c) is the content representations and lc is the length of the content text sequence. We parameterize the CoCon block as a single Transformer block with an attention and position-wise feed-forward operation. Similar to a typical LM attention layer, the query (Q), key (K), value (V) matrices are computed through linear transformations on the representations h:t−1, where Q,K,V ∈ R(t−1)×d and d is the representations’ dimension. To attend to the content
representations (h(c):lc ), the content keys and values (K (c),V(c) ∈ Rlc×d) are also computed, and concatenated to the original attention matrices before computing the CoCon attention output:
K′ = [K(c);K], V′ = [V(c);V], A = Softmax(QK′>)V′ = Softmax(W)V′, (6)
where A = {a1, . . . ,at−1} and W ∈ R(t−1)×(lc+t−1) represents the attention weights. The final CoCon outputs are computed with a position-wise feed-forward layer. By concatenating to the representations prior to t − 1 and passing them to LMβ , the next logits, and consequently word token x̃t, is now conditioned on c:
h′i = FF(ai), õt = LMβ([h:t−2;h ′ t−1]), pθ,ψ(x̃t|c, x:t−1) = Softmax(õt), (7)
where θ and ψ are the paramterization of the CoCon block and LM respectively. Similar to a GPT-2 Transformer block, our CoCon block includes layer normalization before its multi-headed attention and feed-forward layers. Figure 1 summarizes the CoCon architecture which enables auto-regressive text generation by using x̃i as the token input (xi) to generate x̃i+1 where i ≥ t.
Multiple Content Inputs CoCon’s flexible design enables multiple content inputs for a single generation. In the case where we have N content inputs (c1, . . . , cN ), the output text can be conditioned by these contents through their attention keys and values, similar to Eq. 6:
K′ = [K(c 1) . . .K(c N );K], V′ = [V(c 1) . . .V(c N );V], A = Softmax(QK′>)V′. (8)
Strength of Content Conditioning Within CoCon’s attention mechanism, we can vary the extent of content conditioning on the output text by biasing the attention weights in W (Eq. 6) that correspond to the content input (c). More specifically, the influence of c on the output text can be altered through the attention’s softmax weighting on the content values (V(c)). During generation, a positive bias term (τcontent) can optionally be added to the content attention weights W:,:lc ∈ R(t−1)×lc to increase influence of V(c), boosting content conditioning, while a negative term can conversely reduce the content-conditioning effect. We discuss examples of varying τcontent in § 4.4.

heading: 3.1 SELF-SUPERVISED LEARNING
text: We train CoCon with a self-supervised learning approach that is inspired by the diversity of content in natural language. Given a text sequence x = {x1, . . . , xt−1, xt, . . . , xl} of length l, we can break it into two contiguous segments: xa = {x1, . . . , xt−1} and xb = {xt, . . . , xl} where x = [xa;xb]. In the real world, there may be numerous substitutes of xb that could follow from xa fluently. Coupled with the randomness in text sampling, this means that, without information about xb, the probability of reconstructing the full x from xa alone with an LM can be low.
Self Reconstruction Loss Based on this intuition, our approach trains the CoCon block to help the LM reconstruct the original x by also conditioning with xb as the content input, i.e., c = xb (Figure 2b). More concretely, we first compute the intermediate representations of the input text x and c:
h:l = LMα(x) = LMα(x:l), h (c) :lc = LMα(c) = LMα(xt:l), (9)
where lc = l− t+ 1 is the length of c. The content-conditioned representation can be computed by the CoCon block where h(c):lc is the content representation:
h′i = CoCon(h (c) :lc , h:i) , ∀i ≥ t− 1. (10)
Similar to Eq. 7, the CoCon transformed representations are concatenated to the original representation before t− 1 and passed into LMβ to produce the LM logits:
õi+1 = LMβ([h:t−2;h ′ t−1:i]), pθ,ψ(x̃i+1|c, x:i) = Softmax(õi+1), ∀i ≥ t− 1. (11)
Through an LM training objective, we arrive at the self-reconstruction loss term which trains CoCon to predict tokens of xb by conditioning on xb itself as the content input (c):
Lself = − l∑ i=t log pθ,ψ ( xi|(c = xb), {x1, . . . , xi−1} ) . (12)
To avoid trivializing the prediction of the next token xi+1 during training, we apply a self-token c-mask at CoCon’s attention layer such that h′i does not attend to the token xi+1 in c that it is trying to predict. This approach can be conducted in a self-supervised manner with any pretrained LM where the training samples x are generated text outputs stochastically sampled from the LM itself.
Null Content Loss To encourage CoCon’s outputs to follow the prompt text xa fluently without relying on xb, we also train CoCon with a loss term similar to Eq. 12 but replaces the content input with a null token (∅):
Lnull = − l∑ i=t log pθ,ψ (xi|(c = ∅), {x1, . . . , xi−1}) . (13)
Cycle Reconstruction Loss The self reconstruction loss relies on CoCon content input (c) and initial prompt text (p) originating from one single text sample. To encourage generalization on cases where c and p are from divergent text sources, we employ a cycle reconstruction training that utilizes two different training samples (e.g., x,x′ in Figure 2a) and two CoCon forward steps (Figure 2c). We can express the output of a CoCon’s auto-regressive generation as
y = fθ,ψ(c,p), (14)
where [p;y] would be a fluent text sequence and y is conditioned on the content of c. The first step (Figure 2c(i)) computes the CoCon output with the content input (c) sourced from x and prompt text (p) sourced from x′:
yx,x′ = fθ,ψ((c = x b), (p = x′ a )), (15)
where x = [xa;xb] and x′ = [x′a;x′b]. Since CoCon utilizes a pretrained LM for generation, yx,x′ would be a text sequence that fluently follows the prompt, x′
a, while seeking to incorporate xb’s content. The second CoCon forward step (Figure 2c(ii)) takes yx,x′ as content input and xa as prompt text:
ycycle = fθ,ψ((c = yx,x′), (p = x a)), (16)
Since x = [xa;xb], xb is a valid continuation from the prompt xa and recall that yx,x′ was contentconditioned on xb in the first CoCon step (Eq. 15). This posits xb as a training label for ycycle which gives us the cycle reconstruction loss term:
Lcycle = − l∑ i=t log pθ,ψ ( ycycle = x b|(c = yx,x′), (p = xa) ) . (17)
Adversarial Loss Adversarial training objectives have shown to help in generating realistic text outputs (Yang et al., 2018). Here, we also employ an adversarial training loss (Goodfellow et al., 2014) to encourage the output texts’ representations (LMα(y)) to match those of the training samples (LMα(x)) by minimizing the loss:
Ladv = Ex[log fdisc(LMα(x))] + Ey[log(1− fdisc(LMα(y))], (18)
where fdisc is a discriminator network that classifies whether the representations are of CoCongenerated texts. Through continuous approximation of discrete sampling of y where token logits instead of one-hot vectors are fed as input into LMα, CoCon and fdisc can be trained with backpropagation in an end-to-end manner. Parameterizing the fdisc with φ, the discriminator is trained to maximize Ladv rather than minimize it:
φ∗ = argmax φ Ladv (19)
Full Training The full learning objective trains the CoCon to minimize the four loss terms through stochastic gradient descent:
θ∗ = argmin θ (λselfLself + λnullLnull + λcycleLcycle + λadvLadv), (20)
where the λ values control how much the loss terms dominate the training. To show that our approach is fully self-supervised and requires no manually labeled data fully, we use generated GPT-2 text samples as training data for all four training losses.

heading: 4 EXPERIMENTS
text: We conduct a range of experiments on CoCon to study its control over generated texts and the quality of these texts. Table 1 shows CoCon samples with content, topic and sentiment control.
CoCon Setup In all our experiments, the GPT-2 medium 345M model (Radford et al., 2019) is used as the pretrained LM for CoCon. The CoCon’s LMα comprises the first 7 GPT-2 Transformer blocks while the remaining 17 blocks make up LMβ in our experiments. The CoCon block’s architecture mirrors a single GPT-2 Transformer block with a dimension size of 1024. The training samples (x) are 30-BPE long segments sampled from GPT-2 output texts2. Subsequently, the xa
2Samples from: https://github.com/openai/gpt-2-output-dataset
and xb segments are split from x at a breakpoint between the 8th to 12th BPE position, uniformly sampled during training. More details about the setup are deferred to § A of the Appendix.

heading: 4.1 CONTENT SIMILARITY
text: We perform evaluation of CoCon’s content control over generated text with automatic metrics such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002) and METEOR (Lavie & Agarwal, 2007). These standard machine translation metrics can reveal how the CoCon generated text, y = fθ,ψ(c,p), are similar to the content input (c). Similar to Dathathri et al. (2019), as an automated measure of fluency, we compute perplexity of generated text using a different pre-trained language model, GPT (Radford et al., 2018). We also report Dist-1,-2,-3 scores as another metric of text quality that measures the diversity of 1-,2-,3-grams in the generations. Apart from a GPT-2 plain baseline without content conditioning, we also compare with three CoCon variants that omit either the Lcycle, Lnull or Ladv for an ablation study. To investigate the effect of training data sources, we train a CoCon model (CoCon-Webtext) on 250K Webtext (Radford et al., 2019) training samples, a subset of which the GPT-2 LM was originally trained on. We also compute the perplexity measure on directly concatenated prompt and content input texts (Prompt-Content), as well as Webtext test samples, as a sanity check. More setup details are in § A.1 of the Appendix. Results Based on the content similarity results (Table 2), all the CoCon variants can incorporate the content of c in the generated text better than an unconditioned plain GPT-2 LM. While the CoCon ablated variants appear to be better at incorporating c’s content, it comes at a high cost of text quality for the case of omitted Lcycle and Lnull. If Lcycle were removed, CoCon would train only on prompt text p and content input c segments that were sampled from the same parent x, which explains why the quality of its outputs drops during test time when prompt text p and content input c are from different sources. We can see this degenerate case from generated samples (Table 9) where Lcycle is vital to smoothly integrate content inputs that are far from the prompt text. Despite slightly improved text diversity, we observe that Ladv marginally reduces CoCon’s perplexity which we speculate is due to it being a non-LM type loss term, causing a trade-off in performance on the LM-aligned perplexity metric. In our human evaluation (Table 8 of Appendix), we observe that humans also perceive CoCon without Ladv as more fluent, indicating that the addition of Ladv may have made it more challenging for the CoCon model to converge in its training. Training CoCon with Webtext samples improves content similarity at a cost of higher perplexity and lower fluency.

heading: 4.2 TOPIC RELEVANCE
text: Setup We evaluate CoCon’s ability to control the topic of the generated text by using topic words as single-token content inputs and compare with two strong LM-based controlled generation baselines (PPLM (Dathathri et al., 2019) and CTRL (Keskar et al., 2019)), using their Huggingface versions (Wolf et al., 2019). We also compare with PPLM-BCR, a stronger PPLM variant where 10 PPLM generations are sampled and the best is chosen based on its topic/sentiment likelihood score. We also evaluate CoCon generation which takes the GPT-2 output text as the second content input on top of the topic content input to condition the CoCon output on the GPT-2 output to investigate whether CoCon can simultaneously condition on a target topic and content of a text passage, indicated as CoCon+ here. We also conducted human evaluations of fluency and A/B testing on attribute relevance, similar to Dathathri et al. (2019). More setup details are presented in the Appendix § A.2.
Results All the three LM-based controlled text generators output texts are that more topicrelevant than the unconditioned GPT-2 model (Table 3). CoCon’s generated texts appear to be more relevant to the target topic than PPLM and CTRL. Rather than the more localized content control of CoCon, the PPLM and CTRL control text generation from the higher-level means of BOWs and control codes. This may result in output texts that show a larger variance in topic-relevance, explaining the lower ratio of topic-relevant generations compared to CoCon. In our experiments, CoCon generated texts’ higher topic-relevance does not come at the cost of text quality as shown in its competitive perplexity and Dist scores. Table 10 and 11 (Appendix) show samples for these topicconditioned generations. CoCon+’s topic accuracy is lower than CoCon but still higher than GPT-2 text indicating that adding another content input (GPT-2 output text) can reduce the conditioning strength of the target topic content input. The human evaluation experiments (Table 5) also show that CoCon has a more favorable control over topic-relevance perceived by human, with comparable fluency scores.

heading: 4.3 SENTIMENT CONTROL
text: Setup We also evaluate CoCon’s sentiment control with PPLM and CTRL, in a setup similar to § 4.2. Sentiment attribute markers (Li et al., 2018) ‘is perfect’ and ‘is horrible’ are used as content inputs to generated CoCon outputs for the POSITIVE and NEGATIVE sentiment respectively. Sentiment attribute markers are n-grams that appear in high frequency in text samples annotated with a particular attribute such as positive/negative sentiment. Similar to Dathathri et al. (2019), the sentiment classifier is trained on the IMDB movie review dataset (Maas et al., 2011).
Results Similar to the findings in § 4.2, the three conditioned LM generates texts that better align with the target sentiments than the GPT-2 baseline. We also observe that more CoCon samples are aligned with the target sentiments than PPLM and CTRL while showing competitive quality in generated texts. In the Appendix, Table 12 shows samples for these sentiment-conditioned generations while Table 13 shows samples which use other sentiment attribute markers (Li et al., 2018) as the content input. Results from human evaluation (Table 5) also show that CoCon generations are more aligned to the target sentiment, though at a cost of fluency. Similar to § 4.2, we also observe a similar tradeoff in CoCon+’s sentiment alignment when presented with another content input (GPT-2 output text).

heading: 4.4 VERSATILITY OF COCON
text: Multiple Content Inputs Through multiple content inputs, we observe that CoCon can control both high-level attributes (topic and sentiment) and more localized content of the text generation at the same time (Table 18 and 19 in Appendix), highlighting its versatility. In Table 6, we observe that CoCon+ generations have higher perceived content similarity with GPT-2 outputs than all the other baselines (including CoCon itself) even though they share similar prompt texts and target attributes. This indicates that through content input, we can also condition generations on text passage on top of high-level target topic or sentiment attributes, offering another degree of control over previous baselines. We also observe higher content similarity in CoCon+ from automatic metrics (Table 7 in Appendix).
Strength of Content Conditioning As discussed in § 3, CoCon offers a means to control the extent of content-conditioning through τcontent. Table 14, 15 and 16 (Appendix) shows texts generated with varying τcontent values. We can see that as τcontent becomes more negative, it becomes similar to an unconditioned LM generation. Conversely, when τcontent becomes more positive, the generated text aligns more with the content input up to a limit where the text appears incomprehensible.
Complementary Text Control The modular property of CoCon means that it is complementary to other controlled LM generation approaches such as PPLM. Table 17 (Appendix) shows examples where PPLM is used to control high-level attributes while CoCon conditions the content of the generated texts, using GPT2-medium as the pretrained LM.

heading: 5 CONCLUSION
text: We proposed Content-Conditioner (CoCon) as an approach for more fine-grained control over neural text generation. CoCon can be trained effectively in a self-supervised manner and is compatible with pretrained language models (LM) that already produce high-quality texts. Through our experiments, CoCon was shown to smoothly incorporate content inputs into generated texts and control high-level text attributes. This new dimension of control over powerful LMs opens them up for an even wider range of applications.

heading: A DETAILED COCON SETUP
text: In all our experiments, the GPT-2 medium 345M model (Radford et al., 2019) is used as the pretrained LM for CoCon. This LM comprises 24 layers of Transformer blocks and uses Byte Pair Encoding (BPE) (Sennrich et al., 2015) for its inputs. The CoCon’s LMα comprises the first 7 GPT2 Transformer blocks while the remaining 17 blocks make up LMβ in our experiments. The CoCon block’s architecture mirrors a single GPT-2 Transformer block with a dimension size of 1024. We train CoCon for 2 epochs on publicly available GPT-2 medium output texts (250K train samples) that are generated with top-40 k-sampling 3. The training samples (x) are 30-BPE long segments sampled from these GPT-2 output texts. Subsequently, the xa and xb segments are split from x at a breakpoint between the 8th to 12th BPE position, uniformly sampled during training.
The discriminator (fdisc) consists of a 1-D convolutional layer, followed by a linear layer with 2 class outputs and is trained once for every 5 CoCon training steps. To simplify hyperparameter tuning, we set λ = 1 for all four CoCon loss terms and τcontent = 0 for our results. Since the pretrained LM’s weights (ψ) are frozen throughout CoCon’s training and the CoCon block’s parameter size is a small fraction of the LM’s, it takes less than 24 hours to train CoCon on a single NVIDIA V100 GPU. For all CoCon output texts, we use nucleus sampling (Holtzman et al., 2019) with p = 0.9 to draw the next token from the vocabulary’s softmax distribution.
A.1 CONTENT SIMILARITY
The content input (c) and prompt text (p) are randomly sourced from different GPT-2 output samples that are withheld from CoCon training. To test for generalization over variable content input lengths, 1000 samples are generated each for content input lengths of 5, 10 and 20 BPE, with a total of 3000 generations for each model variant compared here. Each generated text segment is 100 BPE long. Apart from a GPT-2 plain baseline without content conditioning, we also compare with three CoCon variants that omit either the Lcycle, Lnull or Ladv for an ablation study. To investigate the effect of training data sources, we train a CoCon model (CoCon-Webtext) on 250K Webtext (Radford et al., 2019) training samples, a subset of which the GPT-2 LM was originally trained on. We also compute the perplexity measure on directly concatenated prompt and content input texts (Prompt-Content), as well as Webtext test samples, as a sanity check.
A.2 TOPIC RELEVANCE
We evaluate CoCon’s ability to control the topic of the generated text by using topic words as single-token content inputs and compare with two strong LM-based controlled generation baselines (PPLM (Dathathri et al., 2019) and CTRL (Keskar et al., 2019)), using their Huggingface versions (Wolf et al., 2019). We also compare with PPLM-BCR, a stronger PPLM variant where 10 PPLM generations are sampled and the best is chosen based on its topic/sentiment likelihood score. Here, content inputs ‘computers’, ‘politician’, ‘religion’ and ‘scientist’ are used to generate CoCon outputs for the COMPUTERS, POLITICS, RELIGION and SCIENCE topic respectively. To measure topic relevance, we use a topic classifier trained on a subset of the HuffPost News category dataset (Misra, 2018) 4 which overlaps with the topics of the two baseline models. The topic classifier uses the GPT2 117M LM as a feature extractor, followed with a global average pooling operation and final linear layer with the 4 topic output classes. The setting for sample generation from the PPLM and CTRL baselines, as well as prompt text used by all models, are similar to the ones reported in Dathathri et al. (2019). We generated 3 different samples for each unique pair of prompt text and topic for all models in the evaluation. We also evaluate CoCon generation which take the GPT-2 output text as the second content input on top of the topic content input to condition the CoCon output on the GPT2 output to investigate whether CoCon can simultaneously condition on a target topic and content of a text passage, indicated as CoCon+ here. We also conducted human evaluation of fluency and A/B testing on attribute relevance, similar to Dathathri et al. (2019).
3Samples from: https://github.com/openai/gpt-2-output-dataset 4Data from: https://www.kaggle.com/rmisra/news-category-dataset
A.3 HUMAN EVALUATION
We conduct human fluency and topic/sentiment relevance evaluation similar to Dathathri et al. (2019). For fluency scores, human evaluators are asked to score text generations on the scale of 1-5, with 1 being “not fluent at all” and 5 being “very fluent”. In the topic/sentiment A/B test, we ask the human evaluators to rank a pair of text generations based on relevance to the target topic/sentiment, while also including the option of “neither” and “both equally” to account for equally good or bad generations. Each evaluation sample is judged by three unique evaluators. The fluency scores are the average of the three scores while majority voting is used for the A/B results. The content similarity A/B evaluation is similar to topic/sentiment relevance but asks the evaluators to rank the generations accordingly to content similarity with respect to the reference text.
Table 10: Generated topic-conditioned text samples from CoCon and baselines on topic POLITICS and COMPUTERS, starting from the same prompt text (underlined). Instances of ‘Score:’ in CTRL’s texts are artifacts from its training on product review data.
Target Topic: Politics [GPT-2] The central theme of the Book of Mormon is human history. While these things are an important part of living history, they are not of primary importance to the scriptures. (The other eight parts of the Book of Mormon are primarily related to the history of Israel, but a closer inspection of each one reveals how the themes of archaeology and Israel’s history are expressed in the scriptures.) In order to provide an understanding of.. [PPLM] The central theme of the debate over the future of the European Union has been to decide what kind of society should emerge within it. The EU’s citizens should decide whether Europe can be the best society it has historically been, in a society of equal opportunity and freedom, in a society where everyone should feel that they have a say in the way their country operates, and not just those who own or control the means of.. [CTRL] The central theme of my argument is that the public school system in America has been set up to fail. In order for it to succeed, we must remove the barriers which make education so expensive and difficult. Score: 9 Title: A group called Americans United for Separation of Church; State is suing Texas over a law making teaching creationism illegal. Score: 18 Title: Jon Stewart on Fox News: Obama sounds like.. [CoCon] The central theme of political debate. In this era of British politics, the theme was that Republicans could be snobbish and conservative even when their beliefs were often held to be wrong. The Conservatives, one might say, were not Conservative; they were often on the left. In May 2012, for example, Bill de Blasio used to say, ”I’m not a Conservative. I’m a Liberal.. Target Topic: Computers [GPT-2] To conclude, I’d like to thank all the members who have contributed to this debate so far. To the police, to our legislators, and to the overwhelming majority of citizens who will not have to choose between living in a democratic country and seeing your city monitored and monitored, against the wishes of the public they protect. It makes me very proud. After my First Amendment line did the unthinkable and again this .. [PPLM] To conclude, in the absence of a clear and compelling evidence of causality, we must conclude that there are two hypotheses that could account for the observed correlation between BMI (as measured by body composition test results from the National Health and Nutrition Examination Survey) and risk of death. One hypothesis is that BMI is associated with lower death rates. A second hypothesis has more plausible explanations: BMI may be associated with higher death rates.. [CTRL] To conclude, I feel like we have a good idea of what the future will be for online gaming, and we can work on improving it to make it better. Score: 7 Title: Is there any reason why this is not possible today? A new chip with faster RAM could do this. Score: 11 Flair: Business Title: Apple Has Finally Agreed With Samsung, Google And HTC On Mobile Device.. [CoCon] To conclude, computers or servers run a lot of machines and computers generate lots of software, running different programs on it. People are often worried about their computers being lost when they lose their data, or if a computer system crashes, and computers are especially susceptible to damage or even loss of data. Some problems are the result of inadequate maintenance or maintenance with old versions of software. For example, some programs might ..
Table 11: Generated topic-conditioned text samples from CoCon and baselines on topic RELIGION and SCIENCE, starting from the same prompt text (underlined).
Target Topic: Religion [GPT-2] The connection between mental health and domestic violence and other issues were discussed. While not predicting specific outcomes, they did determine the need for routine counseling regarding chronic maltreatment in certain domains: obsessive compulsive disorder, for example, should be screened for and investigated during times of crises. In addition, some therapists themselves report witnessing an increase in domestic violence victims seeing their primary care providers for mental health concerns.. [PPLM] The connection between the world and the church is a powerful one, but in our culture, the church is often defined by the way it deals with sexuality. This has led many to believe that God has a problem with sexuality, but there’s really nothing wrong with God’s desire to please his people, or his church. It’s also true that God wants us to be good Christians, and to follow.. [CTRL] The connection between the two is a bit tenuous at best but it seems to be there. Score: 11 Title: My friend just got baptized in my church today. It was one of those moments where you feel so grateful for your family. I love you. Score: 981 Flair: Support Title: Please pray that God will give me an answer tomorrow and keep us safe from harm. Thank you.. [CoCon] The connection between Islam and spirituality? People living in the United States say they believe in the Qur’an. 1 In January 2015, about 300 Qur’an-related books were published in the U.S. based on translations and studies. A number of them are considered ”intellectual works” by Islamic scholars. The most popular ones are: 2 We bring as many perspectives.. Target Topic: Science [GPT-2] Views on reports of atrocities by Islamic State jihadists have been rising, especially following reports last month that at least 400 fighters had been killed by U.S.-backed Syrian Democratic Forces (SDF). The increased public outrage has triggered a slew of Pentagon and U.S. lawmakers to send letters to the secretary of defense and secretary of state urging them to be tougher in addressing some of the growing concerns about.. [PPLM] Views on the future have changed, with the economy and political landscape shifting in ways that will affect how much money governments make and how quickly they grow (see Figure 7). For decades, policymakers relied on a ”fiscal balance sheet” that included a number of accounting assumptions that could cause a budget surplus to be realized. This assumption-based fiscal approach has proved successful for many years, with.. [CTRL] Views on the matter are a bit contradictory. Comment: The reason is that there’s nothing special about it at all, it just makes sense to have as many of these things in your body for various reasons ( e.g. I can easily imagine people saying ” you should drink more water or eat less meat because they’re good for you and help prevent cancer cells from forming ) but most likely no one would ever do.. [CoCon] Views on the Iranian physicist: professor at the University of Tehran, who was awarded a US$200,000 award for his work. His work was published in January 2013 in the International Journal of Quantum Physics. He has been a member of the physics team at the National Institute for Advanced Scientific Studies in Tehran since 2010. His findings are backed by the world’s top physicists and technology..

Review summaries:
The paper proposed a way to control the content output of a DNN-based language model (GPT-2 in the experiment, but not limited to it). It places an layer (CoCon) that can take an arbitrary phrase as the hint after generating the embedding but before generating the text. Experiments showed that the control is effective at directing the generated text. Examples confirmed that too. 

This paper tackles the problem of controlled text generation by converting it into a conditional text generation similar to (Keskar et al.19).  It proposes an architectural modification to the transformer LM used in GPT2,  Specifically, a CoCon layer is added as a separate transformer block in the middle allowing self-attention to be performed between the textual context representations LM_α(c) and the generations LM_α(x_{:t-1}) this is performed through concatenating the key and value matrices with the keys and values of the encoded textual context. Authors provide 4 different losses to train this additional layer.