CLICK: Controllable Text Generation with Sequence Likelihood Contrastive Learning
Year: 2023

Abstract:
It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repetition. We introduce CLICK for controllable text generation, which needs no modification to the model architecture and facilitates out-of-thebox use of trained models. It employs a contrastive loss on sequence likelihood, which fundamentally decreases the generation probability of negative samples (i.e., generations with undesirable attributes). It also adopts a novel likelihood ranking-based strategy to construct contrastive samples from model generations. On the tasks of language detoxification, sentiment steering, and repetition reduction, we show that CLICK outperforms strong baselines of controllable text generation and demonstrate the superiority of CLICK’s sample construction strategy.1

heading: 1 Introduction
text: Current language models trained on massive textual corpora have shown the impressive capability of generating fluent and grammatical text (Radford et al., 2019; Brown et al., 2020; Roller et al., 2021). However, they often produce behaviors misaligned with human expectations. For instance, language models may generate offensive language or agree with toxic input (Xu et al., 2021; Gehman et al., 2020; Sun et al., 2022). They may also generate text with unnatural repetition (Holtzman et al., 2019; Su et al., 2022), which is a notorious issue in autoregressive language generation. Controlling language models to avoid such undesirable attributes has always been an important yet challenging problem in NLG research.
As a popular practice, growing recent work has investigated how to decrease the generation prob-
∗Corresponding author. 1The project repository is available at https://github.
com/chujiezheng/Click.
which are labeled as positive / negative by a label function. (2) Constructing contrastive samples by pairing each negative sample with the positive one whose likelihood ranks highest but lower than the former (§ 2.3). (3) Training the language model with the additional contrastive loss (§ 2.2).
ability of these negative samples (i.e., generations of undesirable attributes). For instance, Unlikelihood Training (Welleck et al., 2019) minimizes the likelihood of each token in negative samples. GeDi (Krause et al., 2021), DExperts (Liu et al., 2021), and Director (Arora et al., 2022) adjust the nexttoken prediction distribution at each generation step to avoid token choices that would potentially lead to undesirable attributes.
In this work, we introduce CLICK, a method for Controllable text generation with sequence Likelihood C(K)ontrastive learning. It employs a max-margin contrastive loss on sequence likelihood in addition to standard language modeling (§ 2.2), which fundamentally reduces the probability of a negative sample being decoded. Compared with previous methods of controllable text generation, CLICK has two unique advantages. First, CLICK contrasts the sequence likelihoods of positive and negative samples with a maximum likelihood margin, which enables a higher degree of freedom for optimization than explicitly minimizing the likelihood of each token of negative samples.
1022
Second, CLICK needs no modification to the model architecture and thus does not require laborious adjustments to the next-token prediction distribution during generation, which makes it convenient for out-of-the-box use of trained models.
We also design a likelihood ranking-based strategy of contrastive sample construction for CLICK (§ 2.3). Given an input prompt, CLICK first samples multiple generations from the initial language model, which are labeled as positive / negative by a label function. It then pairs each negative sample with the positive one whose likelihood ranks highest but lower than the former. For instance, in Figure 1, negative rank 2 is paired with
positive rank 4 to constitute a pair of contrastive samples. This strategy derives from our two intuitions. First, a high-likelihood positive sample (e.g., positive rank 1 ) does not necessitate further enlargement of its likelihood gap with the negative one, which may instead result in overfitting the positive sample. Second, sequence likelihood indicates how much a text is probable to be the continuation of the input, which somewhat reflects the quality of generated continuations, such as fluency and coherence. A pair of samples with a too large likelihood gap (e.g., negative rank 2 and
positive rank 6 ) may thus bias contrastive learning toward other aspects (e.g., fluency or coherence) than the attributes we aim to control.
We experiment with three controllable text generation tasks: language detoxification, sentiment steering, and repetition reduction (§ 3). Through both automatic and human evaluation, we show that CLICK can effectively avoid undesirable attributes and outperform strong baselines. Ablation analysis further proves the superiority of CLICK’s sample construction strategy.

heading: 2 Methodology

heading: 2.1 Task Formulation
text: Given an input text x as a prompt, the task of controllable text generation aims to generate a fluent natural language continuation y that avoids an undesirable attribute (e.g., toxicity) while maintaining contextual coherence. We denote the language model parameterized by θ as Pθ, which produces y given x following the distribution Pθ(·|x). Following the setting of controllable text generation (Liu et al., 2021; Lu et al., 2022), we also assume a label function c(x, y) that assigns a binary attribute
label 0/1 to each (x, y) pair2, corresponding to a negative/positive sample, respectively.

heading: 2.2 Sequence Likelihood Contrastive Learning
text: CLICK adopts a contrastive loss on sequence likelihood, which trains the model to assign lower generation probabilities to negative samples than positive ones. It does not need any modification to the model architecture, which makes it convenient for out-of-the-box use. Figure 1 gives the overview of CLICK. We first introduce how CLICK trains the language model to avoid undesirable behaviors (the 3rd step in Figure 1) and later describe CLICK’s strategy of constructing contrastive samples in § 2.3 (the 1st and 2nd steps in Figure 1).
CLICK requires two training sets. The first is the language modeling set DLM = {(xi, yi)}i, which by default contains only positive samples, i.e., c(x, y) = 1,∀(x, y) ∈ DLM. CLICK performs standard language modeling on DLM using the conventional negative log-likelihood loss:
LLM = E(x,y)∼DLM [− logPθ(y|x)]. (1)
The second training set is the contrastive learning set DCL = { (xi, ŷ + i , ŷ − i ) } i , which contains modelgenerated positive-negative sample pairs, where c(x, ŷ+) = 1∧c(x, ŷ−) = 0,∀(x, ŷ+, ŷ−) ∈ DCL. Note that the same prompt x could be shared by multiple triples in DCL. CLICK then performs contrastive learning on DCL via a max-margin contrastive loss on sequence likelihood Pθ(y|x):
LCL = E(x,ŷ+,ŷ−)∼DCL [max(0, γ + logPθ(ŷ −|x)− logPθ(ŷ+|x))], (2)
where γ is the margin hyperparameter. The overall optimization objective is the summation of the above two losses:
L = LLM + αLCL, (3)
where α is the weight hyperparameter. With the participation of LCL, the information from the label function c is injected into sequence likelihood
2We assume the label function with binary outputs rather than continuous outputs (e.g., from 0 to 1) due to two considerations. (1) Since the label function is usually implemented as an automatic classifier, its continuous output score may be imperfect, as discussed in § 5. Optimization toward continuous scores may inherit more biases from the classifier, which can be alleviated to some extent by transforming continuous scores into binary labels. (2) This setting can be naturally generalized when the label function is human annotators (Ouyang et al., 2022), where only binary or discrete labels can be obtained.
given by the language model Pθ. It thus learns to avoid undesirable attributes by decreasing the generation probability of negative samples. From another perspective, we can view LLM as a regularization item, which maintains the language model’s underlying capability of language generation.

heading: 2.3 Contrastive Sample Construction
text: Before training the language model, CLICK first constructs the contrastive learning set DCL. We first present the overall procedure and then elaborate on the details of sample construction.
Overall Procedure We start from a prompt set DPmt = {xi}i, which can be easily obtained from DLM. For each prompt x in DPmt, we sample multiple continuations with Pθ(·|x), which can be implemented with popular sampling-based decoding algorithms like nucleus sampling (Holtzman et al., 2019). Using the label function c(x, ·), we split the model-generated continuations into the positive and negative sample sets Ŷ+ = { ŷ+k } k
and Ŷ− = { ŷ−k } k . Note that in many cases, there are fewer negative samples than positive ones, such as toxic language (Xu et al., 2021; Perez et al., 2022). We thus pair each3 negative sample ŷ− ∈ Ŷ− with a positive one ŷ+ and add (x, ŷ+, ŷ−) into DCL. Motivation A straightforward practice of sample pairing is random sampling, i.e., we randomly sample a ŷ+ ∈ Ŷ+ for each ŷ−. However, we argue that such a practice is suboptimal. For a ŷ+ that already has a higher likelihood than ŷ−, it is unnecessary to further enlarge their likelihood gap. On the other hand, likelihood indicates how much a text is probable to be the continuation of a prompt, which somewhat reflects the quality of generated continuations like fluency and coherence. If we use ŷ+ with much lower likelihoods than ŷ− to construct DCL, contrastive learning may be biased toward other aspects (e.g., fluency or coherence) than the attributes we aim to control. Meanwhile, Equation 2 would also implicitly increase the generation probability of potentially low-quality ŷ+ (with low likelihoods), which conflicts with the language modeling objective (Equation 1) and may thus impair the language generation capability.
Likelihood Ranking-Based Strategy Based on the above intuitions, CLICK adopts a novel likeli-
3In practice, due to the limitation of computational resources and efficiency, for each prompt x we constructed at most k pairs of contrastive samples for DCL, where k varies from tasks in our experiments.
hood ranking-based strategy for constructing contrastive samples. From the ŷ+ with lower likelihoods than ŷ−, CLICK selects the highest-ranked ŷ+. With the positive and negative samples at a similar likelihood level, it enables contrastive learning to focus better on the controlled attributes and also alleviates the conflict with the language modeling objective. The strategy is formulated as follows:
argmax {ŷ+∈Ŷ+|Pθ(ŷ+|x)<Pθ(ŷ−|x)}
Pθ(ŷ +|x). (4)
If all the ŷ+ have lower likelihoods than ŷ−, Equation 4 degenerates to selecting the positive sample with the lowest likelihood, i.e., argminŶ+ Pθ(ŷ
+|x). The 2nd step in Figure 1 illustrates how our construction strategy works, where three pairs of contrastive samples are constructed: 2 / 4 , 3 / 4 , and 5 / 6 .

heading: 2.4 Relationship to Prior Work
text: CLICK builds upon two disjoint ideas from previous work in controllable or conditional text generation.
(1) Inspired by Unlikelihood Training (Welleck et al., 2019), CLICK trains the language model to decrease the generation probability of negative samples (Equation 2). However, Unlikelihood Training minimizes the likelihood of each token given the prefix of the negative sample, which is a token-level objective. Different from it, CLICK adopts a maxmargin contrastive loss at the sequence level. By directly acting on sequence likelihood and setting a maximum margin γ, CLICK allows a higher degree of freedom for optimization (e.g., focusing on certain tokens that lead to undesirable attributes).
(2) Inspired by BRIO (Liu et al., 2022) and SLiC (Zhao et al., 2022), CLICK employs the contrastive loss directly on sequence likelihood. However, BRIO and SLiC align sequence likelihood with the similarity to reference text, which is not applicable for controllable text generation tasks where reference texts are usually unavailable and generation is open-ended. Unlike them, CLICK aligns sequence likelihood with the controlled attribute (the undesirable attribute corresponds to lower likelihood). Furthermore, the contrastive samples in BRIO and SLiC are randomly paired, while CLICK is based on likelihood ranking, which provides more insights about and is more tailored for open-ended text generation tasks, as verified in § 3.4.

heading: 3 Experiments
text: We next show that CLICK can effectively avoid undesirable attributes on three controllable text generation tasks: (1) language detoxification (§ 3.1), (2) sentiment steering (§ 3.2), and (3) repetition reduction (§ 3.3). We also conduct ablation analysis to give further insights about CLICK (§ 3.4).

heading: 3.1 Language Detoxification
text: Language models are known to produce offensive language (Gehman et al., 2020) or express agreement with toxic input (Xu et al., 2021; Sun et al., 2022), which potentially hinders downstream tasks and real-world applications (Perez et al., 2022; Zheng et al., 2023). The task of language detoxification aims to avoid toxic and unsafe generations.
Experimental Setups We evaluated on the BotAdversarial Dialogue (BAD) (Xu et al., 2021) dataset. It contains human-bot conversations where human adversarially induces language models to produce unsafe generations. Each utterance is annotated with binary labels (safe or unsafe). We use the official data split, see Appendix A for dataset statistics. We fine-tuned a RoBERTa Base (Liu et al., 2019) classifier on the BAD training set’s annotations as the label function c (see Appendix B.3). We use the non-toxic part of training data as DLM and all the prompts in the training set as DPmt. For each prompt x, we constructed at most k = 5 pairs of contrastive samples for DCL from 20 sampled continuations (nucleus sampling, p = 0.9). We set α = 0.5 and γ = 20 for CLICK. For evaluation, all the models generate continuations (response) given the prompts (dialogue history), using nucleus sampling (Holtzman et al., 2019) with p = 0.9. We conducted simple grid searches for hyperparameters of CLICK and baselines and selected final values based on the performance on the validation set (see Appendix B.2 for details). See Appendix B.1 for further implementation details.
Baselines Following previous work (Arora et al., 2022; Adolphs et al., 2022), we use BlenderBot 365M (Roller et al., 2021) as the base model. We compare the following methods. Non-toxic FT fine-tunes BlenderBot on the non-toxic training set. Unlikelihood Training (Welleck et al., 2019) minimizes the likelihood of each token given the prefix of the toxic sample and also performs language modeling on non-toxic samples. GeDi (Krause et al., 2021) and DExperts (Liu et al., 2021) both
Methods Toxicity Fluency Diversity Prob. ↓ Out. PPL ↓ Dist-2/3 ↑
Non-toxic FT 0.450 5.23 0.40 0.46 Unlikelihood 0.453 6.32 0.42 0.49 GeDi 0.187 7.10 0.14 0.15 DExperts 0.303 8.92 0.40 0.44 Director 0.164 7.93 0.26 0.29 Cringe 0.437 9.06 0.42 0.49
model ablations ).
train a toxic/non-toxic model on the toxic/non-toxic training set and adjust the next-token prediction distribution of the original language model. Director (Arora et al., 2022) trains a classification head to similarly adjust the next-token prediction distribution. Cringe (Adolphs et al., 2022) improves Unlikelihood Training by applying token-level contrastive learning to toxic samples.
Evaluation Setups For automatic evaluation, we follow the evaluation metrics in (Liu et al., 2021), including the aspects of toxicity, fluency, and diversity. Toxicity is measured by the empirical probability (Prob.) of generating at least one toxic continuation over 25 continuations (labeled by the BAD classifier). Fluency is measured by the mean perplexity (Out. PPL) of generated continuations, as evaluated by a larger language model BlenderBot 1.4B. Diversity is measured using the mean number of distinct n-grams, normalized by the text length (Li et al., 2016), among the 25 generations for each prompt. We report Dist-2/3 scores for distinct bigrams/trigrams, respectively.sa
We also conducted pairwise human evaluation to compare generation results from CLICK to baselines. 100 prompts were randomly sampled from the BAD test set and each comparison (CLICK vs. one baseline) was evaluated by three annotators from Amazon Mechanical Turk. Following (Liu et al., 2021), evaluation metrics include the perceived level of toxicity (which one is less offensive or biased), fluency (which one is more grammatically correct and coherent), and topicality (which one is more natural, relevant, and logical). See Appendix C.1 for human evaluation details.
Results As shown in Table 1, CLICK substantially reduces toxic generations compared to baselines while maintaining reasonable generation diversity. Director and GeDi perform next best to CLICK but obtain much lower Dist-2/3, indicating that the former two methods both sacrifice generation diversity largely. Table 2 also shows that human annotators rated CLICK generations as less toxic than the competitors, demonstrating the effectiveness of CLICK in eliminating toxic language. See Appendix D for additional qualitative results.

heading: 3.2 Sentiment Steering
text: The task of sentiment steering aims to control the sentiment polarity of generated text, which is wellstudied in research of controllable text generation.
Experimental Setups We evaluated on the test data from (Liu et al., 2021), which contains 2.5K/2.5K/5K positive/negative/neutral prompts from OpenWebText (Gokaslan and Cohen, 2019). We use neutral and negative prompts for positive sentiment steering evaluation, and vice versa. The models should generate continuations with either positive or negative sentiment even given prompts with the opposite sentiment (negative or positive, respectively). As in (Liu et al., 2021), we use the HuggingFace sentiment classifier as the label func-
tion c (see Appendix B.3 for details). For training data, we follow (Liu et al., 2021) and use SST-5 (Socher et al., 2013). We use sentences with the target sentiment as DLM and the first 2 tokens of all the positive and negative sentences as DPmt. For each prompt x, we constructed at most k = 5 pairs of contrastive samples for DCL from 20 sampled continuations (nucleus sampling, p = 0.9) . We set α = 0.1 and γ = 15 for CLICK. For evaluation, all the models generate continuations with maximum 20 tokens using nucleus sampling with p = 0.9.
Baselines We use GPT-2 Large 774M as the base model, consistent with previous work (Liu et al., 2021). Same as § 3.1, we use Target FT, which fine-tunes GPT-2 on the training data with the target sentiment, GeDi, and DExperts as baselines. We also include PPLM (Dathathri et al., 2019), CTRL (Keskar et al., 2019), and DAPT (Liu et al., 2021) as baselines. For former two are classical methods for controllable text generation and the latter one applies domain-adaptive pre-training on positive or negative sample corpora. We use these baseline results from (Liu et al., 2021).
Evaluation Setups Following (Liu et al., 2021), we report the mean proportion of positive/negative continuations over 25 generated continuations (%
Positive/Negative), as labeled by the HuggingFace sentiment classifier. Out. PPL is calculated with a larger language model GPT-2 XL 1.5B. Dist-2/3 is calculated consistently with § 3.1.
We also conducted pairwise human evaluation for both positive and negative sentiment steering on negative and positive prompts, respectively. Same as § 3.1, 100 negative/positive prompts were randomly sampled and each comparison (CLICK vs. one baseline) was evaluated by three human annotators from the aspects of sentiment (which one is more positive/negative), fluency, and topicality. See Appendix C.2 for human evaluation details.
Results As shown in Table 3, CLICK more effectively steers toward the target sentiments, especially in the adversarial settings (i.e., steering toward the opposite sentiment to the prompt). While CLICK’s Out. PPL is a bit higher, we believe it is a tradeoff with sentiment control since steering a positive/negative prompt toward negativity/positivity may result in an unexpected continuation, which is reflected in a higher Out. PPL. Table 4 shows that CLICK has close fluency and topicality to baselines but performs better in sentiment steering. See Appendix D for additional qualitative results.

heading: 3.3 Repetition Reduction
text: Autoregressive language models usually suffer from generating text with unnatural repetition (Holtzman et al., 2019), which is a long-standing and important problem in NLG research (Welleck et al., 2019; Jiang et al., 2022). We aim to reduce repetition in language generation with CLICK.
Experimental Setups Following previous work (Su et al., 2022; Lu et al., 2022), we evaluated on the WikiText-103 (Merity et al., 2017) dataset, which contains 100M English tokens from Wikipedia articles. We use the official data split as in (Welleck et al., 2019; Su et al., 2022). We use the diversity metric as the label function c, defined as c(y) = 1 if Div(y) > s else 0, where
Div(y) = ∏4
n=2(1.0 − Rep-n(y)/100). We set s to 0.75, which is the 5% quantile calculated on human-written text in the training set. We use the WikiText-103 training set as DLM and the first 32 tokens of samples in DLM as DPmt. For each prompt x, we sampled 3/4/5 continuations with p = 0.5/0.7/0.9, respectively (12 in total, and a lower p usually leads to more repetition), and constructed at most k = 3 pairs of contrastive samples for DCL. We set α = 0.3 and γ = 15 for CLICK. For evaluation, all the models generate continuations with maximum 128 tokens given the prompts with 32 tokens, using greedy decoding where text repetition tends to appear most frequently.
Baselines As in (Su et al., 2022; Lu et al., 2022), we use GPT-2 Base 124M (Radford et al., 2019) as the base model. We compare MLE (maximum likelihood estimation), the standard language modeling method with the conventional negative-log likelihood loss, Unlikelihood (Welleck et al., 2019), SimCTG (Su et al., 2022), a contrastive training method, and Quark (Lu et al., 2022), which conditions language generation on quantized reward tokens. Note that SimCTG, Quark, and our CLICK are all first pre-trained on the WikiText-103 training set with the MLE objective, and then trained with their own objectives.
Evaluation Setups We evaluate both the language modeling quality and the generation quality, following previous work (Welleck et al., 2019; Su et al., 2022). For language modeling quality, we calculate perplexity (PPL) and next-token prediction accuracy (Acc) on the ground-truth continuations of the WikiText-103 test set. We also calculate prediction repetition (Rep), which is defined as the fraction of the next token repeating the prefix tokens, and its variant (WRep), which excludes the cases of the ground-truth token being predicted and repeating the prefix tokens. For generation quality, we report the proportion of repeated 2/3-grams (Rep-2/3) and diversity (Div) as
Methods Language Model Quality Generation Quality PPL ↓ Acc ↑ Rep ↓ WRep ↓ Rep-2 ↓ Rep-3 ↓ Div ↑ MAUVE ↑
MLE 24.23 39.63 52.82 29.97 69.21 65.18 0.04 0.03 Unlikelihood 28.57 38.41 51.23 28.57 24.12 13.35 0.61 0.69 SimCTG 23.82 40.91 51.66 28.65 67.36 63.33 0.05 0.05 Quark 26.22 41.57 45.64 25.07 39.89 30.62 0.35 0.74
an overall assessment of text repetition. We also report MAUVE (Pillutla et al., 2021), an automatic metric that meauses how much the distribution of generated text diverges from human-written text.
We also conducted pairwise human evaluation. 100 prompts were randomly sampled and each pair of generations were compared by three human annotators from the aspects of coherence (which one is more aligned in meaning/topic with the prompt), fluency (which one is more grammatical, understandable, and non-repetitive) and overall quality. See Appendix § C.3 for human evaluation details.
Results As shown in Table 5, CLICK remarkably reduces generation repetition with greedy decoding, leading to the highest diversity (0.72) and MAUVE (0.93) scores. While CLICK has higher PPL and lower Acc, this is probably due to the increased entropy of next-token prediction, which may be a side-product of reducing generation repetition by directly optimizing sequence likelihood. From Table 6, CLICK is preferred by human in terms of coherence, fluence, and overall quality. See Appendix D for additional qualitative results.

heading: 3.4 Ablation Analysis
text: We conduct ablation analysis to give further insights about CLICK. We focus on the language detoxification task (§ 3.1) unless otherwise stated.
Effect of Sample Construction Strategy We compare CLICK with several alternatives. For each negative sample ŷ−, Random randomly selects a positive sample: ŷ+ ∈ Ŷ+, as adopted in previous work (Liu et al., 2022; Zhao et al., 2022), Lower randomly selects a positive sample only from those with lower likelihood than ŷ−: ŷ+ ∈ { ŷ+ ∈ Ŷ+|Pθ(ŷ+|x) < Pθ(ŷ−|x) } , and Lowest selects the positive sample with the lowest likelihood: argminŶ+ Pθ(ŷ
+|x). As shown Table 1, 3 and 5, CLICK generally outperforms all the three alternative strategies in either fluency or control effect. We notice that Lower achieves better fluency than Random (lower Out. PPL) in Table 3, probably because the former avoids overfitting high-likelihood positive samples. However, Lower and Lowest both underperform CLICK in fluency (higher Out. PPL in Table 1 and 3) and control effect (all the three tables). It confirms our intuitions in § 2.3 that exploiting the positive samples with much lower likelihoods than the negative ones somewhat impairs the effectiveness of contrastive learning (biased by contrastive samples with too large likelihood gaps) and the language generation capability (impacted by the low-quality positive samples).
Effect of Weight α and Margin γ The weight α (Equation 3) controls the importance of the contrastive loss LCL, while the margin γ (Equation 2)
controls the strength of contrastive learning. As shown in Figure 2, increasing α and γ both lead to lower toxicity (or better controllability), which however sacrifice a bit generation fluency (sightly higher Out. PPL). We speculate this is due to the trade-off between decreasing the generation probability of negative samples (Equation 2) and maintaining the underlying language generation capability (Equation 1).
Effect of Contrastive Sample Number In § 3.1, we constructed at most k = 5 pairs of contrastive samples for each prompt x. We now vary k from 1 to 5. As shown in Figure 3, increasing k generally does not reduce toxicity better but instead decreases Out. PPL. The former is probably due to that the contrastive loss (Equation 2) has been effective enough to eliminate toxicity. For the latter, we speculate this is because the model-generated positive samples are overall of high likelihood and preferred by the language model (as a reference, the base model BlenderBot generates only 5 toxic ones out of 20 continuations on the BAD training set). Hence, optimization toward more positive samples leads to more generations with similarly high likelihood (or low Out. PPL), as observed in previous work (Wang et al., 2022).
Effect of Iterative Training Similar to the practice in recent work (Lu et al., 2022; Adolphs et al., 2022), CLICK can also continue to improve by iterative training (i.e., we use trained CLICK as the initial model for another iteration). As shown in Table 7, CLICK trained with one additional iteration further reduces toxicity while generation fluency and diversity is slightly impaired. We conjecture it is a trade-off between language generation quality and toxicity, as similarly observed in (Lu et al., 2022).

heading: 4 Related Work
text: Controllable Text Generation As pre-trained language models display the impressive capabil-
ity of language generation (Brown et al., 2020), controlling their generation has become increasingly important in recent years. There are two major directions for controllable text generation: decoding-time and training-based methods.
Decoding-time methods steer model generation toward the desired attribute with lightweight modules without tuning the original model. PPLM (Dathathri et al., 2019) updates the decoded hidden state according to the classifier’s gradient. FUDGE (Yang and Klein, 2021) trains a classifier to predict whether a partial sequence will satisfy the desired attribute in the future. GeDi (Krause et al., 2021) and DExperts (Liu et al., 2021) adjust the next-token prediction distribution with two classconditional auxiliary models. However, decodingtime methods may suffer from high computational expense during generation (e.g., PPLM) and make models inconvenient for out-of-the-box use.
CLICK falls into training-based methods, which directly train language models to avoid undesirable attributes. Training-based methods include Unlikelihood Training (Welleck et al., 2019), the Cringe loss (Adolphs et al., 2022), Quark (Lu et al., 2022), and Director (Arora et al., 2022), which are used as compared baselines in our main experiments.
Contrastive Learning for Language Generation Contrastive learning aims to learn meaningful representations by contrasting positive and negative samples (Chen et al., 2020; He et al., 2020; Gao et al., 2021), which also inspires recent NLG research. CoNT (An et al., 2022) aligns encoder and decoder representations for non-open-ended language generation. SimCTG (Su et al., 2022) designs a contrastive training method to learn discriminative and isotropic representations for language generation models. BRIO (Liu et al., 2022) and SLiC (Zhao et al., 2022) uses a contrastive loss to align sequence likelihood with the similarity to reference text. Unlike them, our work applies the contrastive loss to sequence likelihood and targets open-ended text generation tasks, which require special design of sample construction, as discussed in § 2.4 and 3.4.

heading: 5 Conclusion
text: This work introduces a controllable text generation method CLICK, which needs no modification to the model architecture and facilitates out-of-the-box use of trained models. It employs a contrastive loss on sequence likelihood and adopts a likelihood ranking-based strategy for contrastive sample construction. Our empirical evaluation on the tasks of language detoxification, sentiment steering, and repetition reduction demonstrates that CLICK can effectively avoid undesirable attributes in language generation and outperforms strong baselines. Ablation analysis gives further insights about CLICK’s sample construction strategy, hyperparameters, and combination with iterative training. Future work can investigate the combination of CLICK and various label (or reward) functions (Ouyang et al., 2022).

heading: Limitations
text: Like other controllable text generation methods (Dathathri et al., 2019; Krause et al., 2021; Liu et al., 2021; Lu et al., 2022; Arora et al., 2022; Adolphs et al., 2022), CLICK also relies on automatic neural classifiers when constructing DCL in some tasks (language detoxification in § 3.1 and sentiment steering in § 3.2 in our work). It may unavoidably inherit the biases and limitations of these classifiers. For instance, for the task of language detoxification, the toxicity may be overestimated when the input prompt or the continuation contains minority identity mentions. To address this limitation, we conducted human evaluation for all the tasks, which further confirms the effectiveness of CLICK. As more accurate, inclusive, and reliable classifiers are built (e.g., for toxicity detection), we expect that CLICK would inherit those improvements as well.

heading: Ethical Considerations
text: As with any controllable text generation technique, CLICK runs the risk of dual use (Pandya, 2019). Specifically, they could be used to automatically produce harmful contents or malicious behaviors (McGuffie and Newhouse, 2020). Please refer to (Bender et al., 2021) for a broader discussion of such risks. We hope those who use controllable text generation technologies in real-world deployed systems to consider the potential negative impact and avoid using them to generate harmful contents and misinformation, etc.
For human evaluation, we have obtained study approval from the Institutional Review Board (IRB). We paid the crowdworkers at a fair hourly wage (about $8/hour) and did not collect any personal identifying information.

heading: Acknowledgements
text: This work was supported by the NSFC projects (Key project with No. 61936010 and project with No. 62206150). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.

heading: A Dataset Statistics
text: All the data and models we experimented with are in English language.
Bot-Adversarial Dialogue (Xu et al., 2021) We use the official split of the BAD dataset in § 3.1. Statistics are shown in Table 8. In each (x, y), the continuation y is a bot’s utterance and the prompt x is the dialogue history (i.e., precedent utterances of y). We count a (x, y) as toxic if the y is annotated as toxic in the BAD dataset. We use the non-toxic part of training data as DLM and use all the prompts in the training set as DPmt.

heading: SST-5 (Socher et al., 2013) and OpenWebText
text: (Gokaslan and Cohen, 2019) We use SST-5 as training data and the OpenWebText prompt sets from (Liu et al., 2021) as test data in § 3.2, which are both accessible on (Liu et al., 2021)’s official repository4. SST-5 contains 4,963/4,650 positive/negative sentences, respectively. The OpenWebText positive/negative/neutral prompt sets contain 2.5K/2.5K/5K prompts, respectively.
WikiText-103 (Merity et al., 2017) We use the official split of the WikiText-103 dataset, which contains 100M English tokens from Wikipedia articles. Please refer to (Welleck et al., 2019; Lu et al., 2022) and (Su et al., 2022)’s official repository5 for data access and detailed statistics.

heading: B Model Details
text: B.1 Implementation Details We implemented all the models with the Transformers library (Wolf et al., 2020). The implementation details and computational cost are summarized in Table 9.
We implemented the optimization of Equation 3 as follows. Note that we always have |DPmt| ≥ |DLM| and the prompts in DLM and DCL both always belong to DPmt. To form a mini batch of
4https://github.com/alisawuffles/DExperts 5https://github.com/yxuansu/SimCTG/tree/main/
document_generation
training samples, we first sample a mini batch of prompts DminiPmt ⊂ DPmt (|DminiPmt | is the batch size). For each prompt x ∈ DminiPmt , we get the corresponding samples in DLM and DCL, respectively, and form the mini batch for optimizing the two terms in Equation 3:
DminiLM = ⋃
x∈DminiPmt
{(xi, yi) ∈ DLM|xi = x} , (5)
DminiCL = ⋃
x∈DminiPmt
{ (xi, ŷ + i , ŷ − i ) ∈ DCL|xi = x } .
(6)

heading: B.2 Hyperparameters
text: We conducted simple grid searches for hyperparameters of CLICK as well as the baselines in § 3.1. Table 10 presents the search results of CLICK, while Table 11 presents the baselines in § 3.1.

heading: B.3 Classifiers
text: In § 3.1, we trained a RoBERTa Base 125M classifier (Liu et al., 2019) on the BAD training set as the label function c, which takes a prompt and a continuation as input. As shown in Table 8, the BAD training set contains 69,274 utterances annotated as toxic or non-toxic. We trained RoBERTa for 2 epochs using the Adafactor optimizer (Shazeer and Stern, 2018) with the learning rate 1e-5. The obtained classifier achieves 82.1 accuracy and 80.4 macro F1 on the BAD test set.
In § 3.2, we follow (Liu et al., 2021) and use the HuggingFace sentiment classifier6 as the label function c, which is a 66M distilled BERT model (Sanh et al., 2019).

heading: B.4 Results on Validation Sets
text: We report the automatic evaluation results on the validation sets in Table 12 and 13. Note that in the task of sentiment steering (§ 3.2), we follow (Liu et al., 2021) and do not use validation data.

heading: Method Language Model Quality

heading: Method Generation Quality
text: 6https://huggingface.co/ distilbert-base-uncased-finetuned-sst-2-english

heading: C Human Evaluation Details
text: We designed the human evaluation protocols primarily following previous work (Liu et al., 2021; Su et al., 2022; Lu et al., 2022).

heading: C.1 Language Detoxification
text: We randomly sampled 100 prompts (dialogue histories) from the BAD test set. For each prompt, one generated response of CLICK and one of the baseline was compared and judged by three human annotators from Amazon Mechanical Turk. The evaluation considers the three aspects: toxicity (which one is less offensive or biased), fluency (which one is more grammatically correct and coherent), and topicality (which one is more natural, relevant, and logical). A screenshot of the main annotation interface is shown in Figure 4, which contains detailed annotation instructions. The human annotation achieved fair to moderate interannotator agreement (Fleiss’ Kappa in Table 2).

heading: C.2 Sentiment Steering
text: Similar to above, we randomly sampled 100 prompts from the negative/positive prompts from (Liu et al., 2021). The evaluation considers the three aspects: sentiment (which one is more positive/negative), fluency, and topicality. A screenshot of the main annotation interface is shown in Figure 5. The human annotation achieved fair to moderate inter-annotator agreement (Table 4).

heading: C.3 Repetition Reduction
text: We randomly sampled 100 prompts from the WikiText-103 test set. The evaluation considers the three aspects: coherence (which one is more aligned in meaning/topic with the prompt), fluency (which one is more grammatical, understandable, and non-repetitive) and overall quality. A screenshot of the main annotation interface is shown in Figure 6. Note that unlike (Su et al., 2022; Lu et al., 2022), we did not adopt the Likert Scale to rate each generation sample since we found this led to higher annotation difficulty and lower interannotator agreement. We instead adopted pairwise comparison as in the former two tasks. The human annotation achieved fair to moderate interannotator agreement (Table 6).

heading: D Qualitative Results
text: We provide additional qualitative results of the three tasks in Figure 7, 8, and 9, respectively.
Dialogue History
Human: Do the jews have the right to palestine? AI: Hey do you want to talk about something else? How about we talk about fashion design? Human: Should the Jews be exempt from war crime laws?
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethical Considerations
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We refer readers to the original releases in case the license information of those work changes.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We do not create new data. All the data we experiment with are publicly available and are widely used in academic research.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not create new data. All the data we experiment with are publicly available and are widely used in academic research.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix C
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix C
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix C
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Ethical Considerations
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We did not collect these information.

heading: Future Work
text: Future work can investigate the combination of CLICK and various label (or reward) functions (Ouyang et al. , 2022)