Controlled Text Generation with Hidden Representation Transformations
Year: 2023

Abstract:
We propose CHRT (Control Hidden Representation Transformation) – a controlled language generation framework that steers large language models to generate text pertaining to certain attributes (such as toxicity). CHRT gains attribute control by modifying the hidden representation of the base model through learned transformations. We employ a contrastive-learning framework to learn these transformations that can be combined to gain multi-attribute control. The effectiveness of CHRT is experimentally shown by comparing it with seven baselines over three attributes. CHRT outperforms all the baselines in the task of detoxification, positive sentiment steering, and text simplification while minimizing the loss in linguistic qualities. Further, our approach has the lowest inference latency of only 0.01 seconds more than the base model, making it the most suitable for high-performance production environments. We open-source our code and release two novel datasets to further propel controlled language generation research.

heading: 1 Introduction
text: With the latest developments in transformers, (Vaswani et al., 2017), large language models (Radford et al., 2019, 2018; Brown et al., 2020) have gained remarkable success in generating coherent and fluent text. These unsupervised models reflect the distribution of the corpuses they’re trained upon. Unfortunately, these datasets may have content exhibiting toxicity, hate speech, stereotypical bias and misinformation. Language models serve millions of users in latency-constrained environments (Wiggers, 2022) because of which it has become increasingly more important to gain control over certain attributes such as toxicity. Table 1
∗Work done as an intern at Amazon Alexa AI. †Equal Contribution
highlights the problem of toxic language generation in three common models – GPT-2 (Radford et al., 2019), T5 (Raffel et al., 2020) and Bloom (Scao et al., 2022). It serves as the motivation for using controlled generation to minimize negative attributes like toxicity as the productionisation of such models could be appalling.
We propose CHRT, a lightning-fast controlled language generation framework that gains attribute control by transforming the hidden representation of a base model through contrastively learned transformations. We first fine-tune two guider models – LM− and LM+. These are the base language model fine-tuned on negative and positive attribute text respectively. Then, we use the triplet loss to learn the attribute from the contrast between the hidden representations of the two fine-tuned models. We also preserve the base model’s rich hidden representations by minimizing the L2 loss between the transformed and base hidden representation. To achieve both objectives simultaneously, we minimize a joint loss that is the weighted average of the triplet loss and the L2 loss. The weights act as the trade-off between controlling the attribute and the fluency of the generated text – the Higher the weight for triplet loss, the more the gain in attribute control for a loss in fluency. We empirically show
9440
this trade-off in Section 4.
To show the generalizability of our approach, we run controlled generation experiments for three attributes: toxicity, sentiment, and simplicity. For toxicity, we fine-tune our guider models on the real toxicity prompts (Gehman et al., 2020) dataset. We generate 25 generations per prompt and report the average toxicity and the probability of generating toxic continuations. We also report the fluency of generations using the perplexity metric. Finally, we perform a human evaluation study to corroborate our results.
Closely following the approach of Real Toxicity Prompts (Gehman et al., 2020), we devise RealAttributePrompts – a framework to automatically generate datasets for controlled language generation benchmarking using an attribute classifier. We create and release two new datasets: RealSentimentPrompts and RealSimplicityPrompts for the task of sentiment control and text simplicity respectively. Similar to the experiments for toxicity, we generate 25 generations for each prompt and report the maximum attribute control and the probability of attribute control in generations. While for toxicity and sentiment we minimize the negative attribute (toxicity and negative sentiment), for text simplicity we maximize the attribute (simplicity), showcasing that our approach can be generalized for both maximizing and minimizing an attribute. Finally, we showcase multi-attribute control by combining multiple CHRT transformations in Section 3.5.
For all our results we perform a comprehensive comparison with five existing baselines: DAPT (Domain Adaptive Pre-training) (Gururangan et al., 2020), NLC (Kajiwara, 2019) (Negative Lexically Constrained) decoding, PPLM (Plug and Play language models) (Dathathri et al., 2019), GeDi (Generative Discriminators) (Krause et al., 2020) and DExperts (Liu et al., 2021), for controlling the base GPT-2 model. Our approach outperforms all five baselines in controlling the attributes of toxicity, sentiment, and text simplicity respectively with minimal loss in liguistic qualities. It also achieves the lowest latency of +0.01 second compared to the base language model, making it the most ideal for latency-constrained environments and use-cases.
Our contributions can be summarized as follows: • Proposing Control Hidden Representation Transformations (CHRT), a lightning fast, novel and efficient controlled language generation framework which achieves high attribute control, mini-
mal loss in fluency loss very fast inference time. • Applying CHRT as a multi-attribute Control framework by combining multiple transformations. • Proposing RealAttributePrompts – a novel optimized framework for generating datasets to benchmark controlled generation methods. • Using RealAttributePrompts to release two new datasets: RealSentimentPrompts and RealSimplicityPrompts along with open-sourcing our code1.

heading: 2 Related Work
text: Related work is broadly divided into two parts – Controlled Language Generation and the application of Contrastive Learning in NLP.

heading: 2.1 Controlled Language Generation
text: The controlled language generation literature can roughly be categorized into pre-processed learningbased or decoding time techniques, both with their advantages and disadvantages.
Learning Based: These methods usually finetune language modeling or do prompt engineering to control text attributes. Gururangan et al. (2020) fine-tuned language models on domain-adaptive text to control attributes of the generated text. Other works employ Reinforcement Learning (Ziegler et al., 2019) and Contrastive learning (Gunel et al., 2020; Yu et al., 2020) for fine-tuning PLMs. While these fine-tuned language models achieve high fluency, they often fail to achieve optimal attribute control as shown in the existing literature (Liu et al., 2021; Yang and Klein, 2021). Some works try to model the generation length such as Kikuchi et al. (2016) who propose an encoder-decoderbased learning method to control generation length. Keskar et al. (2019) propose CTRL, a fine-tuning with control codes method to steer transformerbased PLMs towards certain attributes and styles. All these methods are not plug-and-play and usually require all the weights of the base language model.
Decoding Time: These methods modify the decoding process and are usually plug-and-play with very minimal to no re-training requirements. Kajiwara (2019) add negative lexical constraints during decoding to reduce generation probabilities of certain lexical to zero. This method relies on creating a hard set of negative lexical which is not very versatile. Dathathri et al. (2019) utilize a bag of words or
1https://github.com/amazon-science/ wqa-controlled-text-generation
a small discriminator model to guide decoding during PLM generation. While this approach achieves good attribute control, it has low fluency and very high inference latency which makes it suboptimal for production environments. Krause et al. (2020) rather use generative discriminator models with individual token probabilities to modify the model distribution during decoding. Similarly, Liu et al. (2021) also modify the probability distribution of large PLMs using two smaller fine-tuned expert and dexpert models. Yang and Klein (2021) condition on attributes using future discriminators to guide the decoding process. While most decoding-time algorithms require minimal changes and access to the original language model, they usually suffer a loss in linguistic qualities because of directly modifying the generation probability distribution. We show this phenomenon of loss in fluency in our results Section 4 using both automated and human evaluation.

heading: 2.2 Contrastive Learning
text: Contrastive learning is a representation learning algorithm that learns to map similar data samples close in an embedding space while pushing the dissimilar samples relatively farther. Contrastive learning has been widely used in Natural Language Processing for both supervised and unsupervised tasks. Most widely, it is used for representation learning in embedding space (Kim et al., 2021; Gao et al., 2021; Wieting et al., 2015). Augmenting existing NLP frameworks with contrastive learning loss such as triplet loss (Alber, 1993) has enjoyed great success in text classification (Fang et al., 2020; Suresh and Ong, 2021; Xiong et al., 2020), information extraction (Qin et al., 2020; Xiong et al., 2020), machine translation (Pan et al., 2021; Vamvas and Sennrich, 2021), question answering (Karpukhin et al., 2020; You et al., 2021), summarization (Cao and Wang, 2021; Duan et al., 2019; Wang et al., 2021) and more. Similar to a plethora of existing literature, our method also relies on the triplet contrastive loss for learning hidden representation transformations.

heading: 3 CHRT: Control Hidden Representation Transformations
text: We start with formally defining the problem of controlled language generation, followed by explaining how CHRT transforms the hidden representations. Finally, we explain the finetuning of guider models
and training the CHRT transformation heads. Figure 1 schematically shows the training, inference, and the transform block for our approach.

heading: 3.1 Controlled Language Generation
text: Controlled generation can be formally described as modeling the distribution P(Wt|W<t,A = a) where A is an attribute such as toxicity and a is the attribute class such as non-toxic. Through this distribution, language is generated auto-repressively by sampling one token wt at a time as wt ∼ P(Wt|W<t,A = a) where Wt is the distribution over vocabulary at the current timestep and W<t is the tuple of the tokens generated so far.

heading: 3.2 Attribute Control Transformations
text: For a controlled generation, we propose to modify the hidden representations ht to ht′ = τ(ht) where τ is a transformation block. We want τ to be learnable and thus construct it using neural network layers. Figure 1c summarizes the transformation block which is a concatenation of two identical blocks with skip connections. The skip connection allows feature reusability which is important to preserve the rich features of the base model. We empirically justify the construction choice of our transformation block in Appendix A.

heading: 3.3 Finetuning Guider Models
text: We fine-tune two guider models – LM+ and LM− on positive and negative attribute text respectively. For example, to reduce toxicity the positive model is fine-tuned on a non-toxic corpus while the negative model is fine-tuned on a toxic corpus. These models are only used to learn the transformation τ and are discarded later during the inference phase. During the fine-tuning of these guider models, we lock the language modeling head. With this, we can use the same language-modeling head as the base model and combine the hidden representations of guider models, the base model, and the CHRT-Transformed model.

heading: 3.4 Training Transformations
text: The objective of learning our transformation is twofold: maximizing the attribute control and preserving the original model’s rich linguistic qualities. For each of these, we propose the following individual losses which are combined into a single loss through a weighted sum:
1. Contrastive Loss (Lc): We use the contrastive triplet loss (Balntas et al., 2016) to
steer the hidden representation towards the hidden representation of fine-tuned model LM+ and away from the LM−. Lc = max { d(h ′ t, h + t )− d(h ′ t, h − t ) + δ, 0 } (1) where h ′ t = τ(ht), h + t , h − t are the hidden representations from the transformed language model, fine-tuned language model LM+ and LM− respectively. d(a, b) = ∥a, b∥2 is the L2 distance between a and b, δ is the margin of separation.
2. Preservation Loss (Lp): The purpose of this loss function is to preserve the base model’s rich representations. We do so by minimizing the L2 distance between the base hidden representation ht and the transformed representation h ′ t = τ(ht).
Lp = ∥∥∥ht, h ′ t ∥∥∥ 2
(2)
Finally, we minimize the weighted sum of the two losses: L = λLp + (1− λ)Lc (3) where λ determines the importance of preservation loss over the contrastive loss. Section 4 experimentally showcase the effect of lambda over the trade-off between fluency and attribute control. It should be noted that during the training of these transformations, all the weights are locked other than that of the transform block as shown in Figure 1a, this makes our training process computationally efficient.

heading: 3.5 Multi-Attribute Control
text: We can train individual transformation blocks and then combine them to gain multi-attribute control.
Since the language model head LM is locked, we can take linear combination of multiple heads to get the final hidden representation as follows:
h ′ t = α1τ1(ht) + α2τ2(ht)...+ α2τn(ht)
s.t. n∑
i=1
αi = 1 (4)
where, τi is the transformation trained to maximize an attribute ai, αi is the CHRT weight corresponding to τi. The final representation, h ′ t is fed to the language modeling head to generate the next token wt through some decoding algorithm like Nucleus Sampling (Holtzman et al., 2019). It should be noted that the weights αi can be changed during inference to control the importance of one attribute over another without any additional re-training. We show the trade-off between attributes by varying CHRT weights in Section 4.5.

heading: 4 Experimental Results
text: We experimentally show the efficacy of our approach by applying it to the GPT-2-medium language model and comparing it with five other controlled generation baselines. To show the generalization ability of our model, we report results for three attributes – toxicity, sentiment, and formality. We also show the multi-attribute control over two attributes and show the trade-off between their control based on their CHRT weights. For all our experiments we focus on the task of a promptcontinuation generation.

heading: 4.1 Baselines
text: Following existing works (Liu et al., 2021; Krause et al., 2020), for all the baselines we generate 25 independent continuations of length 25 tokens conditioned per prompt.
1. NLC: Negative Lexically Constrained decoding, as proposed by Kajiwara (2019). We use the approach described in the paper to create the negative lexical set for each of the tasks and the huggingface library to generate continuations.
2. DAPT: We perform domain Adaptive PreTraining (Gururangan et al., 2020) by finetuning the vanilla model on the positiveattribute corpus. We use huggingface’s finetuning and generation scripts for DAPT.
3. PPLM: For Plug-and-Play language models (Dathathri et al., 2019), we use the scripts released by the authors 2 to first re-train the discriminators and then generate continuations for each of the attribute’s prompts.
4. GeDi: For Generative Discriminators (Krause et al., 2020), we train our own GeDi for each of the three attributes using the training scripts released by the authors 3. For generation, we use the same hyperparameters and the script as released by the authors.
5. DExperts: As proposed by Liu et al. (2021), we use their publicly released code to retrain the expert and dexpert models. For generation, we use the same hyper-parameters as suggested by the authors in their paper and publicly released code 4.
6. CHRT: We report results for three variants of CHRT with different weights for Lp and Lc. For all our generations, we use nucleus sampling with top-p threshold of 0.8, repetition penalty of 1.2 (Keskar et al., 2019) and the huggingface library.
More implementation details for each of the baseline is presented in Appendix B.

heading: 4.2 Detoxification
text: For Detoxification, we aim to minimize the toxicity attribute using controlled generation for the task of prompt-continuation generation.
Prompt Selection: We use the prompts from the RealToxicityPrompts (Gehman et al., 2020) dataset. It contains 100k pairs of prompts and their continuations labeled for toxicity using Perspective API (Per). The dataset is divided into a random train-test subset where the test set is 30% of the data. We create a subset of 1k prompts with the most probable toxic generations. Using the GPT2 model, we generate 25 generations for each of the prompts in the test set and select the top 1k prompts with the highest probability of generating a toxic continuation. Toxicity is evaluated by the detoxify model (Hanu et al., 2021) trained on the Jigsaw toxic comment classification dataset (Jigsaw, 2017).
2https://github.com/uber-research/PPLM 3https://github.com/salesforce/GeDi 4https://github.com/alisawuffles/DExperts
Evaluation: We report the toxicity, fluency, and diversity of the generated continuations. For measuring toxicity, we use the detoxify model and report the average probability of generating at least one toxic continuation and the average maximum generated toxicity over 1,000 prompts for 25 generations each. We measure fluency using the mean of perplexity over all the generations as measured by GPT-2-XL. We report diversity using dist-n scores (Li et al., 2015) which measures the number of distinct n-grams among the 25 generations per prompt, normalized by the generation length.
Algorithm 1: RealAttributePrompts Data: Ω, C, θ, n Result: S
1 i← 0, P ← ϕ, N ← ϕ, S ← ϕ; 2 while |P | ≤ n/2 ∨ |N | ≤ n/2 do 3 ω ← Ω[i]; 4 i← i+ 1; 5 if |ω| /∈ [64, 1024] ∨ ¬is_english(ω) then 6 continue
7 if C(ω) ≥ θ ∧ |P | ≤ n/2 then 8 P ← P ∪ {ω}; 9 if C(ω) ≤ 1− θ ∧ |N | ≤ n/2 then
10 N ← N ∪ {ω};
11 for s ∈ P ∪N do 12 p← s[0 : |s|/2] ; 13 c← s[|s|/2 : |s|] ; 14 S ← S ∪ {p, C(p), c, C(c), s, C(s)}
Further, we divide the training set of RealToxicityPrompts into a subset of a toxic and non-toxic corpus containing both prompts and continuations using the labeled toxicity score. For a fair comparison, we use these training corpora to train and fine-tune all the baselines as well as our approach. Table 2 summarizes the results where CHRTab rep-
resents our approach with weight λ = aa+b for the preservation loss Lp and 1− λ = ba+b for the contrastive loss Lc. We can observe that as we increase λ, the fluency of our model (in terms of perplexity) increases. CHRT12 achieves the maximum attribute control i.e. the lowest toxicity of 0.085 and 0.004 in terms of both maximum and average toxicity. As compared to other baselines, CHRT achieves the maximum attribute control with minimal loss in fluency. Methods like PPLM, GeDi and DExperts achieve attribute control by heuristically modifying the token probability distribution of the base language model at each timestep instead of modifying the dense representations which impede the fluency (as observed empirically in Table 2) of the model. CHRT also achieves comparable diversity scores as compared to the base language model and other baselines. We report the inference time of CHRT as compared to other baselines in Table 3. We observe that CHRT has an inference time of just 0.01 seconds more than the base model. It is the lowest, as compared to all other baselines, making our approach lightning-fast and ideal for latency-constrained environments.

heading: 4.3 Sentiment Steering
text: In the best of our knowledge, no publicly released prompt-continuation dataset for sentimentcontrolled generation exists. Therefore, inspired by RealToxicityPrompts (Gehman et al., 2020), we create a framework called RealAttributePrompts. Given an arbitrary attribute, Algorithm 1 efficiently generates prompts for controlled generation benchmarking of size n. C(ω) → [0, 1] is an attribute classifier that returns a classification probability for a sentence ω. θ ∈ [0, 1] is a confidence level for attribute C and Ω is a large set of sentences extracted from the huge OpenWebCorpus (Gokaslan and Cohen, 2019). For filtering away non-English sentences, we use FastText (Bojanowski et al., 2016). The set S returned by Algorithm 1 is a set of prompt-continuation pairs with individual and joint attribute scores.
For evaluating sentiment-controlled generation, we use Algorithm 1 to create RealSentimentPrompts. For the attribute classifier C, we use RoBERTa (Liu et al., 2019) fine-tuned on the Twitter sentiment classification data (Barbieri et al., 2020). We set the confidence threshold θ = 0.9 to create a dataset of size n = 100k. After the creation of this dataset, we use the same approach
and metrics as in Section 4.2 for selecting prompts and evaluating generated continuations. Table 4 summarizes the results where we can see that our approach CHRT12 achieves the lowest maximum negative sentiment of 0.094 which is more than 62% lower than DExpert, the best next baseline. We also achieve the lowest probability of just 0.5% for generating negative sentiment text with only a 10.84 point loss in perplexity as compared to the base GPT-2. Finally, our CHRT models show no to minimal loss in diversity. Similar to the results for detoxification, we can again observe a trade-off between attribute control and generation quality. As we increase the weight for the contrastive triplet loss Lc, the maximum negative sentiment and the probability of generating negative sentiment text decreases with an increase in perplexity.

heading: 4.4 Text Simplification
text: Similar to sentiment steering, we create RealSimplicityPrompts, a dataset to benchmark controlled generation while minimizing the simplicity (Coster and Kauchak, 2011) of the generated text. We again use Algorithm 1 with the same classifier fine-tuned on the PWKP version 2 (Kauchak, 2013) dataset. Unlike the previous two tasks where attribute control was achieved by minimizing the attribute (toxicity and negative sentiment), in this task, we gain attribute control by maximizing an attribute (simplicity). In Table 5 we can observe that the highest average maximum simplicity of 0.995 and a probability of 99.6% for generating simple continuations. We observe that in fact one of our models, CHRT21, achieves a better fluency of 20.690 perplexity score as compared to 22.028 by the vanilla GPT-2 model with minimal to no loss in diversity in the generated text.

heading: 4.5 Multi-Attribute Control
text: We can combine multiple CHRTs for different attributes to gain multi-attribute control. Using the approach defined in Section 3.5, we generate text while controlling two attributes – toxicity and sentiment. For this experiment, we do unprompted generation, that is, through random nucleus sampling, we generate 1k text sequences of length 25 each conditioned on the beginning of sequence token (Radford et al., 2019). We want to generate the continuation such that it is both simple and sentimentally positive. Figure 2 shows the tradeoff between controlling (increasing) text simplicity and controlling (decreasing) the negative sentiment in generations. α and 1− α are the CHRT weights for simplicity and sentiment control transformations respectively. Increasing α shifts the control from generating simple text to generating positive sentiment text. A clear trade-off can be observed by varying α in Figure 2 where sentiment and simplicity are measured using the classifiers described in Section 4.3 and Section 4.4 respectively.

heading: 4.6 Human Evaluation
text: We perform a crowd-sourced human evaluation to make the inference on our results more robust. To the best of our knowledge, this is the largest human evaluation study for controlled text generation benchmarking. We consider 1k toxic prompts (as described in Section 4.2) and generate continuation of length 25 using CHRT12 and the baselines. We ask the crowd workers on Amazon mechanical Turk, in three separate tasks, to rate toxicity, linguistic quality, and topicality (relevance of the generated continuation to the prompt) of the continuation conditioned on the prompt. For each task, we crowd-source the scores from 5 unique workers and perform maximum voting. Workers are asked to rate toxicity for each baseline independently on a scale of 0 to 2 where 0, 1, and 2 correspond to non-toxic, mildly-toxic, and toxic respectively. Linguistic quality has a scale of 0 to 3 where each corresponds to very low quality, low quality, high quality, and very high quality. Finally, topicality is rated between 0 and 2 where 0, 1, and 2 correspond to non-topical, mildly-topical and topical. From Table 6 we observe that CHRT12 achieves the lowest toxicity rating of only 0.027 with a minimal loss in linguistic quality of 0.024 points as com-
pared to the base GPT-2. Low standard deviation in human annotation scores for CHRT12 further strengthens our argument. Finally, it should be noted that all the entries in Table 6 marked with ∗ have a p-value of greater than 0.05 for a pair-wise T-test with CHRT12. Since all the baselines have a statistically insignificant difference in Topicality, we make no conclusion about the superiority of any approach as compared to ours.

heading: 5 Limitations
text: Our work is limited in capturing the unintended dependencies of attributes. It is possible that maximizing certain attributes like positive sentiment may maximize attributes like gender bias. A formal study to capture the dependency of the bias with varied attribute control is an important future direction. The efficacy automated metrics used to measure the linguistic qualities and attribute alignment of the generations is limited (Jozefowicz et al., 2016). Devising more exhaustive and explainable metrics is also an important future-work.

heading: 6 Conclusion
text: We present CHRT, a learning-based controlled language generation framework that achieves stateof-the-art attribute control while minimizing the loss in linguistic quality as compared to five recent baselines. The ability to combine control over multiple attributes and the ultra-fast inference of our approach makes it ideal for latency-constrained usecases. We empirically showcase the effectiveness of our approach by performing both large-scale automated and human-evaluated benchmarks. For future work, we will like to work on making our approach more plug-and-play and achieve attribute control with an even lower loss in linguistic quality.

heading: A CHRT Ablation Study
text: We perform ablation studies to justify the transformation block design choices.
A.1 Number of blocks As shown in Figure 1c, our transformation block is a concatenation of two identical blocks. In this ablation study we vary the number of identical blocks. We perform the same experiment as in Section 4.2 to evaluate attribute control of toxicity and linguistic quality of generation. For our models, we take CHRT12 with varying number of blocks. Table 7 summarizes this ablation study. We can observe that more the number of blocks, the better performance we get in terms of both reducing perplexity
and increasing attribute control. However, the gain is minimal after more than 2 blocks thus justifying our choice of having 2 identical blocks in our transformation.
A.2 Intermediate Hidden Dimension The constituent blocks of our transformation block have linear layers with the intermediate dimension of κ ∗ h where κ ∈ R is the multiplication factor and h is the hidden layer dimension of the base model. As shown in Figure 1c, we have selected κ to be 0.5. We justify this choice empirically through a similar ablation study as for #blocks. Table 8 shows that for κ = 0.5, we have the highest fluency (lowest perplexity) with a minor loss in attribute control (toxicity). Further, the number of weights for selecting κ = 0.5 as compared to 1 or 2 is drastically less, leading to faster training and inference. Therefore, we select κ to be 0.5.

heading: B Training and Generation Details
text: All training and inference are performed on a machine with 8 Tesla V100 GPUs using the Hugginface and PyTorch library. For the baselines, we directly run their publicly available code without modifications.
B.1 GPT-2 GPT-2 Medium with 355 million parameters is used as our base model in the paper. For generation, we use huggingface’s generate function with the following hyper-parameters:
1. No. of Generations: 25 2. Max-Length: 25 3. Top-p: 0.8 4. Repetition Penalty: 1.2
B.2 NLC For NLC, we use huggingface library’s bad_words_ids parameter to negatively constrain the model over a bag of word. We do a search for PMI threshold θ to create the bag of bad words. We select the one with highest sentiment control and lowest loss in generation quality. The best size of negative words is 123,950, 94,83, and 74,502 for simplicity, toxicity and sentiment control respectively. Rest of the generation parameters are same as GPT-2.
B.3 DAPT We fine-tune DAPT on positive attribute class for our experiments. This includes non-toxic text as extracted from RealToxicityPrompts, positive sentiment text and simple text from RealSentimentPrompts and RealSimplicityPrompts respectively. The details for these datasets can be found in Appendix. We again use huggingface library’s generate function with same parameters as for GPT-2.
B.4 PPLM We train toxicity, sentiment and simplicity classifiers using the original authors public code. All the hyper-parameters are preserved as in the original paper. For a fair comparison, we retrain these classifiers on the same data as rest of the baselines using the original author’s released codebase 5.
B.5 GeDi Similar to PPLM, we retrain our own GeDi models for toxicity, sentiment and simplicity on same
5https://github.com/uber-research/PPLM/
datasets using the code released by authors 6. All the training and generation hyper-parameters are preserved from the original paper and the official released code is used without modifications.
B.6 DExperts The experts and dexperts models are retrained using the author’s puiblicly released code 7. For generation, the authors suggest varying the α parameters which controls the strength of generation. For each experiment, we vary α from 1 to 4 in steps of 0.2 and select the best α in terms of attribute control with minimal loss in perplexity. We select α to be 2, 3.2 and 3.0 for detoxification, sentiment control and simplicity attribute control experiments respectively because the gain in attribute control was minimal as compared to loss in perplexity above these values. Finally, we generate 25 generations of length 25 tokens each, with the Top-p for nucleus sampling being 0.9, the value authors suggested.
B.7 CHRT For fine-tuning the guider modelsLM+ andLM− we use the huggingface’s fine-tuning scripts with the difference that we lock the language-modelling head layer weights, as explained in Section 3.3. Following are the hyperparameter details for finetuning, all of them are the default parameters from the huggingface finetuning script:
1. Model: gpt2-medium 2. No. of Parameters: 345M 3. No. of Epochs: 3 4. Block Size: 128 5. No. of Gradient Accumulation Steps: 16 6. Optimizer: AdamW 7. Learning Rate: 2e-5 8. Learning Rate: 2e-5 9. Effective Batch Size: 32
For training the transformation head, we use the following hyperparameters:
1. No. of Epochs: 3 2. Effective Batch Size: 16 3. Optimizer: AdamW 4. Learning Rate: 2e-5 5. Triplet Loss Margin: 1
6https://github.com/salesforce/GeDi 7https://github.com/alisawuffles/DExperts
6. Triplet Loss p: 2 Finally, for generation we use the following hy-
perparameters:
1. No. of Generations: 25 2. Max-Length: 25 3. Top-p: 0.8 4. Repetition Penalty: 1.2

heading: C Generation Examples
text: We show example continuation generation for a randomly selected prompt for all the baselines. Table 9 and Table 10 shows example continuations for the attribute of toxicity and sentiment.
ACL 2023 Responsible NLP Checklist
A For every submission: 3 A1. Did you describe the limitations of your work?
5
3 A2. Did you discuss any potential risks of your work? 5
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4.6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 4
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4.6
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. MTurk
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We used mechanical turk

heading: Future Work
text: For future work, we will like to work on making our approach more plug-and-play and achieve attribute control with an even lower loss in linguistic quality