Focused Prefix Tuning for Controllable Text Generation
Year: 2023

Abstract:
In a controllable text generation dataset, there exist unannotated attributes that could provide irrelevant learning signals to models that use it for training and thus degrade their performance. We propose focused prefix tuning (FPT) to mitigate the problem and to enable the control to focus on the desired attribute. Experimental results show that FPT can achieve better control accuracy and text fluency than baseline models in single-attribute control tasks. In multiattribute control tasks, FPT achieves comparable control accuracy with the state-of-the-art approach while keeping the flexibility to control new attributes without retraining existing models.

heading: 1 Introduction
text: Controllable text generation aims to generate text associated with a specific attribute. For example, given an attribute TOPIC = sports and a prompt “There is,” a model is supposed to generate a continuation whose TOPIC is sports, such as “There is a tennis match ...”.
In datasets for the controllable text generation task, there exists the annotated attribute, and we call it an explicit attribute (e.g. the TOPIC attribute in the AGNews dataset). In addition to the explicit attributes, the datasets tend to have their own tendency. For example, up to 98% of training data pieces in the IMDb dataset exhibit “TOPIC = sci/tech”, while up to 94% of training data pieces exhibit “SENTIMENT = negative”.1 We call the tendency an implicit attribute (e.g. the TOPIC attribute in the IMDb dataset).
The existence of the implicit attributes could degrade the performance in controlling for an explicit attribute when models are trained on the datasets. Since implicit attributes are of dataset-level and related to undesired explicit attributes, the probability
1The models used for classification are from (Gu et al., 2022).
of generating content with the implicit attributes is first likely to increase. When the text with the implicit attributes was generated, the probability of generating content with other undesired explicit attributes would increase, and the text with them might be generated next. As a result, as shown in Table 1, the model generates content with a high implicit attribute relevance but a low desired explicit attribute relevance (e.g. Vanilla Prefix Tuning (Li and Liang, 2021)). In contrast, if the model generates content with a low implicit attribute relevance, it will have a high desired explicit attribute relevance (e.g. DExperts (Liu et al., 2021). We call this phenomenon attribute transfer.
To mitigate the effect of the attribute transfer, we propose focused prefix tuning (FPT), which makes the generation focused on the desired explicit attribute. FPT uses specific and general prefixes to encode the explicit and implicit attributes, respectively. FPT combines the control power of the two prefixes via logits manipulation at inference time. Experimental results show that FPT achieved better control accuracy and fluency in single-attribute control tasks. In multi-attribute control tasks, FPT can achieve comparable performance with the state-ofthe-art approach. Moreover, we show, since FPT enables the training of each attribute prefix individually, we can incrementally add new attributes without retraining all prefixes.
1116

heading: 2 Related Work

heading: 2.1 Controllable Generation
text: Methods for controlling text generation have rapidly developed (Ficler and Goldberg, 2017; Dathathri et al., 2020; Madotto et al., 2020; Chan et al., 2021). Keskar et al. (2019) trained a large transformer model to generate contents conditioned on up to 55 attributes. However, the cost of training such a model is too high.

heading: 2.2 Prefix Tuning
text: Parameter-efficient fine-tuning (PEFT) methods, such as prompt tuning (Lester et al., 2021) have become particularly significant in driving various natural language processing tasks to reduce the high training cost. Prefix tuning (Li and Liang, 2021) is one of the PEFT methods that steers pretrained models (Radford et al., 2019; Lewis et al., 2020) by applying an additional continuous vector embedding before every activation layer. Qian et al. (2022) proposed a contrastive prefix tuning method that improves its performance by utilizing the relations between attributes. However, they focused only on attributes explicitly annotated and ignored the effect of implicit attributes.

heading: 2.3 Inference-time Methods
text: Inference-time methods (Mireshghallah et al., 2022; Yang and Klein, 2021; Dathathri et al., 2020; Madotto et al., 2020), a lightweight approach without updating the parameters, have been used for controllable text generation. To enhance controllability, Krause et al. (2021) proposed a method to combine the computed classification probability distributions. Liu et al. (2021) found that directly applying probability distributions from language models is a simple but effective approach to control generated texts. Inspired by their work, we propose a method that uses probability distributions from language models to remove the effect of implicit attributes.

heading: 3 Focused Prefix Tuning
text: The task of controllable generation is, given a sequence of prompt tokens x<t and an attribute ATTR = val (e.g. TOPIC = sports), to generate a sequence of tokens as a continuation x that conforms to both the prompt and specified attribute.

heading: 3.1 Vanilla Prefix Tuning
text: In controllable text generation, a prefix can steer a pre-trained model parameterized by θ to generate texts under a specific attribute value ATTR = val. In particular, vanilla prefix tuning (Li and Liang, 2021) prepends a set of continuous vectors before every activation layer of the pre-trained transformer. The continuous vectors are referred to as the prefix Hattr=valϕ , which is parameterized by ϕ.
During training, we freeze the pre-trained model’s parameters θ and update only the prefix parameters ϕ to optimize the following objective:
− ∑
x∈Dattr=val logP (xt|x<t, Hattr=valϕ , θ), (1)
where Dattr=val is the subset of the entire dataset D whose attribute ATTR is val.
Following Li and Liang (2021), we initialize the prefix Hϕ with the activation of actual tokens from the pre-trained model’s vocabulary.

heading: 3.2 Specific and General Prefixes
text: The prefix in vanilla prefix tuning captures an explicit attribute in a dataset by training it on the subset dataset Dattr=val. To capture only implicit attributes while ignoring any explicit attributes, we propose to train another prefix on the entire dataset D. To distinguish the two prefixes, we refer to the prefix trained on Dattr=val as a specific prefix and that trained on D as a general prefix.
The specific prefix is the same as the prefix in vanilla prefix tuning, so we still use Equation 1 to update its parameters. To update the general prefix’s parameters, we optimize the following objective:
− ∑
x∈D logP (xt|x<t, Hgenlϕ′ , θ), (2)
where Hgenlϕ′ represents the general prefix, which is parameterized by ϕ′.

heading: 3.3 Inference-time Logits Manipulation
text: As shown in Figure 1, FPT suppresses the probability of words with implicit attributes in the generated text by combining logits zattr=val steered by the specific prefix and logits zgenl steered by the general prefix via logits manipulation at inference time. For example, when generating text with the attribute TOPIC = sports, the probability of words with implicit attributes (e.g. “impossible” with SENTIMENT = negative) would be suppressed. During
inference, at each step t, we first make two forward runs respectively with the specific and general prefixes to obtain their logits, zattr=valt and z genl t . Since zattr=valt encodes both the explicit and implicit attributes while zgenlt encodes mostly the implicit attributes, we use a subtraction operation at the logits level to suppress the probability of words with implicit attributes:
P (xt|x<t, ATTR = val) = P (xt|x<t, Hattr=valϕ , Hgenlϕ′ , θ) = softmax(αzattr=valt − (α− 1)zgenlt ), (3)
where α is a hyperparameter that can be interpreted as the strength for the control of implicit attributes. Following Liu et al. (2021), we respectively set α and α − 1 as the weight of zattr=val and zgenlt to make the ratio of logits after the logits manipulation equal to 1.
To ensure the fluency of generated texts, we follow Liu et al. (2021) to use top-p filtering to remove the tokens that have low scores in advance before logits manipulation. In particular, we modify the logits produced by the specific prefix by calculating the top-p vocabulary Ṽ and setting all the logits outside Ṽ to −∞:
z̃[v] = { z[v], if v ∈ Ṽ −∞, if v /∈ Ṽ . (4)
Therefore, the logits manipulation in Equation 3 is updated as follows:
P ′(xt|x<t, ATTR = val) = softmax(α ˜zattr=valt − (α− 1)zgenlt ). (5)
The token at step t is then selected by ancestral sampling from P ′(xt|x<t, ATTR = val).

heading: 3.4 Multi-attribute FPT
text: FPT is also applicable to the multi-attribute control task, where we aim to control multiple different attributes at the same time. Similarly, we first train the specific prefix for each attribute. Then, we adapt logits manipulation to the multi-attribute task as follows:
P ′(xt|x<t,{ATTRi = vali}1≤i≤K)
= softmax( ∑K
i=1 zattrit ), (6)
where K is the number of different attributes. Each zattrit is the combination of the logits from the corresponding specific prefix and general prefix. Since applying top-p filtering to every attribute could possibly result in an empty Ṽ , we apply the filtering only to the first attribute:
zattrit =    α ˜ zattri=valit − (α− 1)zgenlit , if i = 1
αzattri=valit − (α− 1)zgenlit , otherwise (7)

heading: 4 Single-attribute Control Experiments

heading: 4.1 Models
text: GPT-2 (Radford et al., 2019): We used the public checkpoint of GPT-2 Medium as the most common baseline.2 DExperts (Krause et al., 2021): A fine-tuning method applying logits manipulation in the inference step. GeDi (Krause et al., 2021): A method combining the classification probabilities for possible next tokens in the inference step. Vanilla prefix-tuning (Li and Liang, 2021): The common prefix-tuning method. Contrastive prefix-tuning (Qian et al., 2022): A strong baseline that takes into account the relationship between attributes.
2The checkpoint of GPT-2 Medium is from https://huggingface.co/gpt2-medium.
We also set up one variant of FPT: Contrastive FPT: Applying contrastive prefix tuning to train specific prefixes. We also set an ablated model that uses the logits of the frozen GPT-2 instead of the logits from the model guided by our general prefix.

heading: 4.2 Experimental Settings
text: Following previous work (Krause et al., 2021; Qian et al., 2022), we evaluated the models on a topic control dataset AGNews (Zhang et al., 2015) and a sentiment control dataset IMDb (Maas et al., 2011). We score the sentiment relevance using HuggingFace’s sentiment analysis classifier (Liu et al., 2019) trained on 15 datasets. For scoring topic relevance, we trained the classifier that obtained comparable results to what was reported. Perplexity was used to evaluate text fluency. Bias (|relevance score − 50|) is how much the relevance of implicit attributes deviated from unbiased relevance (50). We set TOPIC = science as the implicit attribute in the sentiment control generation, and SENTIMENT = negative as the implicit attribute in the topic control generation. Prompts from Chan et al. (2021) were used to generate continuation samples. We generated 20 samples for each attribute and prompt. More details are listed in Appendix A.1 and A.2.

heading: 4.3 Experimental Results
text: As shown in Table 2, in the single-attribute control tasks, Contrastive FPT achieves higher attribute
relevance than prefix tuning-based baselines while having lower bias scores. This indicates that the generated texts are well controlled under the target explicit attribute without transferring by implicit attributes. In FPT, the perplexity score is the best among control-based baselines. The ablation experiment suggests that the proposed general prefix is essential for attribute control.
Table 3 shows the generation samples of SENTIMENT = positive from our models and baselines. In the FPT based model, there are more words with desired explicit attributes in generated texts, while there are more words with undesired explicit attributes contained in the baselines. More generation samples are given in Appendix B.

heading: 5 Multi-attribute Control Experiments

heading: 5.1 Models
text: In the multi-attribute control experiments, we added Distribution Lens (Gu et al., 2022) as a strong baseline. It searches for the intersection space of multiple attribute distributions as their combination for generating.

heading: 5.2 Experimental Settings
text: To explore the ability of FPT in the mult-attribute control task, we added a toxic comment dataset3 for toxicity control. We used additional Google Per-
3https://www.kaggle.com/c/jigsaw-toxic-commentclassification-challenge/
spective API4 to evaluate the relevance of toxicity. Since it is meaningless to generate toxic content, so we only apply the non-toxic attribute in this task. We chose the first attribute as the topic attribute because we found that the filtered vocabulary size in logits manipulation of a topic attribute is larger than the other attributes (sentiment and nontoxic). The prompts used for generating samples are the same as in the sentiment control task. For each prompt, we generated 20 samples per attribute combination. More details are listed in Appendix A.3.

heading: 5.3 Experimental Results
text: Table 4 shows that our method can obtain comparable performance with the state-of-the-art approach. Distribution Lens, however, requires aggregating the datasets of all attributes to train its prefixes. If they hope to add a prefix to control a new attribute,
4https://www.perspectiveapi.com/
they have to retrain all the prefixes. In contrast, FPT trains a prefix for each attribute individually and enables new attribute control prefixes to be added incrementally without retraining existing ones.

heading: 6 Conclusion
text: We proposed FPT, a prefix tuning-based method, to mitigate the effect of attribute transfer. FPT could encode implicit attributes in a dataset by a general prefix and use it to suppress the attribute transfer via inference-time logits manipulation. Results in the single-attribute control experiments showed that, with FPT, the generated texts can be more effectively controlled under the desired attribute with higher text fluency. Experimental results in the multi-attribute control suggested that FPT can achieve comparable performance to the state-ofthe-art approach while keeping the flexibility of adding new prefixes without retraining.

heading: 7 Limitations
text: Although FPT shows better control ability, there are two points that need to be improved in the future. First, as in Gu et al. (2022), we need to select hyperparameter α to balance between the control ability and fluency in generated texts. Second, as shown in Table 5, although the time cost of FPT is lower than that of GeDi, it is higher than those of other prefix tuning-based methods and grows approximately linearly by a factor of 2 × number of attributes.

heading: A Experiment Setting Details
text: All the experiments are conducted on the basis of a GPT-2 Medium model. We freeze the parameters of the GPT-2 model when training all the prefixes. The length of all prefixes is set equal to 10. The GPU used for all training is a P40.
A.1 Topic Control
Following the previous work (Qian et al., 2022), we use half of the data pieces in the AGNews dataset to obtain the general prefix and specific prefix. The number of specific prefixes for this task is 4 (e.g. worlds, sports, business, and science). We set epochs to 10 and the batch size to 8. We use AdamW as the optimizer and set the learning rate to 1e-4. To balance the performance between fluency and controllability, the hyperparameters α for generation are set to 1.1 and the top-p is set to 0.8. The average training time for each prefix is 3 hour for 1 GPU. Following Gu et al. (2022), the classifier is trained on the Deberta model (He et al., 2021), which is used to compute attribute relevance in this task.
The prompts for evaluation: “In summary,”, “This essay discusses”, “Views on”, “The connection”, “Foundational to this is”, “To review”, “In brief ”, “An illustration of ”, “Furthermore”, “The central theme”, “To conclude”, “The key aspect”, “Prior to this”, “Emphasized are”, “To summarize”, “The relationship”, “More importantly”, “It has been shown”, “The issue focused on”, and “In this essay”.
A.2 Sentiment Control
Following the previous work (Qian et al., 2022), we use half of the data pieces in the IMDb to get the general prefix and specific prefix. The number of specific prefixes for this task is 2 (e.g. positive and negative). We set the batch size to 8, and the number of epochs to 50. We use AdamW as the optimizer, and the learning rate is set to 2e-5. To balance the performance between fluency and controllability, the hyperparameter α for generation is set to 3 and the top-p is set to 0.8. We spend 4 hours on average for each prefix.
The prompts for evaluation: “Once upon a time”, “The book”, “The chicken”, “The city”, “The country”, “The horse”, “The lake”, “The last time”, “The movie”, “The painting”, “The pizza”, “The potato”, “The president of the country”, “The road”, and “The year is 1910”.
A.3 Multi-attribute Control For the non-toxic attribute, we use 10,000 pieces of non-toxic labeled data to train the specific prefix. Then use another 20,000 pieces randomly sampled from the whole dataset to train the general prefix. In the multi-attribute control task, we set the batch size to 8 for training the non-toxic specific prefix and general prefix. We use AdamW as the optimizer, and the learning rate is set to 1e-4. To balance the performance among attributes from different aspects, the combination of hyperparameters for generation is:
To decide the first attribute, we choose 20 different prompts as input and obtain the filtered vocabulary sizes of different attributes. The average sizes of filtered vocabularies are shown in Table 7. We choose the attribute with the largest filtered vocabulary size in logits manipulation. When new attributes are added, this method can be used to decide the first attribute.
The prompts used for evaluation: “Once upon a time”, “The book”, “The chicken”, “The city”, “The country”, “The horse”, “The lake”, “The last time”, “The movie”, “The painting”, “The pizza”, “The potato”, “The president of the country”, “The road”, and “The year is 1910”.

heading: B Generated Samples
text: The more samples generated by our models and baselines are shown in Table 8, 9, 10, 11.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
7 A2. Did you discuss any potential risks of your work? Our work is a foundational research and does not contain potential risks. Our experiments are fair.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 and Section 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4 and Section 5

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 and Section 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used open-source datasets, so there is no problem of anonymization.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4, Section 5 and Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4, Section 5 and Appendix A
C 3 Did you run computational experiments? Section 4 and Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Although FPT shows better control ability, there are two points that need to be improved in the future. First, as in Gu et al. (2022), we need to select hyperparameter α to balance between the control ability and fluency in generated texts. Second, as shown in Table 5, although the time cost of FPT is lower than that of GeDi, it is higher than those of other prefix tuning-based methods and grows approximately linearly by a factor of 2 × number of attributes