PCFG-Based Natural Language Interface Improves Generalization for Controlled Text Generation
Year: 2023

Abstract:
Existing work on controlled text generation (CTG) assumes a control interface of categorical attributes. In this work, we propose a natural language (NL) interface, where we craft a PCFG to embed the control attributes into natural language commands, and propose variants of existing CTG models that take commands as input. In our experiments, we design tailored setups to test the model’s generalization abilities. We find our PCFG-based command generation approach is effective for handling unseen commands compared to fix-set templates. Further, our proposed NL models can effectively generalize to unseen attributes (a new ability enabled by the NL interface), as well as unseen attribute combinations. Interestingly, in model comparisons, the simple conditional generation approach, enhanced with our proposed NL interface, is shown to be a strong baseline in those challenging settings.

heading: 1 Introduction
text: With the advancement of large-scale pretraining, language models (LM) are now able to generate increasingly more realistic text (Radford et al., 2019; Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Smith et al., 2022; Thoppilan et al., 2022). Therefore, how to control the generation of LMs has become an important research topic. In controlled text generation (CTG), a series of works (Keskar et al., 2019; Dathathri et al., 2020; Krause et al., 2021; Yang and Klein, 2021; Liu et al., 2021; Yu et al., 2021; Li et al., 2022, inter alia) propose model frameworks to generate text conditioned on some desired (user-specified) attribute a. These attributes, which depend on the datasets of interest, could be topic, formality, sentiment, etc.
An important assumption behind this controlled generation setting is that the attributes are chosen from a fixed set (i.e., they are treated as categorical random variables). Although this setting is convenient, it seriously limits the applications of the
CTG system: (1) Since the attribute set is fixed during training, it would be impossible for the model to generalize to unseen options if used as-is. (2) This interface is not very human-friendly, because it could be difficult for users to navigate through the (possibly long) lists of options. Motivated by these limitations, in this work we propose a natural language interface for CTG, illustrated in Figure 1. With this change of interface, the input to the CTG model changes from one-hot vectors to natural language commands (for short, commands). To efficiently train this system and enable it to generalize, we design a probabilistic context-free grammar (PCFG) to embed categorical attributes into a diverse set of natural language commands.
Using natural language instruction has been explored in recent work (Sanh et al., 2021; Wei et al., 2022; Mishra et al., 2022; Reif et al., 2022; Schick and Schütze, 2021). Our work differs from theirs in (1) We focus on the task of CTG as opposed to the performance on cross-task generalization, and design tailored scenarios for evaluation. (2) We introduce PCFG for command generation, which has
295
not been explored by previous work. We discuss this relationship in more detail in Section 2.
The change of interface brings several immediate benefits: (1) Natural language inputs enable the system to generalize to unseen attribute options (as long as they can be expressed in natural language). (2) Unlike fixed-set template sentences in previous works, the PCFG can generate diverse natural language variation during training, which we will show is crucial for generalization. (3) The input process becomes more natural and interactive to a human user, and it can be linked with, for example, a speech recognition module.
With this new interface, we propose variants of several existing CTG systems that take commands as input, and design experiments to compare different CTG models under tailored scenarios. We briefly summarize our main contributions below:
• We propose a PCFG-based natural language interface for controlled text generation. The natural language interface enables zero-shot generalization on control attributes unseen during training, a capability previously impossible due to the fixed-set assumption.
• We show that training with commands generated by a PCFG is an effective method for increasing natural language variation over using fixed-set templates, allowing natural language CTG models to better generalize to commands unseen during training.
• We test the proposed natural language CTG models on settings where the models need to generalize to unseen attributes and attribute combinations. Surprisingly, the simple conditional generation approach is shown to be a strong baseline in these challenging setups.

heading: 2 Related Work
text: Controlled Text Generation In open-ended text generation, a series of approaches have been proposed to control the generation to satisfy certain attributes (e.g. topic) (Keskar et al., 2019; Dathathri et al., 2020; Krause et al., 2021; Yang and Klein, 2021; Liu et al., 2021, inter alia). Some of these studies utilize a trained classifier to guide the generative model towards the desired attribute, while others use a smaller LM to reweight LM logits. Very recently, Li et al. (2022) focus on controlling more complex attributes such as syntactic structure with a non-autoregressive LM. Another line
of work conducts CTG via prompt learning (Clive et al., 2022; Yang et al., 2022). These work assume a fixed set of control attributes.
Our NL interface is more related to Yu et al. (2021), which uses an attribute alignment function to embed attribute words into a hidden representation that guides LM generation. The attribute alignment function does not assume attribute tokens are from a fixed set, so it is possible to do inference on an attribute token not seen in training. Keyword2Text (Pascual et al., 2021) shift the distribution over vocabulary toward words that are semantically similar to control keywords in a discriminatorfree manner, thus does not assume a fixed set of keywords. Besides attribute control, lexically constrained decoding (Post and Vilar, 2018) has also been used to enforce certain key phrases to be included in the generation (Mao et al., 2020). Different from these work which uses keywords, we utilize PCFG to construct fully-natural-language sentences as commands.
Instruction Following A recent series of work proposes to describe NLP tasks in natural language, and use the task description as an instruction to promote zero-shot generalization for LMs (Sanh et al., 2021; Wei et al., 2022, inter alia). Such task descriptions are manually created, detailed definitions of NLP tasks, which contain explanations about input, output, emphasis, and possibly a small number of demonstrative examples. InstructGPT (Ouyang et al., 2022) uses an RL policy to improve LM’s capability to follow user instructions.
Although our work resembles these works in the form of natural language instructions, we note several important differences. First, existing works focus on general instruction following that is applicable to a very broad range of tasks and evaluate on generalization capabilities across tasks. We specifically consider the use of NL commands in the CTG setting and compare variants of CTG models in tailored test scenarios. Moreover, previous works in natural language instruction employ a fixed number of templates for each task, whereas we craft a PCFG that can generate a diverse set of command sentences to serve as templates. We show the effectiveness of our PCFG over fixed-set templates in subsequent experiments in Section 5.1. Finally, prompting models with NL instructions fails for moderately sized LMs without any modifications Li and Liang (2021). Thus, it is non-trivial to adapt NL instruction to smaller models.

heading: 3 Framework
text: The goal of controlled text generation is to model the conditional distribution P px|aq so that the generated text x satisfies the desired attributes a. a could include multiple attributes (e.g., topic and length), and we will use ai to denote the ith attribute. In the standard categorical setting, the attribute ai are from a fixed set of pre-defined options. We assume there are m attributes of interest (m ď 2 in our experiments). In the next few sections, we describe the PCFG that we craft to embed the categorical attributes, and our proposed NL variants of several existing CTG systems.

heading: 3.1 Embedding Attributes into Commands
text: We embed categorical attributes into natural language commands with a PCFG.1 We favor PCFG due to its ability to generate diverse NL variations expressing the same control semantics. For simplicity, most of the probability weights are set to uniform. In this section, we will describe it at the high-level, and more details and the full set of rules are provided in Appendix C. Table 1 is a concrete example of how a command describing an AG news article with a sports topic could be generated by our PCFG. We clarify that while the PCFG is used for training and testing in our work, the end user will not need to use it, as the model can generalize to unseen commands (Section 5.1).
Our command generation has three steps. First, a template with m attribute slots is generated by the PCFG. We design the PCFG to generate templates that “ask” the system to generate text with some attributes and domains. We first sample a top-level seed template from ROOT that determines high-level sentence structure (e.g., [PLS] [HEAD-FORM] a [TEXT-FORM] [LABEL-SEG]), then fill in sentence segments with PCFG rules (e.g., [HEAD-FORM] will be substituted by “generate”). These sentence segments are neither domain nor attribute specific and thus can be used regardless of the attributes. In contrast to writing a set of fixed templates, our PCFG has multiple levels of rule and can greatly improve NL variation.
Next, we verbalize the domain media D, attribute a, and attribute name A into natural language by crafting PCFG rules that transform them into words or phrases. Considering the fact that different words could have similar meanings in natural
1Note that our command generation process is not strictly a PCFG, but it is very close.

heading: 1. PCFG-based template generation

heading: 2. Verbalize

heading: 3. Postprocess
text: language, these mappings could be one-to-many to further improve NL variation. For instance, news about “business” can also be described as “commerce”, and “very negative” is similar to “terrible”.
Finally, we conduct a postprocessing step to correct simple grammar errors, e.g., “a AG news article” would be corrected as “an AG news article”.
In our preliminary attempts, we attempted to train a conditional neural LM for command generation, instead of using a PCFG. Although the neural model has better diversity, the stochastic nature of sampling makes the attribute embedding inaccurate. Besides, training such a neural LM would require a large amount of (attribute, command) paired data. Therefore we turn to a PCFG approach as it has guaranteed accuracy, with decent diversity.

heading: 3.2 Models
text: In this section, we first review some existing CTG models. For the new NL interface, we propose natural variants of the models which take commands as input. All models are based on a pretrained autoregressive LM, denoted by Pb.

heading: 3.2.1 PrefixLM
text: A direct method to model the conditional distribution P px|aq is to encode the attribute as a prefix and finetune the base model to generate x conditioned on the prefix. In the standard categorical attribute setting, we randomly initialize an embedding vector for each attribute and feed the corresponding embeddings as the prefix. Multiple attributes are arranged in a pre-defined order.
PrefixLM-NL The NL variant of PrefixLM is straightforward. We just use the command as the prefix. No extra parameters need to be added.

heading: 3.2.2 Future Discriminator Controlled Generation (FUDGE)
text: FUDGE (Yang and Klein, 2021) decomposes the conditional distribution using Bayes’ rule according to Equation 1:
Pfudgepxi|x1:i´1, aq 9 Pbpxi|x1:i´1qPclspa|x1:iq. (1)
It involves training a future discriminator to predict whether the generated prefix x1:i will lead to a full generation that satisfies the attribute a. Following FUDGE’s original formulation, we assume different attributes are conditionally independent and train a discriminator P pak|x1:iq for each attribute ak. We then use their product as the probability that all attributes are satisfied, i.e., P pa1, . . . , am|x1:iq “ śk P pak|x1:iq.
As we consider attributes with multiple options (e.g., 4 topics or 5 sentiments), the FUDGE discriminator for a single attribute is a multiclass classification model that predicts the conditional distribution P pa|x1:iq over all possible options of attribute a. FUDGE-NL In order to enable FUDGE to handle natural-language commands, we utilize a binary alignment discriminator to judge whether the generated text aligns with the command. Given a command c, let yc P t0, 1u be a binary variable that denotes whether the prefix x1:i aligns with the command. Control is achieved by generating from the conditional distribution P pxi|x1:i´1, yc “ 1q that the alignment property is satisfied. We modify FUDGE’s decomposition as Equation 2:
Pfudge-nlpxi|x1:i´1, yc “ 1q 9 Pbpxi|x1:i´1qPclspyc “ 1|x1:iq. (2)
Pclspyc “ 1|x1:iq is modeled by a binary classifier trained on a dataset of command and generation prefix pairs tpc, x1:iqu. To create this data, for a given example text x with attributes a, we first apply our PCFG to generate a true command cpos. We then randomly flip one (or both) of the attribute in a, and generate a false command cneg. By pairing cpos and cneg with x, we obtain the positive/negative training data for the discriminator. In practice, we concatenate the command and generation prefix (separated by a special [SEP] token) and feed it as input to the alignment discriminator.
FUDGE-Binary One major difference between FUDGE and its NL variant is that the discriminator
is always binary for FUDGE-NL due to the alignment objective. This inspires us to propose a binary variant of the FUDGE model, FUDGE-Binary, which operates with the categorical interface. Similar to FUDGE-NL, we use a binary variable ya to denote whether x1:i aligns with attribute a, and modify the decomposition as:
Pfudge-binpxi|x1:i´1, ya “ 1q 9 Pbpxi|x1:i´1qPclspya “ 1|x1:iq. (3)
FUDGE-Binary’s discriminator will always make a binary prediction even if there are more than two options for a single attribute. Since attributes are still from a fixed set, we use a single classification model but attach a separate classifier head for each option. During training, the classification head Wa˚ that matches the correct attribute a˚ receives a correct label y “ 1, and all other classification heads tWaua‰a˚ receive label y “ 0. At test time, we select the classification head Wa base on the desired attribute a to predict the alignment probability P pya “ 1|x1:iq. Although this variant is a simple modification from the original FUDGE, empirically we find it to achieve stronger performance in the categorical interface.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: We utilize two popular text classification datasets for our experiments: AG News and Yelp Review.2 For each dataset, we consider two control attributes: label and length. The label attribute is extracted from the classification label, i.e., topic labels for AG News and sentiment labels for Yelp Review. There are 4 topics {world, sports, business, science/tech} in AG News and 5 sentiment classes ranging from most positive to most negative in Yelp Review. The length attribute is created by dividing the dataset to nlen length ranges so that number of training examples in each length range is balanced. We use nlen “ 3 for AG News and nlen “ 5 for Yelp Review. We refer readers to Appendix A for details about dataset preprocessing.

heading: 4.2 Evaluation Metrics
text: We measure the generation performance in three aspects: control accuracy, quality, and diversity. In our experiments, we find that different variants of models mostly perform comparably on quality or
2Obtained from Hugging Face Datasets.
diversity aspects. Therefore, we will mainly focus our discussion on control accuracy.
Control Accuracy To evaluate the effectiveness of the control, we consider three types of control accuracy: LABEL ACCURACY refers to the accuracy that the generation satisfies the classification label, i.e., topic classification accuracy on AG News and sentiment classification accuracy on Yelp. This metric is computed by a RoBERTa classifier finetuned on the corresponding classification dataset. LENGTH ACCURACY refers to the accuracy that the generation’s tokenized length lies within the predefined length range. COMPOSITIONAL ACCURACY is the accuracy that both label and length attributes are satisfied.
Text Quality We consider two metrics to measure the quality of the generated text. GPT-NEO PERPLEXITY (G-PPL): we finetune the GPT-Neo1.3B model3 on the corresponding datasets (without the labels), and report the perplexity of the generated text given by it. BLEU score: we randomly sample 100 examples from the AG News or Yelp test set as the reference, and compute the 4-gram BLEU score.
Diversity We measure diversity of the generated text using 4-gram TEXT ENTROPY (Zhang et al., 2018). That is, treat the generated token frequency as a discrete distribution, and compute its entropy.

heading: 4.3 Model Instantiation
text: Here we describe the implementation of models mentioned in Section 3.2. We use the Hugging Face transformers library (Wolf et al., 2020) and adapt from FUDGE’s released code.4
For all models, we produce generation by top-k sampling with k “ 20 unless otherwise stated. PrefixLM variants We finetune a GPT-2 (Radford et al., 2019) small model without any modification (except for adding necessary special tokens) for both PrefixLM and PrefixLM-NL. At test time, we feed the desired attributes or command sentences as the prefix and evaluate on the continuation produced by the model.
FUDGE variants The backbone language model Pb for FUDGE models is a GPT-2 small model finetuned on the corresponding dataset, using the
3A publicly-available replication of GPT-3 obtained from https://huggingface.co/EleutherAI/gpt-neo-1.3B.
4Our code and data will be released in the public version of this manuscript.
same data available at discriminator training. That is, under the zero-shot setting, we use the same data configuration to finetune the backbone LM.
For FUDGE and FUDGE-Binary, we train two discriminator for each of the label (topic or sentiment) and length attribute; FUDGE-NL use a single alignment discriminator to handle commands.
Each discriminator for FUDGE and FUDGE-NL is a GPT-2 small model followed by a single linear classification layer (with different numbers of output classes). The discriminator for FUDGE-Binary is a GPT-2 small model followed by multiple linear classification layers, with each one corresponding to an option for the label or length attribute. Each classification layer makes a binary prediction about whether the generation prefix satisfies the particular option of the attribute.

heading: 5 Experiments
text: We design experiments to test natural language CTG models’ generalization capabilities, where the models need to generalize to (1) unseen commands (2) unseen attribute options (3) unseen combinations of attribute options. Additionally, we compare natural language CTG models with their categorical counterparts under the standard full-data setting to test whether the NL interface would degrade the model’s performance.

heading: 5.1 Generalization to Unseen Commands
text: A key challenge introduced by the new interface is the diversity of natural language: commands with different surface forms can have the same underlying semantic. Thus we design a set of experiments to test natural language CTG models’ ability to generalize to commands unseen during training. Specifically, we compare the effectiveness of our proposed PCFG with commands generated by fixset templates, as adopted in previous works (Sanh et al., 2021; Wei et al., 2022; Mishra et al., 2022).
To create a setup similar to previous work, we hand-crafted 20 diverse templates for each dataset. This is already twice the number of templates used in Wei et al. (2022) and comparable to the number of seed templates in our PCFG. We denote models trained on this set of templates by “-T20” suffix. We also explore a stronger version of fixset templates by doubling the number of templates, totaling 40 templates for each dataset, denoted by “-T40” suffix. We test the above models on 20 hand-crafted unseen templates that are different
from both the PCFG and fixed-set templates, and compare results with our proposed PCFG-based models, denoted by “-PCFG” suffix.
The results in Table 2, show that when conditioning on unseen commands, both the PrefixLM-NL and FUDGE-NL models with PCFG have notably better controllability compared to fixed-set template models. The above experiments provide empirical evidence that our PCFG can effectively improve the model’s generalization ability on natural language variation within commands.

heading: 5.2 Generalization to Unseen Attributes
text: CTG models with categorical attributes can only control a fixed set of attribute options. It is impossible for these models to control unseen attribute options without re-training due to architecture constraints (e.g., FUDGE trains a classifier with a fixed number of labels). In contrast, our proposed NL interface naturally allows CTG models to generalize to unseen options by embedding novel attributes into an NL command using a verbalizer phrase unseen during training, as long as the novel attributes could be described in natural language. In this section, we conduct experiments to test our PCFGbased natural language CTG models’ capabilities to generalize control to unseen attribute options.
Experimental setup In this section, we control a single attribute (topic) for ease of presentation. Although it is possible to also experiment on the length attribute, they are similar in nature. For an attribute with n classes (e.g., 4 different topics), we create n zero-shot data splits and delete examples from one of the n classes (i.e. the zero-shot class)
completely during training. We test on both the zero-shot and other seen classes separately and report the average result over all n splits. We conduct zero-shot experiments on the AG News dataset.
Adding extra data Since natural-language CTG models do not assume the attribute is from a fixed set of options, it is possible to train the model to control attributes by using extra data with different attribute options. This is another capability enabled by our NL interface, previously unavailable due to the fix-set assumption. We experiment training the models on the zero-shot AG News split along with similar datasets in the news domain, aiming to test whether the model can learn from extra data and generalize to a wider range of attribute options. We utilize three extra news topic classification datasets: News Popularity, News Category (Misra, 2022; Misra and Grover, 2021), and the Inshorts News dataset.5 Topics that overlap with AG News are removed. We refer readers to Appendix A for more details. For these datasets, we use the same PCFG as AG News. When mixing multiple datasets during training, we follow Raffel et al. (2020) and use examples-proportional mixing to control the relative frequency of examples from each dataset. We set the artificial limit of each extra dataset to the size of the original AG News dataset.
The zero-shot results are shown in Table 3. Since the categorical interface does not allow unseen categories, we introduce a no-control baseline by finetuning the base LM with the same zero-shot data and producing generations from it directly without control. Both FUDGE-NL and PrefixLM-NL beat
5Obtained from Hugging Face Datasets and Kaggle.
this baseline. We observe that the simple PrefixLM-NL approach outperforms FUDGE-NL by a large margin in both zero-shot data and zero-shot + extra data setting. Moreover, as measured by both perplexity and BLEU, PrefixLM has higher generation quality as well. While there is still a large gap between the zero-shot and non-zero-shot label accuracy, the extra data approach managed to double the zeroshot accuracy in both NL models, showing the generalization potential of the natural language interface. Qualitatively (shown in Table 8 to Table 11), we found that in cases where the output has the wrong topic, there are still signs that the generation is guided by the command. For example, when we zero-shot on the world topic, we obtain text about sports with multiple country names.
Backbone unblock experiment Due to the nature of the zero-shot experiment, we also block examples of the zero-shot class from the finetuning data of the backbone language model Pb. As a comparison, we try finetuning Pb with full data, while still blocking the zero-shot class from prefix or classifier training, which mimics the setting
where only unlabeled data is available. Results are shown in Table 3 as the “-unb” models. We observe a large performance boost for the FUDGE-NL model. This shows that extra unsupervised data is also helpful for control generalization.

heading: 5.3 Generalization to Unseen Attribute Combinations
text: In this section, we design experiments to test whether the models can generalize to unseen combinations of attributes to test their compositional generalization abilities. We describe our setup for AGNews below, which is similar to Yelp.
Following Lake and Baroni (2018), for each split, we select one of the topic classes (e.g., sports) as the non-compositional class, and for all training samples with this class, we do not include length in attributes or commands (i.e., the model never see combinations of sports and any length attribute in training). Note that the combinations of length attributes and other topics classes are kept (e.g., the model still sees combinations of business and short length). At test time, we set the topic to be the non-compositional class and randomly sample
the length attribute to control. We run experiments across all n possible compositionality splits and report the averaged result.
Results are shown in Table 4, with qualitative examples available in Table 12 to Table 15. We focus on the accuracy gap between this compositionality setting and the full-data setting. PrefixLM-NL has little trouble generalizing to unseen attribute combinations as indicated by the small gap. However, FUDGE-NL performed poorly on generalizing to unseen attribute combinations. Not only did FUDGE-NL’s compositional accuracy drop by a large margin, but it also produced low-quality text.

heading: 5.4 Full-data Setting
text: In the full-data setting, we train the models on all data of the AG News or Yelp review dataset, with the purpose to test whether the new NL interface would degrade the model’s performance. This is the regular setup for existing works on CTG except that we aim to control two attributes simultaneously instead of one. The results for the full-data setting are shown in Table 5, with qualitative examples available in Table 6 and Table 7 in the appendix.
Performance comparison between the NL and categorical interface We notice that the generated text quality and diversity between different models are similar in the full-data setting. While PrefixLM-NL and its categorical variant PrefixLM have similar control accuracy on both datasets, FUDGE-NL consistently outperforms the original FUDGE setup. In either case, the performance of the NL variant is on par with its original model, suggesting our NL interface does not degrade CTG performance in the full-data setting. Somewhat surprisingly, FUDGE-Binary outperforms FUDGENL and the original FUDGE model, especially on
the Yelp dataset where the classification is more difficult. The reason could be that the task of the binary classification is less noisy than the multiclass classification, which leads to stronger control.
Performance across model families Across two datasets, FUDGE-based models outperform PrefixLM models, with the exception that FUDGE does not beat (but is comparable to) PrefixLM on Yelp. This is largely consistent with previous results that discriminator-based CTG approaches can achieve higher controllability than conditional LMs (Yang and Klein, 2021, inter alia). However, as we show in the previous sections, its performance is inferior in the settings requiring NL generalization.

heading: 6 Conclusion
text: In this work, we propose a natural language interface for CTG, where we craft a PCFG to embed categorical attributes into natural language commands. We propose variants of existing CTG models that take commands as input. We design tailored experiments to test the natural language CTG model’s generalization capabilities. We show that our PCFG-based command generation approach is effective for handling unseen commands compared to fix-set templates. Additionally, our proposed NL models can effectively generalize to unseen attributes, an ability newly enabled by the NL interface. Finally, we find the simple PrefixLM approach shows robust generalization ability with the NL interface and outperforms FUDGE-based models, demonstrating significant modeling challenges and potentials with this new interface. We hope our work could motivate further research into this challenging interface for CTG.
Limitations
In this section, we point out several limitations restricted by the scope of our work. While the PCFG we create has decent diversity and is guaranteed to be accurate in embedding attributes, they are still rule-based and could not cover all the variations in natural language.
The natural language interface brings modelling challenges. The CTG model is now required to first extract salient information from the command sentence, while in the original categorical interface they are provided directly.
In this work, we have focused our experiments on PrefixLM and FUDGE. While these approaches are representative, there are still other relevant models we did not test. For instance, guiding the generation of an LM with a smaller LM (Liu et al., 2021), or prompt-based CTG approaches such as Yang et al. (2022). It would also be interesting to test how other models perform under the NL interface.
Finally, while we experiment with controlling more than a single attribute in a single CTG model, in principle a NL command could be more complex and fine-grained. For example, it is possible to describe detailed semantic or syntactic constraints in a command sentence, and we leave those to future work.
Ethics Statement
We acknowledge controlled text generation is potentially capable of generating harmful outputs such as producing offensive languages or hate speech. However, it is also shown in previous work that controlled text generation techniques can achieve text detoxification if used properly (Dathathri et al., 2020; Krause et al., 2021). When changing the control interface from a categorical setting to natural language commands, we are giving the user a larger freedom of input. Thus, extra care should be taken when deploying naturallanguage controlled text generation models to the general public to avoid malicious user inputs.

heading: A Dataset Details
text: A.1 Main datasets
Yelp Review This is a dataset of user-written reviews for Yelp. It is a text classification dataset where the 5-sentiment labels are inferred from 1 to 5 stars given to the review. For each star, there are 130,000 training examples and 10,000 testing examples. In total, there are 650,000 training examples and 50,000 testing examples. We limit text length to 200 after tokenization. After this preprocessing step, there are 450,773 training and 34,620 testing examples, for a total of 485,393 examples. We sample a validation set from the train set with about the same size as the test set, and create a final dataset with 415,901/34,872/34,620 train/val/test examples.
The label attribute for Yelp Review is constructed from the 5 sentiment labels, which we verbalize as {very negative, negative, neutral, positive, very positive}. For the length attribute, we create 5 length classes {very short, short, medium-length, long, very long} with cut-offs 43,72,104,144 so that number of training examples in each length class is balanced. The dataset is obtained from https://huggingface. co/datasets/yelp_review_full.
AG News This is a news topic classification dataset with 4 topics {world, sports, business, science/tech}. The news text used is the title and description. For each topic, there are 30,000 training examples and 1,900 testing examples, for a total of 120,000 training and 7,600 testing examples. We limit text length to 256 after tokenization. After this pre-processing step, there are 119,955 training and 7,599 testing examples, for a total of 127,554 examples. We sample a validation set from the train set with about 10% of the original train set size, and create a final dataset with 107,959/11,996/7,599 train/val/test examples.
We use the topic labels as the label attribute, while adding alternative names for the labels. For the length attribute, we limit text length to 256. Because the text length in AG News is concentrated in a narrow range, we create 3 length classes {short, medium, long} with cut-offs 43 and 56 to make the number of training examples in each class balanced. The dataset is obtained from https://huggingface.co/datasets/ag_news.
A.2 Extra data
News Category The News Category dataset contains about 200K news headlines and short descriptions between 2012 and 2018 obtained from HuffPost. The advantage of this dataset is that it has a wide variety of topics, thus making the corresponding template very diverse. The list of topics and corresponding article counts is shown in Listing 1. We remove topics that has overlap with AG News: THE WORLDPOST, WORLDPOST, WORLD NEWS, SPORTS, BUSINESS, SCIENCE, TECH. The dataset is obtained from https://huggingface.co/ datasets/Fraser/news-category-dataset.
News Popularity The News Popularity in Multiple Social Media Platforms dataset is a dataset of social media sharing data of news articles about economy, microsoft, obama, and palestine. We use the concatenation of the headline and short_description fields as the news text. The size of this dataset is around 93K. The dataset is obtained from https://huggingface. co/datasets/newspop.
Inshort News The Inshort News dataset is a dataset of news with topics sports, politics, entertainment, world, automobile, and science. We remove the topics that has overlap with AG News: sports, world, science. The filtered dataset contains about 5K examples. The dataset is obtained from https://www.kaggle. com/datasets/kishanyadav/inshort-news.

heading: B Experiment Details
text: B.1 Training
On AG News, we use an Adam optimizer with a learning rate 0.00005 and train 10 epochs to train the PrefixLM models as well as FUDGE discriminators. On Yelp Review, we use an Adam optimizer with a learning rate of 0.0001 and train 5 epochs. We conduct all experiments on a single NVIDIA Tesla V100 GPU with 32GB memory. The training time of each model depends on the particular setup, but is within 24 hours for all models. The number of trainable parameters for the PrefixLM, PrefixLM-NL, and FUDGE-NL model is approximately 120M.
The number of trainable parameters for FUDGE and FUDGE-Binary is approximately 120M for each of label or length attribute model, and approximately 240M in total.
Listing 1: News Category dataset topics with corresponding number of examples. POLITICS: 32739 WELLNESS: 17827 ENTERTAINMENT: 16058 TRAVEL: 9887 STYLE & BEAUTY: 9649 PARENTING: 8677 HEALTHY LIVING: 6694
QUEER VOICES: 6314 FOOD & DRINK: 6226 BUSINESS: 5937 COMEDY: 5175 SPORTS: 4884 BLACK VOICES: 4528 HOME & LIVING: 4195 PARENTS: 3955 THE WORLDPOST: 3664 WEDDINGS: 3651 WOMEN: 3490 IMPACT: 3459
DIVORCE: 3426 CRIME: 3405 MEDIA: 2815 WEIRD NEWS: 2670 GREEN: 2622 WORLDPOST: 2579 RELIGION: 2556 STYLE: 2254 SCIENCE: 2178 WORLD NEWS: 2177 TASTE: 2096 TECH: 2082
MONEY: 1707 ARTS: 1509 FIFTY: 1401 GOOD NEWS: 1398 ARTS & CULTURE: 1339 ENVIRONMENT: 1323 COLLEGE: 1144 LATINO VOICES: 1129 CULTURE & ARTS: 1030 EDUCATION: 1004
The FUDGE models have an extra backbone language model that is kept frozen during discriminator training. The size of this backbone language model is approximately 120M. Backbones are first fine-tuned on corresponding classification datasets with a learning rate of 0.0001 for 5 epochs.
B.2 Hyperparameter choice under different settings
We find that the experimental results are not particularly sensitive to training hyperparameters such as learning rate and batch size. At testing, the FUDGE conditioning strength hyperparameter λ does have a notable effect on control accuracy. We report results with λ that gives the highest control accuracy while maintaining text quality. For the FUDGE model family (FUDGE, FUDGE-Binary, FUDGE-NL), we set λ “ 14 on the full-data and low-resource experiments, and λ “ 6 on zero-shot experiments. On compositionality experiments, we set λ “ 6 for AG News and λ “ 4 for Yelp Review. We set a smaller λ for zero-shot and compositionality settings because a larger λ in these cases leads to a significant increase in repetition. Following FUDGE’s original setup, we consider only the top 200 possible output tokens when modifying the LM logits for computational efficiency.

heading: C Command PCFG Details
text: The full template for the AG News and Yelp Review datasets are available in Listing 2 and Listing 3. We briefly explain important elements of the custom PCFG syntax below:
• We first randomly sample a template in the <templates> section. These are templates with attribute slots which will be filled later. Besides attribute slots, there are other nonterminals in the template that corresponds to sentence segments. Rules for these elements are written in the <variables> sections.
• Rules in the <variables> sections are compressed PCFG where rules with the same LHS are grouped together in a single line. They constitute the verbalization of domain names, attribute names, as well as a variety of sentence segments to increase the diversity of the PCFG.
• To verbalize the label attribute, the <label> section contains the mapping from categorical class indices to verbalized class names. Since the mapping could be one-to-many, different verbalizations of the same attribute class is separated by a comma.
• To verbalize the length attribute, the <length> section contains length cut-off values with the corresponding verbalized length level names, having similar syntax with the <label> section. An example with tokenized length l will be treated as the longest length level such that the corresponding cut-off does not exceed l.

heading: D Qualitative Examples
text: We show qualitative examples for different experimental settings in Table 6 to Table 15.
Listing 2: PCFG template for AG News <variables> [TEXT-CLASS] AG news, AG news [TEXT-FORM] [TEXT-CLASS], [TEXT-CLASS], [TEXT-CLASS] article, piece of [TEXT-CLASS], [TEXT-CLASS]
report, [TEXT-CLASS] item, AG newspaper article [HEAD-FORM] give me, generate, tell me about, show, show me, fetch me, output, I need, I want,
need, I request, write [TOPIC-NOUN] topic, topic, theme, focus [TOPIC-NOUNED] topic, topic, themed, focused, related [TOPIC-PREP] about, related to, concerning, regarding, pertinent to [TOPIC-UPDATEWORD] updated, informed [TOPIC-SEG] [TOPIC-PREP] [TOPIC], [TOPIC-PREP] [TOPIC], that is [TOPIC-PREP] [TOPIC], that is [
TOPIC-PREP] [TOPIC], that can keep me [TOPIC-UPDATEWORD] with [TOPIC] [TOPIC-BESEG] [TOPIC-PREP] [TOPIC], [TOPIC-PREP] [TOPIC], [TOPIC-PREP] [TOPIC], can keep me [TOPIC-
UPDATEWORD] with [TOPIC] [PLS] please, , [COMMA-PLS] / please, , # use '/' as comma (escaped) [BEFORE-BE] let it, make sure to, I want it to
<length> 43 short, concise, very short, pretty short, extremely short, extra short 56 medium-length, normal-length 256 long, lengthy, very long, pretty long, extremely long, extra long
<label> [TOPIC] 0 the world, the world, the globe, international matters 1 sports, sports, sporting events 2 business, business, commerce 3 science, science, technology, technology, tech
<templates> # label and length [HEAD-FORM] a [LENGTH] [TEXT-FORM] [TOPIC-SEG] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be [LENGTH] and [TOPIC-BESEG] . [PLS] [HEAD-FORM] a [TEXT-FORM] [TOPIC-SEG], and I need it to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] [TOPIC-SEG] , and [BEFORE-BE] be [LENGTH] [COMMA-PLS] . [HEAD-FORM] a [TEXT-FORM] . I want the [TOPIC-NOUN] to be [TOPIC], and length to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] . I want the length to be [LENGTH], and [TOPIC-NOUN] to be [TOPIC] . [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be not only [LENGTH] but also have a [TOPIC-NOUN] on [TOPIC]
. # label only [HEAD-FORM] a [TOPIC] [TOPIC-NOUNED] [TEXT-FORM] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TOPIC] [TOPIC-NOUNED] [TEXT-FORM] . [HEAD-FORM] a [TEXT-FORM] [TOPIC-SEG] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] [TOPIC-SEG] . [PLS] [HEAD-FORM] a [TEXT-FORM] . Let it have a [TOPIC] [TOPIC-NOUN] . [HEAD-FORM] a [TEXT-FORM] . Let it have a [TOPIC] [TOPIC-NOUN] [COMMA-PLS] . [HEAD-FORM] a [TEXT-FORM] . I want the [TOPIC-NOUN] to be [TOPIC] . # length only [HEAD-FORM] a [LENGTH] [TEXT-FORM] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be [LENGTH] . [PLS] [HEAD-FORM] a [TEXT-FORM] , and I need it to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] , and [BEFORE-BE] be [LENGTH] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] , and [BEFORE-BE] be [LENGTH] . [HEAD-FORM] a [TEXT-FORM]. I want the length to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM]. I want the length to be [LENGTH] [COMMA-PLS] .
Listing 3: PCFG template for Yelp Review <variables> [TEXT-CLASS] yelp review, yelp review, yelp comment [TEXT-FORM] [TEXT-CLASS], [TEXT-CLASS], [TEXT-CLASS] article, [TEXT-CLASS] passage, [TEXT-CLASS]
paragraph, [TEXT-CLASS] piece, piece of [TEXT-CLASS], yelp review chapter, [TEXT-CLASS] item [HEAD-FORM] give me, generate, tell me about, show, show me, fetch me, output, I need, I want,
need, I request, write [SENT-NOUN] tone, sentiment, attitude, mood [SENT-PREP] with, with, with, that has, / which has, of [SENT-SEG] [SENT-PREP] a [SENT] [SENT-NOUN] [PLS] please, , [COMMA-PLS] / please, , # use '/' as comma (escaped) [BEFORE-BE] let it, make sure to, I want it to
<length> 43 very short, pretty short, extremely short, extra short 72 short, concise 104 medium-length, normal-length 144 long, lengthy 200 very long, pretty long, extremely long, extra long
<label> [SENT] 0 very negative, terrible, very bad, extremely negative 1 negative, bad 2 neutral, unopinionated 3 positive, good, promising 4 very positive, very good, excellent, splendid, extremely positive
<templates> # label and length [HEAD-FORM] a [LENGTH] [TEXT-FORM] [SENT-SEG] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be [LENGTH] and having a [SENT] [SENT-NOUN] . [PLS] [HEAD-FORM] a [TEXT-FORM] [SENT-SEG], and I need it to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] [SENT-SEG] , and [BEFORE-BE] be [LENGTH] [COMMA-PLS] . [HEAD-FORM] a [TEXT-FORM] . I want the [SENT-NOUN] to be [SENT], and length to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] . I want the length to be [LENGTH], and [SENT-NOUN] to be [SENT] . [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be not only [LENGTH] but also have a [SENT] [SENT-NOUN] . # label only [HEAD-FORM] a [SENT] [SENT-NOUN] [TEXT-FORM] [COMMA-PLS] . [PLS] [HEAD-FORM] a [SENT] [SENT-NOUN] [TEXT-FORM] . [HEAD-FORM] a [TEXT-FORM] [SENT-SEG] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] [SENT-SEG] . [PLS] [HEAD-FORM] a [TEXT-FORM] . Let it have a [SENT] [SENT-NOUN] . [HEAD-FORM] a [TEXT-FORM] . Let it have a [SENT] [SENT-NOUN] [COMMA-PLS] . [HEAD-FORM] a [TEXT-FORM] . I want the [SENT-NOUN] to be [SENT] . # length only [HEAD-FORM] a [LENGTH] [TEXT-FORM] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] . [BEFORE-BE] be [LENGTH] . [PLS] [HEAD-FORM] a [TEXT-FORM] , and I need it to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM] , and [BEFORE-BE] be [LENGTH] [COMMA-PLS] . [PLS] [HEAD-FORM] a [TEXT-FORM] , and [BEFORE-BE] be [LENGTH] . [HEAD-FORM] a [TEXT-FORM]. I want the length to be [LENGTH] . [HEAD-FORM] a [TEXT-FORM]. I want the length to be [LENGTH] [COMMA-PLS] .

heading: Future Work
text: For example, it is possible to describe detailed semantic or syntactic constraints in a command sentence, and we leave those to future work