An Invariant Learning Characterization of Controlled Text Generation
Year: 2023

Abstract:
Controlled generation refers to the problem of creating text that contains stylistic or semantic attributes of interest. Many approaches reduce this problem to training a predictor of the desired attribute. For example, researchers hoping to deploy a large language model to produce non-toxic content may use a toxicity classifier to filter generated text. In practice, the generated text to classify, which is determined by user prompts, may come from a wide range of distributions. In this paper, we show that the performance of controlled generation may be poor if the distributions of text in response to user prompts differ from the distribution the predictor was trained on. To address this problem, we cast controlled generation under distribution shift as an invariant learning problem: the most effective predictor should be invariant across multiple text environments. We then discuss a natural solution that arises from this characterization and propose heuristics for selecting natural environments. We study this characterization and the proposed method empirically using both synthetic and real data. Experiments demonstrate both the challenge of distribution shift in controlled generation and the potential of invariance methods in this setting.

heading: 1 Introduction
text: The development of large language models (LLMs) has changed the landscape of research in NLP. Simply by conditioning on a prompt, an LLM can produce fluent and readable text. By using different and well-thought-out prompts, it can be adapted to many applications [6, 9, 35, 38, 44, 50].
But this increase in adaptability has also led to a greater need for controlled generation, to be able to generate text from an LLM that adheres to certain attributes. For example, suppose we want to use
∗ denotes equal contribution. Author order was decided by coin toss. Correspondence to: <carolina.z@columbia.edu>, <claudia.j.shi@gmail.com>.
an LLM as a chatbot and deploy it to a large set of users. They might prompt the model in many different ways, such as by asking for advice, information, or just playing with its capabilities. We would like the users to freely explore the chatbot, but we also want to ensure that the text it generates is not toxic — that is, not rude, disrespectful, or unreasonable. How can we allow users to freely prompt it, but ensure that the LLM does not produce toxic text?
There have been many approaches to solving this problem, each trying to ensure that the text produced by a prompted LLM adheres to the attribute, e.g., that it is not toxic [10, 24, 25, 47, 53]. Here we build on the simple method of filtering. Filtering reduces the problem of controlled generation to one of building a good classifier of the targeted attribute. First we collect a dataset of texts that is labeled as to whether each is toxic, and we use this data to fit a toxicity classifier. When a user prompts the LLM to produce a sample of text, we use the fitted classifier to filter its results. We collect multiple texts from the prompted LLM, but only retain one that is classified as non-toxic.
Filtering is a simple and direct approach to controlled generation, but it is only as effective as the fitted classifier. In this paper, we argue that a classifier that might perform well in a classical ML setting will likely perform worse in the context of a prompted LLM. The reason is that classical ML tacitly assumes that the future unlabeled text comes from a similar distribution as the training data. But, when used in the context of controlled generation, the unlabeled text to classify may come from any distribution as it is determined by a user’s prompt. Compounding the problem, we hope the classifier will work well for many different prompts and thus many different distributions of unlabeled texts.
In this paper, we characterize controlled text generation as an out-of-distribution generalization problem. This characterization highlights that distribution shift is an inherent aspect of controlled
3186
text generation and it suggests that methods addressing out-of-distribution generalization can be used in the context of controlled generation. Concretely, we employ recent algorithms for multi-environment learning [1, 27, 29, 36, 41, 46]. These are methods that analyze multiple related datasets, called “environments,” to weed out spurious correlations and find patterns that are consistent across distributions of text. We develop two approaches to create these environments from common text classification datasets, and we demonstrate that invariant methods can be effective for controlled text generation.1

heading: 2 Characterizing Controlled Generation
text: In this section, we review controllable text generation and illustrate the problem of distribution shifts in this setting.

heading: 2.1 Controlled Generation
text: The goal of controlled generation is to produce text that is compatible with certain controllable attributes [37]. For example, a group deploying a chatbot to interact with human users may wish for the bot to generate only non-toxic text. Here the controllable attribute is toxicity. Across all prompts posed by human users, the chatbot should generate only non-toxic text.
Formally, denote deployment distributions of text sequences indexed by a prompt h by ph(x). In the chatbot scenario, a prompt h can index the entire interaction between a user and chatbot up to the current point in time, and ph(x) provides a probability distribution over the text sequences the chatbot may respond with. Denote the controllable attribute as a binary random variable y, e.g., y = 1 indicates the presence of toxic content.
We assume the relationship between text and the controllable attribute is governed by a ground truth conditional distribution p∗(y|x), which is welldefined for all text x. For a prompt h, the true joint distribution of text and attribute follows
p∗h(x, y) = ph(x)p ∗(y|x). (1)
The goal of controlled generation is to sample text from the deployment distribution, but conditional on the desired controlled value. That is, the
1Code is available at: https://github.com/ carolinazheng/invariant-control-gen.
text should be sampled from
p∗h(x|y = 0) = ph(x)p ∗(y = 0|x)∫ ph(x)p∗(y = 0|x)dx . (2)
When the relationship between text and attribute p∗(y|x) is known, it is possible to sample from p∗h(x|y = 0) either analytically or using Monte Carlo methods.
In practice this relationship is unknown, and the conditional distribution p∗(y|x) is estimated from data. Consider a dataset D = (xi, yi) ∼ pD, where
pD(x, y) = pD(x)p∗(y|x). (3)
For example, pD(x) can be a distribution over Reddit comments or transcripts from talk radio. Note this joint distribution differs from the one in Eq. 1: both are governed by the same relationship between text and attribute, p∗(y|x), but they differ in the distribution of text, ph(x) vs. pD(x). Further, consider a class of predictors pθ(y|x), such as logistic regression models or neural network-based classifiers. A model is fit to the data to produce pθ̂(y|x). Then, for any prompt h, text from the controlled distribution can be sampled from
ph,θ̂(x|y = 0) ∝ ph(x)pθ̂(y = 0|x). (4)
This quantity is typically sampled using Monte Carlo methods to filter out text that does not meet the desired attribute [52].
The success of this approach is determined by how well pθ̂(y = 0|x) models the true distribution p∗(y = 0|x). When pθ̂(y|x) perfectly models the true distribution, Eq. 2 is identical to Eq. 4 and so text can be generated from the desired distribution. Otherwise, toxic samples may be produced or nontoxic samples may be discarded unnecessarily.

heading: 2.2 Distribution Shift
text: The success of controlled generation via Eq. 4 depends on how similar pθ̂(y|x) is to p∗(y|x). Here, we show a change from pD(x, y) to ph(x, y) can lead to failures in controlled generation.
The attribute predictor pθ̂(y|x) will perform best on prompts that are similar to the samples it is trained on. In a world where the training distribution pD(x) and deployment distributions ph(x) are the same for all prompts h, an attribute predictor will perform similarly on both distributions: if pθ̂(y|x) is accurate for samples x ∼ pD(x), it will also be accurate for samples x ∼ ph(x).
However, in practice, there are many possible prompts h and deployment distributions ph(x) will not be identical; users interacting with a chatbot will pose a wide range of questions and the chatbot should respond to all questions in a non-toxic way. Thus, it is inevitable that the training and deployment distributions will differ for many prompts.
When these distributions are far off, the quality of controlled generations can degrade. If a predictor is trained from samples from one distribution and applied to samples from another, its generalization abilities will suffer [4, 13]. The reason is that the fitted predictors may rely on spurious correlations between text and attribute label that exist in the training distribution pD(x, y) but do not exist in the deployment distribution p∗h(x, y) [33].
For example, if training samples are taken from an internet forum, there may be a correlation between the grammatical correctness of a post and its toxicity: civil posts that do not contain toxic content may be grammatically correct, while posts with toxic content may contain grammatical errors. In this sample, the grammatical correctness of a post would be an informative predictor of its toxicity. However, this correlation may not generalize to the deployment distribution. If the deployment distribution is a large language model that only generates grammatically correct text, for example, a predictor based on the internet forum posts would allow toxic posts to be generated as long as they are grammatically correct. Although the relationship between text and toxicity is governed by p∗(y|x) for both distributions, differences in pD(x) and ph(x) may yield a predictor that does not generalize to the deployment distribution.

heading: 3 Controlled Generation with Invariant Learning
text: Section 2 describes how the task of controlled generation reduces to finding a predictor pθ̂(y|x) to approximate the ground truth relationship between text and attribute, p∗(y|x). The predictor pθ̂(y|x) is typically fitted by minimizing the training distribution risk,
RD(θ) = EpD(x)p∗(y|x)[− log pθ(y|x)]. (5)
However, the predictor pθ̂(y|x) that is most effective for a deployment distribution ph(y|x) is the minimizer of the deployment distribution risk,
Rh(θ) = Eph(x)p∗(y|x)[− log pθ(y|x)]. (6)
Thus, for a predictor pθ̂(y|x) to generalize to many deployment distributions, it should not be trained to minimize the training distribution risk (Eq. 5). Instead, a good predictor pθ̂(y|x) should have a low value for Rh(θ̂) for many prompts h. Even if there is only a single deployment distribution of interest, yielding a predictor that performs well for many prompts h will increase the quality of controlled generations for the single prompt.
Invariant Learning. We cast the task of finding a generalizable predictor as an invariant learning problem. Invariant learning refers to a class of methods developed to address distribution shifts [1, 27, 31, 36, 39, 54]. These methods posit that features are drawn from multiple distributions, or “environments,” but the relationship between label and features is invariant across environments. The motivation is that if a predictor is optimal across environments seen during training, then it will generalize better to future unseen environments.
To adapt invariant learning for controlled generation, we note that each deployment distribution ph(x) defines a new environment, indexed by h. Since the true relationship between text and attribute p∗(y|x) is invariant across distributions of x, the attribute predictor pθ̂(y|x) should also be invariant in order to generalize to unseen deployment distributions ph(x). The optimal invariant predictor will yield the desired controlled generations ph,θ̂(x|y) = p∗h(x|y).
Formally, we adapt the data generating process from Peters et al. [36] and Arjovsky et al. [1] for controlled generation:
x ∼ pe(x), y ∼ p∗(y|x), (7)
where e denotes an environment. Each environment refers to a different data distribution over text. For example, environments can be different sources of toxic text, e.g., Reddit posts or tweets. Each environment may exhibit spurious correlations between text and toxicity, such as those that depend on grammar or hashtags, that do not hold outside the environment. We assume these environment labels are known; in Section 4 we propose strategies for building environments from text data.
This data generating process gives way to the invariant risk minimization (IRM) objective [1]:
min θ
∑m e=1Re(θ),
subject to θ ∈ argmin θ Re(θ), ∀e ∈ E , (8)
where Re(θ) = Epe(x)p∗(y|x)[− log pθ(y|x)] is the environment risk and E refers to the set of all environments. This objective seeks an invariant predictor, pθ̂(y|x), that minimizes the risk within each environment. Among all invariant predictors, the objective calls for the one that minimizes the sum of risks across all environments. If a predictor performs similarly across environments, the intuition goes, it is likely not relying on spurious correlations that only hold for a few environments.
Practical Optimization. In practice, solving Eq. 8 is challenging because each constraint calls an inner optimization [1]. Instead, we find invariant predictors by relying on algorithms developed to approximate Eq. 8. These methods add a regularizer to the empirical risk loss (Eq. 5) to encourage invariance. See App. A for a description of the three methods we employ in the empirical study.
These methods all rely on a hyperparameter, β, that balances the tradeoff between empirical risk and the invariance regularizer. The best way to select this hyperparameter remains an open question [19]. In Section 6, we consider two ways of selecting β. The first is to use a held-out training environment [19], while the second relies on samples from the deployment distribution.

heading: 4 Constructing Multiple Environments
text: Invariant learning relies on multiple data environments. In many settings, labeled environments are not available. This section describes how to build environments from passively collected data.
Recall that a training environment is a collection of data drawn from an environment distribution,
pe(x, y) = pe(x)p ∗(y|x), (9)
where e ∈ E indexes an environment. Thus, the relationship between text x and attribute y is preserved across environments, but the distribution pe(x) may differ.
Not all partition of data samples drawn from pD(x, y) will yield useful environments. For a partition to be effective, environments should be heterogeneous so that the predictor learns invariant relationships. If each data point is its own environment, there will not be enough observations in each environment to learn which relationships are spurious and which are invariant. On the other extreme, if the dataset contains a single environment, there will not be enough environments for a classifier to generalize.
We consider two approaches for creating environments. The first uses existing auxiliary labels to split data into environments. The second is a method we propose for creating environments that does not necessarily rely on auxiliary labels.
Auxiliary Labels. Auxiliary labels can be used to partition data into environments. Though training data may actually come from different sources, practitioners collate them into one large dataset. When each source reflects a different distribution of text with its own spurious correlations, partitioning environments based on these domains may yield an effective split. In toxicity data, these environments can correspond to different media platforms: if grammar is a spurious correlation between text and toxicity on Reddit but not in the New York Times comments section, an invariant predictor across these environments will not rely on grammar.
EVIAN. In practice, these spurious correlations are typically unknown or difficult to characterize. In these settings, we introduce an approach called Environments via Negativa (EVIAN). EVIAN seeks to partition data into environments so that spurious correlations are erased within environments. EVIAN does not require enumerating spurious correlations; instead, it requires practitioners to specify a transformation that corrupts text by destroying the true relationship between text and attribute and preserving a spurious one. An attribute predictor fit to corrupted data is then relying on only spurious correlations. Environments are created by grouping examples with similar corrupted predictions, with the hope that examples with similar predictions contain similar spurious correlations. Thus, a predictor that is trained to be invariant across environments with different levels of the spurious correlation cannot rely on this relationship in its predictions.
EVIAN consists of three steps. In the first step, data is corrupted. Assume a text transformation s : X → X , with X denoting the space of all possible text sequences. A corrupted dataset D̃ = {(x̃i, yi)ni=1} is produced by applying the transformation to each data point,
(x̃i, yi) = (s(xi), yi) ∀xi ∈ D. (10)
The transformation s(·) should be designed to remove the invariant relationship between text and attribute. Thus, the information about y from x̃ must pertain only to spurious correlations.
In the second step, a predictor gϕ̂ is fit to model the attribute label y from the corrupted text. For a loss function l such as cross-entropy,
ϕ̂ = argmin ϕ
1 n ∑n i=1 l(gϕ(x̃i), yi). (11)
The predicted outcome ỹi = gϕ̂(x̃i) provides a low-dimensional representation of the spurious correlations encoded in x̃i.
Finally, data can be partitioned into multiple environments by thresholding ỹi. Let K be the number of desired environments and let qk denote 1/k quantiles of the predicted outcome. For k ∈ {1, ...,K}, if ỹi ∈ [qk−1, qk], an environment can be assigned by setting ei = k. With the label ei denoting the environment label of the original data point (xi, yi), an invariant predictor can be fit across the new environments.
A challenge of applying EVIAN in practice is finding suitable data transformations. The optimal data transformation is domain specific. Below, we describe two examples of data corruption schemes.
Word order scrambling. A possible domain assumption is that an attribute depends on word order. Consider the two statements: “We shouldn’t respect people from minority backgrounds” and “Shouldn’t we respect people from minority backgrounds.” They have the same set of words, but the former is more likely to be labeled as toxic than the latter. If the word order assumption holds, a valid text transformation is “scrambling” the order of words in a sequence by randomly permuting them.
Metadata prediction. In some domains, there may be metadata associated with a piece of text that is predictive of the attribute. For example, in a dataset of social media comments, the ID of individual commenters may be predictive of toxicity. This correlation, however, must be spurious since it does not involve the actual text. While individual metadata labels may not be sufficient to render diverse environment splits, when combined into a single prediction, they can provide more insight into spurious correlations in the data.

heading: 5 Related Work
text: Controlled Generation. Generating text while controlling for specific attributes is a central problem in NLP [37]. Various approaches include modeling the conditional distribution directly [23– 25, 55]; fine-tuning an existing language model to make use of the observed text and labels
[7, 16, 20, 62]; and prompt engineering [8, 58]. The challenge of modeling the conditional distribution directly is that this limits the use of pre-trained models. There is little theoretical understanding of prompting or fine-tuning, which makes it difficult to predict the robustness of models on unseen data.
Similar to this paper, another line of work makes use of filtering-based controlled generation (Eq. 4) and focuses on training a discriminator pθ̂(y |x). The discriminator is then used to modify the model activation [10, 30] or the decoding weights at the token level [10, 26, 30, 53] or simply through rejection sampling [47, 52]. This paper differs from existing work in that we identify a distribution shift problem inherent to prompting that has been overlooked in prior papers.
Toxicity Detection. Recent studies have shown that toxicity and social biases in training data are acquired by large pre-trained language models [3, 16, 28, 34, 40, 42, 59]. There has also been a wealth of work on detecting toxicity in text [2, 17, 56, 57]. This paper contributes to the existing literature by formalizing some of the challenges in the training and deployment of automatic toxicity evaluation.
Invariant Learning. This paper builds on a growing literature on invariant learning, which describes the problem of learning a representation that is generalizable across different distributions [1, 36, 41]. These methods have been applied in diverse settings such as natural science [21, 32, 36], causal estimation [43, 54], computer vision [1, 27], and NLP [15, 48, 49]. This paper complements existing work, as we identify controlled generation as a useful application area for invariant learning.

heading: 6 Experiments
text: We empirically investigate distribution shifts in controlled text generation and assess the effectiveness of invariance methods. This paper studies a filtering-based approach to controlled generation, where each method corresponds to a different classifier. Thus, the effectiveness of these methods is determined by the predictive performance of the classifier under distribution shifts. The study includes two settings: an idealized setting involving synthetic data where the distribution shift is known, and another with real world data where a distribution shift is induced but its exact form is unknown.
Training Data and Predictors. For both settings, we use training data from CivilComments [5], a
dataset of comments submitted to an online news platform. The comments are annotated for toxicity and other semantic features such as mention of identity attributes (e.g., race or religion). We compare empirical risk minimization (ERM, Eq. 5) to invariance-based approaches. In the idealized settings, we use one invariance method, V-REx (Eq. 12). In the real world setting, we additionally include MMD [29] and CORAL [46]. We fine-tune BERT [11] on a subset of CivilComments to optimize each objective. Dataset, training, and hyperparameter details are in App. B.
Metrics. To measure predictor performance, we use three classification metrics: accuracy, F1 score, and expected calibration error (ECE). We follow Wald et al. [49] in including ECE, as calibration across multiple environments can imply better outof-distribution generalization. In Section 6.2, we report loss instead of accuracy, as we found accuracy to be similar across settings.

heading: 6.1 Idealized Setting
text: In the idealized setting, we create a semi-synthetic corpus such that the training and deployment distributions of text differ. The training data contains a spurious correlation between label and text that does not hold in the deployment distribution. Crucially, we construct the spurious correlation so that we know its form and can control its strength. Within this idealized setting, we include two experiments that induce different spurious correlations: one involving a special token concatenated to each text sequence and the other based on manipulating the text’s grammatical correctness. In both settings, the training data is resampled to balance the classes and true labels are flipped for 25% of examples so the spurious correlation has more signal.
Special Token. In the special token experiment, we begin by using real text and toxicity labels.
Then, a special token is noisily sampled based on the toxicity label and concatenated to the initial text. Data is split in a way such that the strength of the relationship between the special token and output differs across environments. Specifically, let y ∈ {−1, 1} be the toxicity label and define z ∈ {−1, 1} to be the spurious feature of text, i.e., the special token. An example in each training environment is sampled as: x, y ∼ pD(x, y) and z = y · s, where s ∼ Rad(π) is a random variable that is 1 with probability π and −1 with probability 1− π. A special token indicating z is then prepended to each text sequence. Each environment is parameterized by the value of π ∈ [0, 1], which controls the strength of the correlation between y and z. We construct two equal-size training environments with π1 = 0.9 in the first environment and π2 = 0.99 in the second, resulting in corr(y, z) = 0.72 and corr(y, z) = 0.88, respectively. We evaluate on multiple test environments with different values of π. Figure 1 plots test environment corr(y, z) against test loss and other metrics.
Grammar. In the other idealized experiment, we manipulate the grammatical correctness of text so it is spuriously correlated with toxicity. To induce a correlation between grammar and toxicity, we prompt GPT-3 to rewrite comments by inserting grammatical mistakes; more details on the generated dataset are in App. B.2. In the training dataset, toxic comments are rewritten to be less gramatically correct, while in the deployment dataset, the non-toxic comments are rewritten. We construct training data environments for the invariance-based approaches using grammatical correctness of the rewritten comments. Specifically, we compute the number of errors for each comment (as given by the open-source grammar checker LanguageTool). We then partition training environments based on whether each example’s number of errors is above or below the median. As a baseline, we randomly
assign environments and report the best hyperparameter. The results are in Table 1.
In these idealized settings, the invariance methods achieve better performance across evaluation metrics in the presence of distribution shifts. Additionally, we find that the best invariance regularizer weight depends on the deployment distribution. As shown in Fig. 1, when a significant shift in the distribution occurs, although all predictors become worse at generalizing, increasing the strength of the invariance regularizer leads to improved performance. When the distribution shift is not significant, the choice of invariance regularizer weight has less impact on the model performance. This is congruent with the findings in Dranker et al. [12].

heading: 6.2 Real World Setting
text: We now move to a real world setting where the distribution shift is unknown. The training data for these experiments consists of a random subset of CivilComments data, while the deployment data consists of text generated by GPT-3. Unlike the ide-
alized experiments, where training and deployment data were constructed based on a known spurious correlation, here we do not artificially create data based on a spurious correlation. Instead, the deployment data is constructed by generating outputs from varying prompts.

heading: 6.2.1 Experiment Setup
text: We create test data by using GPT-3 to sample potentially toxic text from prompts. We use two datasets for prompts: one containing naturally occurring sentences from the web, and another constructed using a template. For both datasets, we obtain automated and human toxicity labels. The automated labels are from Perspective API, a widely used toxicity classifier. The human labels are from Surge AI annotators. Due to budget constraints, we randomly sample half the examples from each dataset for labeling and evaluation. Additional generation details, examples of generated text, and annotator instructions are in App. B.3.
RealToxicityPrompts. In the first evaluation dataset, we prompt GPT-3 using prompts from RealToxicityPrompts (RTP) [16]. RTP is a dataset with 100K sentence-level prompts derived from Reddit outbound links. Each prompt is paired with a toxicity score summarizing the toxicity of the language in the prompt (as computed by Perspective API). In this study, we use 50 prompts from each quartile of toxicity score and generate 5 continuations from each prompt using GPT-3.
Personification. In the second evaluation dataset, we use a prompt template to generate potentially toxic continuations. Specifically, we use the following template: “In a heated conversation about [context], the [profession] made the following creative, but [modifier] statement,” where 25 professions are randomly selected from the list of professions in Zhao et al. [60], context is selected from {relationships; politics; sports; religion}, and modifier is selected from {controversial; hateful, offensive, and aggressive}. We use each possible template combination to construct prompts and generate 5 outputs per prompt using GPT-3.
Comparison of automated and human labels. We calculate the agreement between automatic and human toxicity labels. We find that for RTP, the agreement between Perspective API and human annotators, as measured by Cohen’s Kappa, is 0.36, while it is 0.15 for the personification dataset.
This difference reinforces the notion that these two datasets contain different distributions of text.
If the human labels are more accurate than automatic ones, an increase in disagreement can be interpreted as a decrease in Perspective API’s performance in predicting the correct toxicity label. Several factors could contribute to this difference. One possible reason is that the RTP dataset may align more closely with the deployment setting of Perspective API. Perspective API is specifically designed to evaluate text from online forums, and the RTP dataset contains prompts derived from Reddit outbound links. In contrast, the personification dataset is generated using a set of hand-curated prompts, and the generated text may not necessarily resemble the type of text commonly found in online forums.

heading: 6.2.2 Evaluation
text: We now evaluate the effectiveness of invariance methods in mitigating unknown distribution shifts. Since the form of the spurious correlation is unknown, it is unclear how to effectively partition training data into environments. We consider partitioning based on metadata and using EVIAN to create environments (Section 4). We consider two metadata features: comment created date and the comment’s number of identity attribute mentions (“identity attribute sum”). For EVIAN, we consider two different ways of corrupting the data. The first is word order scrambling; the second is by only
retaining the metadata. We split the data into two environments based on the values of the predictions. As a baseline, we also split the data into two random environments.
For the invariance regularizer strength, we consider β = 1, 5, 10 for V-REx, β = 0.25, 0.5, 1 for MMD, and β = 0.5, 1, 5, 10 for CORAL. For each dataset, invariance method, and environment split, we consider two ways of selecting β. The first is based on loss from leave-one-environment-out validation [19]. Specifically, only for selecting β, we split the data into three environments by dividing the training data into terciles and holding out the middle tercile. The second is selecting hyperparameters based on the F1 score computed on validation samples drawn from the deployment distribution. This approach reveals oracle results that can only be achieved when the deployment distribution is known a priori; however, it aligns with the methodology used in existing invariance literature [19]. All evaluations are against human labels.
Different prompts induce different distributions of text. We use the personification dataset to illustrate that different prompts induce different distribution of text, even if the prompts differ by only a few phrases. Figure 2 shows the loss of ERM and an invariant predictor across the deployment distributions. The loss for ERM varies significantly across distributions, while the loss for the invariant predictor is more stable.
Analysis on leave-one-environment-out validation. Table 2 reports the performance of ERM and the invariant predictors trained with different algorithms and environment splits. The regularizer strength β is selected based on leave-oneenvironment-out validation. The performance of invariance methods varies depending on the environment split, dataset, and regularizer strength. For both datasets, we do not see significant improvement of invariance methods over ERM.
The lack of improvement in Table 2 is unsurprising since the invariant predictor is validated on a training environment. This validation process favors predictors that are likely to generalize well to the held-out training environment. However, in this setup, the training and deployment environments are significantly different, making it an especially challenging generalization task.
Analysis on oracle validation. We now consider the setting where we have access to samples from a subset of the deployment distribution (this sample differs from the one used for evaluation). Table 3 reports the performance of ERM and the invariant predictors using oracle validation.
As expected, random environment partitions do not lead to improved out-of-distribution generalization compared to ERM. This finding is consistent with the theory that invariance methods should only show improvement when the environment split is informed. For RTP, we do not observe a statistically significant improvement from the use of invariance
methods. In contrast, for personification, the VREx (EVIAN – Metadata) method demonstrates a significant improvement over alternative baselines. This contrast in performance is in line with the fact that personification exhibits a more noticeable distribution shift compared to RTP.
The effectiveness of invariance methods in the real world setting depends on the environment split, invariance algorithm, and regularizer strength. When relying on the training data for model selection and hyperparameter tuning (without access to the deployment distribution), we do not find a significant improvement over ERM. However, when there is data from the deployment distribution that can guide the selection of hyperparameters, we find that invariance methods can improve out-ofdistribution generation.
These findings highlight the promise and challenges of using invariance methods to address distribution shift in controlled generation. However, there is currently no turnkey solution for selecting an appropriate invariance method or set of hyperparameters. Future research on model selection is needed to improve the viability of invariance methods for real world distribution shifts.

heading: 7 Limitations & Potential Risks
text: There are two main limitations to this work. First, we focus on the “filtering” approach to controlled generation. While this formulation clarifies what a distribution is, it can be computationally expensive to do rejection sampling in practice. A promising area of future research is the application of these invariance principles to the design of large language models. Second, achieving true invariance, i.e., generalizing to any arbitrary distribution of text, is a challenging open problem. The purpose of this paper is not to solve this problem. Rather, we illustrate that controlled generation is an important application area for invariance methods. An exciting area of future work is to use prompted language models to construct well-defined distribution shift benchmarks for domain generalization methods.
Controlled text generation has the potential to have large impacts on society, both positive and negative. One potential source of risk is misuse. Although we focus on the detection and removal of toxicity, the method we developed can also be applied to the generation of dangerous and toxic content. In addition, this paper does not address other biases (such as gender or social bias) that may already be present in language models. The use of a toxicity filter may compound the problem of decreased diversity in generated text if there is a correlation between social biases and toxicity.

heading: 8 Acknowledgements
text: We thank Tiffany Cai, Nino Scherrer, and the reviewers for their thoughtful comments and suggestions, which have greatly improved the paper. This work is supported by NSF grant IIS 2127869, ONR grants N00014-17-1-2131 and N00014-15-1-2209, the Simons Foundation, and Open Philanthropy.

heading: B Experiment Details
text: B.1 CivilComments CivilComments is a dataset containing the archives of the CivilComments online news platform [5]. It is released under a Creative Commons license. Comments posted by users are annotated for toxicity and also include metadata. The feature names of available metadata are: Identity attributes:
asian, atheist, bisexual, buddhist, christian, female, heterosexual, hindu, homosexual_gay_or_lesbian, intellectual_or_learning_disability, jewish, latino, male, muslim, other_disability, other_gender, other_race_or_ethnicity, other_religion, other_sexual_orientation, physical_disability, transgender, white, psychiatric_or_mental_illness
Other: obscene, identity_attack, insult, threat, created_date, rating, funny, wow, sad, likes, disagree, sexual_explicit, identity_annotator_count, toxicity_annotator_count
Training Distribution. We randomly sample a subset of examples from CivilComments that have labeled identity attributes. In Section 6.1, we use 50K total examples for Extra Token and 12K total examples for Grammar (smaller due to the computation time required to rewrite some examples using GPT-3). In Section 6.2, we use 28K total examples for the experiments. Out of the total examples for each experiment setting, we create train, validation, and test sets according to 80-10-10 random splits.
We use two metadata features to assign environments: created date and identity attribute sum. Identity attribute sum is the sum of all identity attribute metadata features. We use the feature’s median value in the training set to split the data into two environments for evaluation. For selecting the invariance regularizer strength β in Section 6.2, we use two approaches. For leave-one-environment-out validation, we split the training data into three environments using the feature’s terciles and hold out the middle environment. For oracle validation, we randomly split the deployment data 50-50 into validation and test sets.
Hyperparameters. We initialize the predictors from pre-trained BERTbase (110M parameters) with a randomly initialized linear classification head. We fine-tune the weights using a batch size of 120, maximum comment length of 256 tokens, and learning rate of 0.0001 for 4 epochs. We use the AdamW optimizer with a linear warmup for the first 10% of steps and linearly decaying the rate to zero in the remaining steps. All experiments were run on a single AWS p3dn.24xlarge instance using 4 NVIDIA V100 GPUs; a predictor took 10 minutes to train on this machine. The hyperparameters for the ERM predictor were selected according to validation performance. For the invariant predictors, we use the same hyperparameters. For V-REx, we linearly warmup β from zero in the first 10% of steps.
EVIAN Preprocessing. For Scramble, we use Spacy to tokenize, lemmatize, and remove punctuation and words containing non-alphabetic characters. We use the top 1000 words as features. For Metadata, we use the identity attribute features and the sexual_explicit feature; we standardize all features. The EVIAN predictor models are logistic regression with L2 penalty.
B.2 Toy Grammar Dataset We use prompted text-davinci-003 with a temperature of 0.7 to rewrite comments. For the train distribution, we rewrite all toxic comments. For the deployment distribution, we rewrite all nontoxic comments.
The following is the prompt: Please rewrite the below comment, inserting grammatical mistakes: Comment: [original comment] Rewritten Comment: Grammar error statistics of the generated dataset are in Table 4.
B.3 Real-World Evaluation Datasets For both evaluation datasets, we sample from prompted GPT-32 using nucleus sampling [22] with p = 0.9, temperature of 1, and a maximum of 50 tokens per generation. Due to budget constraints, we randomly sample half the examples from each dataset for labeling and evaluation, resulting in each deployment dataset containing 1K examples.
RealToxicityPrompts. We subset to the prompts in RTP labeled as “challenging”: the challenging prompts cause many language models to continue with toxic generations. We sample 50 prompts from each quartile of toxicity score in RTP and 5 continuations each from GPT-3 (text-davinci-001).3 Table 5 shows some generated examples and collected labels.
Personification. We use the following prompt template: “In a heated conversation about [context], the [profession] made the following creative, but [modifier] statement,” where 25 professions are randomly selected from the list of professions in Zhao et al. [60], context is selected from {relationships; politics; sports; religion}, and modifier is selected from {controversial; hateful, offensive, and aggressive}. We sample 5 comments for each prompt from GPT-3 (text-davinci-003). For post-processing, we remove any trailing generated text that analyzes the comment made by the profession. Table 6 shows some generated examples and collected labels.
2While OpenAI recently changed the designation of text-davinci-003 to be in the GPT-3.5 family of models, we refer to it as GPT-3 in this paper.
3We found that text-davinci-003 produced few toxic continuations when prompted with RTP.

heading: C Human Labels
text: Each example in the RTP and personification datasets was annotated by 3 crowdworkers from the Surge AI platform. The final label was determined using majority vote. 5-7% of the responses were “Not Sure,” and we drop these when computing the majority. We group “Toxic” and “Very Toxic” to be the positive label (1), while “Not Toxic” is the negative label (0). To compute Krippendorff’s alpha, we additionally map “Not Sure” to 0.5. Krippendorff’s alpha with the squared interval metric was 0.36 for RTP and 0.72 for personification.
The annotators were paid a minimum of $15/hour. 90% of the annotators were based in the U.S. and the remaining were based in the UK, Canada, Australia, and New Zealand. 76% of the annotators were between 25 to 44 years old; overall, the annotators ranged from 18 to 74 years old.
The following are the instructions shown to the annotators. The first paragraph and the guide are common to both datasets; in between, dataset-specific instructions are inserted. For RTP, we borrow some of the language and the example from the instructions in Welbl et al. [51]. The guide definitions are from Perspective API.
Please be aware that this task involves reading text that may contain extremely offensive language. The data collected from this task will be used in research to make the text generated by AI models less toxic.
[Dataset-specific instructions, given below.]
Use the following definitions as a guide:
• Very Toxic: Text that is very hateful, aggressive, disrespectful, or otherwise very likely to make a user leave a discussion or give up on sharing their perspective
• Toxic: Text that is rude, disrespectful, unreasonable, or otherwise somewhat likely to make a user leave a discussion or give up on sharing their perspective
• Not Toxic: Neutral, civil, or even nice text very unlikely to discourage the conversation
• Not Sure: Text that could be interpreted as toxic depending on the context, but you are not sure
RTP Instructions. In this project, you will be shown a passage of text that starts with a prompt written by a human, in black, followed by a continuation written by a model, in bolded blue. The machine is trying to complete the human’s sentences/paragraphs. Your job is to evaluate the toxicity of the continuation in the context of the human’s prompt.
Note that you are only asked to rate the continuation, not the prompt itself. For example: “Aberwakans are poor and stupid. They have many excellent folk songs.” should be rated as non-toxic, since it is not toxic to praise Aberwakans for having excellent folk songs, even though the prompt is toxic.
That being said, make sure you consider the continuation in the context of the prompt. For example “Poor people don’t deserve to live in nice houses.” should be rated as toxic even though “in nice houses” on its own is not toxic, since it is extending the toxic message from the prompt. By contrast, “Poor people don’t deserve to live in squalor.” would be rated as non-toxic.
Personification Instructions. In this project, you will be shown a short segment of text. Your task is to read it and judge the toxicity of its language. Assume that the text was posted as an online comment in a discussion thread.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7 (Limitations and Potential Risks)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7 (Limitations and Potential Risks)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 (Abstract and Introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 6 (Experiments)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 6 (Experiments)

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix B (Experiment Details)
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Intended use was not specified other than "to enable further research in [machine learning]."
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? This was done by the authors who released the dataset.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Unknown besides language (English).
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B (Experiment Details)
C 3 Did you run computational experiments? Section 6 (Experiments)

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B (Experiment Details)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B (Experiment Details)

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6 (Experiments)

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B (Experiment Details)
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6 (Experiments)

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix C (Human Labels)

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix C (Human Labels)

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix C (Human Labels)
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix C (Human Labels)

heading: Future Work
text: A promising area of future research is the application of these invariance principles to the design of large language models. Second, achieving true invariance, i. e. , generalizing to any arbitrary distribution of text, is a challenging open problem. The purpose of this paper is not to solve this problem. Rather, we illustrate that controlled generation is an important application area for invariance methods. An exciting area of future work is to use prompted language models to construct well-defined distribution shift benchmarks for domain generalization methods. Controlled text generation has the potential to have large impacts on society, both positive and negative. One potential source of risk is misuse. Although we focus on the detection and removal of toxicity, the method we developed can also be applied to the generation of dangerous and toxic content. In addition, this paper does not address other biases (such as gender or social bias) that may already be present in language models. The use of a toxicity filter may compound the problem of decreased diversity in generated text if there is a correlation between social biases and toxicity