SlowGuess/NAACL_2021_OCR huggingface dataset:
Kevin Yang
yangk@berkeley.edu
Dan Klein
klein@berkeley.edu
We propose Future Discriminators for Generation (FUDGE), a flexible and modular method for controlled text generation. Given a pre-existing model  $\mathcal{G}$  for generating text from a distribution of interest, FUDGE enables conditioning on a desired attribute  $a$  (for example, formality) while requiring access only to  $\mathcal{G}$ 's output logits. FUDGE learns an attribute predictor operating on a partial sequence, and uses this predictor's outputs to adjust  $\mathcal{G}$ 's original probabilities. We show that FUDGE models terms corresponding to a Bayesian decomposition of the conditional distribution of  $\mathcal{G}$  given attribute  $a$ . Moreover, FUDGE can easily compose predictors for multiple desired attributes. We evaluate FUDGE on three tasks — couplet completion in poetry, topic control in language generation, and formality change in machine translation — and observe gains in all three tasks.
Recent advances in large pretrained language models allow us to generate increasingly realistic text by modeling a distribution  $P(X)$  over natural language sequences  $X$ . The distribution  $P(X)$  may be truly unconditional, as is common in language modeling, or it may model  $P(X|I)$  conditioned on some input  $I$ , as in machine translation or summarization.
We are frequently interested in controlled text generation, the task of generating text conditioned on an additional desirable attribute  $a$  which is not already built into  $P(X)$ . That is, we would like to model  $P(X|a)$  (or possibly  $P(X|I,a)$ ; henceforth we will drop  $I$  from the notation for simplicity). For example,  $P(X)$  may be a pretrained translation model for Spanish inputs  $I$  to English outputs  $X$ , but we may wish to additionally constrain the outputs to possess a new attribute  $a$ , e.g., formality, which we did not optimize for during training.
Unfortunately, once we have already obtained an unconditioned  $P(X)$  defined as the output dis
The distribution of some large generative model  $\mathcal{G}$ , it is nontrivial to add conditioning on a new attribute  $a$  without either training a new model from scratch or fine-tuning with additional data. Although in principle we can trivially sample from  $P(X|a)$  via rejection sampling from  $P(X)$ , rejection sampling may be highly inefficient in practice. On the other hand, while generating according to attribute  $a$ ,  $P(X)$  should be left otherwise intact: in the previous translation formality example, it is pointless to generate formal English outputs if they do not preserve the original Spanish meaning.
In light of these concerns, we propose Future Discriminators for Generation (FUDGE), a flexible and modular method for modeling  $P(X|a)$  which accesses only the output probabilities of the generative model  $\mathcal{G}$  which defines  $P(X)$ . FUDGE learns a binary predictor for whether attribute  $a$  will become true in the complete future, based on an incomplete sequence prefix (Sec. 3). Multiplying the output probabilities of this predictor with  $\mathcal{G}$ 's original probabilities and then renormalizing yields a model for the desired  $P(X|a)$  via Bayes' Rule.
We run experiments on three controlled text generation tasks — couplet completion in poetry, topic control in language generation, and formality change in machine translation — showing our method's broad applicability. Additionally, we demonstrate the modularity of FUDGE by composing multiple attribute constraints in both the couplet and topic control tasks. In our experiments, we find that FUDGE is highly effective at attribute control, outperforming both a baseline which directly fine-tunes  $\mathcal{G}$  and also a strong gradient-based method (PPLM (Dathathri et al., 2019)). Our code is available at https://github.com/yangkevin2/naacl-2021-fudge-controlled-generation.
Ideally, a controlled text generation method should efficiently control for  $a$  while preserving  $P(X)$
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511-3535 June 6-11, 2021. ©2021 Association for Computational Linguistics
as much as possible. Recent work on controlled text generation has greatly advanced our ability to control for a required attribute  $a$  flexibly and cheaply, with varying degrees of modification to the original model  $\mathcal{G}$  which defines  $P(X)$ .
One line of work fine-tunes a pretrained model for a desired attribute (Ficler and Goldberg, 2017; Yu et al., 2017; Ziegler et al., 2019). The result is a class-conditional language model (CCLM). However, it is difficult to isolate the desired attribute from the distribution shift between  $\mathcal{G}$  and the finetuning dataset (Hu et al., 2017; John et al., 2018; Lazaridou et al., 2020), i.e., it is nontrivial to preserve the desirable qualities of the  $P(X)$  modeled by  $\mathcal{G}$ . One may also need to fine-tune separately for each attribute of interest. CTRL (Keskar et al., 2019) partially addresses these issues by providing 55 attribute control codes for a large language model trained from scratch, although this is expensive. Very recently, GEDI (Krause et al., 2020) achieves strong performance by using CCLM generators as discriminators, though it relies on several heuristics. More broadly, text generation models for style transfer (Hu et al., 2017; Lample et al., 2018b; Dai et al., 2019a), summarization (See et al., 2017; Gehrmann et al., 2018; Zaheer et al., 2020), and machine translation (Lample et al., 2018a; Ng et al., 2019; Lewis et al., 2019) can also be viewed as CCLM's for different "attributes."
A second type of approach instead conditions on a desired attribute by backpropagating gradients, either to directly modify model activations (Dathathri et al., 2019; Liu et al., 2020) or to find a trigger string (Wallace et al., 2019, 2020). Such methods often exhibit a high degree of attribute control, and can be used in adversarial attacks (Wallace et al., 2020). In fact, Subramani et al. (2019) show that by carefully modifying the latent state, one can cause the base  $\mathcal{G}$  to produce arbitrary outputs.
A third class of methods, referred to as weighted decoding (WD), assumes access only to  $P(X)$  (i.e.,  $\mathcal{G}$ 's output logits), and operates directly on these logits (Ghazvininejad et al., 2017; Holtzman et al., 2018; Cohn-Gordon et al., 2018; Shen et al., 2019). Compared to other approaches, WD methods are relatively interpretable in how they obtain  $P(X|a)$  from  $P(X)$ , but prior WD implementations have been observed to perform poorly in controlled text generation (See et al., 2019; Dathathri et al., 2019). While FUDGE shares a Bayesian motivation with other WD methods, FUDGE follows the Bayesian
factorization more closely in implementation (Sec. 3). The key distinguishing feature of FUDGE is that it models whether attribute  $a$  will be true in the future, rather than in the present. We find that FUDGE substantially outperforms previous WD approaches in our experiments (Sec. 4.2).
3 Future Discriminators for Generation
We now explain the details of our proposed method, Future Discriminators for Generation (FUDGE), and show that it corresponds to modeling the desired conditional distribution  $P(X|a)$ .
For a given language generation task, assume we have an autoregressive model  $\mathcal{G}$  (e.g., a large pretrained language model) which models  $P(x_{i}|x_{1:i - 1})$  for tokens  $x_{1}\ldots x_{i}$ . Letting  $X = x_{1:n}$  denote a completed sequence,  $\mathcal{G}$  can sample from  $P(X) = P(x_{1:n})$  one token at a time by factoring  $P(X)$ :
$$
P (X) = \prod_ {i = 1} ^ {n} P (x _ {i} | x _ {1: i - 1})
$$
To condition on attribute  $a$ , we instead model  $P(X|a)$ . This requires a model for  $P(x_{i}|x_{1:i - 1},a)$  modifying the previous factorization:
$$
P (X | a) = \prod_ {i = 1} ^ {n} P (x _ {i} | x _ {1: i - 1}, a)
$$
If we model  $P(x_{i}|x_{1:i - 1},a)$  directly, we obtain a class-conditional language model (CCLM). We can learn the CCLM by e.g., fine-tuning  $\mathcal{G}$  depending on the available data, possibly with some structural modification to  $\mathcal{G}$  to accommodate conditioning.
However, FUDGE instead relies on the following Bayesian factorization, exchanging  $x_{i}$  and  $a$  conditioned on  $x_{1:i-1}$ :
$$
P (x _ {i} | x _ {1: i - 1}, a) \propto P (a | x _ {1: i}) P (x _ {i} | x _ {1: i - 1})
$$
The second term is exactly the quantity modeled by the base  $\mathcal{G}$ . It then suffices to model the first term,  $P(a|x_{1:i})$ , with a binary classifier  $\mathcal{B}$  for the attribute  $a$  given a prefix  $x_{1:i}$ . Intuitively, one can view  $\mathcal{B}$  as rescoring or reranking  $\mathcal{G}$ 's original hypotheses.
We emphasize that although  $\mathcal{B}$  takes a prefix  $x_{1:i}$  as input, it predicts whether attribute  $a$  will in the future be satisfied for the completed generation  $x_{1:n}$ . For instance, suppose we are given a dataset of examples  $\{(x_{1:n}, a')\}$  with  $a'$  being the values of binary indicators for the desired  $a$  (i.e., if  $a$  is formality, then  $a'$  is 0 or 1 when  $x_{1:n}$  is informal
Figure 1: Illustration of one decoding step in FUDGE, for an example where the desired attribute  $a$  is formality. A large pretrained model  $\mathcal{G}$  (dark blue) outputs unconditioned probabilities. Our binary predictor (red) predicts whether the eventual completed sequence will be formal for each possible continuation (computed for each candidate  $x_{3}$ , e.g., "want"; holding  $a$  fixed). The probabilities for each  $x_{3}$  are multiplied (purple) and then renormalized to obtain  $P(x_{3}|x_{1:2},a)$ , from which we sample the next token  $x_{3} =$  "prefer."
or formal respectively). For each training example  $(x_{1:n},a^{\prime})$ , we train our classifier  $\mathcal{B}$  using all pairs  $(x_{1:i},a^{\prime})$ ; that is, we construct a separate example from each prefix  $x_{1:i}$  of  $x_{1:n}$ . Our approach contrasts with previous methods such as Dathathri et al. (2019), which greedily optimize for  $a$  on the immediate extension  $x_{1:i+1}$ . One particular benefit is that FUDGE naturally plans for the future: in the example for generating text on the "space" topic in Table 6, FUDGE writes about a "mysterious ship" despite "ship" itself not being in the given "space"-topic bag of words, because "mysterious ship" easily leads into a mention of one of the targeted "space" words ("Earth"). Similarly, in the first couplet completion example in Table 3, FUDGE needs to rhyme with "fear" after exactly ten syllables. After seven syllables, it could reasonably generate the word "clear," but it first generates the adverb "pretty" in order to set up the generation of "clear" as the tenth syllable.
FUDGE's implementation is shown schematically in Figure 1, and is quite simple in practice. FUDGE just needs to learn a  $\mathcal{B}$  (red in Figure 1) sharing tokenization with  $\mathcal{G}$  (dark blue). It then converts  $\mathcal{B}$ 's output into probabilities (red table in Figure 1), and multiplies with the original output probabilities from  $\mathcal{G}$  (dark blue table), to obtain unnormalized probabilities  $P(x_{i},a|x_{1:i - 1})$  (purple table). Finally, renormalizing over the output vocabulary yields the desired distribution  $P(x_{i}|x_{1:i - 1},a)$ . In practice, we operate in the log-probability space for numerical stability.
To improve computational efficiency, we typically choose  $\mathcal{B}$  to be lightweight relative to  $\mathcal{G}$ . We also consider only the top 200 possibilities for  $x_{i}$  according to  $\mathcal{G}$  at each step, as a cheap approxi
mation to the full distribution, and find that this works well in practice. $^{1}$  In each task in Sec. 4, running FUDGE on the test set takes no more than 15 minutes on a single Quadro RTX 6000 GPU.
Finally, as with other controlled generation approaches such as Dathathri et al. (2019), it is likely that augmenting FUDGE with reranking approaches such as rejection sampling could improve output quality at the cost of compute time, although we do not comprehensively evaluate such extensions in this work.
3.1 Advantages and Limitations
We highlight several additional potential advantages of FUDGE compared to directly modeling  $P(x_{i}|x_{1:i - 1},a)$  via e.g., a fine-tuned CCLM:
1. FUDGE requires access only to  $P(X)$  (i.e.,  $\mathcal{G}$ 's output logits) rather than  $\mathcal{G}$  itself.
2.  $\mathcal{G}$  can be freely swapped out for any other model that shares the same tokenization when larger models become available.
3. Given multiple conditionally independent attributes with predictors for each, FUDGE can easily condition on the combination of these attributes in a modular fashion by summing their output log-probabilities (Sec. 4.1, 4.2).
Unfortunately, like previous methods, FUDGE cannot fully guarantee that all outputs possess the desired attribute  $a$ . In FUDGE's case, this is due to the approximation inherent in modeling  $P(a|x_{1:i})$ , as well as only considering the top 200 possible  $x_i$  for computational efficiency.
See Appendix H for ablations on the top-200 pruning.
We run experiments on a range of controlled text generation tasks to evaluate the effectiveness of our proposed method: poetry couplet completion (Sec. 4.1), topic-controlled language generation (Sec. 4.2), and machine translation formality change (Sec. 4.3). For each task we discuss the evaluation setup, the specific details of our method and baselines, and finally experimental results.
4.1 Poetry Couplet Completion
So long as men can breathe or eyes can see,  
So long lives this and this gives life to thee.
Table 1: An example couplet by William Shakespeare. Every second syllable is stressed, following iambic meter, and the last words of each line (see/thee) rhyme.
We begin with English poetry generation, a task that emphasizes well-formedness, and which has been studied in different forms by many previous works (Zhang and Lapata, 2014; Wang et al., 2016; Ghazvininejad et al., 2016, 2017). Our task here is couplet completion. Given the first line of an iambic pentameter couplet (e.g., Table 1), the model must generate a second line which (1) satisfies iambic pentameter, (2) rhymes with the first line, and (3) ends a sentence. The desired attribute  $a$  is defined as possessing all three properties, as evaluated by a rule-based checker  $\mathcal{F}$  (Appendix A). Our test set is a collection of prefix lines of couplets, collected from the ending couplet of each of Shakespeare's 154 sonnets.
Metrics. We consider four metrics.
1. Success, the fraction of couplet completions with the desired attribute  $a$ , as checked by  $\mathcal{F}$ . This is the main metric.
2. Grammaticality, the probability of grammaticality given by a Roberta-based CoLA grammaticality model (Liu et al., 2019; Warstadt et al., 2019), averaged over all outputs.
3. Perplexity of the completion conditioned on the prefix. Following Dathathri et al. (2019), since our models use GPT2-Medium (Radford et al., 2019) as  $\mathcal{G}$ , we evaluate perplexity using GPT (Radford et al., 2018).<sup>2</sup>
4. Distinctness of completions, measured as the number of unique unigrams, bigrams, and trigrams across all samples, divided by the total number of words (Li et al., 2015).
At test time, we decode until the model generates ten syllables followed by an end-of-sentence punctuation mark, or after the eleventh syllable (an automatic failure, since iambic pentameter requires exactly ten syllables).
Overall, because we define  $a$  using a rule-based  $\mathcal{F}$  which is accessible during training, our formulation of couplet completion is a relatively clean task for evaluating the effectiveness of FUDGE.
4.1.1 Method and Baselines
FUDGE Instantiation. The obvious approach is to learn a predictor for  $\mathcal{F}$  directly. However, the three components of  $a$  — meter, rhyme, and sentence-ending — should be roughly independent. Thus we assume conditional independence, and demonstrate the modularity of FUDGE by constructing three separate predictors to be combined at test time:
1.  $\mathcal{B}_1(x_{1:i})$  takes a text prefix  $x_{1:i}$ , and predicts whether the completion  $x_{1:n}$  of prefix  $x_{1:i}$  will be in iambic meter. The model is an LSTM followed by a linear output layer.
2.  $\mathcal{B}_2(x_{1:i},t,r)$  takes prefix  $x_{1:i}$ , the number of syllables  $t$  between  $x_{i}$  and  $x_{n}$  for  $n\geq i$ , and a rhyme sound  $r$ .<sup>3</sup> It predicts whether the completion  $x_{1:n}$  has the rhyme sound  $r$  at the end of token  $x_{n}$ . The model is an LSTM with attention dependent on  $t$  and  $r$ , followed by a shallow feedforward network, and is trained via noise-contrastive estimation (Gutmann and Hyvarinen, 2010).<sup>4</sup>
3.  $\mathcal{B}_3(x_{1:i},t)$  takes prefix  $x_{1:i}$  and the number of syllables  $t$  between  $x_{i}$  and  $x_{n}$  for  $n\geq i$  and predicts whether  $x_{n}$  ends a sentence. The model is an LSTM followed by a shallow feedforward network.
The predictors vary in architecture because  $\mathcal{B}_2$  and  $\mathcal{B}_3$  require inputs other than  $x_{1:i}$  — in truth, they are families of related predictors. We find that performance is not overly sensitive to the particulars of the predictor architectures (Appendix D).
3Two words have the same "rhyme sound"  $r$  if they rhyme according to the CMU Pronouncing Dictionary (Weide, 1998).
4The output logits from  $\mathcal{B}_2$  are unnormalized, but this does not affect FUDGE after they are added to the output logits of  $\mathcal{G}$  and softmaxed for sampling.
See Appendix E for other perplexity measurements.
Method	Correctness	Text Quality	Diversity
Success ↑	Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
G	0	0.52	44.3 ± 42.2	0.35	0.74	0.77
FINETUNE	0.21	0.44	55.8 ± 98.3	0.35	0.74	0.78
PPLM	0	0.54	60.8 ± 66.1	0.40	0.78	0.78
FUDGE	0.44	0.44	70.9 ± 89.4	0.40	0.79	0.78
Shakespeare	0.45	0.29	333.8 ± 418.9	0.44	0.81	0.79
Table 2: Couplet completion results. Success (main metric), grammaticality, perplexity, and distinctness of different methods, tested on 154 prefix lines from Shakespeare sonnets. FUDGE substantially outperforms automated baselines on success and maintains high diversity, although quality unsurprisingly suffers compared to the base  $\mathcal{G}$  due to the difficult constraint  $\mathcal{F}$ . Note Shakespeare's work is often "incorrect" due to the narrowness of our metric  $\mathcal{F}$ ; he also scores poorly on text quality because our evaluation models are intended for more modern English.
To train the discriminators, we sample a dataset of 10 million generations of varied length from GPT2-Medium. From these generations, we sample random subsequences  $x_{1:n}$  of roughly 10 to 30 syllables and truncate  $t \leq 10$  ending syllables. These truncations become inputs  $x_{1:i}$  to the predictors. For simplicity, we did not balance the class labels for e.g., the iambic predictor during training, although it is likely that doing so would improve performance.
At test time, we extract  $r$  from the given first line of the couplet, and initialize  $t = 10$ , updating at each step. We then modify the output logits of  $\mathcal{G}$  by simply adding the log-probabilities from  $\mathcal{B}_1$ ,  $\mathcal{B}_2$ , and  $\mathcal{B}_3$ , demonstrating the ease of composing constraints in FUDGE.
Baselines. We compare to four baselines.5
1.  $\mathcal{G}$ , the original GPT2-Medium.
2. FINETUNE, a CCLM which finetunes  $\mathcal{G}$  on similar inputs to those used for  $\mathcal{B}_2$  in FUDGE. Since it is not obvious how to compose multiple CCLM's for different attributes, we train a single CCLM for all desired properties together. We condition by prefixing the input with (1) whether the last 10 syllables of the original untruncated  $x_{1:n}$  are iambic, (2) the
rhyme sound at the end of  $x_{n}$ , and (3) whether a sentence ends with  $x_{n}$ . A special token is inserted 10 syllables from the end of  $x_{1:n}$ .
3. PPLM (Dathathri et al., 2019), which uses shallow predictors learned from  $\mathcal{G}$ 's top-level hidden layer to modify  $\mathcal{G}$ 's states toward increasing probability of the desired attribute via gradient ascent. We decompose the predictors into the same iambic, rhyme sound, and end-of-sentence predictors as for FUDGE, inserting an additional hidden layer in the shallow predictor when needed to incorporate additional input (the desired rhyme sound and/or number of syllables until end-of-sentence).
4. Shakespeare's original couplet completions.
All non-Shakespeare methods use top- $k$  sampling with  $k = 10$ .
Even though our GPT2-Medium-generated training dataset is completely different from the test domain, and contains essentially zero examples of correct couplets, FUDGE is able to learn the desired attribute. As shown in Table 2, FUDGE greatly outperforms all automated baselines in success rate.
Surprisingly, the PPLM baseline achieves zero success. We find that its iambic and rhyme predictors are very poor, so we hypothesize that the relevant information is not easily extractable from the last hidden layer of  $\mathcal{G}$ . In contrast, FUDGE's predictors operate directly on the raw text.
Funnily enough, FUDGE even matches Shakespeare according to  $\mathcal{F}$ , although this is largely due to the narrowness of  $\mathcal{F}$  and should not be taken se
5A system like Hafez (Ghazvininejad et al., 2016, 2017), which enforces meter and rhyme at each decoding step using a hard constraint, could achieve perfect success rate. However, this approach relies on the meter and rhyme attributes being "prefix-checkable" at the word level: one can guarantee success by simply never selecting a word which immediately violates the constraint. This is often the case for simple rule-based constraints, but not for many other interesting attributes, such as the topic and formality attributes in our subsequent experiments. To preserve generality, FUDGE does not rely on this "prefix-checkable" property, and neither do our baselines.
riously. $^{6}$  Similarly, the grammaticality and perplexity metrics are designed for our automated baselines, and thus assign poor scores to Shakespeare's antiquated and flowery style.
FUDGE also maintains relatively fluent generation despite lower grammaticality and perplexity compared to  $\mathcal{G}$ . See Table 3 for two successful examples. Interestingly, FUDGE also increases diversity compared to  $\mathcal{G}$ , perhaps due to the difficult constraint  $\mathcal{F}$  forcing FUDGE to use lower-probability regions of the base distribution  $P(X)$ .
And even thence thou wilt be stol'n, I fear, for this shall be the end. That's pretty clear.
Or, if they sleep, thy picture in my sight I will be glad to look upon the night.
Table 3: Two examples of successful couplet completions (in purple) generated by FUDGE.
Finally, it is possible (and trivial) to adjust the conditioning strength in FUDGE by multiplying the binary predictors' output logits by a constant. However, this deviates from our Bayesian factorization of  $P(X|a)$ , and we do not do so.
4.2 Topic-Controlled Language Generation
Next, we explore topic control in English language generation. The desired attribute  $a$  is to be on-topic for a given topic, such as science or politics. To facilitate comparison with prior work, we largely follow the setup of PPLM (Dathathri et al., 2019): the model is provided an approximation to the topic at test time, in the form of a bag of on-topic words  $\mathcal{W}$ . The goal is to sample text according to the topic approximated by  $\mathcal{W}$ , starting from a generic prefix. There are 7 topics (space, politics, military, legal, science, religion, and computers) and 20 prefixes, and the model generates 3 80-token<sup>7</sup> samples from each topic-prefix pair, for a total of 420 generations.
Metrics. Unfortunately, we cannot easily construct a rule-based  $\mathcal{F}$  for being "on-topic." Addi