SlowGuess/NAACL_2021_OCR huggingface dataset:
tionally, use rate of words in  $\mathcal{W}$  is a poor metric, because a model can score highly by e.g., simply returning the words in  $\mathcal{W}$ , without generalizing to the full topic that  $\mathcal{W}$  approximates. Instead, we adopt a notion of success which requires the model to generalize the bag  $\mathcal{W}$  to the full topic. The remaining metrics are measures of quality and diversity.
1. Success, the average number of distinct words in a heldout bag  $\mathcal{W}'$  which appear in the model output. Specifically, for each word in  $\mathcal{W}$ , we add to  $\mathcal{W}'$  the closest GloVe (Pennington et al., 2014) word by cosine similarity, such that the new word does not contain (and is not contained by) any word in  $\mathcal{W}$ . (This excludes e.g., most plurals.) Usage of distinct words in  $\mathcal{W}'$  measures the model's ability to generalize  $\mathcal{W}$  to other on-topic words, of which  $\mathcal{W}'$  is a non-exhaustive set. This is our main metric.
2. Grammaticality, identical to the couplet task.
3. Perplexity, identical to the couplet task.
4. Distinctness, defined as in the couplet task. However, it is calculated separately within the 60 generations for each topic, and then averaged over the 7 topics.
Additionally, following the evaluation procedure of prior work such as (Dathathri et al., 2019), we run human evaluations via Amazon Mechanical Turk for FUDGE against each baseline, comparing topic control and fluency. For each pairwise comparison, we ask 3 workers to evaluate each of 420 paired outputs. Workers were asked to mark which generation is more on topic (first, second, both, or neither), and to rate each generation's fluency on a Likert scale from 1 to 5. We report the average fraction of outputs marked as on-topic as well as the average fluency rating for each method.
4.2.1 Method and Baselines
FUDGE Instantiation. Since we model topics as bags of words, FUDGE uses a binary predictor  $\mathcal{B}(x_{1:i},w)$  which takes a prefix  $x_{1:i}$  and word  $w$ , and classifies whether  $w$  appears in the future  $x_{i:n}$  for  $n \geq i$ . (Since it is desirable to stay on topic even after successfully getting on topic, we use  $x_{i:n}$  rather than  $x_{1:n}$ .) Training examples  $(x_{1:i},w)$  are sampled from the same dataset of 10 million GPT2-Medium generations used for the couplet task, and  $\mathcal{B}$  is trained using noise-contrastive estimation.  $\mathcal{B}$
<sup>6</sup> We define  $\mathcal{F}$  using somewhat narrow criteria (Appendix A), which capture only a subset of what Shakespeare considered to be well-written couplets. The purpose of this task is to evaluate FUDGE's ability to satisfy a difficult well-formedness constraint compared to automated baselines, rather than to perfectly capture the human notion of an iambic pentameter couplet. Thus Shakespeare is marked wrong when he (1) uses archaic pronunciations, (2) uses loose rhymes, (3) elides syllables to fit meter, or (4) uses words missing from the CMU Pronouncing Dictionary. See Appendix A.1 for details. Of course, Shakespeare is only included as a whimsical point of reference; our generations obviously do not hold a candle to Shakespeare's originals.
7 All models and baselines use GPT2 tokenization.
Method	On-Topic Success ↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
G	0.22	0.81	37.1 ± 26.9	0.35	0.78	0.92
FINETUNE	0.28	0.74	24.9 ± 13.7	0.29	0.70	0.88
WDEC	0.14	0.59	33.8 ± 33.7	0.16	0.42	0.55
PPLM	0.48	0.78	43.1 ± 23.7	0.35	0.78	0.92
FUDGE	0.59	0.79	40.7 ± 26.3	0.34	0.75	0.91
is a lightweight LSTM-based classifier similar to  $\mathcal{B}_2$  from the couplet task.
At test time, we can compose individual-word constraints if we assume conditional independence between words (although this may be imperfect). Given a bag of  $N$  words  $\{w_{1}\dots w_{N}\}$  and prefix  $x_{1:i}$ , we could condition on all words in the bag appearing in the future by adding all log-probabilities  $\log P(w_1|x_{1:i})\ldots \log P(w_N|x_{1:i})$  to  $\mathcal{G}$ 's logits. However, topic control does not require every word to appear; perhaps some number  $\lambda$  of on-topic words is enough to be "on-topic." Therefore, we model the topic constraint as selecting a random subset of  $\lambda$  words from the original bag, and requiring that only those  $\lambda$  words all appear. Since each of the  $N$  words is selected with probability  $\frac{\lambda}{N}$ , the quantity we add to the base  $\mathcal{G}$  logits is  $\frac{\lambda}{N}\sum_{j = 1}^{N}\log P(w_j|x_{1:i})$  in expectation. In our experiments we use  $\lambda = 4$ , based on a fantasy-topic bag of words used for validation (Appendix C).
Baselines. We compare to four baselines.
1.  $\mathcal{G}$ , the original GPT2-Medium.
2. FINETUNE, which finetunes  $\mathcal{G}$  on the same inputs used for FUDGE. The future word is given as a prefix for conditioning. At test time, we compute logits for each prefix in the given  $\mathcal{W}$  and use the average as the true logits, as an ad hoc way to condition on the full  $\mathcal{W}$ .
3. WDEC, a simple weighted decoding implementation which greedily considers only the immediate next token when optimizing for  $a$ . Instead of using  $\mathcal{B}$ , WDEC just adds a fixed  $\lambda_{\mathrm{WDEC}}$  to the logit for each word in  $\mathcal{W}$ . Note WDEC requires  $a$  to be well-defined at the token level, so it is not easily transferable to certain tasks (e.g., couplet completion).
4. PPLM (Dathathri et al., 2019), which modifies the activations of  $\mathcal{G}$  to make the desired bag of words more likely at the immediate next position. We use their method without reranking for fair comparison.
All methods use top- $k$  sampling with  $k = 10$ , following Dathathri et al. (2019)'s setup.
Table 4: Topic control results. Success (main metric), grammaticality, perplexity, and distinctness for different methods. FINETUNE and WDEC often degenerate into repeating the given bag of words  $\mathcal{W}$ ; this is ill-captured by perplexity, but results in poor grammaticality and distinctness. FUDGE substantially outperforms all baselines on success, including the strong gradient-based PPLM baseline, while preserving high quality and diversity.
Method	Topic	Fluency
G	0.16	4.11
FUDGE	0.78	4.30
FINETUNE	0.24	3.95
FUDGE	0.76	4.22
WDEC	0.49	2.50
FUDGE	0.75	4.21
PPLM	0.45	4.05
FUDGE	0.74	4.16
Table 5: Topic control human evaluations, pairwise comparisons. FUDGE achieves a substantially higher fraction of on-topic outputs compared to each baseline, in addition to higher average fluency (rated 1 to 5).
FUDGE achieves the highest success by a substantial margin (Table 4), and outperforms all baselines on human evaluations in both topic relevance and fluency (Table 5). FUDGE simultaneously preserves high quality and diversity according to automated metrics. Table 6 shows two examples.
Unsurprisingly,  $\mathcal{G}$  performs poorly on success. WDEC and FINETUNE also perform poorly, in success and especially in distinctness. WDEC frequently degenerates into repeating the given words in the bag  $\mathcal{W}$ , despite tuning  $\lambda_{\mathrm{WDEC}}$  (Appendix C).
Space: The issue focused on the original plot, which was about a mysterious ship that would land on Earth, and would lead to humanity's first interstellar expedition. The original plan called for humanity to use the spacecraft to colonize outer space and build the first city on Mars. But this idea fell by the wayside in the final drafts.\n\n
Politics: The issue focused on whether the two institutions were operating within the bounds set by the constitution and the law.\n\nThe Constitutional Court said that both governments "have a duty to ensure the integrity of the electoral process and its effective administration, especially in light of the current political climate that is threatening the functioning of elections"
Table 6: The first output from FUDGE when using the prefix "The issue focused on" for two topics. We used red to highlight words in the given bag of words  $\mathcal{W}$  along with obvious forms (e.g., plurals), and cyan for other on-topic words, including related words not in the heldout bag  $\mathcal{W}'$ . More examples in Appendix J.
FINETUNE also suffers from repetition, which appears to be the result of distribution shift from finetuning. Our fine-tuning dataset was built by sampling directly from the original  $P(X)$  modeled by  $\mathcal{G}$  to mitigate distribution shift, but it is well-known that language model generations are more repetitive than natural language (Holtzman et al., 2018, 2019). We hypothesize that FINETUNE, being finetuned on language model generations rather than natural language, amplifies this repetitiveness. This repetition is reflected in the poor grammaticality for both FINETUNE and especially WDEC. In contrast, FUDGE does not touch the original  $P(X)$ , largely avoiding FINETUNE's distribution shift problem on this task.
Finally, FUDGE outperforms the strong gradient-based PPLM method, despite requiring access only to  $\mathcal{G}$ 's output logits. Non-reliance on gradients means FUDGE is also many times faster than PPLM, which takes a few hours compared to FUDGE's 15 minutes for the full set of 420 generations on our hardware. Sometimes we do not even have gradients: for example, gradients are unavailable in the API for GPT3 at time of writing.
4.3 Machine Translation Formality Change
Finally, we turn to a somewhat more challenging task, changing formality in machine translation — specifically, from informal to formal. Given a source sentence written in an informal and conversational style, the goal is to output a translation which is also more formal. We test on the Fisher and CALLHOME Spanish-English Speech
Translation Corpus (Post et al., 2013), a collection of transcribed Spanish conversations with English translations. Both the source Spanish and target English are highly informal and disfluent. Salesky et al. (2019) augment the Fisher dataset with additional parallel English translations, rewritten to be more fluent (and hence more formal); see Table 7 for an example. Our task is to translate the original informal Spanish to into more formal English. However, we assume that Salesky et al. (2019)'s fluent references are unavailable during training.
entonces de verdad sí sí pero先进技术 tu estudiando para es es digo es más porque es exactamente
Then, if it's business, but then you are a student for a PHD, the Master's is that exactly.
If it's business, then you are a student for a PhD. The masters is exactly that.
Table 7: An example from the Fisher dataset.
Top: The original Spanish transcription.
Middle: The original English translation.
Bottom: Salesky et al. (2019)'s more fluent version.
Metrics. The desired attribute  $a$  is formality, but we cannot sacrifice the source sentence's meaning. The latter requirement makes generation more constrained than in the couplet and topic tasks, so perplexity and distinctness are less relevant. Instead, we use the following:
1. BLEU Score (Papineni et al., 2002), using two of Salesky et al. (2019)'s fluent references per test example. This is our main metric.
2. Formality, the average probability that the model's outputs are formal, according to an evaluator trained on the Family/Relationships domain of the GYAFC formality dataset (Rao and Tetreault, 2018). The evaluator is an LSTM followed by a linear layer.
4.3.1 Method and Baselines
FUDGE Instantiation. We assume that the attribute  $a$ , formality, is conditionally independent from the original conditioning in  $\mathcal{G}$ , i.e., the meaning of the Spanish input. FUDGE uses a binary predictor  $\mathcal{B}(x_{1:n})$  which classifies whether the text starting with prefix  $x_{1:n}$  is written in a formal style.  $\mathcal{B}$  is an LSTM followed by a linear layer, trained on the Entertainment/Music domain of GYAFC.
At test time, FUDGE directly augments  $\mathcal{G}$ 's log-its using log-probabilities from  $\mathcal{B}$ .  $\mathcal{G}$  is a pretrained Marian (Junczys-Dowmunt et al., 2018)
transformer model for Spanish-English. We evaluate both when  $\mathcal{G}$  is fine-tuned on the original Fisher training dataset (i.e., using the original targets, not Salesky et al. (2019)'s more fluent targets) as well as zero-shot with no fine-tuning, which is challenging due to the highly informal and disfluent text.
Baselines. We compare to two baselines.
1.  $\mathcal{G}$ , the original machine translation model.
2.  $\mathcal{G} + \mathrm{ST}$ , a pipeline consisting of  $\mathcal{G}$  followed by a style transfer model. Our style transfer model is T5 (Raffel et al., 2020), fine-tuned on the same GYAFC Entertainment/Music domain that we used to train  $\mathcal{B}$  in FUDGE.
Since we do not assume access to Salesky et al. (2019)'s more formal targets during training, it is difficult to apply PPLM to this task: PPLM's predictor would operate on the pretrained translation model's hidden states, thus requiring a Spanish-English translation dataset with both formal and informal English. We omit FINETUNE for the same reason. In contrast, FUDGE requires only the original English dataset with formality annotations.
All methods use greedy decoding.
Method	G (No fine-tune)	G (Fine-tune)
BLEU ↑	Form. ↑	BLEU ↑	Form. ↑
G	16.98	0.45	22.03	0.41
G + ST	7.87	0.96	9.63	0.97
FUDGE	17.96	0.51	22.18	0.48
As shown in Table 8, FUDGE increases the formality of outputs compared to  $\mathcal{G}$ , even though the test-time formality predictor is trained on a different domain (Family/Relationships, rather than Entertainment/Music). Note that formality unsurprisingly decreases after fine-tuning  $\mathcal{G}$ , simply due to the informality of the fine-tuning dataset. As in
the couplet task, one could adjust the strength of the formality control in FUDGE, although this is unprincipled from the view of modeling  $P(X|a)$ .
Moreover, while FUDGE and  $\mathcal{G}$  achieve similar BLEU after fine-tuning  $\mathcal{G}$ , FUDGE achieves higher BLEU compared to  $\mathcal{G}$  when  $\mathcal{G}$  is not fine-tuned on the Fisher training set. In the latter case, controlling for formality somewhat remedies the struggles of  $\mathcal{G}$  when not fine-tuned on such disfluent text.
In contrast, the  $\mathcal{G} + \mathrm{ST}$  baseline achieves near-perfect formality but less than half the BLEU of  $\mathcal{G}$ , due to the style transfer model overfitting to the GYAFC Entertainment/Music dataset. This is similar to the distribution shift issue that we observed in topic control for FINETUNE, an issue which FUDGE largely avoids. Nevertheless, there remains substantial room for improvement on this difficult task.
Table 8: Machine translation formality results. BLEU (main metric) and average formality for different methods, with and without fine-tuning  $\mathcal{G}$  on the Fisher domain. FUDGE increases the formality of translations compared to the base model  $\mathcal{G}$  while preserving or increasing BLEU score. Conversely,  $\mathcal{G}$  with style transfer overfits to the formality data, resulting in near-perfect formality but losing the original meaning.
Spanish
G
FUDGE
Reference	que era lo que tenía que tienes que hacer
that was what you had to do
That was what you had to do
What's there to do?
Spanish
G
FUDGE
Reference	ah en mi en inglaterra por ejemplo
Ah, in my, in England, for example.
Ah, in England, for example.
In England, for example?
Table 9: Example translations by  $\mathcal{G}$  (fine-tuned on the Fisher dataset) and FUDGE using the same  $\mathcal{G}$ . Original Spanish and Salesky et al. (2019) references also shown. In this setting, FUDGE achieves similar BLEU to  $\mathcal{G}$  while increasing formality. While FUDGE often simply corrects punctuation or capitalization (top), it also makes more complex adjustments (bottom). More examples in Appendix L.
FUDGE is a principled approach to controlled text generation which models  $P(X|a)$  by closely following a Bayesian factorization, thus preserving the base  $P(X)$  as much as possible. FUDGE achieves strong performance on a wide range of different tasks: poetry couplet completion, topic control, and informal-to-formal machine translation. Additionally, FUDGE can easily compose different attributes in a modular fashion: the meter, rhyme, and end-of-sentence constraints for couplet completion, and the individual words within each topic bag for topic control. In principle, FUDGE is applicable to any controlled generation task where we can train discriminators for the desired attribute or attributes.
<sup>8</sup>We nevertheless ran PPLM in a somewhat convoluted setup, but found that it performed poorly (Appendix B).
6 Ethics of Controlled Text Generation
We recognize that strong controlled generation methods have the potential to produce harmful outputs and/or misinformation when used adversarially (Wallace et al., 2019, 2020). However, such methods can also be a powerful tool for mitigating harmful biases learned by large pretrained language models (Radford et al., 2019; Brown et al., 2020), for example by detoxifying language (Dathathri et al., 2019; Krause et al., 2020). Overall, we believe it is still beneficial to continue research into general controlled text generation methods such as FUDGE.
We thank Daniel Fried, David Gaddy, Eric Wallace, Kevin Lin, Nicholas Tomlin, Ruiqi Zhong, and the three anonymous reviewers for their helpful comments and feedback, which aided us in greatly improving the paper. We also thank the authors of Dathathri et al. (2019) for clarifying our questions about their topic control setup. This work was supported by Berkeley AI Research, DARPA under agreement HR00112020054, and the NSF through a fellowship to the first author. The content does not necessarily reflect the position or the policy of the government, and no official endorsement should be inferred.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. arXiv preprint arXiv:1804.05417.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019a. Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.05621.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019b. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and
Rosanne Liu. 2019. Plug and play language models: a simple approach to controlled text generation. arXiv preprint arXiv:1912.02164.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792.
Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. 2016. Generating topical poetry. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1183-1191.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pages 43-48.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. arXiv preprint arXiv:1703.00955.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2018. Disentangled representation learning for non-parallel text style transfer. arXiv preprint arXiv:1808.04339.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, et al. 2018. Marian: Fast neural machine translation in c++. arXiv preprint arXiv:1804.00344.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2018b. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. 2018. Deep-spare: A joint neural model of poetic language, meter and rhyme. arXiv preprint arXiv:1807.03491.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint arXiv:2005.07064.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9031-9041.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. arXiv preprint arXiv:1907.06616.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037.