SlowGuess/NAACL_2021_OCR huggingface dataset:
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the fisher and callhome spanish-english speech translation corpus. In Proc. IWSLT.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535.
Elizabeth Salesky, Matthias Sperber, and Alex Waibel. 2019. Fluent translations from disfluent speech in end-to-end speech translation. arXiv preprint arXiv:1906.00556.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. arXiv preprint arXiv:1902.08654.
Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019. Pragmatically informative text generation. arXiv preprint arXiv:1904.01301.
Nishant Subramani, Samuel Bowman, and Kyunghyun Cho. 2019. Can unconditional language models recover arbitrary sentences? In Advances in Neural Information Processing Systems, pages 15258-15268.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125.
Eric Wallace, Mitchell Stern, and Dawn Song. 2020. Imitation attacks and defenses for black-box machine translation systems. arXiv preprint arXiv:2004.15015.
Qixin Wang, Tianyi Luo, Dong Wang, and Chao Xing. 2016. Chinese song iambics generation with neural attention-based model. arXiv preprint arXiv:1604.06274.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Robert L Weide. 1998. The cmu pronouncing dictionary. URL: http://www.speech.cs.cmu.edu/cgi-bin/cmudict.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, pages arXiv-1910.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-first AAAI conference on artificial intelligence.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062.
Xingxing Zhang and Mirella Lapata. 2014. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670-680.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
A Details of  $\mathcal{F}$  for Couplet Completion
We provide the full details of the function  $\mathcal{F}$  we use to check iambic pentameter, rhyme, and sentence-ending in our couplet completion task. Note that iambic pentameter consists of two components: iambic meter as well as containing exactly ten syllables.
1. Iambic meter: Given a phrase, we obtain the sequence of stresses (0 for unstressed, 1 for stressed, 2 for secondary stress) for each word, according to the CMU Pronouncing Dictionary (Weide, 1998). If any word does not exist in the dictionary (almost never for non-Shakespeare methods) we return False. We treat 2 as ambiguous stress, and additionally change 1 to 2 for any monosyllabic words, i.e. we allow monosyllabic stressed words to be unstressed but not vice versa. Finally, we check that all syllables at even indices (0-indexed) are unstressed or ambiguous, and all syllables at odd indices are stressed or ambiguous.
2. Number of syllables: We count the number of syllables in each word based on the number of stresses according to the CMU Pronouncing Dictionary. If a word does not exist in the dictionary, we estimate the number of syllables by rounding the number of letters divided by 3 to the nearest integer.
3. Rhyme: Two words rhyme if and only if they both exist in the CMU Pronouncing Dictionary and are a perfect rhyme according to the dictionary.
4. Sentence-ending: We check if the output ends with a period, question mark, or exclamation mark.
Of course, both FUDGE and FINETUNE will fit to whatever output is given by  $\mathcal{F}$ . The purpose of the couplet task is to check FUDGE's ability to fit a difficult well-formedness constraint. We simply design an  $\mathcal{F}$  that corresponds to true iambic pentameter rhymes in most cases.
A.1 Shakespeare Evaluation
Shakespeare himself performs somewhat poorly according to  $\mathcal{F}$ , which is designed with the automated baselines in mind, not for Shakespeare. (The
same is true for our grammaticality and perplexity metrics.)
One source of error is words which are out-of-vocabulary for the CMU Pronouncing Dictionary. Such words are almost never generated by either FUDGE or our automated baselines, but appear in a fifth of Shakespeare's lines, resulting in failures on the iambic meter and syllable checks.
Nevertheless, most of Shakespeare's "errors" are the result of real — though slight — deviations from our very strict definitions of meter and rhyme. In particular, he frequently (1) elides syllables to fit meter, and (2) uses loose rhymes; both "error" types are likely exacerbated by differences between archaic and modern pronunciations. The example in Table 10 illustrates both types of "errors." Although such deviations are often acceptable to a human, they are difficult to capture in an automatic metric, and we do not allow such deviations in  $\mathcal{F}$ . Again, Shakespeare is only included as a whimsical point of reference, and not as a serious baseline to be compared to.
But here's the joy; my friend and I are one; Sweet flattery! then she loves but me alone.
Table 10: An example couplet by William Shakespeare, illustrating two common deviations from the narrow definition of correctness we use in  $\mathcal{F}$ . For this example to follow iambic meter, one must read "flattery" in only two syllables. Moreover, "one/alone" is a loose (non-perfect) rhyme, at least in modern English.
B PPLM Baseline in Machine Translation
As discussed in the main text, it is difficult to apply PPLM in our machine translation setup, in which  $P(a|X)$  is learned from an English formality dataset without parallel Spanish. Since  $P(X)$  is a Spanish-English translation model, we must obtain hidden states for training PPLM's  $P(a|X)$  by first "backtranslating" English into Spanish, accessing a second pretrained translator. For this purpose we use a second pretrained Marian transformer from HuggingFace (https://huggingface.co/Helsinki-NLP/opus-mt-en-es). Additionally, we needed to tune their suggested hyperparameters.
During evaluation, we observe that PPLM makes some reasonable modifications for formality compared to the base  $P(X)$ , like changing "hard" to "difficult," but such improvements are also accompanied by occasional disfluencies and/or repetitions (although such problems plague all methods to
some degree). Overall, while PPLM achieves similar BLEU to FUDGE, it is substantially less formal (Table 11).
Method	G (Fine-tune)
BLEU ↑	Form. ↑
PPLM	21.94	0.40
FUDGE	22.18	0.48
Table 11: PPLM baseline in machine translation formality on the fine-tuned  $\mathcal{G}$
FUDGE has essentially one hyperparameter in our topic control task,  $\lambda$ , which controls the strength of conditioning and corresponds to the number of words in the bag which should appear in the future.
To choose  $\lambda$  in topic control, we used a separate validation bag of words (on the topic of fantasy; Appendix K.4) to select a reasonable  $\lambda$  for our main paper experiments ( $\lambda = 4$ ). Unlike in the main paper where we use heldout bags  $\mathcal{W}'$  to measure success, during validation we simply use the original bag. We use a set of 60 generations, considering values ranging from 1 to 6 (Table 12), although the result may be somewhat noisy. Of course, different choices of  $\lambda$  result in different tradeoffs (Appendix G).
We also optimized the conditioning strength  $\lambda_{\mathrm{WDEC}}$  for the WDEC baseline on the same fantasy bag of words, considering values ranging from 1 to 32. We selected the only value (4) which achieved reasonable success without a total collapse in diversity (Table 13), but diversity still collapsed when tested on our seven main test bags of words.
We do not optimize any model hyperparameters in the couplet completion and informal-to-formal translation tasks. LSTM's and feedforward networks are 3 layers (including the output layer of dimension 1) and 300-dimensional unless otherwise specified. They are bidirectional (150-dimensional in each direction) for the couplet rhyme predictor and the topic control future words predictor, and otherwise unidirectional. Attention mechanisms use key-query-value attention. For the rhyme and future words predictors the output hidden state is multiplied element-wise by the embedding of the rhyme sound or future word, then concatenated to the embeddings, before the final feedforward network. Since a selling point of our method is the
lightweight process of constructing and training predictors, noise-contrastive estimation is a natural choice for the rhyme and future word predictors: we avoid softmaxing over the output dimension during training. (This is primarily relevant for the future word predictor, as the number of distinct rhyme sounds is not too large, but we use noise-contrastive estimation for both for consistency's sake.)
For the PPLM baseline, we used step size 0.01 for both couplet completion and MT after tuning, and kept their other hyperparameters fixed. For topic control we simply evaluated their provided generations instead of rerunning their model.
D Ablations on Predictor Architectures
Some variation in predictor architectures is necessary due to the diversity of our tasks (as evidenced by the difficulties in adapting PPLM). Specifically, while our core predictor architecture is word embeddings followed by LSTM and output layer, task-specific architectures vary because some "predictors" are actually families of related predictors. We model such families as a single predictor taking additional input (e.g., rhyme sound in poetry); this is needed in our poetry and topic tasks.
On these two tasks, we provide ablations with more homogenized predictors: additional inputs are simply embedded and concatenated to each input word embedding. The difference is relatively small in both cases (Tables 14 and 15). FUDGE-Mod indicates the ablated version of FUDGE.
E Alternative Perplexity Measurements
On the couplet completion task, we additionally measure perplexity using Transformer-XL (Dai et al., 2019b) and using a GPT model fine-tuned on Shakespearean language as generated by (Lau et al., 2018). We measure using Transformer-XL on the topic control task as well. Relative perplexities between most models remain largely similar when switching between GPT and Transformer-XL, with a few exceptions. Compared to the base GPT, Shakespeare's perplexity naturally decreases while other models' perplexities increase when measured with Shakespeare-finetuned GPT. The highly repetitive and disfluent WDEC baseline is rightly punished for this behavior when measured by Transformer-XL. PPLM also obtains slightly lower perplexity than FUDGE on topic control when
Method	On-Topic Success ↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
FUDGE, λ = 1	0.05	0.80	38.3 ± 32.6	0.36	0.78	0.92
FUDGE, λ = 2	0.10	0.76	31.1 ± 17.1	0.35	0.75	0.91
FUDGE, λ = 4	0.28	0.76	40.1 ± 27.6	0.37	0.77	0.92
FUDGE, λ = 6	0.30	0.72	46.9 ± 29.9	0.38	0.77	0.91
Table 12: Results from 60 samples for FUDGE with different  $\lambda$  on topic control for a validation fantasy-topic bag of words. Note that during validation only, success directly measures use rate of words in the given bag  $\mathcal{W}$ , not a heldout bag  $\mathcal{W}'$  as in the main paper.
Method	On-Topic Success ↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
WDEC, λWDEC = 1	0.02	0.83	34.5 ± 23.5	0.36	0.78	0.91
WDEC, λWDEC = 2	0.02	0.83	34.9 ± 23.8	0.36	0.78	0.91
WDEC, λWDEC = 4	0.57	0.79	34.7 ± 23.6	0.33	0.74	0.86
WDEC, λWDEC = 8	1.90	0.47	14.5 ± 19.2	0.04	0.09	0.12
WDEC, λWDEC = 16	2.32	0.40	8.4 ± 9.6	0.01	0.04	0.06
WDEC, λWDEC = 32	2.35	0.41	7.5 ± 9.1	0.01	0.04	0.06
Table 13: Results from 60 samples for WDEC with different  $\lambda_{\mathrm{WDEC}}$  on topic control for a validation fantasy-topic bag of words. Note that during validation only, success directly measures use rate of words in the given bag  $\mathcal{W}$ , not a heldout bag  $\mathcal{W}'$  as in the main paper.
Method	Correctness	Text Quality	Diversity
Success ↑	Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
FUDGE	0.44	0.44	70.9 ± 89.4	0.40	0.79	0.78
FUDGEmod	0.39	0.43	72.1 ± 66.3	0.41	0.79	0.77
Table 14: Ablation of FUDGE with a modified predictor architecture on couplet completion.
Method	On-Topic Success ↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
FUDGE	0.59	0.79	40.7 ± 26.3	0.34	0.75	0.91
FUDGEmod	0.62	0.77	47.8 ± 51.3	0.33	0.73	0.88
Table 15: Ablation of FUDGE with a modified predictor architecture on topic control.
Method	GPT	TFXL	GPT-Shakespeare
G	44.3 ± 42.2	84.8 ± 111.0	72.9 ± 62.0
FINETUNE	55.8 ± 98.3	76.1 ± 64.6	69.0 ± 92.5
PPLM	60.8 ± 66.1	111.5 ± 150.4	120.0 ± 243.8
FUDGE	70.9 ± 89.4	137.5 ± 170.9	96.2 ± 117.3
Shakespeare	333.8 ± 418.9	1879.5 ± 6088.1	195.1 ± 228.9
Table 16: Different perplexity measurements on couplet completion, using GPT, Transformer-XL (TFXL), and GPT fine-tuned with Shakespearean language (GPT-Shakespeare). Main paper results use GPT.
Method	GPT	TFXL
G	37.1 ± 26.9	34.1 ±	25.2
FINETUNE	24.9 ± 13.7	27.7 ±	15.6
WDEC	33.8 ± 33.7	7802.4 ± 29942.6
PPLM	43.1 ± 23.7	38.7 ±	21.0
FUDGE	40.7 ± 26.3	42.8 ±	46.9
Table 17: Different perplexity measurements on topic control, using GPT and Transformer-XL (TFXL). Main paper results use GPT.
measured by Transformer-XL. Full results in Tables 16 and 15.
F Statistical Significance
In couplet completion, FUDGE outperforms the strongest automated baseline (FINETUNE) on success rate with  $p < 0.0001$  on a McNemar test, pairing the generations for each Shakespeare prefix.
In topic control, FUDGE outperforms the strongest automated baseline PPLM with  $p = 0.04$  using a Wilcoxon matched pairs test, pairing the generations for topic-prefix combinations.
In translation formality, FUDGE's generations are more formal than those of the base  $\mathcal{G}$  with  $p < 0.0001$  according to a paired t-test.
Space: The issue focused on the new, higher level of control that NASA had in the space shuttle program. \n\nThe question of how far the U.S. government can extend its jurisdiction in space was raised," Mr. Smith said.  $\backslash \mathsf{n}\backslash \mathsf{n}$  NASA's role has become increasingly important in the 21st century in part because of the growth in space activities. The space shuttle program began in 1977 with
Politics: The issue focused on how much power each company was willing to use in response to the request.\n\nThe issue of how much power each company was willing to use in response to the request. \n\nComcast has not been forthcoming with any data, such as how often it uses the technology, and what it has paid for it, in order to meet the FCC's mandate to make its own data more accessible.\n\n
Military: The issue focused on the use of force by the armed forces and police, as well as the use of lethal force by civilians.  $\backslash \mathsf{n}\backslash \mathsf{n}$  The bill would require that a shooting occur "with reasonable care," meaning a shot was "justified" under the circumstances of the case and not in retaliation for an act of violence, and that a shooting was "necessary for the safety of the officer or the
Table 18: The first generation by FUDGE using  $\lambda = 2$  on the space, politics, and military topics given the prefix "The issue focused on." Words in the given bag are highlighted in red, and other related words in cyan.
G Effect of Varying Topic Control Strength
Although we use  $\lambda = 4$  for FUDGE in our main paper experiments for topic control, we experiment here with varying the conditioning strength. Specifically, we experiment with  $\lambda = 2$  and  $\lambda = 8$ . The conditioning is unsurprisingly stronger as  $\lambda$  increases, as shown quantitatively in Table 19, although the perplexity increases as well.
We also provide some example generations for  $\lambda = 2$  and  $\lambda = 8$  in Tables 18 and 21, for the same prompts and topics as in Table 6 for  $\lambda = 4$  in the main text. The  $\lambda = 8$  generations remain mostly fluent and interesting, despite their worse grammaticality and perplexity.
H Effect of Varying Candidate Pruning
For computational efficiency, we only feed the top 200 candidates returned by  $\mathcal{G}$  into FUDGE's predictor when predicting each next token. Here, we ablate on this number in our topic control setting, testing 100 and 400 (Table 20).
I Additional Couplet Completion Examples
We provide some additional examples of FUDGE and baselines on our couplet completion task in Table 22.
We also show some unsuccessful examples for FUDGE in 23. Overall, we find that most errors are due to the rhyme and ten-syllable end of sentence constraints, or due to Shakespeare's prefix ending in a word not in the CMU Pronouncing Dictionary (e.g., "prognosticate" in the table). FUDGE also sometimes overgenerates punctuation at the end of a sentence.
J Additional Topic Control Examples
In Tables 24, 25, and 26 we show additional example generations by our method using the same
Method	On-Topic Success ↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
FUDGE, λ = 2	0.49	0.79	38.6 ± 24.1	0.36	0.76	0.91
FUDGE, λ = 4	0.59	0.79	40.7 ± 26.3	0.34	0.75	0.91
FUDGE, λ = 8	0.76	0.74	56.5 ± 39.0	0.35	0.75	0.90
Table 19: FUDGE results for different values of  $\lambda$  on the main 7 topics and 20 prefixes. Success and perplexity both increase as the conditioning strength  $\lambda$  increases. Our main paper experiments use  $\lambda = 4$ .
Method	On-Topic Success↑	Text Quality	Diversity
Grammar ↑	Perplexity ↓	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑
FUDGE, 100	0.54	0.77	48.9 ± 37.9	0.36	0.75	0.91
FUDGE, 200	0.59	0.79	40.7 ± 26.3	0.34	0.75	0.91
FUDGE, 400	0.61	0.77	49.4 ± 56.9	0.35	0.76	0.91
Table 20: FUDGE results for different numbers of candidates fed through FUDGE's predictor. Main paper results use 200.
Space: The issue focused on the size of NASA's satellite telescope that is being used to observe the universe.\nThe telescope will be the world's largest when it is completed in 2022.\nThe US space agency wants to put the telescope into a new orbit around the planet.\nThe Hubble Space Telescope orbits in an elliptical orbit, which puts the telescope into a "cross-path"
Politics: The issue focused on the power, independence and independence of the federal judiciary. In its ruling, the three-judge "progressive" panel of the 10th Circuit of the U.S. Court of Appeals for the 10th Circuit found that the "Supreme Court lacks the requisite power to make decisions on the constitutionality of any particular federal regulation, including the power to make the final determination
Military: The issue focused on the military wing of the U.S. Navy that manages ships to the surface of a seabed, the USS Ponce, which is carrying a guided-missile destroyer. A small accident with the weapon's guidance system forced the carrier's "supergun" system to be deactivated, resulting in a loss of a "kill ship" role from the fleet. The
Table 21: The first generation by FUDGE using  $\lambda = 8$  on the space, politics, and military topics given the prefix "The issue focused on." Words in the given bag are highlighted in red, and other related words in cyan.
Prefix: Pity the world, or else this glutton be,
$\mathcal{G}$  (1) to see that you are not to be ashamed.
CCLM: or else it is a human being. He.
PPLM: ... " He continued, "I have seen, and heard of the
FUDGE: and so I do; and so I do, as be.
Shakespeare: To eat the world's due, by the grave and thee.
Prefix: But flowers distill'd though they with winter meet,
$\mathcal{G}$  : and with their sweet, sweet love; The snow-white rose
CCLM: the flowers are not quite so sweet or sweet.
PPLM: 'Twas a great feast, and merry company all
FUDGE: (I will be here to see her) let my sweet.
Shakespeare: Leese but their show; their substance still lives sweet.
Prefix: And nothing 'gainst Time's scythe can make defence
$\mathcal{G}$  : A better foe to be slain, than Death, and all
CCLM: a better weapon than it is. That defence
PPLM: I am so glad that I have found a new home
FUDGE: worth more. That's how much time is on the fence.
Shakespeare: Save breed, to brave him when he takes thee hence.
Table 22: Additional example couplet completions, comparing  $\mathcal{G}$ , FINETUNE, PPLM, FUDGE, and Shakespeare. "Prefix" indicates the first line of the couplet which generations are conditioned on.
Prefix: Make thee another self, for love of me,
FUDGE: and for the joy of thee, and for love's,...I.,
Shakespeare: That beauty still may live in thine or thee.
Prefix: Or else of thee this I prognosticate:
FUDGE: I, O, Lord Jesus Christ, my Beloved; I am
Shakespeare: Thy end is truth's and beauty's doom and date.
Prefix: But were some child of yours alive that time,
FUDGE: would you still have loved that child for that? I'm...
Shakespeare: You should live twice; in it and in my rhyme.
Table 23: Additional example couplet completions where FUDGE is unsuccessful. In the first, FUDGE fails to rhyme; in the second, "prognosticate" is not in the CMU Pronouncing Dictionary; in the third, FUDGE gets the syllable count wrong.
hyperparameter setting as in the main paper,  $\lambda = 4$ . Specifically, we provide the first generation by FUDGE for 3 separate prefixes for each of the 7 topics. Virtually all examples are clearly on topic, while avoiding repetitiveness.
Additionally, we provide example generations from  $\mathcal{G}$ , FINETUNE and WDEC in Tables 27, 28, and 29 respectively. For PPLM we refer the reader to the examples in the main paper and appendices of Dathathri et al. (2019)'s original work.
Space: More importantly, the E4E-R-E-S-T report finds, the greatest threat to Earth's existence comes from the human race's inability to adapt: "It is clear from E4E's analysis that a lack of knowledge about how to manage an expanding world and to adapt to changing climatic conditions poses a serious challenge to our ability to sustain life on planet Earth.
Politics: More importantly, in an effort to preserve the historical integrity of the state's judicial system, the state also needs to ensure its integrity within the larger American political system through fair, transparent, and competitive elections. In other words, a system based upon meritocracy and equality for all candidates, voters, candidates and parties.\n\nThe American people have a right to know whether the current system for electing state