SlowGuess/NAACL_2021_OCR huggingface dataset:
5. Science: astronomy, atom, biology, cell, chemical, chemistry, climate, control, data, electricity, element, energy, evolution, experiment, fact, flask, fossil, funnel, genetics, gravity, hypothesis, lab, laboratory, laws, mass, matter, measure, microscope, mineral, molecule, motion, observe, organism, particle, phase, physics, research, scale, science, scientist, telescope, temperature, theory, tissue, variable, volume, weather, weigh
6. Religion: absolute, affect, aid, angel, anthem, apostle, archangel, Archbishop, balance, ban, belief, benefit, Bible, bishop, bless, blessing, bliss, bond, bow, Buddhism, canon, Cantor, cathedral, celestial, chapel, charity, choice, Christianity, church, comfort, community, conflict, connection, conquest, conservative, control, conversion, convert, core, counsel, courage, Covenant, creative, Creator, creed, cross, Crusade, Darkness, decision, deity, destiny, Devil, disciple, discipline, discussion, divine, divinity, doctrine, duty, effect, elder, energy, essence, eternal, ethics, event, evidence, exile, Exodus, faith, family, fate, Father, favor, fundamental, gift, glory, God, gospel, grace, growth, guru, habit, hallow, halo, happiness, harmony, healing, Heaven, Hebrew, holy, honor, hope, host, humane, immortal, influence, insight, instruction, issue, Jesuit, Jesus, joy, Judaism, judgment, justice, karma, keen, Keystone, Kingdom, Latin life, light, love, loving, marriage, meaning, mercy, Messiah, minister, miracle, mission, mortal, mosque, movement, music, mystery nature, nun, official, oracle order, organ Orthodox,outlook,pacific,pagan parish participation,pastor,patriarch peace/perception personal,perspective,petition,pilgrim politics,powerpracticeprayer,prelude presencepriest principle privacy prophet protection purpose query quest question quiet radiant,radical,rally,rebirth redemption refuge relationship relative religion religious Revelation ritual roleSacrament,sacred,sacrifice,sage,saint,salvation,sanctuary,savior,sentence,supplementary,satistus,synthesis,sensory sense serious serve serviceSharia sheherd,shrine silence sin society soul source spirit spiritual split statue Sunday support Supreme teaching temple tests text Torah tradition traditional trust unique unity unknown value,vanity virtue vision voice voices watch weight whole wisdom wonder yang yin zeal
7. Computers: algorithm, analog, app, application, array, backup, bandwidth, binary, bit, bite, blog, blogger, bookmark, boot, broadband, browser, buffer, bug, bus, byte, cache, caps, captcha, CD, client, command, compile, compress, computer, configure, cookie, copy, CPU, dashboard, data, database, debug,
delete, desktop, development, digital, disk, document, domain, dot, download, drag, dynamic, email, encrypt, encryption, enter, FAQ, file, firewall,firmware,flaming,flash,folder, font, format, frame, graphics, hack, hacker, hardware, home, host, html, icon, inbox, integer, interface, Internet, IP, iteration, Java,Joystick,kernel,key,keyboard,keyword,laptop, link,Linux, logic, login,lurking,Macintosh,macro,malware,mmedia,memory,mirror, modem, monitor,motherboard,mouse,multimedia,net,network,node, offline,online,OS, option,output,page,…password,paste,…path,piracy,pirate,platform,podcast,portal,print printer, privacy, process, program,programmer,protocol,RAM,reboot,resolution,restore,ROM,root,router,routine,save,scan, scanner,screen,screenshot,script, scroll, security server shell shift,snapshot software spam,spreadsheet,stORAGE,surf,syntax,table,tag, template,trader,toobar,trash,undo Unix/uploadURLuserUI,...username,...utility,version,virtual,virus,web,website,widget,wikiwindowWindows,wireless,wormXML,Zip
K.2 Prefixes
"An illustration of", "Emphasised are", "Foundational to this is", "Furthermore,", "In brief,", "In summary", "In this essay", "It has been shown", "More importantly,", "Prior to this", "The central theme", "The connection", "The issue focused on", "The key aspect", "The relationship", "This essay discusses", "To conclude,", "To review,", "To summarise", "Views on"
K.3 Heldout Bags of Words
Note that our heldout bag construction process yielded two stopwords, which we removed; they are omitted below.
1. Space: actress, aeronautics, broadband, cosmonaut, cosmos, fireball, flyby, galaxies, heavens, interstellar, lander, lunar, mothership, Romulan, room, worlds
2. Politics: appropriated, aristocrats, authorisation, autocratic, capitalist, communist, credibility, cultural, democratic, diplomatic, efforts, energy, excise, exporting, fascist, federal, federated, freedom, gender, ideologies, immediate, imported, income, judge, jurisdiction, leg-
islative, lengthy, minority, Nazism, progressivism, properties, purchase, ratify, referenda, remember, secondary, shortfall, socialist, subsidies, uphold
3. Military: aboard, academies, adjutant, advancing, airmen, allies, argue, armies, armistice, armour, armoury, assets, ATL, aviation, barrageage, batteries, bleeding, bottom, bricks, cadre, camera, capturing, cargo, casing, casualties, citadel, civilian, civilians, clandestine, committee, companies, concern, conquered, cursor, customer, dead, decoding, defensive, deputy, detonated, dormitories, encoding, enemies, engaging, escorting, evacuating, execute, expert, explosion, fatigues, flames, flying, forcing, forming, fought, freedom, frigate, gatling, glider, groan, guerilla, hand-to-hand, highest, hires, honour, howitzer, ICBM, injuries, inundate, invading, Iraq, khaki, knowledge, lace, late, launchers, leaving, lob, longtime, Maj., manoeuvre, medical, militia, naval, offensively, offices, operation, paragraph, personnel, persuade, pirate, pistol, policeman, propel, proposal, public, pump, rear, relinquish, rescuing, rifle, rifleman, riflemen, rifles, rocket, sabotaging, samurai, scouts, secluded, seige, Sgt., ship, shoulders, significant, skipper, skirmish, sloop, sonar, stationed, strategic, strategy, subsidiary, sunk, sword, taken, team, tensions, terms, threat, tribute, victory, visor, wear, won zone zoning
4. Legal: accusation, acquittal, admit, aggrieved, agreement, alleging, amendment, appearing, appellant, asserted, assertion, assualt, authority, burglary, championship, convicted, conviction, criminal, custodial, debatable, decision, defensive, democratic, deposited, deputies, disagree, discoveries, dispute, disputes, edict, embezzlement, enforcement, ethical, event, exams, families, federal, felonies, findings, folder, forgive, heard, homicide, immune, incarcerated, inept, injunctive, inmates, innocence, insolvency, insolvent, investment, judgment, judicial, jurors, knowing, land-use, leave, legislative, liability, litem, maintain, major, malpractice, mandamus, mediator, mutual, negligent, objecting, offender, pants, parties, passageways, pixels, police, property, prosecuting, prosecution, proxies
purchase, quash, regulations, repress, requesting, rescind, reservation, respondent, restaurant, revelation, reversing, rulings, second-degree, sentencing, sitting, solicitor, statutory, step-by-step, sued, sworn, testify, track, transcribed, treasurer, waived, whether, widget, wrongful, wrongs
5. Science: action, astronomical, bacterium, bone, clinical, component, compounds, electron, electrons, evolved, flow, fuels, genomics, gravitational, humidity, hypotheses, idea, increasing, jug, ligand, magnesium, mathematics, measuring, microscopy, molecular, nothing, observatory, observing, parameter, phone, physicist, physiology, pounds, rain, reason, renewable, scaling, scientific, siphon, statutes, stored, studies, system, tests, theories, transition, warming
6. Religion: Adventure, Almighty, Always, Answer, Appeals, Aramaic, Assistance, Association, Atlantic, Attorney, Balancing, Baptist, Basilica, Baskets, Best, Buddhist, Bunyan, Calculator, Calvary, Catholic, Catholicism, Charitable, Charities, Chen, Cognition, Communities, Compassion, Connery, Constantinople, Contemporary, Cosmic, Cost, Court, Creativity, Criminal, Crisis, Cure, Curriculum, Dangerous, Database, Date, Death, Deities, Demon, Determining, Dharma, Diocese, Double, Dreams, Echoes, Economic, Elegant, Emanuel, Empires, EOS, Episcopal, Epistle, Ethical, Eucharist, Everlasting, Excel, Existence, Factors, Fallen, Families, Fervor, Focus, Foods, Forums, Freedom, Glad, Glorious, Heart, Heavy, Hell, Help, Him, Honour, Hospital, Hypothesis, Impact, Implications, Influencing, Injunction, Intel, Invitations, Involvement, Jewish, Judas, Judgement, Kenichi, Kiss, Kombat, Lamp, Laughter, Learning, Leviticus, Liberal, Liberation, Lisa, Lives, Lord, Loss, Lust, Maker, Mandir, Marital, Married, Mary, Masjid, Meditation, Melody, Merrell, Metatron, Methodist, Militant, Mind, Mirror, Modernity, Morality, Mother, Motivation, Muhammad, Mutual, Mysteries, Mystical, Nanak, Natural, Network, ODST, Oneness, Outreach, PDF, Piano, Policy, Political, Pope, Practicing, Praise, Preview, Prime Prostitute, Provider, Punishment, Purchase Pure Qi Queries Radiance Rallies Reiki
Reincarnation, Remote, Renewable, Resurrection, Rev., Rites, Safety, sanctuaries, Saturday, Saviour, Scrolls, Sculpture, Secular, Secure, Self, Sermon, Serving, Shadows, Shari'a, Shinto, Significance, Silent, Sonata, Songs, Spangled, Spanish, SPCA, SQL, St., Stevie, Suites, Supply, Sweet, SWF, Talmud, Templar, Terrier, Testament, Testing, Thank, Theology, Thyme, Tie, TransCanada, Truth, Uncharted, Understanding, United, Venue, Videos, VoIP, Volume, Vote, Wetlands, Wiccan, Worship
7. Computers: 512MB, allows, Android, article, attribute, autocomplete, automatically, back-up, barcode, beach, binaries, button,  $\mathrm{C + + }$ , caching, cake, camera, capabilities, casing, chairs, change, cheat, chew, choice, click, coder, compiling, components, computation, computing, confidentiality, configuring, connections, console, copies, counterfeiting, creating, crucial, customer, CyanogenMod, cyber, debian, decimal, decrypt, decryption, deflate, deleting, demo, detect, developing, dialog, dialup, direction, disc, display, DNS, DSL, DVD, e-mail, edit, educational, elements, encyclopedia, Excel, execute, extract, Firefox, fixes, flames, Frequently, functionality, gamepad, garbage, glass, gmail, GPU, guest, hats, house, identifier, infected, initialize, inkjet, input, interactive, interview, ISP, iterative, Jacket, journalists, jQuery, latency, layout, little, logon, lurks, Macs, mails, mainboard, memories, mice, must, notebook, off-line, on-line, operand, original, overflow, packet, pane, paper, parasite, parsing, pasting, PDF, phishing, php, pixels, point-in-time, popup, prev, profit, pull, query, reasoning, rectangular, redirect, rename, restart, restoring run-time, sailor, saving, search, secure shoe, sidebar, signal, sites Solaris spyware step stored storing subdirectory taxi telephoto text tools topic torrent touchpad typeface Ubuntu update usb utilities visuals VPN wifi workstation writer XP XSLT
K.4 Validation Fantasy Bag of Words
beast, Cerberus, demon, dragon, fairy, Frankenstein, ghost, giant, Godzilla, horror, hydra, imp, monster, mummy, ogre, orc, savage, spirit, sprite, titan, troll, undead, unicorn, vampire, witch, zombie
L Additional Machine Translation Formality Examples
We provide some additional examples of FUDGE against baselines on our machine translation formality task in Table 30.
Spanish: ah pero muy ventsajoso que qua a qué quó que haces en el en nada yo temen trabajo en el en en el operapero pero yo soy un sobr es
$\mathcal{G}$  : Ah, but very advantageous, what, what do you do in the, in the opera? I work in the, in the opera, but, but I'm an envelope
$\mathcal{G} + \mathbf{ST}$  : I work in the, in the opera, but I'm an envelope. I am very advantageous. What, what, what do you do in the?
FUDGE: Oh, but it's very advantageous, what, what do you do in the opera? I'm afraid of working in the opera, but I'm an envelope
Reference: I also work in the library but i am a shelver
Spanish: la información que que tenemos todo es propaganda enteroces es portante ver otros versiones de lo que está pasando en el mundo no solamente de la version de las disuasion de bush
$\mathcal{G}$  : the information that, that we have is all propaganda, then is important to see other versions of what's happening in the world, not only the version of Bush's deterrence
$\mathcal{G} + \mathbf{ST}$ : The information that we have is all propaganda. It is important to see other versions of what is happening in the world, not only the version of Bush's deterrence.
FUDGE: The information that we have is all propaganda, so, it's important to see other versions of what's happening in the world, not only the version of Bush's deterrence
Reference: The information we get is all propaganda, it's important to see other versions of what happens in the world
Spanish: y está un poco fácil verdad
$\mathcal{G}$  : And it's a little hard, right?
$\mathcal{G} + \mathbf{ST}$ : It's a little hard, right? Is that a bit of a hard thing to do with it? I'm not sure.
FUDGE: And it's a little difficult, right?
Reference: It's a bit hard, to tell you the truth
Table 30: Additional example translations, comparing  $\mathcal{G},\mathcal{G} + \mathrm{ST}$  , and FUDGE.
M Software
All models are implemented in PyTorch (Paszke et al., 2019), and pretrained models  $\mathcal{G}$  are obtained from HuggingFace (Wolf et al., 2019). Specifically, the Marian translation model is https://huggingface.co/Helsinki-NLP/opus-mt-es-en.