SlowGuess/NAACL_2021_OCR huggingface dataset:
Linguistics, pages 1790-1801, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32-42, Minneapolis, Minnesota. Association for Computational Linguistics.
Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. arXiv preprint arXiv:1901.05287.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725-1744, Online. Association for Computational Linguistics.
Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105-1117, Minneapolis, Minnesota. Association for Computational Linguistics.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Melbourne, Australia. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In CogSci 2018, pages 690-695.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Marten van Schijndel and Tal Linzen. 2020. Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty. PsyArXiv.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations.
Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731-4736, Brussels, Belgium. Association for Computational Linguistics.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, page 1707-1713.
Thomas Wolf. 2019. Some additional experiments extending the tech report "assessing BERTs syntactic abilities" by Yoav Goldberg. Technical report, Technical report.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Charles Yu, Ryan Sie, Nicolas Tedeschi, and Leon Bergen. 2020. Word frequency does not predict grammatical knowledge in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4040-4054, Online. Association for Computational Linguistics.
A Additional Plots
Scores at Cut-offs (Large Models)
BERT Large uncased (EW)
BERT Large uncased (MW)
BERT Large cased (EW)
BERT Large cased (MW)
RoBERTa Large (EW)
RoBERTa Large (MW)
GPT2 XL (EW)
Figure 2: The plots above show the EW and MW scores as a function of Top-  $p$  and Bottom-  $p$  percentile cutoffs for BERT-base-uncased, BERT-large-cased, RoBERTa-large and GPT2-XL. In general, as the percentile increases, so does the score, though RoBERTa and BERT's EW scores are quite stable.
Simple
VP coordination
Across prepositional phrase
Across subject relative clause
Across object relative clause
Across object relative (no that)
In object relative clause
In object relative (no that)
BLiMP
Top
Bottom
GPT2 XL (MW)
In a sentential complement
Total Model Probability Accounted For
Figure 3: Above are plots of the proportion of the models' probability mass take up by the inflections of the verb lemmas we used. The y-axis is the total probability mass the lemmas account for and the x-axis is the percentile cutoff. Note that even when considering all of the lemmas, (at  $p = 100\%$ ) there is probability mass not covered by our inflections. This probability mass is often put on other inflections of verbs (e.g. past-tense verbs) or other vocabulary items.
Percentage of Templates with no Lemmas at Cut-Off
Figure 4: Above is the proportion of the templates in the datasets where models assign no probability mass to lemmas in the top or bottom  $p\%$  of the their distributions. The y-axis is the proportion of lemmas that are rejected (i.e. values closer to one mean that the scores are calculated based on fewer templates). The x-axis is again the percentile cutoff. Note that the bottom-most cutoffs often has a large proportion of invalid lemmas, so these scores are based on fewer lemmas.
B A Subset of Verb Lemmas
The 1971 lemmas from COCA and The Penn Treebank we used are available below (Davies, 2008; Marcus et al., 1993):
abandon, abate, abide, abolish, absorb, abuse, accede, accelerate, accept, access, acclaim, accommodate, accomodate, accompany, accomplish, accord, account, accrue, accuse, achieve, acknowledge, acquire, acquit, act, adapt, add, address, adhere, adjust, administer, admit, adopt, adorn, advance, advantage, advise, affect, afford, aggravate, agonize, agree, aid, aim, air, alert, alienate, allay, allege, alleviate, allocate, allow, allowed, ally, alter, amalgamate, amass, amaze, ample, amend, amortize, amount, amplify, analyze, anchor, announce, answer, antagonize, anticipate, apologize, appeal, appear, appease, append, applaud, apply, appoint, appraise, appreciate, approach, approve, are, argue, arise, arm, arouse, arrange, arrest, arrive, articulate, ask, assassinate, assemble, assert, assess, assign, assimilate, assist, associate, assuage assume, assure, atone, attach, attack, attain, attempt, attend, attention, attest, attract attribute auction, audit, audition, augment, authorize autograph average avert avoid await back backfire bail balance,balk balloon ban band banish bank bankroll bankrupt bar bargain base bash batter battle bear beat become bedevil beef begin behave belie believe bellow belong bend benefit bet bid bill bite blackmail blame blast bleed blend bless blink block blow blunder blunt blur board boast bode bog bolster bomb boom boost born borrow bother bottle bounce bow brace brag branch brave brazen breach break breathe breed brew bribe bridge brief bring broadcast broaden browbeat brush buck buckle budge buffer buffet build bumble bump buoy burn bury butt buttress buy buy-back buzz bypass calculate call calm cancel cap capitalize capture care careen caricature carry cash cast catapult catch cater cause caution cease cede celebrate cement centralize certify challenge change channel characterize charge chart chase chat chauffer cheat check cheer chew chill chisel choke choose chop churn cinch circulate circumvent cite claim clamp clarify flash classify clean cleanse clear click climb cling clip clobber close cloud clutter coach coax code co-exist cohere co-host coincide collaborate collapse collect combat combine come command commemorate commend comment
commercialize, commit, communicate, compare, compel, compensate, compete, compile, complain, complement, complete, complicate, comply, compound, comprise, compromise, compute, computerize, con, conceal, concede, conceive, concentrate, concern, conclude, condemn, condone, conduct, confer, confirm, confiscate, conform, confront, confuse, congratulate, connect, connote, conquer, consent, conserve, consider, consist, console, consolidate, constitute, constrain, construct, construe, consult, consume, contact, contain, contemplate, contemporize, contend, content, continue, contract, contrast, contribute, control, convene, convert, convey, convict, convince, cook, cool, cooperate, coordinate, cope, co-produce, copy, corner, corral, correct, correspond, co-sponsor, cost, cough, count, countenance, counter, counteract, counterprogram, court, cover, crack, craft, crank, crash, crawl, creak, create, credit, crest, criminalize, crimp, criticize, crop, cross, crumble, crunch, crush, cry, cuff, curb, cure, curl, curry, curtail, cushion, cut, dabble, damage, damp, dampen, dance, dare, dash, date, deal, debate, debunk, debut, deceive, declare, decline, decrease, deduct, default, defeat, defect, defend, defer, define, deflect, defraud, defuse, degenerate, delay, delegate, deliberate, delight, deliver, demand, demilitarize, demobilize, democratize, demolish, demonstrate, denude, deny, depart, depend, depict, deposit, depress, deprive, derail, deregulate, describe, desert, deserve, design. designate. desist. destabilize. destroy. detect. deter. deteriorate. determine. detract. develop. devise. devote. dial. dictate. die. differ. differentiate. dig. digest. dignify. dilute. diminish. dip. direct. disagree. disappear. disappoint. disarm. disassemble. disassociate. disband. discard. discern. disclose. discomfit. disconnect. discontinue. discount. discourage. discover. discredit. discuss. disdain. disengage. disguise. dish. dislike. dismantle. dismiss. disobey. disparage. dispatch. dispel. dispense. displace. display. dispose. disprove. dispute. disqualify. disregard. disrupt. dissipate. dissociate. dissolve. dissuade. distance. distinguish. distort. distract. distribute. disturb. diverge. diversify. divert. divest. divide. do. doctor document.dole dominate don donate double doubt downsize draft drag drain drape draw dream dress drift drill drink drive drop drown drum dry dump duplicate dwarf earmark earn ease eat eavesdrop ebb echo eclipse economize edge edit educate effect elaborate elect eliminate elon
gate, emasculate, embark, embarrass, embellish, embrace, emerge, emote, empathize, emphasize, emphaticize, employ, empty, emulate, enable, enact, encapsulate, encompass, encounter, encourage, end, endanger, endeavor, endorse, endure, enforce, engage, engineer, enhance, enjoin, enjoy, enlarge, enlist, enroll, ensue, ensure, entail, enter, entice, entitle, entrench, entrust, envision, equal, equate, equip, eradicate, erase, erect, erode, erupt, escalate, escape, establish, estimate, evade, evaluate, evaporate, even, evolve, exacerbate, exaggerate, examine, exceed, except, exchange, exclude, excorciate, excuse, execute, exempt, exercise, exhaust, exhibit, exist, exit, exorcise, expand, expect, expedite, expel, experience, expire, explain, explode, exploit, explore, export, expose, express, expunge, extend, extinguish, extort, extract, extricate, fabricate, face, facilitate, factor, fade, fail, fall, falsify, familiarize, fantasize, fare, farm, fashion, favor, fear, feature, feed, feel, fend, ferret, ferry, fester, fetch, field, fight, figure, file, fill, finance, find fine fine-tune, finger, finish, fire, firm, fit, fix, flash. flatten flaunt flay,flee,finch,fip float,flood flounder flourish flow fluctuate flush fly focus fog foil fold follow fool foot force forecast. foreseeforfeit forging forgive forgo form. formulate,foster frame,franchise free freeze freight,fret,frighten,frolic,frustrate,fuck,fudge fuel fulfill function fund funnel furnish further gain galvanize gamble garden garner gasp gather,gauge gender generalize generate get give,glamorize glaze glean glide gloat gloss glut go gon gore govern grab grace grant grapple grasp grimace ground group grow growth guarantee guard guess guide gut guzzle hack halt halve hammer hamper hand handicap handle hang happen harass harm hasten hate haul haunt have head heal hear heat hedge heed heighten help herald hesitate hide highlight hinder hinge hint hire hit hoe hold holler,homer hone honor hope host house hum hurry hurt identify idle ignite ignore illuminate illustrate imagine impact impair impede implement implicate imply import impose impound impress imprison improve impugn incarcerate inch include incorporate increase increased incur indemnify indicate indirect induce indulge industrialize infiltrate inflame inflate inflict influence inform,infringe,infuriate,influence ingest,ingratiate,inhibit,initiate,inject injureinnovate insist inspect inspire install instill institute insulate
insure, integrate, intend, intensify, interconnect, interest, interfere, interpret, intervene, interview, intimate, introduce, invade, invent, invest, investigate, invite, invoke, involve, irk, iron, isolate, issue, itemize, jack, jeopardize, join, joke, jolt, judge, juggle, jump, junk, justify, keen, keep, key, kick, kidnap, kill, kiss, knock, know, kowtow, label, lack, lag, land, languish, lash, last, laugh, launch, launder, lay, lead, lean, leap, leapfrog, learn, lease, leave, lecture, legislate, legitimize, lend, lengthen, lessen, let, level, levy, liberalize, license, lie, lift, light, lighten, like, limit, line, linger, link, liquefy, liquidate, list, listen, live, load, loan, lobby, locate, lock, lodge, log, look, loom, loose, loosen, loot, lose, love, lower, lunch, lure, mail, maintain, make, man, manage, mandate maneuver, manipulate, manufacture, march, mark, market, marry, marvel, massage master, match, materialize, matter, mature, maul maximize mean measure mediate meet meld melt mention merge merit mesh migrate militate milk mimic mince mind minimize mirror misinterpret misrepresent miss mistreat mitigate mix moan mobilize model moderate modernize modify mollify monitor monolocate mop mortgage motivate mount move mow mull multiply muse muster mute nail name narrow navigate naysay need negotiate net network nod nominate normalize notch note notice nudge nullify nurture obfuscate object obscure observe obtain obviate occasion occupy occur offend offer offset omit ooze open operate oppose opt order organize originate oust outfit outflank outfly outlast outline outpace outperform outsell outshine out-smart outstrip outtrade outweigh overbid overcome overemphasize overhaul overlook overpower overpurchase overreact override overrule oversee overstate overthrow overturn overwhelming owe own pace pack package paint pair pale palm pan panic parachute parallel parcel park parry part partake participate pass patronize pave pay peak pedal peddle peer penalize penetrate perform permit perpetuate persist persud peruse petition phase pick piece pile pin pinch ping pinpoint pit pitch placate place plague plan plant play plea plead please plot plow pluck plug plummet plunge plur pocket point police polish poll pollinate pollute ponder pop popularize populate portend portray pose position possess post postpone pot pounce pound pour practice pray precede preclude predict predispose
pre-empt, prefer, premiere, prepare, prepay, prereregister, presage, prescribe, present, preserve, press, pressure, pretend, pre-try, prevail, prevent, price, privatize, probe, proceed, process, proclaim, prod, produce, profile, profit, program, progress, prohibit, project, prolong, promise, promote, prompt, prop, propel, propose, prosper, protect, protest, prove, provide, provoke, prune, publicize, publish, pull, pummel, pump, punch, punish, purchase, pursue, push, put, puzzle, qualify, quantify, quarrel, quell, question, quiet, quit, quiz, quote, rage, raid, rain, raise, rally, ramp, range, rank, rat, ratify, rationalize, rattle, rave, reach, react, read, readmit, reaffirm, realign, realize, reap, reappraise, rearm, rearrange, reason, reassert, reassess, reassume, reassure, reauthorize, rebound, rebuild, rebut, recall, recapture, receive, reckon, reclaim, recognize, recommend, reconcile, reconnect, reconsider, reconstruct, record, recoup, recover, recraft, recreate, recycle, redden, redeem, redefine, redesign, redevelop, radial, rediscover, redo, redound, redraw, reduce, re-emerge, re-enter, reestablish, re-establishment. re-evaluate, re-examine, refer, refinance, refine. reflect. refocus. refocuses. reform. refrain. refund. refurbish. refuse. refute. regain. regard. regenerate. register. regret. regroup. regulate. rehabilitate. reignite. reimburse. reimpose. rein. reinforce. reinstate. reinvent. reinvest. reinvigorate. reject. rejoin. rejuvenate. rekindle. relate. relaunch. relax. release. relieve. relinquish. relish. relocate. rely. remain. remake. remark. remedy. remember. remind. remove. reneges. renegotiate. renew. renounce. renovate. rent. reopen. reorganize. repair. repatriate. repay. repeal. repeat. repel. replace. replaster. replenish. replicate. reply. repond. report. reposition. reposess. represent. reproduce. repurchase. request. require. reschedule. rescind. rescue. research. resell. resemble. resent. reserve. reset. reshape. reshuffle. reside. resign. resist. resolve. resort. respect. respond. rest. restart. restate. restore. restrain. restrict. restructure. result. resume. resurrect. retail retain. rethink; retire; retreat; retrieve; retrofit; retry return; reunite; revamp; reveal; reverberate; reverse review; revisit; revisit; revive; revoke; revolutionize; revolve; reward; rewrite; rid; ride; ring; rise; risk; rival; rock; rolil; roll; roost; root; rotate; round; row; rub; rubber-stamp; ruin; rule; run; rush; sabotage; sacrifice; safeguard; safety; salvage; sap; satisfy; saturate; save; savor; say; scale; scan; scape; scare; schedule; school; scold; score; scorn; scour; scout; scrap; scream; screen; scrutinize; scurry; scuttle
seal, search, secure, seduce, see, seek, seem, segregate, seize, select, self-reinsure, sell, send, sense, sensitize, separate, sequester, serve, service, set, settle, sever, shadow, shake, shall, shape, share, sharpen, shave, shed, shell, shield, shift, shine, ship, shirk, shock, shoe-horn, shoot, shop, shore, horn short, shorten, shoulder, shout, shove, show, shower, shrink, shun, shut, shy side, sidetrack, sift sign, signal, signify, simplify, sing, sink, sit, ski skid, skim, skip, skipper, slack, slam-dunk, slash sleep, slide, slip, slog, slow, slump, smash, smell smile, smoke, smooth smother, smuggle, snatch sniff, soak, soar, sob, socialize socke soften solicit solidify solve soothe sort sound sour sow spare spark spawn speak specialize specify speculate speed spell spend spill spin split sponsor spot spotlight spread spring sprout spruce spur spurn spy square squeeze stabilize stack staff stage stain stall stampedep stanch stand standardize star stare start starve stash state staunchstave stay steal steer stem step sterilize stick stifle stimulate stipulate stir stock stockpile stomach stop store strafe straighten strain stray streamline streetspeak strengthen stress stretch strike strip stroll structure struggle study stumble subcontract subject sublet submit subordinate subpoena subscribe subsidize substantiate substitute subtract subvert succeed suck sue suffer suffice suggest suit sum summarize summon supersede supervise supplement supply support suppose suppress surface surge surpass surprise surrender surround survey survive suspect suspend sustain swallow swamp swap sway swear sweat sweeten swell swing switch synthesize tackle tag take talk tamper tandy tap target tarnish taste tax teach team tear tell tend tender term terminate test test-drive testify thank the think thrash threaten thrive throw thwart tick tie tighten tilt time tiptoe tolerate top topple torment torpedo torture toss total totter touch tough toughen tour tout tower trace track trade traduce trail train transact transfer transform translate transmit transplant transport trash travel tread treat trend trick trickle trigger trim triple trivialize trust try tumble turn twitch unblock uncover undercut undergo underlie underline undermine underperform underpin underscore understand undertake underwrite undo undulate unfold unite unleash unload unmask unplug unravel unveil unwind update upgrade uphold upset urge use
usurp, utilize, vacate, vacillate, vacuum, value, vanish, vary, vault, veer, vent, venture, verify, veto, view, violate, visit, visualize, vitiate, voice, void, volunteer, vote, wad, wade, wage, wail, wait, waive, wake, walk, wall, wan, wander, wane, want, ward, warm, warn, warrant, wash, waste, watch, water, weaken, wear, weather, wedge, weigh, weight, welcome, were, whack, whip, widen, will, wimp, win, wind, wipe, wire, wish, withdraw, withhold, withstand, witness, wonder, woo, work, worry, worsen, wound, wrap, wreck, wreck, wrest, wrestle, wring, write, yank, yield, zero, zip, zoom
The rest of the lemmas from Essay (2015) are available here: https://patternbasedwriting.com/1/Giant-Verb-List-3250-Verbs.pdf.