SlowGuess/NAACL_2022_OCR huggingface dataset:
Tom Michael Mitchell, William W. Cohen, Estevam R. Hruschka, Partha P. Talukdar, Bo Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, Jayant Krishnamurthy, N. Lao, Kathryn Mazaitis, Thahir Mohamed, Ndapandula Nakashole, Emmanouil Antonios Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard C. Wang, D. Wijaya, Abhinav Gupta, Xinlei Chen, Abulhair Saporov, Malcolm Greaves, and Joel Welling. 2015. Never-ending learning. Communications of the ACM, 61:103 - 115.
Yannis Papanikolaou and A. Pierleoni. 2020. Dare: Data augmented relation extraction with gpt-2. ArXiv, abs/2004.13845.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15-25, Vancouver, Canada. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of ai through gamification.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and
Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In International Conference on Learning Representations.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1008-1025, Online. Association for Computational Linguistics.
Hongming Zhang, Daniel Khashabi, Y. Song, and D. Roth. 2020a. TransOMCS: From linguistic graphs to commonsense knowledge. In *IJCAI*.
Hongming Zhang, Xin Liu, Haojie Pan, Y. Song, and C. Leung. 2020b. Aser: A large-scale eventuality knowledge graph. Proceedings of The Web Conference 2020.
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7579-7589, Online. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097-1100.
We conduct human evaluations on Amazon Mechanical Turk using the template of Figures 4,5. Workers are presented with ATOMIC-style triples, replacing relations with natural language templates (e.g. HinderedBy becomes "can be hindered by"). 3 annotators rate each triple, with options for acceptability: "always/often", "sometimes/likely", "farfetched/never", "invalid", or "too unfamiliar to judge". The first two are considered "accepted", the second two "rejected" and the final is "no judgement". For reporting acceptance rates, and training a critic model, we only distinguish between "accepted" and not "accepted".
Workers are compensated  $0.17 per task (i.e. completing all questions in the evaluation template Figures 4,5). We estimate an upper bound of 30s to complete a single task, which gives an hourly rate of$ 20.4. Workers are selected based on an Amazon Mechanical Turk qualification, specifically filtering for workers with high accuracy on past knowledge base triple evaluations. We follow the same setup for all evaluations, besides number of annotators. This setup is shown to result in consistent and reliable annotations, with an inter-annotator agreement given by Fleiss' kappa (Fleiss, 1971) of 40.8 when evaluating with 3 annotators, in §3.4.
B Using Alternate Models as Knowledge Sources
One natural question that arises from the strong performance of symbolic knowledge distillation is whether other sources of knowledge (i.e. language models) would similarly benefit from this method. In this section, we particularly measure the capacity of other language models to serve as the "loose teacher" which generated the base knowledge of the resulting corpus.
We expand our study beyond GPT-3 here (the model used in our work), to include 2 contemporary large language models, GPT-J (Wang and Komatsuzaki, 2021) and T5-11B (Lester et al., 2021) finetuned for language modelling. For knowledge generation (verbalization) we follow the same procedure as §3 along with simple adjustments to improve quality. We are investigating the effect of the critic on knowledge precision here, so we also include  $\mathrm{ATOMIC}_{20}^{20}$  to probe the usefulness of automatic filtering for human-authored knowledge.
For each knowledge source, we follow the human evaluation setup in §3.4 to obtain quality an-
notations of 2000 examples, with 1 annotation per example. This follows a similar setup to §4—indeed, we are replicating the earlier critic experiments but at a smaller scale (2000 annotations vs. 10000) to allow for more knowledge sources. For each knowledge source, we randomly split into sizes of 1400/300/300 for train, dev, and test sets. We follow §4 to train a critic model for each knowledge source.
We plot different thresholds (\% of corpus filtered) against the resulting precision (proportion of corpus that is judged to be "valid" knowledge) in Figure 3, and give numbers at various sizes in Table 7. One striking aspect is that a critic model can raise the precision of any of these knowledge sources to approximately  $90\%$  while retaining  $30\%$  of the original corpus size. While this discards a significant portion of the original generated knowledge, it raises the exciting prospect of using more cost-effective models at a large scale to generate strong commonsense corpora like  $\mathrm{ATOMIC^{10x}}$ . GPT-J and T5-11B can both be run locally by researchers, unlike GPT-3 which uses a pay-per-generation API. Thus, one can imagine producing a large and high-quality corpus like  $\mathrm{ATOMIC^{10x}}$  at a lower cost by instead generating a larger volume of knowledge from such an accessible model, and simply filtering to a greater extent.
Another interesting aspect is how the various knowledge sources diverge. Under little to no critical filtering (i.e. corpus size = 1.0), the precision of various knowledge sources is widely spread. Before applying a critic, quality of knowledge source is very important. Indeed, precision is ordered by cost of generation: human  $\mathrm{ATOMIC}_{20}^{20}$  has the highest precision while being the most expensive, followed by GPT-3 (used here) which is pay-per-generation, and finally the two publicly available models. Another point of divergence is for extreme filtering (at approximately  $20\%$  of the original corpus size. All knowledge sources but GPT-3 plateau at approximately  $90\%$  accuracy, while GPT-3 rises towards  $100\%$ . Indeed, this supports our use of GPT-3 in this work, as a high-quality automatic knowledge source.
C Critic Model
We train binary classifiers (critics) for human acceptability using RoBERTa-Large (Liu et al., 2019), fine-tuning all parameters, along with a 2-layer MLP on the [CLF] representation. We conduct
Knowledge Source	Precision at Corpus Size (%)
100	90	80	70	60	50	40	30	20	10
\( \text{ATOMIC}_{20}^{20} \)	84.0	86.3	87.9	89.0	88.3	88.7	91.7	90.0	90.0	90.0
GPT-J	71.7	76.7	81.7	83.8	86.7	88.0	88.3	87.8	93.3	90.0
T5-11B	64.7	66.7	70.8	74.8	79.4	84.7	89.2	92.2	91.7	93.3
GPT-3 curie	79.3	81.5	85.0	86.2	88.3	90.7	91.7	90.0	98.3	100.0
Table 7: Knowledge precision at various corpus sizes (from  $100\%$  to  $10\%$ ) based on filtering by the critic model. Precision is calculated by human annotation of valid or invalid knowledge. We consider 4 knowledge sources, as described in Appendix B. This corresponds to the data plotted in Figure 3.
Figure 3: Precision resulting from the critic step from §4, with various thresholds. We include corpora generated by GPT-3 (ATOMIC $^{10x}$ ), GPT-J, T5-11B, and humans (ATOMIC $^{20}$ ). Without filtering (corpus size = 1.0), different corpora have a variety of precisions. As more examples are filtered by the critic, precision rises significantly demonstrating the strong value of the critic step.
a small grid search on the validation set finding batch size 128, dropout .1, and Adam (Kingma and Ba, 2015) learning rate 5e-6 to be effective. We use early stopping and decay learning rate on validation performance plateauing, to maximize  $R@80\%$  on the validation set. We find RoBERTa pretrained on MNLI (Williams et al., 2018) effective, outperforming other options. As well, we substitute randomly-sampled names in for person designations "X"/"Y". We include as a baseline an unsupervised filtration metric inspired by (Davison et al., 2019): they propose a model estimate of PMI to score mined commonsense triples. In our case, we use Negative Log-Likelihood (NLL) and token-mean-NLL from GPT-3 itself.
The validation precision/recall of our best performing model, the baselines, and the in-optimal hyperparameter configurations are given in Figure 6. Once fixing our model, we applied it to the test set (also in Fig 6), verifying that it generalizes to ATOMIC $^{10x}$  entries. Overall, our trained critic model is more effective than the baselines in identifying high and low quality teacher generations at all levels of precision and recall. This result demonstrates that a small amount of human supervision can consistently help to correct GPT-3's mistakes.
D ATOMIC $^{10x}$  Generation Prompts
We include example prompts for all generations we do, from Table 8 to 15. Note that elements of generation prompts are randomized for each batch. For event generation, the few-shot examples and order are randomly sampled from a seed set of 100 high-quality examples from  $\mathrm{ATOMIC}_{20}^{20}$  in each batch. For inference generation, the natural names used for PersonX and PersonY are randomly sampled from a small predefined set of names.
(WARNING: This HIT may contain adult content. Worker discretion is advised.)
If the data is good, it's good. If bad, then bad. Please annotate as you see not worrying about how many of each label you find yourself assigning! If you understand the words but the Phrases or the complete assertion makes poor sense, please mark as INVALID. Thank you!
You will evaluate how often assertions are true. Each assertion is comprised of 3 parts: Phrase A, Relation, Phrase B
Phrase A, Phrase B
Short phrases. May describe objects, object properties, events, actions, etc.
How  $A$  relates to  $B$ .
For each assertion, determine how true it is:
always/often
sometimes/likely
farfetched/never
too unfamiliar to judge
Always or quite often true.
Sometimes is true or true for some people. -or- Likely true.
False or farfetched, at best. -or- Unlikely to be true.
This assertion makes no sense (i.e., "what does this even mean?!).
Cannot make a fair evaluation. Unfamiliar with one or both of the phrase.
If you see "nothing in particular" for Phrase B, assess Phrase B in context:
- Sometimes certain actions can simply be responded to by doing nothing!
- Other times, doing nothing in particular is simply a weird or unlikely reaction to something.
See examples under tricky relations tagged with (nothing in particular example)
Please report any prejudiced or inappropriate language:
Profane or offensive content (NSFW, R-rated material etc)
- Prejudiced assumptions or derogatory language that villainizes people.
HOWEVER, please note, not all negative content is derogatory especially if Phrase B is intrinsically what Phrase
A means. For example:
criminals are characterized by committing crime is OK.
$\rightarrow$  This isn't necessarily villianizing people since "criminal" means "a person who has committed a crime".
homeless are characterized by being lazy is prejudiced.
There are many reason a person is rendered homeless. This is a gratuitous prejudice about homelessness.
- Material that people may find disturbing, off-putting, or improper
A couple NOTES:
- Please be forgiving of spelling or grammatical errors
- If the terms are too obscure or you don't know the truth of the fact at the top of your head, it is okay to mark it "too unfamiliar to judge". If you can answer (e.g., based on likelihood), please provide a response.
Tricky Relations (click to expand/collapse)
Figure 4: Page 1 of template used for human evaluation.
1) PersonX approaches PersonY's aunt, as a result, PersonX feels, awkward How often does the assertion hold true?
always/often sometimes/likely faretched/never invalid too unfamiliar to judge
This fact is true but outdated
I would count this as an inappropriate, prejudiced or offensive material
2) PersonX asked PersonY out on a date, can be hindered by, PersonX is still dating Sarah. How often does the assertion hold true?
3) PersonX fails to go home, as a result, PersonX, is grounded How often does the assertion hold true?
4) PersonX makes her own clothes, as a result, PersonX feels, artistic. How often does the assertion hold true?
always/often sometimes/likely farfetched/never invalid too unfamiliar to judge
5) PersonX notices PersonY's response, can be hindered by, PersonX is distracted by the music. How often does the assertion hold true?
You must ACCEPT the HIT before you can submit the results.
Figure 5: Page 2 of template used for human evaluation.
1. Event: PersonX unwraps PersonY's hands
2. Event: PersonX overcomes evil with good
3. Event: PersonX is fed up with the present situation
4. Event: PersonX breaks PersonX's back
5. Event: PersonX calls no one
6. Event: PersonX never gets angry
7. Event: PersonX does not learn from PersonY
8. Event: PersonX refuses to touch PersonY's hands
9. Event: PersonX looks at flowers
10. Event: PersonX unloads an atomic bomb
11. Event:
Table 8: Prompt for head generation.
Figure 6: Precision vs. recall of our critic model on the human labelled validation set. The best trained models are labelled, and other hyper-parameter settings are shown as faded lines. We also include generation negative log-likelihood (nll) and token-wise mean nll as cutoff measures—these perform much worse than the supervised model.
Next, how are people seen in each situation? Examples:
Situation 1: Devin bullies Jean.
Devin is seen as dominant.
Situation 2: Jamie moves to another city.
Jamie is seen as adventurous.
Situation 3: Sydney changes Ryan's mind.
Sydney is seen as influential.
Situation 4: Lindsay writes a story.
Lindsay is seen as creative.
Situation 5: Rowan covers Pat's expenses.
Rowan is seen as wealthy.
Situation 6: Lee takes time off.
Lee is seen as carefree.
Situation 7: Riley advises Noel.
Riley is seen as informed.
Situation 8: Adrian bursts into tears.
Adrian is seen as depressed.
Situation 9: Hunter deals with problems.
Hunter is seen as responsible.
Situation 10: Sam follows Charlie.
Sam is seen as suspicious.
Situation 11: Alex makes Chris wait.
Alex is seen as
Table 9: Prompt for generating xAttr.
Next, what do situations make people do? Examples:
Situation 1: Devin gets a divorce.
As a result, Devin dates someone new.
Situation 2: Jamie lifts weights.
As a result, Jamie has sore muscles.
Situation 3: Sydney takes Ryan to a bar.
As a result, Sydney gets drunk.
Situation 4: Lindsay decides to hire a tutor.
As a result, Lindsay gets better grades.
Situation 5: Rowan buys Pat drinks.
As a result, Rowan is thanked by Pat.
Situation 6: Lee hears bad news.
As a result, Lee begins to cry.
Situation 7: Riley buys a chocolate bar.
As a result, Riley gets change.
Situation 8: Adrian does a lot of work.
As a result, Adrian gets mental fatigue.
Situation 9: Hunter attends a concert.
As a result, Hunter hears a new song.
Situation 10: Sam gets the job done.
As a result, Sam gets more responsibilities.
Situation 11: Alex makes Chris wait.
As a result, Alex
Table 10: Prompt for generating xEffect.
For each situation, describe the intent. Examples:
Situation 1: Devin gets the newspaper.
Devin intends to read the newspaper.
Situation 2: Jamie works all night.
Jamie intends to meet a deadline.
Situation 3: Sydney destroys Ryan.
Sydney intends to punish Ryan.
Situation 4: Lindsay clears her mind.
Lindsay intends to be ready for a new task.
Situation 5: Rowan wants to start a business.
Rowan intends to be self sufficient.
Situation 6: Lee ensures Ali's safety.
Lee intends to be helpful.
Situation 7: Riley buys lottery tickets.
Riley intends to become rich.
Situation 8: Alex makes Chris wait.
Alex intends
Table 11: Prompt for generating xIntent.
Next, we will discuss what people need for certain situations. Examples:
1. Before Devin makes many new friends, Devin has to spend time with people.
2. Before Jamie gets a date, Jamie has to ask someone out.
3. Before Sydney changes Ryan's mind, Sydney has to think of an argument.
4. Before Lindsay gets a job offer, Lindsay has to apply.
5. Before Rowan takes a quick nap, Rowan has to lie down.
6. Before Lee tries to kiss Ali, Lee has to approach Ali.
7. Before Riley rides Noel's skateboard, Riley has to borrow it.
8. Before Adrian eats the food, Adrian has to prepare a meal.
9. Before Hunter watches Netflix, Hunter has to turn on the TV.
10. Before Sam has a baby shower, Sam has to invite some friends.
11. Before Alex makes Chris wait, Alex has
Table 12: Prompt for generating xNeed.
Next, how do people feel in each situation? Examples:
Situation 1: Devin lives with Jean's family.
Devin feels loved.
Situation 2: Jamie expects to win.
Jamie feels excited.
Situation 3: Sydney comes home late.
Sydney feels tired.
Situation 4: Lindsay sees dolphins.
Lindsay feels joyful.
Situation 5: Rowan causes Pat anxiety.
Rowan feels guilty.
Situation 6: Lee goes broke.
Lee feels embarrassed.
Situation 7: Riley has a drink.
Riley feels refreshed.
Situation 8: Adrian has a heart condition.
Adrian feels scared about their health.
Situation 9: Hunter shaves Avery's hair.
Hunter feels helpful.
Situation 10: Sam loses all of Charlie's money.
Sam feels horrible.
Alex feels
Table 13: Prompt for generating xReact.
Next, what do people want in each situation? Examples:
Situation 1: Devin mows the lawn.
Devin wants to take a shower.
Situation 2: Jamie is going to a party.
Jamie wants to take an Uber home.
Situation 3: Sydney bleeds a lot.
Sydney wants to go to the ER.
Situation 4: Lindsay works as a cashier.
Lindsay wants to find a better job.
Situation 5: Rowan gets dirty.
Rowan wants to do a load of laundry.
Situation 6: Lee stays up all night studying.
Lee wants to rest.
Situation 7: Riley gets Noel's autograph.
Riley wants to tell some friends.
Situation 8: Adrian sees Taylor's point.
Adrian wants to agree with Taylor.
Situation 9: Hunter leaves Avery's bike.
Hunter wants to keep the bike safe.
Situation 10: Sam wants a tattoo.
Sam wants to find a tattoo design.
Situation 11: Alex makes Chris wait.
Alex wants
Table 14: Prompt for generating xWant.
Next, what can hinder each situation? Examples:
Situation 1: Devin makes a doctor's appointment,
This is hindered if Devin can't find the phone to call the doctor.
Situation 2: Jamie rubs Wyatt's forehead,
This is hindered if Jamie is afraid to touch Wyatt.
Situation 3: Sydney eats peanut butter,
This is hindered if Sydney is allergic to peanuts.
Situation 4: Lindsay looks perfect,
This is hindered if Lindsay can't find any makeup.
Situation 5: Rowan goes on a run,
This is hindered if Rowan injures her knees.
Situation 6: Lee takes Ali to the emergency room,
This is hindered if Ali has no health insurance to pay for medical care.
Situation 7: Riley spends time with Noel's family,
This is hindered if Noel's family doesn't like spending time with Riley.
Situation 8: Adrian moves from place to place,
This is hindered if Adrian can't afford to move.
Situation 9: Hunter protests the government,
This is hindered if Hunter is arrested.
Situation 10: Sam has a huge fight,
This is hindered if Sam does not like confrontation.
Situation 11: Alex makes Chris wait,
This is hindered if
Table 15: Prompt for generating HinderedBy.