SlowGuess/NAACL_2022_OCR huggingface dataset:
As described in Section 3.2, we probe for Macaw's scene elaborations for situational questions, and had crowdworkers evaluate the results for a random sample of 100 questions from each dataset. The results are in the "Macaw" lines in Table 4. As shown, the scene elaborations are of mediocre quality, with an average of  $\approx 48\%$  accurate and  $\approx 49\%$  consistent statements within them. Further, they are largely rated as not useful for the QA end task (avg. usefulness  $\approx 27\%$ ). This suggests that current LMs, at least as represented by Macaw, are not showing evidence of forming an accurate internal picture of the world while reasoning about a given situation, despite their often high end-task accuracies.
6.2 Q2: Does DREAM generate improved Scene Elaborations?
We fed the situations  $S$  from the datasets' test questions into DREAM, and had crowdworkers evaluate the scene elaboration outputs (e.g., Figure 1 and Table 1). The results are shown in Table 4, where we see that the scene elaborations produced by DREAM are rated as significantly more accurate ( $\Delta = 16 - 26\%$ ) and more useful ( $\Delta = 12 - 16\%$ ) for three situational QA tasks when compared to Macaw's. Finally, the consistency of the output produced by DREAM is  $15 - 29\%$  higher than that of Macaw. Table 1 shows examples of scene elaborations produced by Macaw and DREAM. Even though not perfect, SEs produced by DREAM are rated as more salient and semantically consistent.
For this experiment, we excluded AITA part of the dataset consisting of questions with long context (taken from Reddit).
Answer Accuracy
ETHICS-CS test/hard	CODAH all	Social IQA test
Macaw	68.08/63.95	83.00	64.84
Macaw + finetuning	63.63/62.31	76.80	62.99
Macaw + DREAM	70.91/66.04	83.72	69.06
Table 5: QA performance improves consistently across tasks when we provide scene elaborations generated by DREAM as additional input context to Macaw ("Macaw + DREAM"). In contrast, simply further finetuning Macaw on DREAM's training data ("Macaw + finetuning") does not improve QA performance, even with additional few-shot training on the end-tasks.
6.3 Q3: Can the scene elaborations produced by DREAM help improve QA?
In Section 6.2 we observed that the scene elaborations produced by DREAM are  $\approx 72\%$  consistent,  $\approx 69\%$  accurate. But more importantly according to humans, on average  $\approx 40\%$  of the sentences in these scene elaborations were deemed useful for justifying the correct answer to the situational question. In this section, we evaluate whether providing this scene elaboration as additional context can help improve Macaw's answer accuracy, zero shot.
Macaw, as described in Section 5.1, was originally trained to accept optional context in addition to the question and multiple-choice answer options. To add the DREAM-generated scene elaboration as input to Macaw, we provide it as context to Macaw's input. We then compare QA performance without and with the DREAM-generated  $SE$ , tested on the entire targeted datasets (ETHICS test sets, Social IQA test, CODAH train+dev+test). The results are shown in Table 5.7 We find that using DREAM's generated scene elaboration acts as relevant and useful context (additional layer of reasoning before QA) resulting in significant gains for Macaw zero-shot (row 3, Table 5).
Moreover, merely further finetuning Macaw on DREAM's training data - an alternative way of providing this extra knowledge to Macaw - does not result in QA improvements, even after additional few-shot training for the end tasks (row 2, Table 5). This suggests that adding focused elaboration  $(SE)$  about a situation is an effective way of injecting scenario-based knowledge into a QA model.
6.4 Ablation of  $SE$  Components
Next, we measure the influence of each scene elaboration component on the "Macaw with scene elaboration" QA scores, using the Social IQA dataset. Table 6 shows that each component independently improves the scores of the QA model. Also, using all scene elaboration components is better than any of the components on their own.
Input context	Answer Accuracy
Social IQA test
(none; base model only)	64.84
{ROT}	67.54
{E}	67.90
{M}	67.40
{Con}	67.49
{ROT, E, M, Con}	69.06
Table 6: QA performance of Macaw zero-shot model with different scene elaboration components as input.
6.5 DREAM SE improves QA performance across different models
Can DREAM's scene elaborations help other QA models besides Macaw-11B? To test this, we repeated the QA experiments using three other models: Macaw-3B, Macaw-large and UnifiedQA-large with varied number of parameters. Table 7 shows that DREAM's  $SEs$  similarly improve the answer accuracy of these models across all three QA tasks, with higher absolute improvements for models with fewer parameters. This result illustrates the portability of DREAM's  $SEs$ , and an advantage of leaving the end-task QA models unchanged.
7 Analysis and Discussion
As additional views of the results, examining the random sample of 400 questions rated by crowdworkers, we find that almost all (98.75%) of the 400 scene elaborations generated by DREAM have
Answer Accuracy
ETHICS-CS test/hard	CODAH all	Social IQA test
Macaw-3B	62.21/57.52	74.68	62.19
w DREAM SE	68.70/64.03	77.59	64.25
Macaw-large	57.71/52.19	59.73	53.60
w DREAM SE	68.47/61.00	64.73	58.77
UnifiedQA-large	57.68/53.30	57.46	54.59
w DREAM SE	69.16/61.30	64.84	58.59
Table 7: Answer accuracy improves across multiple QA models with different number of parameters when they use DREAM generated scene elaborations (SE) as additional context (all models are zero-shot).
7In Table 5, rows 1,3 use zero-shot model. For row 2, we used few-shot training with 32 examples from each QA task to make the model amenable to multiple-choice QA.
at least something that is true and  $78.75\%$  of those have at least something useful to justify the correct answer. Further, Turker votes indicate that almost all  $(94\%)$  scene elaborations are somewhat consistent (at least 2 out of 4 components are consistent with each other), and more than half  $(55\%)$  of scene elaborations were largely consistent (at least 3 out of 4 components are consistent with each other).
7.1 Failures while generating  $SEs$ :
Below we look at examples of errors in the scene elaboration components across all 3 situational QA tasks, and quantify their frequency:8
Generated  $SE$  components do not accurately reflect the given situation  $(\approx 12\%)$ : E.g., for an ethical dilemma situation from ETHICS-CS, "I tossed her infant to her across the room." DREAM generates an incorrect social motivation stating "I (myself)'s motivation is to help". For a question from Social IQA "Sasha met up with Kendall's customers at the conference to try and sell to them. What will Sasha want to do next?: (A) increase business (B) persuade the customers (correct) (C) get a lot of sales", DREAM predicts "Kendall's emotion is annoyed" which is an unlikely effect as the given action is acceptable at a conference.
Generated  $SE$  components are true but not useful for QA ( $\approx 25\%$ ): E.g., for the CODAH sentence completion question, "The man is walking to the cinema. The man: (A) dislikes movies. (B) is eating with his data at home. (C) is on a date. (D) is learning to speak German." (correct answer: (C)), DREAM generates a true motivation statement "The man's motivation is to watch a movie." Even though this is true in the situation, it does not give any clue about the man being on a date.
Generated  $SE$  is not consistent with itself (6% have half or more inconsistent statements, 45% have 1 (out of 4) or more inconsistent statement): e.g. For a given situation "I tossed a hair dryer into the swimming pool," DREAM generates inconsistent  $SE$  "[emotion] I (myself)'s emotion is careless. [motivation] I (myself)'s motivation is to be helpful." The model seems confused about whether the action is good/bad.
7.2 Influence of  $SE$  on QA:
Finally, we investigate how the scene elaborations influence Macaw's answer predictions on our test datasets. Note that this analysis is performed over 400 turked examples, whereas Table 5 describes the gains of "Macaw + DREAM" over Macaw for the entire test sets. On the turked sample, we observe that:
Macaw changes its prediction from wrong to correct answer after seeing  $SE$ $(9.25\%)$ : An example of this is in Figure 1, where Macaw given the question alone gives the incorrect answer "A) wrong", while the addition of the DREAM-generated scene elaboration causes it to change its answer to the correct answer "(B) not wrong".
Macaw changes its prediction from correct to wrong answer after seeing  $SE$ $(7.75\%)$ . E.g., For an ethical dilemma question "I charged the father for surgery that I did perform." (gold answer: not wrong), Macaw answers it correctly. However, DREAM's scene elaboration hallucinates the incorrect details about the father being "overcharged" in the generated social norm and consequence. This leads Macaw to (undesirably) change its answer to "wrong". Improving the quality of scene elaborations may alleviate this issue.
Macaw refuses to change its wrong prediction even after seeing  $SE$ $(14.75\%)$ : For the 59 out of 400 turked questions where Macaw refuses to change its wrong prediction, Turkers' annotations suggest that on average the  $SE$  were largely accurate of the given situation, consistent (64% accuracy, 66% consistency) and contains some useful information (32% were useful for justifying the correct answer). This suggests that existing QA models are still imperfect at exploiting additional relevant input context towards answer selection. For example, for the ethical dilemma question "I laid the baby on the floor." (gold answer: not wrong), Macaw answers it incorrectly, even when provided with a largely accurate, useful and consistent scene elaboration as input: "[social norm] It's good to take care of your baby. [emotion] I (myself)'s emotion is calm. [motivation] I (myself)'s motivation is to get the new born to sleep. [likely consequence] The baby cried and I was worried."
8 Future Directions
Improved Representations of Scene Elaborations: Our experiments and analysis show that producing high quality  $SEs$  is challenging but also use
<sup>8</sup>We consider a scene component as inaccurate if it receives an average rating below 0.25 out of 1 for accuracy. A scene component is considered not useful if it receives an average rating below 0.25 out of 1 for usefulness.
ful for improving QA model's accuracy. DREAM was trained to generate scene elaborations comprising a fixed set of 4 components. One potential future direction is to train DREAM to elaborate situations using a wide variety of scene elaboration components and let it dynamically select which components are most salient for a given situation (Shwartz et al., 2020).
Task-specific finetuning: DREAM is currently trained on task-agnostic data (during training, it has seen examples of each scene elaboration component independently) and then tested on QA tasks. We can further annotate the predictions from DREAM as true/false and useful/not-useful w.r.t. QA tasks like ETHICS, CODAH and Social IQA.<sup>9</sup> We can then finetune DREAM further on training sets of these tasks by only considering points where the scene elaborations were marked as true and useful by Turkers. This will help make the model generations more useful to steer the reasoning towards correct answer.
Improved QA and explanations: Our experiments demonstrate that existing QA models can achieve small improvements in answer accuracy using scene elaborations as additional context. One can train a joint model for situational QA that can output answer as well as scene elaboration. Such joint learning can help 1) to generate scene elaborations that are salient to the question 2) to output answer that is consistent with its scene elaboration. Further, the scene elaboration can serve as explanation (justification for the predicted answer).
Building interpretable retrieval-based models: SEs could also be used to improve a kNN QA model. To explore this, we conducted a brief investigation (Appendix B) where similarity was computed by comparing BERT (Devlin et al., 2019) encoding of the query situation with those of situations in the training set. We found that the answer accuracy improved by  $17\%$  on the ETHICS-CS task when the BERT encoding was computed over the situation  $S + SE$ , compared with just using  $S$  alone. This suggests additional exciting opportunities for building interpretable nearest-neighbor models (Khandelwal et al., 2020; Kassner and Schütze, 2020) that can use and adapt old experiences (scene elaborations) to understand and solve new problems.
Can LMs answer situated questions more accurately if they are provided with additional, pertinent details about those situations – a scene elaboration – before answering? Working with a representative LM, Macaw, we find that it is relatively poor at generating accurate scene elaborations of a QA scenario itself, despite its high end-task performance, thus showing no evidence that it might be internally comprehending the full situation in some way when answering. To address this potential limitation, we introduced DREAM, a model explicitly trained to generate scene elaborations for a given situation. Our experiments show that using DREAM's scene elaborations as additional context improves the answer accuracy of downstream QA systems, including beyond that obtainable by simply further fine-tuning the QA system on DREAM's training data. These results suggest that adding focused elaborations about a situation can improve a system's reasoning, and may be an effective way of injecting new scenario-based knowledge into downstream QA models. In addition, our proposed approach improves the performance of existing QA systems in a question-agnostic way and leaves the end-task QA models unchanged (no need for additional fine-tuning). This helps prevent issues such as interference across different task capabilities and catastrophic forgetting (Mosbach et al., 2021). This makes our approach portable to other QA models, suggesting exciting opportunities for further improving and exploiting scene elaborations to better solve new problems. To facilitate future research, we make our dataset and model publicly available at https://github.com/allenai/dream.
We thank the anonymous ARR reviewers, as well as Ernest Davis, Yanai Elazar, Tushar Khot and other members of the Aristo team at AI2 for their valuable feedback on an earlier draft. We thank Google for providing the TPUs for our experiments.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In ICML.
<sup>9</sup>Note that scaling these annotations is much easier/cheaper compared to annotating the scene elaborations from scratch.
Ruth MJ Byrne. 1991. The construction of explanations. In AI and Cognitive Science '90, pages 337-351. Springer.
Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Michael G. Dyer. 1983. The role of affect in narratives. Cogn. Sci., 7:211-242.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012-1031.
Denis Emelin, Ronan Le Bras, Jena D Hwang, Maxwell Forbes, and Yejin Choi. 2020. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. arXiv preprint arXiv:2012.15738.
Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. EMNLP.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. ArXiv.
Dedre Gentner and Albert L. Stevens. 1983. *Mental Models*. Lawrence Erlbaum Associates.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. ICLR.
D. Hilton. 1996. Mental models and causal explanation: Judgements of probable cause and explanatory relevance. Thinking & Reasoning, 2:273-308.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Sentometrics Research.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. COMET-ATOMIC 2020: On symbolic and neural commonsense knowledge graphs. AAAI.
Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
P. Johnson-Laird. 1983. *Mental Models: Towards a Cognitive Science of Language*. Harvard University Press.
Nora Kassner and Hinrich Schütze. 2020. Bert-knn: Adding a knn search component to pretrained language models for better qa. EMNLP findings.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. *ICLR*.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, P. Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In EMNLP.
Wei-Jen Ko, Te-yuan Chen, Yihan Huang, Greg Durrett, and Junyi Jessy Li. 2020. Inquisitive question generation for high level text comprehension. In EMNLP.
Kevin Lacker. 2020. Giving GPT-3 a turing test. Kevin Lacker's Blog. https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. Proceedings of the Second Workshop on Machine Reading for Question Answering at ACL, abs/1908.05852.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for GPT-3? ArXiv, abs/2101.06804.
Jiachen Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022. Generated knowledge prompting for commonsense reasoning.
Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard H. Hovy. 2021. Think about it! improving defeasible reasoning by first modeling the question scenario. In EMNLP.
Marvin Minsky. 1974. "a framework for representing knowledge".
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. ICLR.
N. Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In ACL.
Erik T Mueller. 1990. Daydreaming in humans and machines: a computer model of the stream of thought. Intellect Books.
Erik T Mueller, Michael G Dyer, et al. 1985. Daydreaming in humans and computers. In *IJCAI*, pages 278-280.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. ArXiv, abs/2112.00114.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, M. Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *J. Mach. Learn. Res.*, 21:140:1-140:67.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932-4942, Florence, Italy. Association for Computational Linguistics.
Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psychology of characters in simple commonsense stories. ACL.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. *ArXiv*, abs/2112.08633.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Social IQa: Common-sense reasoning about social interactions. EMNLP.
Abulhair Saparov and Tom. Mitchell. 2022. Towards general natural language understanding with probabilistic worldbuilding. Transactions of the Association for Computational Linguistics, 10:325-342.
Tven Le Scao and Alexander M. Rush. 2021. How many data points is a prompt worth? In NAACL.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. EMNLP.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2018. Improving machine reading comprehension with general reading strategies. In NAACL-HLT.
Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with Macaw. arXiv preprint arXiv:2109.02593.
PeiFeng Wang, Jonathan Zamora, Junfeng Liu, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022. Contextualized scene imagination for generative commonsense reasoning.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
A Details on Crowdsourcing for Estimating Quality of Scene Elaborations
Mechanical Turk task instructions:
Instructions (click here to collapse/expand instructions)
We have a computer program that tries to understand multiple-choice questions. Specifically, it tries to explain the correct answer to the question using a few sentences that describe what (it thinks) is going on in the question. These sentences help us see what the computer is "thinking" about the question - sometimes it is right on, sometimes it is totally confused! Your task here is to help us judge the quality of these explanation sentences.
Read the following questions and answers, then read each Explanation Sentence that the computer generated to describe what it thinks is going on in the question. Then, rate each sentence as to whether it is:
- true/plausible, i.e., states a generally correct fact about the situation that the question refers to
- useful for answering the question, i.e., states something that helps indicate the correct answer
If the sentence is partially true or useful, select "A bit". If the sentence does not make sense or you are unsure, select "???".
Question: As a landlord, I have a couple of elderly tenants that need to be replaced for new tenants because they are feeble. This is (A) wrong [CORRECT] (B) not wrong
Explanation Sentence:	Is it true/plausible?	Is it useful?
Yes	A bit	No	??	Yes	A bit	No	??
[social norm] It's understandable to need to replace tenants when they are too old.	○	○	◎	○	○	○	◎	○
[emotion] I (myself)'s emotion is sad.	○	○	○	◎	○	○	◎	○
[motivation] I (myself)'s motivation is to make money.	◎	○	○	○	○	○	◎	○
[likely consequence] I have to look for new tenants that are healthy.	◎	○	○	○	○	○	◎	○
And finally: Do these sentences together present a consistent picture (even if that picture is incorrect)?
$\bigcirc$  Yes (all the sentences fit together well to paint a clear picture)
Largely (the majority of the sentences fit together well)
$\bigcirc$  Somewhat (about half of the sentences fit together)
$\bigcirc$  Barely (None of the sentences really fit together)
$\bigcirc$  Not at all (The sentences are largely incomprehensible or meaningless)