SlowGuess/NAACL_2022_OCR huggingface dataset:
Yuling Gu*, Bhavana Dalvi Mishra, Peter Clark
Allen Institute for AI, Seattle, WA
{yulingg,bhavanad,peterc}  $@$  allenai.org
When people answer questions about a specific situation, e.g., "I cheated on my mid-term exam last week. Was that wrong?", cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the "scene". To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM's training data. These results suggest that adding focused elaborations about a situation can improve a system's reasoning about it, and may serve as an effective way of injecting new scenario-based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters.<sup>1</sup>
Cognitive science has long promoted the formation of mental models - coherent, constructed representations of the situations we encounter - as central to understanding and question-answering (Johnson-Laird, 1983). Drawing loosely on this
Figure 1: Given a situation S, our system DREAM generates an elaboration of the situation - a "scene elaboration"  $SE$  - envisioning details of what might be happening in S. Given a question Q about S, we find that a SOTA QA system (Macaw) answers Q more accurately when  $SE$  is provided as additional input.
idea, but without making any claims of how language models (LM) reason internally, our goal is to investigate if a LM can answer situated questions about social situations more accurately if they are provided with additional, pertinent details about those situations before answering. To explore this goal, we first investigate how well a representative state-of-the-art language model can generate such scene elaborations zero-shot, by asking it probing questions. We find that such zero-shot elaborations are generally poor quality. To overcome this, we train a new model, called DREAM, $^2$  to specifically answer these elaboration questions, resulting in higher quality answers (more accurate, useful, and consistent). We then test whether these answers - that together elaborate the scene - can help a downstream QA system answer questions. We find that they can, improving the answer accuracy beyond that obtainable by simply further fine-tuning the
$^{2}$ Dynamically wRitten ElAborations to iMprove question-answering
*First author was a student at University of Washington at the time of submission of this paper. This is work done during an internship at Allen Institute for AI.
<sup>1</sup>We make our dataset and model publicly available at https://github.com/allenai/dream.
Human Language Technologies, pages 1115 - 1127
QA system on DREAM's training data.
Figure 1 illustrates this approach. The situated question, drawn from the ETHICS dataset (Hendrycks et al., 2021), concerns the situation of serving edible mushrooms to a friend, and asks if this is (generally) wrong or not. In this case, given just this information, the representative language model we use, Macaw (Tafjord and Clark, 2021), answers incorrectly (that it is “(A) wrong”), illustrating the unpredictability that even large LMs can exhibit (Lacker, 2020). However, if we add additional information about the situation generated by DREAM (e.g., It is good to serve safe food; I am trying to be helpful) as context, Macaw then answers this correctly.
Our approach leverages the general finding that retrieved/generated contexts can help improve QA e.g., (Rajani et al., 2019; Wei et al., 2022), in two important ways. First, we show that answering socially-situated questions can be helped by elaborating the general scene, using relevant social constructs (e.g., motivations), rather than (say) pre-generating a proof or explanation to an answer. Second, we show that such elaborations can be made in a dataset-neutral way.
Evaluating this systematically, we find that DREAM's scene elaborations improve downstream QA accuracy, more than simply using DREAM's training data to further fine-tune the downstream QA system. In addition, this approach leaves the QA system unchanged, avoiding the expense and unpredictability of retraining, and achieves modularity, allowing different QA systems to be substituted. Our contributions are thus:
1. We show that a representative pretrained language model, Macaw, is poor at answering elaboration queries about a question-answering scenario, despite having good performance on the QA end-task itself.
2. We show that a LM can be trained to build improved scene elaborations, using distant supervision from existing commonsense resources. Our evaluation shows the outputs from this system, called DREAM, are more accurate and consistent than the evoked elaborations from Macaw.
3. Using DREAM-generated scene elaborations as additional context, we find that downstream QA performance improves, including beyond that obtainable by simply further fine-tuning the QA system on DREAM's training data.
Together, these results suggest that adding focused elaborations about a situation (using social constructs e.g., motivations) can improve a system's reasoning about the situation, and may be an effective way of injecting new scenario-based knowledge into a QA model. Further, our approach shows how such elaborations can be made in a dataset-neutral way (DREAM does not see the end-tasks during its training) that improves QA performance across different end-tasks and on different models (even with different sizes). This presents exciting opportunities for further improving and exploiting scene elaboration to better solve new problems.
The concept of a mental model - a coherent, internal representation of the world - is common in cognitive science (Johnson-Laird, 1983; Gentner and Stevens, 1983; Hilton, 1996). It suggests that people solve situated problems by elaborating a mental picture of the situation, including elements that may be peripheral to a specific answer, rather than by constructing a deductive proof from a few key facts to an answer (Byrne, 1991). Recently, Saparov and Mitchell (2022) tried creating an internal "mental model" using a set of axioms that deductively explain the observations. Other studies in AI have attempted to identify such elements by studying what questions people naturally ask when reading text (Ko et al., 2020) or viewing images (Mostafazadeh et al., 2016). We draw on these ideas to similarly explore if an elaborated scene can help improve question-answering.
Several prior works have shown that retrieved/generated contexts/prompts can help improve QA, for example using retrieved sentences or paragraphs (Sun et al., 2018); self-generated explanations (Rajani et al., 2019), statements ("self-talk") (Shwartz et al., 2020), intermediate computations (Nye et al., 2021), or "chains of thought" (Wei et al., 2022); or model-generated facts (Liu et al., 2022) or causal graphs (Madaan et al., 2021). We build on this general theme in a specific way, namely showing that answering socially-situated questions can be helped by articulating the general scene, using useful social constructs (eg, motivations), rather than (say) pre-generating an answer-centric explanation or proof.
Our approach can be viewed as a form of prompt engineering (Scao and Rush, 2021), where we pre
Query situation	Macaw's Scene Elaboration	DREAM's Scene Elaboration
[CODAH]
[Q, S] The woman and her daughter were happily strolling through the park. The woman: Gold Ans: was smiling.	[social norm] if you want to be happy, be kind
[emotion] The woman's emotion is joy. Daughter's emotion is joy.
[motivation] Her legs' motivation is the need to move forward.
[likely consequence] the woman will trip and fall	[social norm] It's good to spend time with your children.
[emotion] Woman's emotion is happy.
[motivation] Woman's motivation is to spend time with her daughter.
[likely consequence] was able to enjoy the company of her daughter.
[ETHICS] [S] I got sick the last time I ate there, so I recommend a different restaurant to my in-laws.
[Q] Reaction: this is (A) wrong (B) not wrong Gold Ans: not wrong.	[social norm] if something makes you sick, don't eat there
[emotion] In-laws' emotion is anger. Food's emotion is disgust.
[motivation] Bacteria's motivation is to make more bacteria. My motivation is I don't want them to get sick.
[likely consequence] they will eat there once and get sick	[social norm] It's good to make recommendations to others.
[emotion] I (myself)'s emotion is responsible.
[motivation] I (myself)'s motivation is to protect others.
[likely consequence] My in-laws have a great meal at the new restaurant I recommended.
Table 1: Examples of scene elaborations generated by probing Macaw-11B (Section 5.1) and as generated by DREAM. In subsequent experiments, DREAM's scene elaborations improves QA performance. Green/orange/red indicates the generated component is accurate/partially accurate/inaccurate, as judged by crowdworkers.
fix a question with an elaborated scene. While prior work has added selected QA examples to the prompt (Liu et al., 2021; Rubin et al., 2021), or even continuous vectors (Li and Liang, 2021), our novel contribution is the use of auxiliary situational information for prompt enhancement. Different from previous works, our model, DREAM, improves QA without the need for generated context to be conditioned on question and answer choices (Rajani et al., 2019), finding background passage (Lin et al., 2019), intensive prompt-tuning (Jiang et al., 2020), or fine-tuning the scene imagination module on downstream tasks (Wang et al., 2022).
3 Scene Elaborations (SE)
We focus on the task of situated reasoning about social situations where the input consists of (a textual description of) a situation S, and question Q testing the model's understanding of the situation (both explicitly mentioned and implicitly indicated facts). The output is the answer A to the question. Figure 1 shows an example. Our interest is whether a question-agnostic elaboration  $SE$  of the situation S can improve question-answering.
3.1 Representing Scene Elaboration
For simplicity, we define a scene elaboration  $SE$  of situation  $S$  as a 4-tuple  $\{M,E,ROT,Con\}$  that provides details about  $S$  along four key conceptual dimensions, where each element is represented as text (typically a single sentence), prefixed with an identifier indicating its dimension. The four dimen
sions are as follows:
1.  $M$ : motivation of character(s) before  $S$ .
2.  $E$ : emotion of character(s) after  $S$ .
3.  $ROT$ : general Rule of Thumb (ROT) about whether action described in  $S$  is socially acceptable or not (also known as social norm).
4. Con: likely consequence of action in  $S$ .
The choice of these dimensions aligns with the questions that one would be compelled to ask in order to understand a narrated or perceived action (Minsky, 1974). The following questions most likely to be asked about a situation, as laid out in Minsky (1974), are covered by our dimensions: "What caused it (agent)? What was the purpose (intention)? What are the consequences (side-effects)? Whom does it affect (recipient)?"
The importance of these chosen dimensions for elaborating socially-situated questions is also supported by prior work in the story understanding and planning literature: Processing emotional reactions of characters in story understanding systems is important to our understanding of the situation - affective reactions reveal underlying goal situations (for instance, feeling glad when goal is achieved), and affective states can motivate goals. e.g. "Why did Richard get drunk? Richard was upset by almost having run over an old man." (Dyer, 1983). Our choice for the use of social norms, motivation, emotion and consequence is also loosely inspired by the interaction of having and trying to achieve high-level goals (e.g. being love-giving), emotions (e.g. surprise, anger), and daydreaming
goals (e.g. reversal, rehearsal). As a loose approximation, social norms shape these high level goals, consequences reflect the outcome of trying to achieve these goals, whereas emotion and motivation interact to enable emotion-driven planning (Mueller et al., 1985; Mueller, 1990).
For situated questions that are not socially oriented, e.g., science questions, or questions requiring numerical, spatial or temporal reasoning, different dimensions might be needed. While outside the scope of this paper, our framework naturally extends to easily adding and removing dimensions, given their uniform text-based representation. We discuss this further in Section 8.
3.2 Probing for Scene Elaboration
Arguably, it may be redundant to provide  $SE$  as additional input to a QA model, if that QA model can already infer those details itself from  $S$ , i.e., build  $SE$  itself. To explore this, we probe our QA model using the following four questions along the four dimensions of interest:
- What is [ENTITY]'s motivation?
- What is [ENTITY]'s emotion?
- What is a rule of thumb relevant here?
- What is likely to happen next?
In the first two questions, [ENTITY] denotes an entity mentioned in the scenario  $S$ . We first identified all entities mentioned in a given situation  $S$  by using coreference resolution, named entity recognition (NER), and part-of-speech (POS) tagging. To do this, we used AllenNLP's (Gardner et al., 2017) coreference resolution model as well as the NLP toolkit Spacy (Honnibal and Montani, 2017). If more than one entity is found in  $S$ , then the question is asked for each entity in turn, and the answers concatenated together. In addition, for these two questions, each answer (e.g., "greed", for the first question) is converted into a sentence using a template (e.g., "John's motivation is greed.") so the information is fully captured. The last two questions are asked directly. The answers are gathered into a single structure (e.g., see Table 1).
4 Our Model: DREAM
In addition to probing, we also explore whether we can train LMs to build improved scene elaborations, and whether they can improve QA performance. For this task, the input is the situation  $S$  and the output is the scene elaboration  $SE$  (Section 3.1).
4.1 Training Data
We use three existing commonsense resources to construct a training dataset for learning scene elaborations:
1) Story Commonsense (Rashkin et al., 2018)
2) Social Chemistry (Forbes et al., 2020)
3) Moral Stories (Emelin et al., 2020)
Statistics about these data sources, and which dimension(s) they contribute to the training data along with examples are shown in Table 2. We call the dataset the "Scene Elaborations Dataset".
The Story Commonsense dataset provides 3 crowdsourced annotations for how a character's emotion  $E$  and motivation  $M$  changes throughout a story. We create multiple training examples from each such story using the "sentence", "character", "emotion" and "motivations" fields in the dataset to create mappings: (A)  $S \longrightarrow E$ : situation (a sentence in the story) to emotional response of a character after the sentence and (B)  $S \longrightarrow M$ : situation to motivation of character before the sentence. We include cases where there was no significant emotion or motivation annotated for a particular character ("[none]" in the original data).
In the Social Chemistry dataset, we use the "situation" and "rot" (rule of thumb) fields to create mapping  $S \longrightarrow ROT$ : situation to most relevant social norm. Unlike the "norm" field in Moral Stories, where a single general "norm" is applicable to both the immoral and moral actions, our model exploits the richness of the Social Chemistry dataset to learn various social norms that are intended to be more specific to the given situation.
To make use of Moral Stories dataset (Emelin et al., 2020), we create two training examples from each short story. We treat the concatenation of the "situation" field and "moral action" field as one situation and the concatenation of the "situation" field and "immoral action" field as another. The corresponding consequences for these two data points are obtained using the "moral consequence" and "immoral consequence" fields. Differing from just generating a "likely consequence" (found in the COMET dataset (Hwang et al., 2021)), this
4 In a rare case when the situation is very short and has no person entity e.g. 'This winter is very cold.', no question is asked to Macaw for emotion and motivation. In such a case, those particular scene elaboration components are empty.
5The two templates are "[ENTITY]'s motivation is [ANSWER]", "[ENTITY]'s emotion is [ANSWER].
Source
Dataset	Question	Answer
Situation	SE component	Training Size
Social
Chemistry	smacking an airplane seat to intimidate a child.	ROT	23K	You shouldn’t scare people.
reporting someone for cheating.	ROT	It is good to report any cheating that you see.
Story
commonsense	Rick saw an insect he never saw before.	E	17.5K	Nick’s emotion is amazed.
Mike was at a burger restaurant.	M	17.5K	Mike’s motivation is to eat.
Moral Stories	Sally is starting a new school today. Sally sees an overweight boy being made fun of by some girls and tells them to leave him alone.	Con	20K	The boy appreciates Sally standing up for him and the two become good friends.
Sally is starting a new school today. Sally sees some girls making fun of an overweight boy and joins in and laughs with the others.	Con	The boy has his feelings hurt and Sally feels guilty afterwards.
Table 2: Examples of datapoints from the Scene Elaborations Dataset.
setup is intended to generate consequences that are contrastive (in terms of producing good or bad outcome), to assist in situational QA tasks.
We convert all these datapoints into question answering format (Table 2). E.g., During training, DREAM sees a question like '[SITUATION] smacking an airplane seat to intimidate a child. [QUERY] social norm' and it is trained to generate answer 'You shouldn't scare people'. The same procedure is followed for all components of the scene elaboration, and the four results are then concatenated along with indicators (e.g., "[emotion]") indicating each result's component.
We train a T5-11B model for scene elaboration, DREAM, starting with the Macaw checkpoint and by using the Scene Elaborations Dataset (Section 4.1). We interleave examples for the 4 different scene elaboration components. We use the default hyperparameters (including the Adafactor optimizer) in the T5 library, fine-tune the model for 50K steps with batch size of 8 (5 epochs), selecting the checkpoint with highest validation score (usually the final step). Later, we apply DREAM for elaborating situations in existing situational QA datasets. Examples of elaborations are in Table 1.
We conduct experiments to address three questions, using Macaw as our representative LM:
Q1. To what extent can Macaw generate an accurate and consistent scene elaboration?
Q2. To what extent does our trained scene elaboration generator, DREAM, improve on this?
Q3. Can the scene elaborations produced by DREAM help improve QA?
5.1 Representative LM: Macaw
Our representative LM, Macaw, is an off-the-shelf, state-of-the-art, T5-based question-answering system (Tafjord and Clark, 2021). Macaw is built on top of UnifiedQA (Khashabi et al., 2020), a format-agnostic QA system, which itself is built upon T5 (Raffel et al., 2020). Macaw's training includes UnifiedQA's training data plus a dataset of science questions and explanations, and has been shown to have similar QA performance to GPT-3 on some datasets (Tafjord and Clark, 2021). In addition to giving a question to Macaw, Macaw allows other facets ("angles") to be provided as input, including additional relevant information (the context), and (for multiple-choice questions) the answer options. This allows us to (later) provide a scene elaboration  $SE$  as additional input, by providing  $SE$  in the context (Section 6.3). We use the 11B version of Macaw. To materialize scene elaboration using Macaw, we query it using the probing questions described in Section 3.2.
5.2 Test Datasets
We evaluate the probed and DREAM-generated scene elaborations on three different situational QA datasets, zero-shot (statistics in Table 3). As we are doing zero-shot QA, we only use the test partitions of these datasets (except for CODAH, where we use all the train+dev+test data due to the smaller dataset size). For the ETHICS dataset, we use the commonsense partition (hence "-CS"). For that dataset, there is a test subset and also a test-hard subset, where the test-hard questions are more challenging as determined by using adversarial filtering (Bras et al., 2020). We track scores on both test and test-hard for this dataset.
Dataset	Train	Dev	Test/Test-hard
CODAH
(Chen et al., 2019)	1665	556	555
ETHICS-CS
(Hendrycks et al., 2021)	13910	-	3885/3964
Social IQA
(Sap et al., 2019)	33410	1954	2224
5.3 Metrics
To evaluate quality of the probed/generated scene elaborations  $(SE)$ , we use human evaluation using (mechanical Turk) crowdworkers. Workers rate each of the four components (each typically a sentence) of the scene elaboration along 2 dimensions:
SE accuracy: this metric checks if the component in  $SE$  is true w.r.t. the situation described in the question. Each component gets a score of 1 (yes) / 0.5 (a bit) / 0 (no).
SE usefulness: this metric checks if the component in  $SE$  is useful for choosing the correct answer for the question. Each component gets a score of 1 (yes) / 0.5 (a bit) / 0 (no).
In addition, workers rate the complete  $SE$  along the following dimension:
SE consistency: this metric measures what fraction of sentences in the entire  $SE$  are consistent with each other, independent of whether they support the correct answer. Each explanation gets a score of 0 (not consistent)/0.25 (barely consistent)/0.5 (somewhat consistent)/0.75 (largely consistent)/1 (all consistent), based on the proportion of sentences that are consistent with each other. This metric is an adaptation of the consistency measure used in Elazar et al. (2021).
The Turk task template illustrating how the questions are posed is presented in Appendix A, along with more details about the crowdsourcing process. We collect and average three worker scores for each question. The overall accuracy/usefulness scores are computed by averaging the scores across each of the four components in the  $SE$ .
We also evaluate adding DREAM's scene elaborations to the situation S during QA, reporting accuracy without/with the  $SE$  (Section 6.3).
Table 3: Statistics for the situational QA datasets used. Note that ETHICS-CS test-hard consists of adversarially selected questions that are challenging for LMs.
Dataset	Model	Quality of Scene Elaboration
%Acc	%Useful	%Cons
ETHICS-CS test	Macaw	52.23	29.48	56.74
DREAM	67.77	43.71	72.17
ETHICS-CS test-hard	Macaw	49.85	28.90	52.75
DREAM	67.98	41.21	73.50
CODAH test	Macaw	44.94	22.29	42.42
DREAM	68.18	34.13	66.58
Social IQA test	Macaw	46.96	25.34	45.42
DREAM	72.77	41.44	73.92
Table 4: DREAM produces significantly better scene elaborations compared to Macaw-11B with probing for situations in the three situational QA tasks in terms of accuracy, usefulness and consistency metrics.
6.1 Q1: How good are Macaw's Scene Elaborations of a situation S?