COLING 2022
Oct. 12-17, 2022 @ Gyeongju, South Korea

Possible Stories: Evaluating SituatedCommonsense Reasoning under Multiple Possible
Scenarios
Mana Ashida (Yahoo Japan Corpora1on)* maashida@yahoo-corp.jp
Saku Sugawara (Na1onal Ins1tute of Informa1cs) saku@nii.ac.jp*work done while at Tokyo Metropolitan University.

Background
Commonsense reasoning is a fundamental intelligence acquiredwith humans, and researchers are interested in if it is learned by
the models.
Many benchmarks have been proposed to measure machine’scommonsense reasoning ability, but the current state-of-the-art
models’ performance is comparable to human.

2

Background
Natural language understanding (NLU) benchmarks incorporating
several possibilities to one question, along with when it becomesplausible are still rare.
● SituatedQA (Zhang and Choi, 2021), objective facts changing
across time and place.
● Moral Stories (Emelin et al., 2021), human behaviors either
normative or divergent.

3

Our Goals● Creating a resource to evaluate commonsense reasoning undermany different types of conditions.
● Probing how well the current SoTA models perform, comparedwith human.

4Situated Commonsense Reasoning
Task
Context: what to do during the holidays.

🤔

Situation 1

Ending 1🏖

🌞Ending 2

Situation 2🏥🌪

Situation 3

😷

Ending 3🏠5Dataset Creation
ROCStories (Mostafazadeh et al., 2016)
A collection of short stories consists of five sentences.
● Beginning and ending are clear.
● Not too generic, not too specific.
● Stories on everyday situations.

6

Dataset Crea5on
1. Alternative Ending Collection
2. Question Writing
3. Validationa. Question-Answer
b. Content7

Possible Stories Overview● 4.5k questions to 1.3k passages, 3.45 questions per passage.
● The average # of tokens:○ 14.2 for question, 15.3 for options→ longer than RACE (Lai et al., 2017) or CosmosQA (Huanget al., 2019)● Dataset split:○ train (75%), dev (10%), test (15%)○ Dev and test set contain the examples generated by theworkers who perform very well.

8Experiments● unsupervised○ fine-tune with RACE
● supervised○ fine-tune with Ours

Model and human performances on our dataset.
(∗) indicates that the model is fine-tuned onRACE.

9Case StudyContext: A family going on a trip in the summer and made new friends.
Options:
A. They kept in touch with their friend even after they went home.
B. At the end of the day the kids got into a fight with each other andwere happy to leave. ✅
C. The Smith’s decided they’d visit a new beach every year, and they
made tons of new friends.
D. They went home though and the kids never saw their friend again.
⏪ DeBERTa-large

Question: Which ending involves the most conflict?

10

Analysis of Annotation Artifacts
Annotation artifacts (Gururangan et al., 2018; Gardner et al.,2021)
Statistical patterns between inputs and output labels found in thecrowdsourced datasets.

11

Analysis of Reasoning Types● Classifying questions into nine reasoning types.
● Possible Stories dataset contains questions with following types
which are often absent in existing datasets.

12Conclusions● We propose a situated commonsense reasoning task and create amultiple-choice QA dataset. (accessible at https: //github.com/niicl/possible-stories.)● We discover that current strong pretrained language models struggle
to solve our task when training data are unavailable.
● We show that our dataset contains minimal annotation artifacts inthe answer options and has many challenging questions that require
counterfactual reasoning and an understanding of characters’
motivations and reactions, readers’ perceptions, and fictional
information.
13