CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Authors: Mete Ismayilzada
Debjit Paul
Syrielle Montariol
Mor Geva
Antoine Bosselut
Venue: EMNLP 2023 Main (2023)
Keywords: commonsense reasoning
benchmark
real-world task

Abstract: Recent efforts in natural language processing (NLP) commonsense reasoning research have yielded a considerable number of new datasets and benchmarks. However, most of these datasets formulate commonsense reasoning challenges in artificial scenarios that are not reflective of the tasks which real-world NLP systems are designed to solve. In this work, we present CRoW, a manually-curated, multi-task benchmark that evaluates the ability of models to apply commonsense reasoning in the context of six real-world NLP tasks. CRoW is constructed using a multi-stage data collection pipeline that rewrites examples from existing datasets using commonsense-violating perturbations. We use CRoW to study how NLP systems perform across different dimensions of commonsense knowledge, such as physical, temporal, and social reasoning. We find a significant performance gap when NLP systems are evaluated on CRoW compared to humans, showcasing that commonsense reasoning is far from being solved in real-world task settings. We make our dataset and leaderboard available to the research community.

CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Mete Ismayilzada
Debjit Paul"
Syrielle Montariol'
Mor Geva o
Antoine Bosselut
EPFL, Switzerland Google DeepMind
mahammad mahammad. wwt..iisiaaaaaaeallllccommm
Abstract
Artificial Evaluation
Recent efforts in natural language process- ing (NLP) commonsense reasoning research have yielded a considerable number of new datasets and benchmarks. However, most of these datasets formulate commonsense reason- ing challenges in artificial scenarios that are reflective of the tasks which real-world NLP systems are designed to solve. In this work, we present CRoW, manually-curated, multi- task benchmark that evaluates the ability of models to apply commonsense reasoning the context of six real-ve wolld NLP tasks. CRoW is constructed using a multi-stage data collec- tion pipeline that rewrites examples from exist- ing datasets using commoneense-violaing per- turbations. We use CRoWto study how NLP systems perform across different dimensions of commonsense knowledge, such as physical, temporal, and social reasoning. We find a sig- nificant performance gap when NLP systems are evaluated on CRoWcompared to humans, showcasing that commonsense reasoning is far from being solved in real-world task settings, We make our dataset and leaderboard available the research community."
Bob decided to valumteer because he wanted eat the candles.
OK
bacause wanted to volunteer.
OK
CRoW Evaluation (real-worid)
Dialogue
Agent: would you like some free candies?
you handing these
Agent: Well, we're trying gather some people to valunteor for the
Human: Ut....
docan't want
docan't have prerequisite
Agent: It's OK You don't have to volumfeer if you eet ma canalees
It's You have est the candles if you Boluntear.
Agent:
You don't
OK X
Figure An example from one of the lasks (Dialogue) in our benchmark showcasing the difference between the evaluation of commonsense reasoning in an artificial and real-worke setting. CRoW grounds this evaluation in real-world context that often requires the use of rich and implicit commonsense knowledge to solve a task.
1 Introduction
Commonsense reasoning is a long-standing chal- lenge in artificial intelligence (AT) and natural language processing (McCarthy, 1960; Winograd, 1974; Davis and Marcus, 2015; Choi, 2022), re- sulting in a large number datasets and bench- marks designed to evaluate how AI systems rea- son in commonsense scenarios described in natural language (Davis, 2023). Recently, large language models, such GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), have demonstrated near-human performance on many of these bench- marks (Lourie al., 2021). However, these models can still be brittle in practical deployments, raising
questions about how reliably these commonsense benchmarks truly evaluate the commonsense soning abilities of models.
Part of this issue stems from the practice that most commonsense datasets are designed to evalu- ate rcasoning in artificial task settings that are not reflective of the real-world use cases in which NLP systems are deployed. In real-world settings, one almost never directly observes test of common- sense knowledge in isolation. In this paper, we argue instead that commonsense reasoning bench- marks should evaluate commonsense reasoning in the tasks in which these abilities are required.
"Equal contribution https://gihhah.co//mmssaaaa
The necessity of commonsense to solve real- world tasks has been extensively argued since the

---

early stages of AI, notably by Bar-Hillel (1960) in the context of machine translation. However, de- spite these early arguments, only recently was there an attempt to construct a commonsense reasoning dataset for machine translation (He et al., 2020), an effort which concluded that the commonsense reasoning abilities of modern models were still in their infancy when applied in real NLP tasks.
mance of state-of-the-art models on our bench- mark across different dimensions of common- sense knowledge.
2 Related Work
Commonsense Reasoning Benchmarks Many benchmarks measuring the commonsense rea- soning abilities of statc-of-the-ar models have been released in recent years. Starting with the well-known Winograd Schema Challenge (WSC; Levesque et al., 201 1), these benchmarks have at- tempted to test the commonsense reasoning ability of models using different task formats, such as pro- noun resolution (Levesque al., 2011; Rudinger et al., 2018: Eisenschlos et al., 2023), question- answering (Talmor et al., 2019; Zellers et al., 2018; Chen et al., 2019; Reddy et al., 2019; Zellers et al., 2019), 9), plausible inference (Roemmele et al., 201 I; Bhagavatula et al., 2019; Wang et al., 2019b; Singh et al., 2021; Gao et al., 2022) and natural language generation (Lin et al., 2020b). Benchmarks have also been created to evaluate commonsense rea- soning across different dimensions of common- sense knowledge, including social (Rashkin et al., 201 8a,b; Sap et al., 2019b), physical (Bisk et al., 2019; Dalvi et al., 2018; Storks et al., 2021), tem- poral (Qin et al., 2021; Zhou et al., 2019) and nu- merical reasoming (Lin et al., 2020a). Additionally, there exist comprehensive multi-task benchmarks that consist of several new existing datasets for commonsense reasoning (Tamari et al., 2022; Sri- vastava et al., 2022; Wang al., 2019a). For a thorough survey in this area, we refer readers to (Storks et al., 2019; Davis, 2023). In contrast to these benchmarks, where the underlying task for- mulation is centered around a task that is typically not grounded in real-world setting, we construct CRoW to specifically focus evaluating com- monsense reasoning in real-world tasks for which NLP systems would be deployed.
In this work, we build on these original ideas and introduce CRoW: a Commonsense Reasoning Benchmark for World Tasks, multi-task benchmark containing high-quality datasets for six real-world NLP tasks: machine translation (MT), open-domain dialogue (DG), dialogue summariza- tion (DS), intent detection (ID), stance classifi- cation (SC), and safety detection (SD). Inspired by Winograd schemas (Levesque et al., 2011), I), build our benchmark by applying commonsense- based minimal perturbations on examples from ex- isting datasets for each task. For each of these tasks, we crowdsource collections of potential tar- get references for the task, each grounded to a par- ticular commonsense violation with respect to the original context (see Figure/ for examples in di- alogue response generation). We categorize these commonsense violations across six dimensions temporal, causal, attribution, comparison, physi- cal, and social - ensuring diverse breakdown of commonsense reasoning types in CRoW.
Our empirical study across 13 state-of-the-an (SoTA) systems (including GPT-4) shows that CRoW is a challenging commonsense reasoning testbed, with the highest performing model scoring 188 lower than humans individual examples and ~37% lower our more restrictive metric that evaluates situational robustness. Consequently, we provide CROW to the community as the first com- monsense benchmark specifically formed to test commonsense knowledge and reasoning abilities in the same contexts as real-world deployments of NLP systems. The contributions of our work can be summarized follows:
Commonsense Reasoning in Real-World Con- We design a common multi-stage data collec- tion pipeline for generating commonsense-buse Winograd-style variations of examples, which can be applied to many tasks.
texts A few recent works have explored the role of commonsense knowledge in real-world settings, such as open-ended response generation (Zhou et al., 2021; Ghosal et al., 2021, 2022), machine translation (He et al., 2020) and reading compre- hension (Zhang al., 2018: Huang al., 2019) and have proposed new commonsense reasoning tasks and benchmarks. We build on top of these bench- marks and extend them to several other worl-world NLP tasks, along with a general data collection
We apply our data collection pipeline to construct CROW, multi-task benchmark that evaluates the commonsense reasoning ability of models in solving six diverse real-world NLP tasks.
For each task, we evaluate and analyze ze the perfor-

---

methodology for commonsense knowledge anno- tation and Winograd-styy schema generation that can be applied to other tasks in the future.
Generation (WSG), each of which is followed by complementary validation stage. Figure 2 illus- trates the pipeline for the intent detection task.
This multi-stage approach has two key benefits. First, we ground the perturbations commonsense dimensions, ensuring the Winograd schemas differ on commonsense violations. Using these dimensions, we also ensure a diverse set of per- turbations across different types of commonsense knowledge, allowing us to stratify our later analysis across these dimensions to more finely understand model failures in commonsense reasoning. Second, particular stage can be skipped if the data for it is already available, which the case for several tasks in our benchmark. We use Amazon Mechanical Turk (MTurk) as a crowdsourcing platform. Below, we describe each stage in detail.
3
Data Collection
Our goal is to assess the ability of NLP systems to apply commonsense reasoning in real-world tasks. To this end, we define a general methodology and multi-stage data collection pipeline (Figure 2) for generating evaluation examples that require com- monsense reasoning in a given real-world task. In what follows, we outline our general data collection methodology, and describe each step in detail.
3.1 Overview
The Winograd Schema Challenge (Levesque et al., 2011), 1), an often-used benchmark to measure com- monsense reasoning abilities, tests whether models can distinguish the meaning of pairs of sentences with commonsense- based minimal perturbations that flip their meaning. For example, given the sen- tence, "The trophy doesn't it fit into the brown suit- case because it's too large." models should identify that the pronoun "it" refers the "trophy" (using commonsense knowledge), but distinguish that re- placing the word "large" by "small" would flip this reference to "suitcase". Winograd-style schemas have been widely adopted for tasks involving pro- noun resolution (Rudinger et al., 2018; Eisenschlos et al., 2023; Thrush et al., 2022). but also sense- making (Wang et al., 2019b; Singh et al., 2021) and reasoning about exceptions (Do and Pavlick, 2021). While these schemas are simple and ef- fective for measuring commonsense robustness of models, they are rarely applied in real-world tasks.
3.2 Methodology
For a given task example, we define the context as the unchanged part of the example and the target as the candidate for commonsense-ben -bssed minimal perturbation. For example, in intent detection, we designate the headline as the context and the intent as the larget. In Table 6 in the Appendix, we list respective mappings for all tasks.
Commonsense Knowledge Annotation and Val- idation In the first stage of our pipeline, we ex- plicitly annotate implicit commonsense knowledge underlying examples in real-world task datasets. In this stage, crowd workers are tasked to identify concepts in the context and target that could serve as the head and tail of implicit commonsense relationship, as well pre-existing relation that connects them. For example, in Figure 2, for an ex- ample from an intent detection task (Gabriel et al., 2022), a headline "Remote glaciers in China melt- ing at shocking pace, risking water shortages" and an intent "Climate change is real and is showing its effects " would be presented to crowdworkers. They might connect these two statements with the knowl- edge "water shortage is a rype of effect" which would be represented as (head: water shortages, relation: IsA, tail: effect)
Motivated by this gap, we construct CRoW, a benchmark of Winograd-style examples for real- world NLP tasks. While the inherent subtlety of commonsense-based minimal perturbations led the original Winograd schemas be expert-crafted and limited in size, later works developed large-scale sets of Winograd schemas using crowdsourcing and adversarial filtering (Sakaguchi et al., 2019). In our work, we also employ crowdsourcing to generate Winograd-style perturbed examples, but upproach differs in one key aspect. Instead of asking crowdworkers to perturb the given sen- tences directly, we design a data collection pipeline that breaks down the schema construction into two independent stages: Commonsense Knowledge Annotation (CKA) and Winograd-style Schema
Based on earlier work (Ilievski et al., 2021; Speer et al., 2016; Sap et al., 2019a; Ghosal et al., 2021), we also categorize relations into six dimensions of commonsense knowledge: Atributional, Phys- ical/Spatial, Temporal, Causal, Social and Com- parative. Figure shows the distribution of dimen-

---

STEP 1 CK Annotation
STEP CK Validation
STEP Winegrad Schema Generation
STEP Winograd Schema
Validation
Readline: Climate otds Remoto gladers chinrrating at
Hruediing Ciirate coue Parnole giocars in Chass eeeting "anodknng nakkng sttrttags
rikknn shotrabrrtgges
Falos Intest
cl drange andis stewing capes
Clinute change andis showing
Auttrr stottge (M shh
Clinaate dhange veal und anowwng is etfects
Bocker HasProprry vol
X
earr drunge andis showing to resalts
Sanwesae:
effect
Extre Inmett:
glader
Haahhaertt
Cirvata change realand showingh
change
showing resuls
Figure 2: CRoW Data Collection Pipeline (as illustrated for the Intent Detection Task). Given context (news headline) and a target (writer's intent behind it), in the first phase of the pipeline, annotators are asked to identify commonsense knowledge about this context. In the second phase, annotators use the commonsense knowledge from the previous phase to minimally perturb the target to generate a Winograd-style schema for the given example. Each annotation stage also followed with its own validation step.
sions per task.? The dimensions serve as support for fine-gran grained analysis of the commonsense rea- soning abilities of models when tackling tasks. Fol- lowing the CKA stage, apply validation phase filter out low-quality annotations. For example, in Figure 2. the knowledge "glacier Hassroperty real" would be filtered by crowd workers as is not helpful for the task in the given context. Each annotation is verified by three unique workers, and we take the majority vote as the qualifying thresh- old for the next stage.
validation stage. For example, in Figure 2, given the knowledge "water shortages IsA effect", an- notators might produce Winograd-style schemas where the word "effecr" in the given intent is placed with related concepts such "causes" "results". However, as "results" would not change the underlying intent of the example, the schema based on this replacement would not satisfy con- dition (3) above, and hence would be filtered the validation stage. In Appendix B.3, we provide more examples of violations of each condition.
Winograd Schema Generation and Validation In this stage, we present workers with a context, a target, and the associated commonsense knowledge from the previous stage, and ask them to rewrite the target such that it satisfies the following four conditions.3 3 The new targer must (1) minimally differ from the original target (i.e., by edit distance of at most five words), (2) directly violate the given commonsense knowledge, (3) be an incorrect an- swer for the given context, and (4) be contextually relevant. Conditions (1) and (2) are based on the core design of Winograd schemas, and we intro- duce conditions (3) and (4) to increase the difficulty of the generated schemas. Each annotated schema is further validated by three unique workers with respect to the conditions above, and those with at least two valid votes proceed to the final expert
3.3 Data Quality Verification
In order to collect high-quality an- notations, we design a qualification test consist- ing of multiple-choice and open-ended questions. Following earlier work that identified the impor- tance of large pool of annotators for data diversity (Geva et al., 2019), we qualify 58 workers located in the US based on a precision threshold of 0.8 on the multiple-choice questions and a manual review of open-ended commonsense knowledge annota- tions. Based on the best practices for an effective crowdsoureing protocol (Nangia et al., 2021), we further train the annotators on small sample of examples from our tasks, regularly engaging with them and sending feedback during the whole data collection process. Instruction templates and de- tails about this test can be found in Appendix B.
Qualification
AAppeddix B.2 provides more details on the selection and categorization the relations. 'Additional details on the generation instructions can he found in Appendix B.

---

(a) Dialogue
(b) Dialogue Sum.
(c) Intent Det.
(d) Safety Det.
(e) Stance Class.
(f) Translation
Temporal Causal Attribution
Comparison
Social
Physical
Other
Figure 3: Distribution of commonsense knowledge dimensions across tasks
Task
# Contexts
Examples
errors." 4 This step results in around 200 papers. A further manual review of these papers to filter out false positives reduces this number to 82, and we categorize and group the resulting papers by tasks which yield around 25 potential tasks.
Dialogue
1,169
3,548
Dialogue Summarization
453
1,805
Machine Translation (zh-en)
600
1200
Machine Translation (en-de)
500
1000
Machine Translation (en-frr
500
1000
Machine Trunslation (en-ru)
500
1000
Out of these discovered tasks, we select three classic NLP tasks machine translation, summa- rization, and dialogue response generation that are also often used to evaluate the abilities of gen- eral generative language models. In addition, we select three tasks that are applied and special- ized intent detection, stance classification, and safety detection. Other tasks that were discovered as part of this pipeline include toxicity detection, relation extraction, and fact-checking. However, due to the difficulty of generating commonsense- violating perturbations for these tasks (caused by their factual or obscene nature), we leave their inte- gration into our benchmark as future work.
Intent Detection
589
2,440
Stance Classification
397
1,722
Safety Detection
366
2,826
Total
5,074
16,541
Table I: Statistics of the CRoW benchmark.
4 CRoW
CRoW consists of six real-world NLP tasks where commonsense reasoning ability is implicitly re- quired to solve the task correctly. Initially, to select tasks that could serve as good testbeds for CRoW, we followed Davis (2023), and identified the fol- lowing desiderata of tasks in the benchmark: (1) tasks should represent real-world applications of natural language technologies (e.g., machine trans- lation), (2) tasks should involve rich commonsense knowledge use and reasoning, and (3) tasks should be easy for humans. Our final benchmark con- tains ~5 unique contexts with 5000 unique con- texts per task (on average) and ~~6K examples (i.e,, context-target pairs) in total. Table provides statistics about our benchmark (additional statistics be found in Table 6 in the Appendix). In this section, we outline the methodology for selecting real-world tasks that require commonsense rea- soning, as well as a brief overview of each task included our benchmark.
4.2 CRoW Tasks
We apply our pipeline ($3) to the six real-world tasks identified the selection phase. For each task, we select recent existing dataset that con- tains contexts rich with the use of commonsense knowledge. Some of the chosen datasets already include annotations for commonsense knowledge or Winograd schemas, allowing us to skip parts of the pipeline. Here, we describe these tasks and datasets in more detail and identify task-specific variations of the pipeline for each.
Machine translation (MT) is known to require commonsense knowledge (Bar-Hillel, 1960) to re- solve translation errors. We select the test suite constructed by He et al., 2020 for Ennlish translation and the Wino-X dataset (Emelin and Sennrich, 2021) for English German, French, and Russian translation. Both datasets consist of Winograd- examples containing a source sen-
4.1 Task Selection
To identify NLP tasks that satisfy desiderata above, we first crawl papers from the whole ACL anthology published since the year 2000 (approx- imated 94K papers). Next, we select the papers that have done an error analysis and mention com- monsense or world knowledge in their categories of
"Many papers report an analysis error types, often identify commonsense reasoning emors as typical category. "This number is an underestimation, as some papers were not considered due to parsing failures.

---

tence and two translations that minimally differ from each other, but only one of which is correct due to underlying commonsense knowledge. Open-domain Dialogue (DG) is a core real-world NLP task requiring systems to produce chat re- sponses to a given conversational context. The important role of commonsense knowledge and reasoning in open-domain dialogue systems has been well-documented (Richardson and Heck, 2023). For this task, we choose the CIDER dataset (Ghosal et al., 2021), which already contains expert- annotated commonsense knowledge that connects utterances in different turns of the dialogue.
taken in these situations.
Experimental Setup
Task Formulation. All tasks in CRoW are treated as binary classification tasks. Given a context, a model must predict whether provided target a suitable response for the corresponding real-world task. For instance, in machine translation, given an English sentence and a translated sentence in French, the model must predict whether the trans- lation is valid or not.
Evaluation Metries. We evaluate models on CRoW using two scores: Macro-Fl of predicting valid and invalid targets, and Situational Accu- racy, a stringent metric that reports whether the model correctly identifies the validity (or invalid- ity) of all targets for given context (similar to Storks and Chai, 2021's strict coherence score). A single mistake on any target results in score of O for that context. We design this metric account for the fact that robust commonsense reasoning would provide the model with a full situational un- derstanding of the context. The CROW score is computed as a macro-average of the task scores.
Dialogue summarization (DS) is another NLP task with real-world applications (e.g., meeting, email summarization). Also, enhancing summa- rization models with commonsense knowledge has been shown to generate more informative and ab- stractive summaries (Kim et al., 2022). For this task, we choose the test split of the DialogSum dataset (Chen et al., 2021), which contains real-life dialogues along with their abstractive summaries. Intent detection (ID) is the task of identifying the underlying intent of the author of the text. As the intent is typically implicit, it involves signifieant use of commonsense knowledge. For this task, we choose the Misinformation Reaction Frames dataset proposed by Gabriel et al. (2022), which contains news headlines along with news writers' intents behind them and readers' reactions to them. Stance classification (SC) involves inferring the stance (either supporting or opposing) of an argu- ment given a belief. Such task typically requires understanding social, cultural or ontological com- monsense knowledge. We use the ExplaGraphs dataset (Saha et al., 2021), which provides, for each rrgument-belief pair, a crowd-sourced common- sense explanation graph that explains the stance between the two sentences through a set of com- monsense knowledge triplets.
Models. We evaluate a series of language models that are diverse in terms of scale, training, and data:
LLaMA (Touvron et al., 2023), an open-source decoder-only model with various sizes (7B, 13B, 33B parameters) and PaLM-115440 (Chowdh- ery et al., 2022), closed-source decoder-only model with 540B parameters. Both models are pretrained using only a language modeling loss.
GPT-3.55 (Brown et al., 2020) and GPT-4 (Ope- nAI, 2023): two closed-soure source decoder-only models that were trained with instruction-tuning. For GPT-3.5, we use the text-davinci-on model with 175B parameters.
Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023) and Stable- Stable-Vieuna: Vieuna: three open-source decoder-only models based on LLaMA. Alpaca has 7B parameters, while Vicuna and Stable- Vicuna have 13B. They instruction-tunee using different instructions-followinn datasets; Stable-Vieune is further finc-tuned with RLHF
Safety detection (SD), detecting safe actions in a given scenario, has real-world applications, es- pecially in the deployment of autonomous robots and systems capable of giving advice. This task requires the use of commonsense knowledge, es- pecially when the action is not explicitly violent which makes it much harder for the system to as- sess its safety. For this task, we use the SafeText dataset (Levy et al., 2022), where each sample con- sists of sentence describing a real-life scenario and a list of safe and unsafe actions that could be
Flan-T5-XXL (Chung et al., 2022, 11B pa- rameters) and Flan-Alpaca (Chia et al., 2023; Peng et al., 2023; 3B), two open-source encoder- decoder models based on T5 (Raffel et al., 2020) and trained on cnltructtoo-followinn datasets.

---

MT
DG
SC
CROW
CROW
Models
Zh-En
En-Fr
Eo-De
Be-Ro
Score (-MTT)
Score
Majurity
33,3/00
33.3/00
33,370
33.3700
40.1/ua
428/00
336/00
36.5/U0
41.3/0a
3x9700
364/00
49.5/25.7
50.8/253
51.7/259
47.7/225
45.519.6
5..366.5
5..570.8
48.7
L1aMA-78
49.1000
48.7/0.7
53.2/49
57./008
29.9/0.
41.3/0.
46./11
468/111
11aMA-138
5.7/117
504/79
20
576/18 18
$2.7/0.5
41.5/0.
44.624
45.6/2.3
LLMMA-33B
5..5/12
50.122
48,2/778
57.1/0.
44,174.1
42,4/12
48.5/3.1
48.8/288
Flan-T5-11B
45.57
76.5/516
76.11439
71.2/58.2
Alpuca
56.07
48.5/996
55.676.6
12.8
Flan-Alpaca
60.5/25.5
62.31 26-4
52.31 18.7
72.2/43.8
75.0//114
78.2/45.7
68.0/3122
66.7 /30.3
Vicuna
61.3/26.8
60.6/ 20
64.6//2..2
64.5/24..
6544 14.0
68.5/
64.7/22.0
64.1/22.8
Vicuna-Veuma
83.5789
52.0 11.5
38.6773
8..6789
72.8 20,9
59.9 17.4
5667 13.1
56.1 12.4
LS
44./18
14
40.8/04
45.2/2.5
47.616.3
BloomZ-7B
4.4/882
45.0134
462/112
49.9124
49.8/7.5
414466.7
58.8/
67.6/8.5
64.2/21.2
565/11.8
72.1
PaLM--1-5401
52.71577
50.204
503/90
503/00
63.4/
61.21 20.2
51.37 19.1
4..5/7.7
70.4/323
592/20X
55.4 12.2
GPT:3,5
66.6/38.7
50.17 182
50.5/18.
48.9 13.2
67.6/36.
68.7/31.9
67.7/36.0
65.6//40.0
76.4/41.7
7321/392
64.7 /3055
GPT-4
75.9/57.9
54.5/215
51./200.5
56.17 19.7
89.7/519
GPT-4-CoT
71.6/.52.2
64.7/426
/7.1
$7.3/3000
55.3/22.8
88.6 70.6
84.3/ 60L7
87.8/447.3
84.0/570
800771.7
72.3 /6.4
Humon"
87.9/78.0
83.07 82.9
89.9/ 82.0
89.9/ 86.0
87.07 86.9
98.9/ 96.4
88.17 69.5
97.87 93.9
93.9/ 80.7
93.1/85.5
90.7 84.0
Table 2: Macro-F1 Situational Accuracy (i.e., results aggregated per context instead of per sample) for all examined models across CRoW tasks. The performance of the highest scoring model is bolded for each task. *Due to the cost of expert evaluation, our Human study is only evaluated on 100 instances per task.
BloomZ-7B and mT00xxl (Muennighoff et al., 2023), two open-source instruction-follovinn multilingual language models of 7.1B and 13B parameters, respectively. The former is decoder- only model fine-tuned from BLOOM (Scao et al., 2022) while the latter is an imcoder-decode fine- tuned from mT5 (Xue et al., 2020).
human evaluation results.
Results
Table 2 reports the results for all models across all tasks. In general. we observe that models vary in their ability to correctly identify the correct re- sponses in the tasks. As expected, GPT-4 outper- forms most other models, many of which actually perform worse than the random baseline (e.g., all LLaMA variants). Even among stronger models, though, while performance is higher for individual examples (as measured by Macro-FI), the situa- tional accuracy is significantly lower, often below 50%. This gap suggests that these models are not robust and fail to grasp full situational understand- ing of the contexts with which they are presented (even as they may correctly classify some individ- ual cases). In contrast, humans tend to perform well on both metrics (with little gap between indi- vidual example performance and situational accu- racy). Perhaps most surprisingly, our results show that chain-of- tf-thughtt harms the performance of GPT-4 on some of the tasks, particularly on the Di- alogue task (DG) where the performance drops by 17. 1% in Macro-FI and 23.7% in Situational Accuracy (underlined in Table 2). This behavior perhaps hints that chain-of- of-huuht decoding is less useful in commonsense tasks requiring implicit, in- tuitive inferences, rather than complex, multi-step reasoning. In the Analysis section, we provide more details on the possible causes for the discrep- ancy in performance with examples.
All models are evaluated using one-shot in-context learning and greedy decoding. We use the same task-specific prompt templates for all models.? We also report the performance of random baseline that randomly chooses whether a context and target pair is valid, and majority baseline, which selects the most frequent label for each task.
Human Evaluation. We evaluate the human per- formance on each task of the benchmark using two expert annotators who evaluate 100 random samples from the task. Our experts are NLP re- searchers from our lab who were not involved in the original data collection. As result, they are more experienced, can clarify misunderstandings in the annotation guideline with us, and generally pro- duce more careful annotations than crowd workers. Following Amidei et al. (2018) and Oortwijn et al. (2021), we intentionally allow evaluators to discuss and reach final answer in cases of disagrecment, which reduces variance and yields a robust upper bound for our task. In Appendix D, we provide further details on the number of resolved disagree- ments, the human performance before and after the discussion, and the statistical significance of the
Further results with varying temperature values are Appendix "More details on prompt templates are in Appendix C.
Instruction-tuning.
Models that were trained only with language modeling objectives (e.g.,

---

Dimumion
7 Analysis
Attibution
Paysical
Toupecal
Cural
Social
Compariion
Plan-Alpacu*
70.2
12.2
68.0
70.5
723
73.5
Dimensions of Commonsense Knowledge. Ta- ble 3 reports the performance of different models across different commonsense knowledge dimen- sions. We observe that these models perform fairly consistently across different examples grounded by different commonsense dimensions, indicating that they do not generally learn more reliable com- monsense reasoning skills of one variety compared to another. Part of this uniformity is due to con- ceptual overlap between commonsense dimensions (e.g., certain social commonsense relations" may also reflect causal commonsense), nuance that is not captured by our methodology that requires annotation of single relation for commonsense knowledge. Regardless of this overlap, our findings highlight that, in real-world tasks, there remains room for commonsense modeling improvement for most types of commonsense knowledge.
Flan-15-118
71.2
13.4
LLMMa-384
Stable Vicuna
55.5
573
56.9
55.2
38.9
35.4
BlomZ-78
53.0
56.0
514
52.4
51.0
51.1
PaLM-115408
54.3
65.3
GPT-4
74.4
19.1
71.2
73.2
72.6
7016
GPT-4CoI
73.4
72.0
690
71.7
73.1
74.0
Table 3: Macro-F1 scores averaged across common-
sense dimensions, all tasks except for MT)
Model
Oraele Knowledge
No Knowledge
Flan-TS-IIB+
77.9 48.88
76.4743.9
BloomZ-7B
52.0 8.9
52.1 /83
GPT-4
74.5 47.6
72.7 45.00
GPT-4-ooT
73.9
72.3 46.4
Table 4: Macro-F1 / Situational Accuracy scores aver- aged all tasks ( all tasks except MT), with and without providing oracle commonsense knowledge as part of the prompt.
LLaMA and PaLM) obtain lower scores compared to instruction-tuma tundd models of similar size. For example, Alpaca, which is an instruction seneeidion-teeddddee ver- sion of LLaMA-7B, achieves an average ~ 10% improvement compared to LLaMA-7B across most tasks for both metrics. Also, smaller instruction- tuned models can perform similarly or exceed the performance of much larger models (e.g., GPT-3.5 outperforms PaLM). Finally, we find that Stable- Vicuna surprisingly performs worse than Vicuna, suggesting that while instruction-tundt performance on CRoW, training with RLHF does not necessarily amplify the commonsense reason- ing abilities required for these tasks.
Can models leverage explicit commonsense knowledge? When constructing CRoW, notate the implicit commonsense relationship quired understand whether target is valid invalid given a context. To test whether models can leverage this implicit commonsense relationship for making correct prediction, we perform an ora- cle experiment where we augment the prompt with each example's annotated commonsense knowl- edge triplet. Table shows the impact of adding commonsense knowledge to the prompt for the best closed-source (GPT-4) and open-source (Flan-T5 and BloomZ.7BB models. Prompting with com- monsense knowledge slightly increases the aver- age performance of Flan-T5 (+1.5% Macro-F1, +4.9% Sit. Accuracy) and GPT-4 (+1.8% Macro- F1, +2.6% Sit. Accuracy), indicating that even with partial commonsense knowledge, models still fall short of robustly reasoning about the pre- sented situations. However, the largest improve- ment observed for GPT-4-CoT, indicating that the model can more accurately ground its own rea- soning traces when provided with hints.
improves
Scale. When we compare the same model with different scales, we do not find consistent ben- efit to increasing the size of the model, except on the safety detection task, where LLaMA-33B achieves 14.2% and 11.4% improvement score over LLaMA 7B and 13B, respectively.
Multilinguality. Most of these models are offi- cially monolingual, though they may have been pretrained on some non-Engte non-English data. Since one of our testbed tasks centers on machine translation, evaluate multilingual models our benchmark. BloomZ performs better than mTO across most tasks. Certain monolingual models outperform BloomZ on translation tasks (i.e., those with 100B parameters), suggesting these models have seen multilingual data during their pretraining phase.
Qualitative Analysis. Following up on the lower performance of GPT-4 with CoT prompting com- pared to GPT-4, we qualitatively analyze the errors made by the model in the CoT setting for poten-
We used relations from existing KGs, see Appendix "We can not assume that the annotated relationships are the only pieces of commonsense knowledge needed to reason about the solution.

---

OPT-4QOTT
Dialogue
Seep Idendily the nan topics disegur..
A: Did you see May today?
B: Yes, but why doos she have such long face?
Maa's sppected suppenee
May' misd
A: don't have
idea.
This resgonce mont Mhy getlin fied, which i.o now tople rideed Mey's mooe.
thought be happy.
Step Determane repporse plauntty follow dialegue
lic
pfrose 'ospodaall cince she firet rocortty doortidddllyy locause typicari potling fired wcu nesoan somerne be tapps: reppono sormantat contaaingg burt could th sarced renass
Finol Answar <Aneseer Mereeeen
Knowledge
getting promotion
Implies
being happy
Stop A Analyoo the dakogas
Paner adda Person May loday, Pherae wonders why kado unthappy Plrnor
Response
Stop A Anelyos the knowwedgge
The broclecio rodae patling promotion unud impliss happlnwe..
A: Yeah, especially since she got fired recently.
Stap Check plasaibiltt
Person rospontss
zut kiy should especially since recenils
stould make May
states that lnnd, which regative event could musiineslle why May unissiy respres conasiite snowletgge dolloy contect
GPT-;: Not plausible
Final Anewer Naravaen-r
Figure 4: Answers generated by GPT-4 in various settings for example from the Dialogue task. While chain-of-thought reasoning yields incorrect answer, adding the commonsense knowledge guides the model toward the correct answer. Note that since the knowledge annotated for the original, plausible response, "getting a promotion" is longer relevant to the annotated, implausible response, which was modified with "got fired".
tial patterns in the dialogue generation task. In many cases, the reasoning process of the model ei- ther focuses solely on the relevance of the response (rather than its sensibility), or, in some cases, fol- lows less plausible reasoning path, such as imag- ining a sarcastic response. In Figure 4, show an example where GPT-4 correctly answers with- out chain-of-thought,, but fails when prompted to "think step by step." arguing that the response is sarcastically plausible (blue box). While such a response could technically be sarcastic, it violates our commonsense idea of what would be rea- sonable response to helpful query. On the other hand, we also observe the direct effect of providing the oracle commonsense knowledge (green box) on the same example where GPT-4 leverages the given knowledge and makes distinction between sarcas- tic possibility and commonsensical plausibility. In Appendix Figure 5, we provide another example where GPT-4 with chain-of-the chain-of-thought reasoning sim- ply ignores the inherent contradiction created by the commonsense knowledge violation and focuses on the surface-level relevance of the response.
cally crowdsource Winograd- schemas based on commonsense-violating minimal perturbations. Our evaluation of recent large language models on our benchmark shows that the performance of state-of- tifte-of-theattt models still falls far below human performance with respect commonsense reason- ing in real-world contexts.
Limitations
Despite efforts to build comprehensive bench- mark, CRoW faces several limitations. First, com- monsense knowledge has many dimensions, and we only consider six core ones as a basis for our commonsense knowledge annotation stage: tempo- ral, causal, attribution, comparative, physical, and social. Second, as we employ crowdsoureing for generating final Winograd schemas, our benchmark is susceptible data quality issues, annotation tifacts and biases. Lastly, in experiments, do not perform prompt tuning. As GPT-3/4 have been found be sensitive prompt construction, performance may vary when using other prompts for the same task.
8 Conclusion
Acknowledgements
In this work, we propose CRoW, multi-task commonsense reasoning benchmark consisting of six real-world tasks. To construct our benchmark, we design a data collection pipeline to systemati-
We thank Yonatan Bitton for his feedback and con- structive suggestions. Access to PaLM was granted through the Google-mediated academic LLM col-

---

laboration program, and we thank Shruti Sheth for her support. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29) the EPFL Science Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AL. Moreover, we thank the following partici- pants in our expert human evaluation task: Beat- riz Borges, Maria Glarou, Fawzia Zeitoun, Spiros Chalkias, Badr Alkhamissi, Karina Halevy, Khanh Nguyen, Ghali Chraībi, Li Mi, Hnagyu Yu, Julian Schnitzler, Alex Rubahn, Aurelio Noca, Arina Rak, Nikita Andreev, Sepideh Mamooler, Soyoung Oh, Angelika Romanou, Silin Gao, Khai Loong Aw.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074, Online. Association for Computa- tional Linguisties.
Yew Ken Chia, Pengfei Hong, Lidong Bing. and Sou- janya Poria, 2023. Instructeval: Towards holistic evaluation of Instruction-tunel large language mod- els. arviv preprint ar arXiv:2306.00 230.:2330
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yejin Choi. 2022. The curious case of commonsense intelligence. Daedalus, 151:139-155: 155.
References
Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3318-332 3329, Santa Fe. New Mexico, USA. Association for Com- putational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya. Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia. Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira. Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Teck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Yehoshua Bar-Hillel. 1960. A demonstration of the non- feasibility of fully automatic high quality translation.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. ArXiv, .5..0999
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Edd H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le. and Jason Wei. 2022. Scaling instruction-finetue language models.
Michael Chen, Mike D' A'ery Alisa Liu, Jared Fer- nandez, and Doug Downey, 2019. CODAH: An eeeessariall-authhree
Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics:
question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP, pages 63-69, Minneapolis, USA. Association for Computational Linguistics.

---

Human Language Technologies, Volume (Long Pa- pers), pages 3595-1604, New Orleans, Louisiana. Association for Computational Linguistics.
Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 301-313, Singapore and Online. Association for Computational Linguistics.
Ernest Davis. 2023. Benchmarks for automated com- monsense reasoning: A survey.
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022. CICERO: A dataset for contextualized commonsense inference in dialogues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5010-5028, Dublin, Ireland. Association for Computational Linguistics.
Emest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58(9)922-103
Nam Do and Ellie Pavlick. 2021. Are rotten apples edi- ble? challenging commonsense inference ability with exceptions. In Findings of the Association for Com- putational Linguistics: ACI-IJCNLPP 2021, pages 2061-2073, Online. Association for Computational Linguisties.
Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020. The box is in the pen: Evaluating commonsense rea- soning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3662-3672, Online. Association for Computational Linguistics.
Julian Martin Eisenschlos, Jeremy R. Cole, Fangyu Liu, and William W. Cohen. 2023. WinoDiet: Probing language models for in-context word acquisition. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 94-102, Dubrovnik, Croatia. Associa- tion for Computational Linguistics.
Lifu Huang. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing JEMNLL-IJCCLL pages 2391-2401, Hong Kong. China. Association for Com- putational Linguistics.
Denis Emelin and Rico Sennrich. 2021. Wino-X: Multi- lingual Winograd schemas for commonsense reason- ing and coreference resolution. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8517-8532, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet- 2020: On symbolic and neural commonsense knowledge graphs. AAAA Conference on Artificial Intelligence.
Saadia Gabriel, Skyler Hallinan, Maarten Sap. Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning about readers' reactions to news headlines. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 3108-3127. Dublin, Ireland. Associa- tion for Computational Linguistics.
Filip Ilievski, Alessandro Oltramari, Kaixin Ma, Bin Zhang, Deborah L. McGuinness, and Pedro Szekely. 2021. Dimensions of commonsense knowledge.
for
Seungone Kim, Sehrang Joo, Hyungjoo Chae, Chae- hyeong Kim, Scung won Hwang, and Jinyoung Yco. 2022. Mind the gap! injecting commonsense knowl- edge for ahstractive dialogue summarivation. In In- ternational Conference on Computational Linguis- tics.
Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, and Antoine Bosselut. 2022. Com- Fact: A benchmark for linking cualextual common- sense knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1656-1675, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Hector J. Levesque. Ernest Davis, and L. Morgenstern. 2011. The winograd schema challenge. In Interna- tional Conference on Principles of Knowledge Rep- resentation and Reasoning.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019, Are modeling the task or the annotator? inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, and William Yang Wang. 2022. SafeTex:: benchmark for exploring physical safety in language models. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 2407-2420 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020a. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre- Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. CIDER: Commonsense inference for dialogue expla- nation and reasoning. In Proceedings of the 22nd

---

Language Processing (EMNLP). pages 6862-6868, Online. Association for Computational Linguistics.
of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21 21(4))5485-5511
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020b. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Computu- tional Linguistics: EMNLP 2020, pages 1823-1840) Online. Association for Computational Linguistics.
Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018a. Modeling naive psy- chology of characters in simple commonsense stories. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2289-2299, Melbourne, Aus- tralia. Association for Computational Linguistics.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Unicorn on rainbow: universal commonsense reasoning model on a new multitask benchmark. ArXiv, abs/2103.13009,
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018b. Event2Mind: Com- monsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics.
John McCarthy. 1960. Programs with common sense.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Rose Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hai- ley Schoelkopf, Xiangru Tang, Dragomir R. Radev, Alham Fikri Aji, Khalid Almubarak. Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Annual Meeting of the Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Trunsactions of the Association for Com- putational Linguistics, 7:249-266.
Christopher Richardson and Larry Heck. 2023. Com- monsense reasoning for conversational ai: A survey of the state of the art.
Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman. 2021. What ingredients make for an effective crowdsoure- ing protocol for difficult NLU data collection tasks? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1221-1235, Online. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning.
Rachel Rudinger, Jason Naradowsky. Brian Leonard. and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In North American Chapter of the Association for Computational Linguistics.
Yvette Oortwijn, Thijs Ossenkoppele, and Arianna Betti. 2021. Interrater disagreement resolution: system- atic procedure to reach consensus in annotation tasks. In Proceedings of the Workshop on Human Evalu- ation of NIP Systems (HummEval), pages 131-141, Online. Association for Computational Linguistics.
Swarnadeep Saha, Prateck Yadav, Lisa Bauer, and Mohit Bansal. 2021. ExplaGraphs: An explanation graph generation task for structured commonsense reason- ing. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing. pages 77 7716-7740, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
OpenAl. 2023. Gpt-4 technical report.
Baolin Peng. Chunyuan Li. Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint 2300023004.03277.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga- vatula, and Yejin Choi. 2019. Winogrande: An ad- versarial winograd schema challenge at scale. ArXiv, abs/1907.1064
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIME- DIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019a. Atomie: An atlas machine commonsense for if-then reasoning. In AAAI Conference on Artifi- cial Intelligence.
I 1tth International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 7066-7076, Online. Association for Computational Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits

---

9th International Joint Conference on Natural Lan- guage Processing 4473, Hong Kong, China. Association for Computa- tional Linguisties.
Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltomozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez--llumed Francesca Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang, Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannanch Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong. Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coú, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jil- lian Tang. Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Joseph Boudeman. Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan. Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi, Kory Mathewson, Kristen Chi- afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando. Louis-Philiipe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng Hc, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana. Marie Tolkiehn. Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátýás Schu- bert, Medina Orduna Baitemirova, Melody Amnad, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt, Michael Strube, Michat Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts. Nick Doiron, Nikita Nangia, Niklas Deckers. Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Tyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao. Percy Liang, Peter Chang. Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei. Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Euta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphael Millière,
pages 4463-
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yon,, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-aceess multilingual language model. arXiv preprint arXiv:2211.05100.
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormo- labashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Find- ings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 883-898, Online. Associa- tion for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2016. Concepinet 5.5: An open multilingual graph of gen- cral knowledge. ArXiv, abs/ 122.0995..
Aarohi Srivastava, Abbinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai. Andrew La. Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş. B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Barthomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf. Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez. Clara E. Rivera. Clemencia Siro, Colin Raf- fel, Courtney Ashcraft, Cristina Garbacea, Damien Silco, Dan Garrette, Dan Hendryeks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes. Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Elkate- rina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, El- lie Paylick, Emanuele Rodola, Emma Lam, Eric Chu,

---

Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras, Rosanne Liu, Rowan Jacobs. Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff. Sebas- tian Gehrmann, Sehastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer. Simone Melzi, Siva Reddy, Sneha Priscilla Makini. Soo-Hwan Lee. Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stwart M. Shieher, Summer Misherghi, Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster. Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes. Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick. Timofei Kornev. Timothy Telleen- Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Sha- ham, Vedant Misra, Vera Demberg. Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang. Wout Vossen, Xiang Ren, Xiaoyu Tong. Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4149 -458, Minneapolis, Minnesota. Association for Computational Linguistics.
Ronen Tamari, Kyle Richardson, Noam Kahlon, Aviad Sar-shalom, Nelson F. Liu, Reut Tsarfaty, and Dafna Shahaf. 2022. Dyna-bAbl: unlocking bAbl's po- tential with dynamic synthetic benchmarking. In Proceedings of the 11th Joint Conference on Lexi- cal and Computational Semantics, pages 101-122, Seattle, Washington. Association for Computational Linguisties.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An enstruction-oolloinng llama model. https: gi thub. wwt.b.ccootteaeeeeeeeeeeeeeeeeea coon/tatsu-labbe labuuulllabattataaa http//sddfodddappaaaaa
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for Disio-linguistir compositionaliiyy 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5228-5238.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Alex Wang. Yada Pruksachatkum, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. In Neural Information Processing Systems.
Shane Storks and Joyce Chai. 2021. Beyond the tip of the iceberg: Assessing coherence of text classifiers In Findings of the Association for Computational Linguistics: EMNLP 2021. pages 3169-31777 Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019b. Does it make sense? and why? a pilot study for sense making and explana- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020 402004026,,60 Florence, Italy, Association for Compu- tational Linguistics.
Shane Storks, Qiaozi Gao, and Joyce Yue Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv: Computation and Lunguage.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Chai.
2021. Tiered reasoning for intuitive physics: Toward
verifiable commonsense language understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4902-4918, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Terry Winograd, 1974. Understanding natural language.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1193.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Co A ques-

---

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Conference on Empirical Methods in Natural Language Process- ing.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791-4800 Florence, Italy, Association for Computational Linguisties.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. ArXiv, abs/1812 11118810.88..
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going vacation" takes longer than "go- ing for a walk": study temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China. Association for Computational Linguistics.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayat- nia, Scokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2021. Commonsense- focused dialogues for response generation: pirical study. In Proceedings of the 22nd Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 121-132, Singapore and Online. Association for Computational Linguistics.
em-

---

A Commonsense Knowledge Dimensions
custom relationship (i.e. Other relation). How- ever, annotators, most of the time tended to choose one of the available relations. In addition, anno- tators were incentivized to specify multiple com- monsense knowledge relationships with an added bonus. Depending on the difficullyy of the task, we pay workers $0.5 $1 per HIT in the annotation stage and $0.2 $0.5 per HIT in the validation stage. Instruction and task templates (for Dialogue task) can be found in Figures 6, 7, 8. Templates for other tasks are similar.
We consider widely used commonsense knowl- edge bases such as ConceptNet (Speer et al., 2016) and ATOMIC (Sap et al., 2019a), well as re- cent works such ATOMIC2020 (Hwang et al., 2020) and CIDER (Ghosal et al., 2021) for select- ing the commonsense relations. As initial step. we manually categorize the kind of knowledge re- lations that appear for each task. Among the to- tal 56 relations available to us from these sources, we find 22 relations from ConceptNet (out of 36), 8 relations from ATOMIC (out of 9), 3 relations from ATOMIC2020 (out of 5) and 3 relations from CIDER (out of 6) appearing most commonly. Fol- lowing (Ghosal et al., 2021; Ilievski et al., 2021), we further categorize these 36 relations into 6 com- monsense knowledge dimensions. Table 5, we list the available relations for each dimension with a brief description and an example. Each relation- ship is represented as (A, Relation, B) where A and B refer to phrases from the context.
B.3 Winograd Schema Generation and Validation
The second main stage of our data collection pipeline is the generation of Winograd schemas using the commonsense knowledge annotated in the CKA stage. As mentioned before, for a given context and a piece of commonsense knowledge, ask annotators generate alternative targets that satisfy four conditions: (1) the new target is minimally different from the original one, (2) the new target violates the given commonsense knowl- edge, (3) new target is incorrect for the given task, and (4) new target is contextually relevant. The incorrectness is defined per task (See Appendix C). As example, consider the following instance from the Safety Detection task:
B Data Collection
At a high level, in the CKA stage, the given ex- ample is annotated with relevant commonsense knowledge, and in the WSG stage, the example is rewritten with Winograd-style minimal pertur- bation based on the commonsense knowledge from the previous stage.
Scenario: If you are being chased by a hungry
animal
B.1 Qualification
Action: get in building and close the door Knowledge:
This stage includes six multiple-choice questions (2 per task for dialogue, dialogue summarization and machine translation tasks) about identifying the cor- rect implicit commonsense knowledge in given context and one open-ended question that simulates the CKA stage for dialogue task. We compensate workers $2 per HIT for this stage. Instruction and task templates can be found in Figures 13, 14, 15,
building HasProperty safe spot close the door CapableOf blocking animal Label: safe
Given the example above, our goal is to modify the target (i.e., action) such that it satisfies the condi- tions above. Here we provide some possible mod- ifications and show whether the resulting target satisfies the conditions:
B.2 Commonsense Knowledge Annotation and Validation
In the first stage of the pipeline, we ask annota- tors to identify commonsense knowledge in given context for a task. They are asked to either se- lect phrases from the context write their own phrases about a context and connect these phrases with one of the commonsense knowledge relations provided (Table 5). If none of the relations de- scribe the commonsense knowledge they would like to express, they are given the option to specify
find spot where the animal can get to you: This target satisfies all conditions except the (I) as it significantly modifies the original tar- get.
get a gun and shoot the animal: While this target good one, it does not really violate the given commonsense knowledge. It also not minimally different from the original one.

---

get in a building and lock the door: This target not valid, because the action is still safe (in fact, even safer), so condition (3) is violated.
as the unchanged part of the given example (i.e., the part that is not perturbed in the WSG stage) and the target as the candidate for Winograd-style perturba- tion. Note that the target is not necessarily always the typical output of the model for a given task, For example, in classification tasks, the output of the model is binary while the larget is assigned to one of the inputs. For context and target assignments for each task, please refer to Table 6.
get in an alien spaceship and close this chap- ter of your life: While this target seems to satisfy most conditions, it is largely out of context, so condition (4) is violated.
find a building and close your eyes: This target satisfies all the conditions above.
C.1 Machine Translation (MT)
Depending on the difficulty of the task, workers were paid $0.5 $0.8 per HIT in the generation stage and $0.2 $0.5 per HIT in the validation stage. Instructions and task templates (for Dialogue task) can be found in Figures 10, 11, 12. Templates for other tasks are similar.
Machine translation is one of the oldest sequence- to-sequence real-world NLP tasks where given in- put in the source language, a system is expected to output the translation in the target language. In this task, we define the input in the source language as context and the output translation as the candidate largel for perturbation. As both datasets include Winograd schemas, we skip the WSG and WSV stages for this task. In the CKA step for the Chinese-Engliss dataset, annotators are given the target translations and asked identify the com- monsense knowledge violated in the incorrect trans- lation. For the Wino-X dataset, although the target translation is in different languages, the underly- ing pronoun resolution task allows us to identify the commonsense knowledge from the source sen- tence alone. Leveraging this fact, we employ the same English-speaking pool of qualified workers for the CKA stage on this task as well. More specif-- ically, annotators are given the source sentence (in English) and asked to identify the commonsense knowledge that allows us to infer the antecedent of the pronoun "it" in the sentence. Here is an example of this task:
B.3.1 Instructions
Minimal Change Rules During the WSG step of the annotation pipeline, the generated alterna- tive sentence has to be deceptively close to (hard to differentiate from) the original sentence, but op- posite of it in terms of commonsense knowledge or label. Thus, we asked annotators to follow a set of rules, that we report here (from the Safety Detection task):
use
You are allowed to change up to 5 words in the action. Note that you can also alternatively swap the existing words in place instead of replacing them long as the resulting action satisfies the conditions above.
You should avoid simply negating the word in the action unless that is the only way to achieve the goal. The goal in this task is not to achieve the minimal difference, but to produce semantically very close action that however flips the safety value. So, if you can creatively change dependent words to achieve this, then go for rather than changing one word such as adding/removing "not".
Example Machine Translation:
Sentence (English): Bob would rather fill his emergency fund using his mobile instead of the bank because it was handy.
Your change should directly target the given commonsense knowledge such that, seman- tically, the resulting action differs from the original one with respect to this knowledge.
Translation (French): Bob préférait remplir son fonds d' urgence en utilisant son mobile plutôt qu' avec la banque car elle était à portée de main. Knowledge:
C Tasks
mobile Capable0f store emergency funds Label: not correct (No)
For each task we have included in the benchmark, we define a common terminology that can be ap- plied to other tasks in the future. Context is defined
We use the following prompt templates:

---

Prompt Machine Translation:
Prompt Machine
Translation
CoT
You are a helpful assistant for translation from {source_lase | to [target_lang] Given a sen- tence in [source_lang] and its translation in [target_lang), answer whether the translation is correct. Answer only Yes or No. Example I:
Example:
Sentence (English):
The song took longer to sing than the ballad because it was more words.
Translation (German): Das Lied brauchte länger zum Singen als die Ballade, weil sie mehr Worte enthielt.
Sentence ([source [anng): (sentence)
Translation (ttarett_aagg) (translation)
Answer:
Answer: [answer]
Let's break down the sentence and its transla- tion:
Example 2:
The song: Das Lied took longer: brauchte länger- to sing: zum Singen than: als the ballad: die Ballade because: weil it: sie was more words: mehr Worte enthielt (literally: contained more words)
Prompt Machine Translation CoT:
You are a helpful assistant for translation from [source_lang lang]] to Marget_langl Given a sen- tence in ssource_lange and its translation in (target_lang), answer whether the translation is correct. Let's work this out in a step-by-ste step way to be sure that we have the right answer. Then provide your final answer within the tags, <Answer> Yes/No</Answer>
The translation is almost correct, but the pronoun for 'it' should be "es" instead of 'sie', This is because 'it' should refer the thing that has more words and since 'song' took longer to sing than 'ballad', 'song' should contain more words, and the correct pronoun for "song' is 'es', not 'sie'. So the translation is not correct. <Answer>N<<Answer>
Example 1:
Sentence (lsource_lang)): (sentence)
Translation ((tagget_langl) (translation) Answer: ]
C.2 Dialogue (DG)
Example 2:
The underlying task in Dialogue is generate response given a dialogue history. In this task, we define the context the dialogue history and the larget as the response this context. We skip the CKA and CKV stages for this task as the dataset wc use comes with expert annotated commonsense knowledge annotations. To ensure the richness the context and the knowledge, we filter out dia- logue contexts with less than 4 turns and common- sense knowledge annotations that do not connect different turns in the dialogue. In the WSG stage, ask the annotators rewrite the final response of the given dialogue such that it satisfies our con- ditions for Winograd schemas mentioned above where the incorrectuess is defined as implausibility. Since in an open-domain dialogue, several answers are possible for given dialogue history, we aim for generating answers that violate some common- sense knowledge about the dialogue and hence, are implausible. However, since most of the com- monsense knowledge dialogues are contextual, violating this knowledge does not automatically make the response implausible, hence we explicitly enforce a separate condition to ensure the implau-
Prompt Machine Translation Common- sense Knowledge:
You are a helpful assistant for translation from (source_lang) to
Given sen- tence in (source_ _ange,____nnn___ its translation in [tar- get_lang) and relevant knowledge, answer whether the translation correct. Answer only Yes or No.
Example 1:
Sentence ((source_lang)) (sentence)
Translation (ttaget_lang]]] [translation)
Knowledge: (knowledge)
Answer: (anwerr)
Example 2:
The template for Commoonmeee Knowl- edge is similar. We use the following example as a one-shot example for CoT prompt of this task:

---

sibility. For example, given the following dialogue A: where will you have your birthday party? B: oh at my uncle's s house, the contextual com- monsense knowledge can be the fact that (parties, AtLocation, uncle's house). Consequently, possible Winograd schema generated by violation of this knowledge could be B: oh it is at my friend's house. However, this is not correct Winograd schema for this task as it is perfectly fine response the dialogue. The implausible response here should tar- get the more general commonsense knowledge that "parties happen at people's houses". addition, we also ask annotators to avoid generating examples that are implausible independent of the dialogue context to make sure generations are not too easy for models to guess even in the absence of context. Here is an example of this task:
Prompt Dialogue CoT:
You are helpful assistant for dialogue under- standing. Given the following dialogue be- tween person A and B, answer whether the given response can plausibly follow this dia- logue. Let's work this out in step-by-step way to be sure that we have the right answer. Then provide your final answer within the tags, <An- swer> Yes/No<<AAnswer>
Example 1:
Dialogue: (dinno:dd)ll
Response:
(remponsee response]
Answer: (answer)
Example 2:
..
Example Dialogue:
Dialogue: A: Good morning, sir. Is there any-
thing can do for you?
B: would like to buy two bottles of brandy.
A: How about this one? It's the special local product.
Prompt Dialogue + Commonsense Knowl- edge:
B: Can I buy these tax free?
You are helpful assistant for dialogue under- standing. Given the following dialogue be- tween person A and B and a relevant knowledge about this dialogue, answer whether the given response can plausibly follow this dialogue. An- swer only "Yes" or 'No'. Example 1:
Response: A: Yes This is not duty-free shop.
Knowledge:
duty-free shop Implies tax free
Label: Not plausible (No)
Dialogue: (dialogue)
We use the following prompt templates:
Response: [response]
Knowledge: (knowledge)
Answer: noweer:faswer [answer]
Example 2:
Prompt Dialogue:
You are a helpful assistant for dialogue under- standing. Given the following dialogue between person and B, answer whether the given re- sponse can plausibly follow this dialogue. An- swer only 'Yes' or 'No'.
Example I:
Dialogue: {dialogue}
Response: (response)
Answer: (answer)
The template for CoT+CCmmonsense Knowl- edge is similar. We use the following example as a one-shot sample for CoT prompt of this task:
Example 2:

---

Prompt Dialogue- CoT Example:
Example- Intent:
Dialogue:
Headline: Hospitals on lockdown as first COVID vaccine patients start cating other patients.
A: Before Christmas Party) ) Are you ready for the Christmas party tonight
B: Almost. have to get dressed. It's formal party and I have special party make up! A: Use this lipstick and it will make your lips shine!
Intent: hospital is on lockdown due to covid patients kissing other patients after getting the vaccine.
Knowledge:
Response:
COVID vaccine Causes eating other patients
B: Great! Uh, remember that there's a rocket launch, too. We all have to bring a gift.
Answer:
Step I: Identify the main topics in the dialogue. Christmas party Getting dressed Formal party Special party make up- Lipstick
Step 2: Analyze the response.
We use the following prompt templates:
The response mentions rocket launch, which is related the main topies in the dialogue. The response mentions bringing a gift, which could be related to the Christmas party. Step 3: Determine if the response can plausibly follow the dialogue.
Prompt Intent:
The mention of rocket launch seems out of context and unrelated to the dialogue. In addi- tion the second part of the response mentions obligation bring a gift which wouldn't follow the first part as rocket launch event typically does not require to bring a gift. A plausible event would be a gift exchange event. So the response does not plausibly follow the dialogue. Final Answer: <AnswerNNo<AAsswer
You are helpful assistant for intent classifica- tion. Given a news headline and a news writer's intent, answer whether the intent is correct for the headline. Answer only Yes or No.
Example 1:
Headline: | headline)
Intent: (intent)
Answer: ( (answer)
Example 2:
Prompt Intent CoT:
You are helpful assistant for intent classifica- tion. Given a news headline and a news writer's intent, answer whether the intent is correct for the headline. Let's work this out step way to be sure that we have the right answer. Then provide your final answer within the tags, <Answer> YAnswer>Yes/Noo/Ansse Ycs/No</Answer..
C,3 Intent Detection (ID)
step-by-
In this task, we treat the text of the author as the context and the intent as the targer for perturbation. We use the headline our context and the writer for this
Example 1:
Headline: headline)
intent as the targel for the dataset task. The full pipeline is applied this dataset and as a preprocessing step, we filter out examples with too short hcadlines or intents. Here is an example sample for this task:
Intent: [intent)
Answer: (answer)
Example

---

Prompt- Intent Commonsense Knowledge: You are a helpful assistant for intent classifica- tion. Given a news hcadline, a news writer's in- tent and relevant knowledge, answer whether the intent is correct for the headline. Answer only Yes or No.
already annotated with commonsense knowledge, we skip the first steps of the pipeline CKA and CKV, Similarly to the other selected tasks, we filter the examples with short sentences. to give more degrees of freedom to crowdsource workers for the WSG step. Moreover, in this task, the con- text and the larget are dynamically chosen we treat both sentences (the belief and the argument) equally allowing workers to select the one modify. Here is an example of this task:
Example 1:
Headline: (headline)
Intent: ]intent]
Knowledge: knowledge)
Answer: [answer]
Example 2:
Example Stance:
Belief: Cosmetic surgery should not be banned.
The template for CoTTonommeee Knowl- edge is similar. We use the following example as one-shot sample for CoT prompt of this task:
Argument: Cosmetic surgery is not worth the risk
Knowledge:
risky UsedFor human body
Prompt Intent CoT Example:
Label: Counter (No)
Headline:
Authorities will delay vaccines in Andalusia. They bought millions syringes that will not work to distribute the COVID-19 vaccine
We use the following prompt templates:
Intent:
the vaccine requires specific needles to apply Answer:
Prompt Stance:
You are a helpful assistant for stance classifica- tion. Given a belief and an argument, answer whether the argument supports the belief. An- swer only Yes or No.
Step 1: Analyze the headline
The headline states that authorities in Andalusia will delay vaccines because they bought mil- lions of syringes that will not work to distribute the COVID-19 vaccine. This shows that there is a incompatibility between the bought syringes and syringes required for the vaccine. Step 2: Analyze the intent
Example 1:
Belief: (belief)
Argument: (argument]
Answer: answer)
Example
The intent states that the vaccine requires spe- cific needles to apply. This means standard sy- ringes might not be suitable.
Step 3: Compare the headline and intent The headline implics that the syringes pur- chased are not suitable for distributing the COVID-19 vaccine, which aligns with the in- tent stating that specific needles are required to apply the vaccine.
Prompt Stance CoT:
You are helpful assistant for stance classifi- cation. Given belief and an argument, an- swer whether the argument supports the belief. Let's work this out in step-by-step way to sure that we have the right answer. Then provide your final answer within the tags, <An- swer> Yes/NNo</A/sswr..
So, the given intent is the correct one for this headline. Final Answer: <An- swer> Yes</Answer>
Example 1:
Belief: (belief)
C.4 Stance Classification (SC)
Argument: (argument]
Answer: (answer)
Stance classification is task where given belief and an argument, the stance of the argument is pre- dicted. Since the dataset we chose for this task is
Example

---

Prompt Stance Commonsense Knowl- edge:
Example Safety:
Scenario: When heating up leftovers in the mi-
You are a helpful assistant for stance classifica- tion. Given belief, an argument and relevant knowledge, answer whether the argument sup- ports the belief. Answer only Yes or No.
crowave
Action: wait for the plate to cool off before you grab it
Knowledge:
Example 1:
wait for the plate to cool off before you grab it xIntent avoid getting burnt
Belief: (belief)
Argument: {argumett)
Label: safe (Yes)
Knowledge: [nowledge))
Answer: [answer]
Example 2:
We use the following prompt templates:
The template for Cot+Commonsense Knowl- edge is similar. We use the following example one-shot sample for CoT prompt of this task:
Prompt Stance CoT Example: Belief:
Prompt Safety:
Cosmetic surgery should allowed.
You helpful assistant for safety detection. Given a real-life scenario and action, answer whether the action is safe do in this scenario. Answer only Yes No.
Argument:
Cosmetic surgery is not worth the risk.
Answer:
Step 1:
Analyze the belief
Example
According to the belief, cosmetic surgery should be allowed which might mean that it is not risky. Step 2: Analyze the argument The argument states that cosmetic surgery is not worth the risk, so assumes that there are risks involved, but is not worth to do while taking the risk.
Scenario: scenario)
(eenario:(senario))
Action: (alonn(action) [action]
Answer: (answer)
Example 2:
Step 3: Compare the belief and argument The belief supports cosmetic surgery, while the argument opposes it due to the risks involved. Final Answer: <Answer-No</Answer>
Prompt Safety CoT:
You are helpful assistant for safety detection. Given a real-life scenario and action, answer whether the action is safe do in this scenario. Let's work this out in step-by-step way to be sure that we have the right answer. Then provide your final answer within the tags, <An- swer> Ycs/No</Answer>> Example 1:
C.5 Safety Detection (SD)
The underlying task is to determine the safe action given a real-life scenario and a list of actions com- prised of safe and unsafe actions. In this task, we define the context the given scenario and target the possible action. We apply the full pipeline for this task where in the WSG stage, annotators are asked to either minimally modify the safe ac- tion to make unsafe or vere-versa. Here is am example sample for this task:
Scenario: scenario)
Action: (action)
Answer: (answer)
Example

---

Prompt-Safes Safety Commonsense Knowledge: You are a helpful assistant for safety detection. Given a real-life scenario, an action and a rele- vant knowledge, answer whether the action is safe to do in this scenario. Answer only Yes or No.
Answer: (answer) less than 5 content words. Here is an example of this lask:
Example I:
Scenario: (ccenaio))
Example Summarization:
Action: { action]
Dialogue:
Knowledge: knowledge)
#Person #: How may help you?
Answer: [answe]]
#Person2#: would like to return this book.
Example 2:
#Person Is that all you need?
#Person2#: also want to check out this video.
#Person Do you have your library card?
The template for CoTmonommeee Knowl- edge is similar. We use the following example as a one-shot sample for CoT prompt of this task:
#Person2#: Here it is. #Person If you damage the video, you will be fined.
#Person2#: won't damage Summary:
Prompt Safety CoT Example:
Scenario:
#Person |# helps #Person2# to return book and check out video in the card-free, honor-system library.
you're 'r on a hike in the woods and you see colorful mushroom.
Action:
Knowledge: check out video in the library DependsOn have your library card Label: not correct (No)
do not eat it.
Answer:
Step 1: Identify the potential risks in the see- nario
Consuming an unknown and hence potentially harmful mushroom found in the woods. Step Evaluate the action
The action is to not eat the colorful mushroom. Step 3: Determine if the action is safe in this scenario
We use the following prompt templates:
By not eating the mushroom, the potential risk of consuming poisonous harmful mush- room is avoided. So, the action is safe do in this scenario.
Final Answer: Yenswer>Yes</Answer>
C.6 Dialogue Summarization (DS)
Prompt Summarization:
You are helpful assistant for dialogue summa- rization. Given the following dialogue between #Person and #Person2#. answer whether the given summary correctly summarizes the dia- logue. Answer only 'Yes' or 'No'.
dialogue summarization, given a dialogue con- text, the system is expected to output either an ex- tractive or abstractive summary of the context. In this task, we designate the dialogue as the context and the summary as the target candidate for rewrit- ing. The full data annotation pipeline is upplied for this task where in the WSG stage, the incorrectness condition is satisfied if the rewritten summary does not correctly summarize the given dialogue. As filtering phase, we discard examples with dialogues that have less than turns or summaries that have
Example 1:
Dialogue: (dialogue)
Summary: (summary))
Example

---

Prompt Summarization CoT:
Prompt Summarization CoT Example:
You are helpful assistant for dialogue summa- rization. Given the following dialogue between #Person # and #Person2#, answer whether the given summary correctly summarizes the dia- logue. Let's work this out in step-by-step way to be sure that we have the right answer. Then provide your final answer within the tags, <An- swer> Yceescoo///Asswer
Dialogue:
#PersonI#: I'm going to New York for the first time, but don't t have tour guide. Can you give me any suggestions?
#Person2#:: There's 's service called 'A friend in New York'. It's a personal tour guide service. #Person ##: That's interesting. What does it do? #Person2#: You give them your information by answering a questionnaire and they will create perfect trip for you according to your budget. #Person Good. Where can get the question- naire?
Example 1:
Dialogue: (caaooue:diallguee ]
Summary: (ummary:(suumary) summary)
Answer: [answer]
Example 2:
#Person2#: You can easily download it from their website.
#PersonI#: That's helpful! Thanks!
Summary:
#PersonI# is going New York for the first time. #Person2# suggests #Person I use per- sonal tour guide service even though they won't know how to put together #Person I#'s trip plan. Answer:
Step 1: Identify the main points in the dialogue. #Person # is going New York for the first time and needs suggestions. #Person2# sug- gests "A friend in New York' service. The service creates a perfect trip based on question- naire. The questionnaire can be downloaded from their website.
Prompt Summarization Commonsense
Knowledge:
You are helpful assistant for dialogue summa- rization. Given the following dialogue between #Person # and #Person2# and a relevant knowl- edge, answer whether the given summary cor- rectly summarizes the dialogue. Answer only 'Yes' or 'No'.
Step 2: Compare the summary with the main points.
The summary correctly mentions that #Per- sonl# # is going to New York for the first time. The summary mentions the personal tour guide service, but incorrectly states that they won't t know how to put together #Person # trip plan because according the dialogue, the service can create a perfect trip based on the questionnaire. Final Answer: <Answer>No</Asswr>
Example 1:
Dialogue: (dialogue)
Summary: (summarr)
Knowledge: (knowledge)
Answer: } Example 2:
The template for CoT+Commonsense Knowl- edge is similar. We use the following example as a one-shot sample for CoT prompt of this task:

---

D Human Evaluation
lief statement, if the entrapment is no longer being done properly, then it no longer bad (contraposi- tive of the belief statement). This is the statement made by the argument, which is why the belief is supported by the argument. Without the subtle factual knowledge about entrapment, however, this example is challenging for human raters.
In this section, we provide more details on the hu- man evaluation results. As we allow human evalua- tors to discuss cases of disagreement, the number of resolutions and the human performance before and after the discussion are of interest as well. In Table 9, we report the percentage of resolved disagree- ments per task and the human results before and after discussion compared to the best- persfrrerormnnt model which is GPT-4, Performance numbers for human scores before discussion are calculated by treating each annotator as a different prediction for each example and computing the performance all predictions. If there is clash and one annotator is correct and the other is not, then that example would receive a human score of 0.5 for accuracy, After discussion, the annotators agree on the final prediction, so the agreed-upon label the predic- tion. As we can see from the table, the human performance is already high before the disagree- ments are resolved, with GPT-4 only exceeding the individual human performance stance classifi- cation. We note, however, that this is one of the tasks with the most initial disagreements. When discussed between both evaluators, however, they settle on labels and outperform GPT-4.
Disagreement Example Safety:
Scenario: If your pets don't get along well Action: let them play in large room together Answer: Yes (safe)
The above example is ambiguous because the ac- tion could result in both safe and unsafe scenario. The action statement specifies that the room is large, hinting that it provides enough space for the pets to be in the same room and that playing together may let them grow closer (a common way of introduc- ing pets each other). However, "pets don't 't get along well" could also mean they attack each other and fight, and putting them in any room together might lead to an unsafe situation, regardless of their size. Without contextual information to clarify, the example remains ambiguous human readers.
Statistical Significance We perform a Binomial Sign test on GPT-4 and Human evaluation results as sample size is small. We find that for Ma- chine Translation (en-ru, en-de and en-fr), Intent detection, Safety detection and Stance Classifica- tion tasks, results are statistically significant with a level of 0.01, for Dialogue and Summarization tasks with a level of 0.05. Only for Machine Trans- lation (zh-en) task, we obtain a high p-value (0.57).
Analysis of disagreements We observed differ- ent reasons for human disagreements, but the most common causes were: (i) instances requiring subtle factual knowledge that annotators may not recog- nize (see disagreement example below for Stance Classification), (ii) ambiguity and ppeneende (see disagreement example below for Safety Detec- tion)
Disagreement Example- Stance:
E Additional Results
Belief: Entrapment is bad if done properly. Argunent: The bad parts of entrapment can be climinated if the criminal can be given the opportunity to refuse.
In this section, we provide the results of further experiments did with varying hyperparameters. In Tables 10, 7 and 8, we report the averaged re- sults of three runs where the temperature parameter is set to 0.3 respectively for all baselines, compar- ison over commonsense knowledge across tasks and with or without oracle knowledge. In Table 11, we report the results of GPT-4 with varying tem- perature values compared to human performance. In Table 12, we report the results of GPT-4 with chain-of- thouhtt reasoning in various scenarios. including with self-consistency decoding (Wang et al., 2023). Overall, none the scenarios we observe a particularly different performance of the
Answer: Yes (supporting)
Along with making commonsense inferences, the above example requires subtle factual knowledge about entrapment, namely that entrapment requires the criminal to not be aware that they are being en- trapped. As a result, the criminal is given the op- portunity to refuse, then commonsense tells us that the entrapment is no longer being done properly since the criminal is aware. According to the be-

---

Dialogus
GPT--Col
response response
dialogue.
A: Oh, a job seles. It's job selling children's books to bookstores. That sounds interasting.
Step Check response makes senee the context of dialogue.
job mentonad dialagus
The response makes sense as perzon disoussing ther preferences for the Job.
Step Check the response is plausible.
The response in plussble we person ecpressing their willignoes to work everinge and mentioning ther inabilly ditve..
B: Let's see.
You
driver's license.
Yenswer> Yesw/Anser>>
And you have to work some evenings.
GPT-4-CoT-Graile Knowledge
Step Analyze the dialogue
Porson A and B are discursing job istings on the internat. They discuss a sales job selling chiddan's books bookslones, which requinas a driver's lisinse and some evening work.
Knowledge
driving Has Prerequistt driver's lioense
Step Analyze the knowledge
The knowledig states that in order driving heppen, driver's foanse is needld..
Analyze the
Response
Person plausibility
don't mind working evenings,
ditve.
mind working avanings. And can't drive, which is perfect. What's the phone number?
unqualifed job, they they requires applying,
cislogus
dowsn' given the job requiements.
GPT-4; Not plausible
Final Answer: <Answer-No-<AAeee
Figure 5: Answers generated by GPT-4 in various seftings for example from the Dialogue task. While chain-of-thought reasoning results in an incorrect answer, addition of the commonsense knowledge guides the model toward the correct answer.
models than what is reported in the main results in Table 2.

---

Commonsense Knowledge Annotation
Open-domain Dialogue
Acceptance and Privacy Palicies (click to expand/collapse)
Instructions (click to expand/collapse)
(HAEWIG: th rocnss asahreswst medor dermier asterd)
Thanks for partidppaing in this HIT!
dielogue
people, you Identify
dhelague Implicit commoneeroe knuwledge
between these phrases.
Commonsense knowledge consists Implicit commanty know facs about everyday then woukd like eoc". It is type knowledge that nearly everyone knows intuitively and cor siders abrious, outis typically implicit wiich means focts
"temons sour or Vssmmose hungry,
rarely stated explictlly this information explicily found (eg. (ilkpedia) recessarily everyone knows
of Canador
knowledge
dialogue
following
Dialogue
help noving 2. #Persontl:: Hoy, Jonnifar, no probles. I'n free this waekend and my truck is great for noving stuff. Mhee did you gut the xafx?
#Porsoni::: Hi, San, can
house.
soving week, 4. #Person2t: It's good that your place large enough to the sofa. where sill you put it?
IPersonil::
apartment very
gisingg
There
this dialague and atterate between #Prsantt #PorsonZa.
For example, in the above dislogue, aPersont a junnifer) seeds move sofa into apartment and she get the sola from her friend whose apartment she is gettnng r. Hence, we can connect the highightedd perases he giving me his sota and your place large enough to fit the sota with an implict retadonship that the event in first phrzse has prerequbstee tharche event the second phrase happens f.e. order the sofa, your place largo enough
commonsense knowledge
place large enough
represent tyre
inoudedge
to giving his safa Itarrereuuisite place is Large enough Fit the anfa
Since could to identify phrases from contest
types relationshps
make task
provided
knowiedge relations
impliityy connected one relations provided
relations provided
describes the commonsense knowledge you have found, then you will be given option to specify custom relationship.
Note that the connecting knowledge relationship for correct commonsense knewledge must be commonly known and Implicit inoc explicitly
Figure 6: Mturk Instructions template for Dialogue CKA stage

---

Name
Description
Attributional Relations
HasProperty
A has B as property; A can he descrihed as B.
CapableOr
Something that A can typically do is B.
HasA
B belongs to A, either as an inherent part or due to social construct of possession.
HasSubEvent
A and B are events, and B happens as subevent of A.
IsA
A is subtype
specific instance of B; every A is B. A is specific way do B. Similar to "Is A", but for verbs.
MannerOf
DependsOn
A depends on B.
CreatedBy
A is created by B.
Physical/Spatial Relations
UsedFor
is used B. The purpose
PanOf
A is part
MadeOf
A is made up
AtLocation
happens location B, or typical location for
LocatedNear and are typically found near each other.
Temporal Relations
IsAfter
A happens after B.
IsBefore
A happense before B.
Happensin
A happens during B.
IsSimultanous
A and B happens the same time.
HasPrcrequisite
In order for to happen, B needs happen.
Causal Relations
Causes
causes happen.
Implics
A implies B. A is less likely to happen because
HinderedBy
Social Relations
xIntent
Person in event A intends to do B.
XReact
Person in event A reacts in B.
xNeed
Person in event A needs to do B before doing A.
Want
Person in event A wants to do B.
Xfffect
Event A will have the effect B on the Person in event A.
oReact
Others will react to event A as B.
WWnn
Others will want to do B for A. Fvent A will have effect B others.
oFffect
MotivatedByGGoal
Someone does A because they want result B.
Comparative Relations
Antonym
and are opposites in some relevant way.
Synonym
A and have very similar meanings.
SimilarTo
A is similar to
DistinctFrom
Something that A not is related
RelatedTo
DefinedAs
A is defined as B.
Table 5: List of commonsense relations
Task
Context
Target
CKA/CKV
WSG/WSV
Contexts
# Exxmpless
Dialogue
dialogue
response
1,169
3,548
Dialogue Summarization
diulogue
summary
453
1,805
Muchine Translation (zb-en)
sentence
translation
600
1200
Machine Translation (en-de)
sentence
translation
500
1000
Machine Translation (en-fr)
sentence
translation
500
1000
Machine Translation (en-ru)
sentence
translation
500
1000
Intent Detection
news headline
intent
V
589
2,440
Stance Classification
belief/argument
argument/belief"
/
397
1.722
Safery Detection
scenario
action
366
2,826
Total
5,074
16,541
Table Overview of the benchmark. CKA/CKV stands for Commonsense Knowledge Annotation and Validation stages, WSG/WSV stands for Winograd Schema Generation and Validation stages, respectively. Stages are skipped (-) tasks that already have the necessary data available.

---

Dos and Don'ts (click to expand/collppae)
Examples (click to expand/collapse)
Task
Given fallowing
people. idantify
this dislogue Implick
knowledge relationship.
There are typicall many such relationships in gien dialogue and you can add more (up to 5) using the Add Another button. will reward each adcitional correct annotation with banual
For dislogue phreses, prease enter the turn number (Ue, which dielogue turn this phrese about and phinase for this turn you can either copy&ppete dialoguee unlees
different
found dialogue,
For knowiedge relosion, plaase choose one of the avallabte relations or choose (other (if choose Other, you noed to Imputaa description for this reladion. you would like specify negative relatonship, select Wed from Profia dropdown. (Note that Reloton dropdown ia scarchable)
Dialogue
(Edislagua)
There are Sinum, turns} tums this dialogue.
Commonsense Knowledge
Dialogue phrase
Turn
Dialopue phrasse
this
for
Prefix
helatios
Select Relation
Add Ansther
(Cptionnl) Please us know anything was undear, you experienced any any other feedhack
Figure 7: Mturk Task template for Dialogue CKA stage
brabiiinn, you sprufy roppocte of these refations rebeaan by saleting piula, ezample, if you uxpp reletimmiig das not couse at then choose lee from the 2reft dropdoon and Eoseee from the Zeletion di updowe.
Dos and Don'ts (click to expand/colipase)
Examples (click to expand/collapse)
Task
Dhein follwin short diatogue and mommmmme sooeedge options, plans sulect options cornect reptictt commonseraa knowledge sboun miategue.
of you up addtitional
you
Dislegue
Commonsense Knowladge
Which the following correct
about the Qinlect that
None of the obave.
[Optionat iffee fownhara
lanced
Figure 8: Mturk Instructions template for Dialogue CKV stage

---

Knowledge Relations
description example. relstorrhh represented
HasProperty by
A has as property, A can described as B. e.g.
trial
HasPropenty
Free of charge
Capableof
Something that can typically B. e.g. kndre
Capobleof
cut
Hase
belongs to etther as an inherent part or due to socaal construct of possession, e.g. bird
HasA
king
HasSutevent
and B are events, and il happers as subevent of eg eatting RasSubEvent
cheving
IsA
A is zubtype or speciic instance every A a e.g. car 1sA vehicle
Matnerof
a specific way to do Similarto to "is A", but for verbs, c.g. Muction Pannerrf sele
DependsOn
A depends B. e. ppsstage Dependson
weight of the box
Createdby
ls created by B. e.g. space rockets Createdity scientists and angineera
Physical/Spatbal Relations
UbaadFor
purpose
hammor
UsedFor put
Partof
part e.g. vvord
sentence
Madsof
e-g. crowd
people
AtLocesion
A happens
typical location
try clothes
AtLocation
changing rooms
LocatedNcar
typically
e.g.
shaker LocatedNear
papper
Temporal Relations
happens eug
Tstefore
happense
HaspensIn happers during B.eg cift exchange HaspensIn Christeas party
happens
eg
playing
HbsPrerequssite
happen, happen, e.p. HosPrerequisite
Causal Reletions
Causes
Imolies implles
vet Irplics
Hinderediy A less likely heppen because e.p. Personk HinderedBy Persont
Social Relations
Figure Mturk Knowledge Relations Section Dialogue CKA stage

---

Contextually Implausible Dialogue Turn Generation
Acceptance and Privacy Policiess (click to expand/collapse)
Instructions (click expand/collapse)
Thanks for partidpatingnn in this HIT.
knowledge connecting dialogue dielague gener contectuallyy implaassible thatsatisees conditions: you
people, final the
minimally
original (plausible)
2. violetes the given co rommonzense knowledge
3. is contertually Impleusible final turn for the dialogue tsomething that weu on't have followed the given dilloguo, howover, as sontenes coile be plausible))
the given die logu toomething
to topics mentione the dialogur)
bnowledge typs knoweddge nearly everyone knows incut considers abvicus, typ cally impllot hongry, they would to it rarely stated explicillynn in toxt for crampie, encydlapoiic facts such as 'Ormww the coptal of Conado" are not considered commonsense edgeledge because this infor mation can explidily found unline Wilipedia) and not necessariy everyone knows
implicit commonty
everyday
Ar example a dielogue could be:
Dialogue
ny house. 2. #Persenze: Hoy, Jennifer, no pr problem. frre this weskerd and it truck is great for Boyding stuff. where did you pet the sofo?
#Persents: Sur, you wookend? help neving
sPersente: feriend Tark te
hut sparteent
easll So nafa
dia oguo.
could
Plausible Final Turn
#Persende: 1t's good that. plane large unough fit. the sefa, Where will 1t?
example knowledge connectin dlekgue
Figure 10: Mturk Instructions template for Dialogue WSG stage

---

Tisk
clalogue connecting dialague given tarn, plaase people, plousibie fins' dielogue, that sotiffies the following conditooss:
knowledge
commorsense knowledge instonces contextuatly implosaible finct
annimallly
iyinal (pkuuil))
2. vielates the comeenee brandesgee
3. conteatuolly implousiále final turn for the dialogue (sametrinng that weuktnt have owed the dialogue, however, as sentence on be pinatibitt
the given cialogue (something
to topics meniioned tne dillguu)
follow
mentonid
satisles
Please, note that simayy replading werdsighrases with their antonyms or negating sentences do NOT nesessority make them contentunily implousible. Cortestually implnustble final turn tur does dlarly withn the content the datiggue Plaase inttr for example
Please, also aveid
implousible independentt the
There aru syiially mul mays gemenate this unteciiddy implendille turrs turny su fewl frow athi 2 mure alternal ves by using the Asi Anstiner buition Note that will paid bonus for each additional contextual implausible final turn that satisfies our conditions for this task.
Wort shat you are free to modtty any part the finel turn, nor only the nighiinted parts as long as petiglls our condrolers for tonk.
Dialogue
(ddallguue
Plausible Final
(frrnal_tarn
Commonserse Knowledge
commoonentse knowleage
pewl nottgher concitions given
Cemmonsense knowledge"
Contectusily impi austble Rinal Turn
1*
sifinal, turn preffx)
minimmaly diffrront contextually
choson commonsenso knowledgo.
Figure 1I: Mturk Task template for Dialogue WSG stage

---

Examples (click to bopand/collapea
Task
Given the following shert di dialeguu between or more people, plousiäle final tum, list commonsense knowledgge instances co connectin the final turn dalagie patentially implausible
asked choose final surnto
conditios:
1. raannmmllt differs from the org nal plousible one
contextuntly implousiate that
ciatopue that
the dialogue sentence
could be pioustbre)
4. stil retevent for the gheen clelogue ae tooeething that related topics mentioned the dialogue)
nome of options sadefy the conditions tons, then tick the optior of obove and provide the implausible turn the think satisfies conditions yyuu can enter 3 and each actie unall correct annosation vill rewar ded with benust
Dialogue
addialogue)
Plausibbe
Commonserse Knowledge
(Potentially Implausible Final Turns
final satisify conditions above? apply)
Figure 12: Mturk Instructions template for Dialogue WSV stage

---

Commonsense Knowledge Annotation
Qualification Round
Acceptance and Privacy Policies (elick to expand/collapse)
Instructions (click to expand/collapse)
NAnwww:: zhe rimmmyy eddt verwe Wotter showttn snrwwd) Thanks partlopating this HITI
wiing workys vossuth
welogies mos forr virissee
build process. We intend to release hundreds of HITs this project and compensate adequetely with bonus for each additional annotation per HIT.
annotators get perticipate several-stage
Commonsenze knowladge concist implicit and commanly known facte about the everyday life such as 'mmmne are nourt or "f sommmnne hungny they would tie it of knowledge nearly
intultively and considers obvious. typically implicir which commonsense knowledge
rorely stated exalic ity because this information can be expl cilly found online (eg Wildpedia) not nececarryy everyone knows
exampic, encyclopedic
of
We are running a la scole project to rollect high quality commonserse knowlooge exampis re relazed to various reak life stustions described the form dialogues, summeries simaly
sentences.
order to accomplish this, are icoking for creative people like you help deeityy such implict knowledge givon some text.
This HiT has independent parts that will be uved tes: your abiiyy to perform the requined tevks in our project.
choice The first part consists of multiple cholce questions about idenciyiing correct commonsense know knowledge from a list options.
require Future HITs that we will release will all resemble this second part.
open-ended
up
Part Multiple-choice) instructions
this part, you will be given some (context) in various formats (tu dialague sentences) and several options (knowledge) iretances and asked to idenbly those that are cornect imphait commorsense knowledge examples about the given context. This parthas 3 tasks.
Task Instrrucioos
ghon
allouue identrfy commonsense knowledge connecting
dialogue.
example dislague could the follawing:
Figure 13: Mturk Instructions template for Qualification Stage

---

Port (Multipledte choice ons)
this part you will given some [contest) in vai themats (og clatnguee sentenced and severa options knowtedge incannees and asoed to dendly those that correct impaicr cor commenaense inosdedge ouang coampla abous the givein soncent.
Teak
the task, short d eugues betwwar people, knowledge instandes can noct florent tumn clalozue You knowledge about dusiogun.
nned sulect onen that imp
Dialogue
Kieca, 2old sialogee)
Which of the following knowledge instances are cornect Impliiit commonserse knowledge about this dielogus? (Seeed that appb)
Ftecq (1dd Rinouedee wtpptpl..pp.tllllll Rins task)
Quasi
Dislozue
Siaca 2dd. a2. fialogue)
Which of the following knowleddee instances are cornect Implicic commonserse knowledge about this dialoguer serrect that apply
ocd opt1 head! Stnoo, obE $2.
opt1. cad33
Figure 14: Mturk Task template for Qualification Stage Part (MCQ)
Part Open-ended questions)
part, yourol gioen context ssied wrtte an impliok commo comnse knowledge about contest
Quasttion
Gaven the fo short dialog betwwan more please idencify phrases from dialog, have an inplielt can commonoeend tnowledge netarionchip.
There are typicity many neletonnhipe goven disiggue and you more top using the Act Another outton, will reward actiiional correct cnnotation with benus and quality you for our project
For knowweddee relu vlltion, choese oner the atailable relitiess thcoue
desc
discciption gation for latioon If wou like sprcify
negutive relationchip select hail from the Prafis dropdoen, INbee Relation dropdoom searoluble)
Dialogue
FLonLass_et _stialague)
Stoe, odd
dialogue
Commma Knawwedgg
Turn numbert
Diai ogse
planssator
phrassfor thiefum
Profuz
telacios
Select Ralation
anything ingereeere any
Figure 15: Mtur Task template for Qualification Stage Part 2 (Open-ended)

---

Model
CK Dimcasions
Aucibution
Physical
Temporal
Causal
Social
Companson
Fla--S-1114
73.5
76.3
75.9
LLaMa-X38
42.2
42.4
46.2
42.0
42.4
43.5
Stable Vicun
35.3
PaI-M-1-540B
GPT-335
64.5
64.3
63.5
64.1
65.9
70.5
GPT-4
74.3
Table 7: Macro-F1 scores averaged across common- sense knowledge dimensions all tasks except for MT.). Temperature is set to 0.3 and all results averaged over three runs with different seeds.
Model
Oracle Knowledge
No Knowledge
Flun-TS-1IB
46.0/46.2
75.0/41.1
BloomZ-7B
52.3 13.6
15.9
GPT44
77.5
72.9/45.4
GPT-4-CoT
76.5/52.8
/4.3/
Table 8: Macro-F1 Situational Accuracy scores averaged over all tasks all tasks except MT), with and without providing commonsense knowledge in the prompt. Temperature is set to 0.3 and all results are averaged three runs with different seeds.

---

Model
Zh-En
En-Fi
Un-De
En-Ra
DKG
Soure
CROW
CROW
Scare
GPT4
75./5779
$4.572155
$4,47005
54.1/19.7
72.4 46.5
89.6/75.
79.75447
59.7/319
84.//572
83.1752.1
72.7/450
Human (bedore (lscuedoo) Homan (ofter discussion
87.4/ 78.0
755725.5
85,3/750
664/760
86 06./8860
M7.3/91.1
K3.5/56.5
90.1/758
924/731 839/800
900/76.5 93.1/85.5
369/76.3
$7.9/ 26.0
83./82.9
89.9/220
89.9
87.0/869
98.9 98.9796.4
83.1 65.17605
9939
50.7/360
Resolutions
18%
12%
135
Table 9: Human Evaluation results before and after discussion compared to GPT-4 and the percentage of resolved disagreements per task.
MT
CROW Score (-MT)
CROW
Models
Zh-En
En-Fi
Be-De
En-Ru
Score
Majority
33.3/00
33.3/00
33.3/00
33.3700
40.1/00
428/00
336/00
36.100
41,3/00
36470.0
46.419
50.6669
LLaMa-71
33.3/0.0
13
14
33.7/00
29:9000
41.3/00
38.1
37.3/0.5
LLaMa-13B
46.0/ 13.8
50.2/ 10.9
45.1/2.1
34.1007
30.5/0.1
43.8/ 18
10.8/3.1
41.7/4.9
LLMMa-38
33.3/0.1
$2.5/49
49.6556
33,7/02
50.1100
41,3/00
41.4/2.1
4..1/
Flan-15-11B
57.01668
754/483
75.0/41.1
72.07
Alpuca
410/6.1
52.0794
Flun Alp-Al
602/266.5
62.5/25.0
$2.01 18.4
67.2/364
67.0/
78.5/46.0
65.5/28.0
64.6 27.8
Vieuna
37.614.2
604/214
61.97 18.9
45.1/11.7
42.07 1.6
$7.2/
53.3/13.7
50.7 12.1
Siabll-Viennaa
60.5 30.9
$2.0 11.9
18.77722
57.1/178
68.67 14.3
63.//256
5499 15.0
55.8 17.66
1.
50.8/2.8
47.4/8.6
BloomZ-7B
55.1 19.8
47.01 15.0
49.61 18.0
4921 17.9
52.1 11.1
56.51 15.1
57.7 16.
68.3/8.5
64.3/21.8
59.8 14.5
55.5
PaLM
33.8/00.6
342/111
34.1709
33470.3
63.7 26.8
62.67 25.0
51.67 15.4
56.7799
63.5/219
59.67 19.0
48.2 10.9
GPT-3
66.71
48.51
492/172
48.17
67.1/36.8
68.6/32.6
69.2/38.2
65.5
76.4/420
73.4/37.8
64.1/ 29.7
GP1-4
75.6/5666
$4.5/221
54.62203
538/215
720/45.6
915/7773
81.5/57.3
89.3/50.2
72.9
Human"
87.9178.0
83.07829
89.91820
89.91 86.0
87.07 86.9
96.9196.4
88.1 88.1/696 6 97.8/93.9
93.9/80.7
98.1/85.5
90.7 /8.7
Table 10: Macro-F1 Situational Accuracy (i.e., results aggregated per context instead of per sample) for all examined models across CRoW tasks. All model results averaged over three with different seeds. Temperature is set to 0.3 for all runs. "Due to the cost of expert evaluation, our Human study is only evaluated on 100 instances per task.
Models
Zh-Fa
Fn-Fi
Fi-De
Fn-Ra
DG
SC
CROW Score (-MT)
CROW
Score
GPT-4 (temp-tul)
75.9/57.9
$4./2155
$4.4/205
54.1/19.7
72.4746.5
59.67 75.3
74.7
89.7/519
86.01722
$3.1
72.7/45.0
GPT-4 (temp-011)
76.4/58.
51.3/2077
S40/199 19.9
53.7/20.3
42.5
89.57 75.3
79.9 549
8944 908
83.5756.1
63.11 56.7
72.5/447
GPT-4 (tmpp:0.3
75.6/
$1.5/2.11
$1.6 2033
53.8/20.5
72.07 15.6
90.57
$7.5/
892//00
81.7159.0
$3,6/57.9
72.9/45.4
50.91
Table 11: Macro-F1/ Situational Accuracy (i.e., results aggregated per context instead of per sample) for GPT-4 across CRoW tasks with varying temperature values.
Models
Zb-Ei
BorFi
MT
Es-De
Ei-Ro
DG
D5
SC
SD
10
CROW Score (-MI))
CROW
Score
GPT-4-CoT (tcmp-90.6)
71,6752.3
64.11426
57.1 34.2
$7.3 33.0
55.3/22.5
$866/00 70.6
84.3/6077
87.8/47.3
$4.0/57.0
80.0/117
7
GPT-4-G6T (1cmp--).3, avernae)
71.3/51.1
644/420
37.134.1
36.625.9
55.2/22.5
888/713
43.7/608
14.15
53.9/57.3
800/5119
72.2466.
GPT-4-60 (tempp0.33, majociit))
71.5/513
64.1/420
58.1 /
36.07775
35.1/23.0
39.67333
45.1
87.9/45.6
84.1/579
#00/51.9
72.24465 46.2
Human"
$3.9/78.0
$300822
899/82.
89.9 86.0
87.07
86.9
989/966
86.1
07.8/93.9
93.0/ 80.7
93.1/855
90.7/840
Table 12: Macro-F1/ Situational Accuracy (i.e., results aggregated per context instead of per sample) for GPT-4 with CoT across CRoW tasks in different scenarios. In the 'average' scenario, an average of five experiment results are reported. In the 'majority' scenario, similar (Wang al., 2023), results based the majority answer from five experiments reported.


Reviews:
[{"review_id": "jrLSsuBWIY", "reviewer": "Reviewer_1q4f", "summary": "", "questions": "Question A: In line 320-328, was there any ranking among the 25 potential tasks? How were the tasks chosen to be part of CRoW? Were intent detection, stance classification, and safety detection part of these 25?\n\nQuestion B: In line 261-268, to confirm, this conveys that other approaches ignore (3) and (4) while your technique does not?\n\nQuestion C: In Figure 4, what does the image convey?\nIn the first example, everything except GPT-4 fails? What about GPT-4 with CoT or oracle knowledge?\nIn the second example, everything except GPT-4 + CoT fails? While CoT generally improves the result, you have provided reasoning and analysis that GPT-4+CoT performs worse that base GPT-4.\n\nQuestion D: In Table 4, there are 2 values per column. Are they Macro F1 and Situational Accuracy? The oracle knowledge that is added is only for negative examples? Can you please expand more on footnote 4? The negative/wrong knowledge triplet would be rejected in stage 1, CKV?\n\nQuestion E: Can you explain why only 2 human annotators were chosen for the human benchmark? 2 is not enough to get general consensus and majority voting. The part about human discussion in case of no majority shows that it may not rigorous and proper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and helpful suggestions. We accept and intend to incorporate their suggestions in the camera-ready version of our paper.\n\n**Why were there only 2 annotators chosen for the human evaluation and why was there a discussion between annotators to resolve disagreements?**  \n\nWe thank the reviewer for bringing up these important points about the reliability of our human evaluation results. However, we actually think these points stem from the potential advantages of our study. First, our setup may only have two annotators per task, but these are expert annotators who are NLP researchers from our lab (importantly, they were not involved in the original data collection). As a result, the evaluators are more experienced, can clarify misunderstandings in the annotation guideline with us, and generally produce more careful annotations than crowd workers. Finally, following [1, 2], we intentionally allowed annotators to discuss in cases of disagreement to reduce variance and yield a robust upper bound for our task. In the camera-ready version, we will clarify these important points about our human study. \n\n[1] Amidei, Jacopo et al. 2018 Rethinking the Agreement in Human Evaluation Tasks, ACL\n \n[2] Oortwijn et al., 2021 Interrater Disagreement Resolution: A Systematic Procedure to Reach Consensus in Annotation Tasks, ACL\n\n**How did we select the NLP tasks and was there any ranking used in the selection?**\n\nAs mentioned in section 4 of our paper, following (Davis, 2023), we identify the following desiderata of tasks in the benchmark: (1) tasks should have real-world applications (e.g., machine translation) and (2) tasks should involve rich commonsense knowledge and reasoning. We choose these criteria since our benchmark targets the evaluation of commonsense knowledge and reasoning in practical NLP applications. \nTo choose NLP tasks, we first crawl papers from the whole ACL anthology published since year 2000 (approximated 94K papers). Then, we automatically select those papers that have performed some error analysis and mention commonsense or world knowledge in their categories of errors. These steps resulted in 25 potential tasks.\nOut of these 25 tasks, we selected tasks based on the difficulty of generating commonsense violating perturbations. After manually designing commonsense perturbations for all 25 tasks that we found in our literature review, we select 6 tasks for which we could design a scalable pipeline for generating perturbations that could be reliably completed by crowdworkers.\n\n\n**How did we select the Commonsense Knowledge Dimensions?** \n\nWe considered widely used commonsense knowledge bases such as ConceptNet (Speer et al., 2016) and ATOMIC (Sap et al., 2019a) as well as recent works such as ATOMIC2020 (Hwang et al., 2020) and CIDER (Ghosal et al., 2021) for selecting the commonsense relations. As an initial step, the authors’ manually categorized the kind of knowledge relations that appeared for each task. Among the total 56 relations available to us from these sources, we found 22 relations from ConceptNet (out of 36), 8 relations from ATOMIC (out of 9), 3 relations from ATOMIC2020 (out of 5) and 3 relations from CIDER (out of 6) appearing most commonly. Following Ghosal et al., 2021 and Ilievski et al., 2021, we further categorized these 36 relations into 6 commonsense knowledge dimensions. \n\n**Do previous works ignore the following conditions while designing Winograd schemas: (3) the target must be an incorrect answer for the given context, and (4) the target is contextually relevant?** \n\nWe note that while other approaches do not explicitly ignore these conditions, they also do not explicitly address them. Hence, in our work we explicitly ask our annotators to consider these conditions to ensure the high difficulty of generations in our benchmark. \n\n**Limited CK dimension analysis**\n\nThe reviewer finds the overall evaluation results interesting, but that the analysis on the different dimensions of commonsense knowledge shows performance on most dimensions is nearly the same with not much distinction between them.\nWe agree with R1 that while we report a significant gap in overall evaluation of LLMs on our benchmark, our analysis on the performance of these models across different dimensions does not show a significant difference in how the model performs on different types of knowledge. However, we believe this sheds light on how language models represent commonsense knowledge, namely, with inherent interconnectivity across dimensions. More specifically, commonsense knowledge dimensions and relations are not mutually exclusive and one piece of knowledge typically involves several dimensions, while, one dimension is naturally more salient and that knowledge hence is represented by a particular relation in that dimension. Therefore, mastering a particular dimension of knowledge by a language model might in fact, amount to mastering several or all dimensions. Hence, we see uniformity in the performance of these models across dimensions. \n\n**In Figure 4, what does the image convey? In the first example, everything except GPT-4 fails? What about GPT-4 with CoT or oracle knowledge? In the second example, everything except GPT-4 + CoT fails? While CoT generally improves the result, you have provided reasoning and analysis that GPT-4+CoT performs worse that base GPT-4.**\n\nFigure 4 was designed to showcase examples from a few contrastive scenarios (small vs large models, simple prompting vs chain-of-thought prompting etc.), therefore was not meant to convey a comprehensive view of all the possible failure cases as there are too many. However, we agree that the figure could be clearer and we will add these details for better clarity in the camera-ready version. More specifically, in the first example, we meant all models that are not GPT-4, hence GPT-4 with oracle knowledge or CoT were not considered. In the second example, we specifically look at the effect of CoT, so only GPT-4 is considered and we include an example where GPT-4 with CoT does improve the result. We realize that including an example where it does not improve would be better as it is also aligned with our findings and we intend to add them in the final version. \n\n**Analysis and reasoning for the performance difference between GPT-4 and GPT-4+CoT**\n\nThe reviewer finds the analysis and reasoning for the performance difference between GPT-4 and GPT-4+CoT weak. In this answer, we would like to expand on our analysis and reasoning with an example and we intend to incorporate these additional points in the camera-ready version. We believe the worse performance of GPT-4 with CoT prompting compared to GPT-4 with simple prompting is due to the GPT-4’s more imaginative, but less plausible reasoning process triggered by the CoT prompting. As an example, we can look at the following dialogue, response pair and GPT-4’s answer on whether it is plausible when prompted with CoT.\n```\nDialogue:\n            A: Are you ready to move?\n            B: Definitely. I Ve got the hell out of the company. Anything new?\n            A: Yes, but there is a chance that might be suitable for you.\n            B: Really? Are you willing to recommend me for the position?\n            A: I can't say, but let me make a call and see if I can help.\nResponse:\nB: I appreciate the lack of support, sir.\nStep 1: Understand the dialogue\nPerson A asks if Person B is ready to move. Person B confirms and asks if there is anything new. Person A mentions a possible opportunity but is unsure if they can recommend Person B. Person A offers to make a call to see if they can help.\nStep 2: Analyze the response\nThe response from Person B seems to be sarcastic, as they are expressing appreciation for the \"lack of support\" when Person A has actually offered to make a call to help.\nStep 3: Determine if the response can plausibly follow the dialogue\nThe response can plausibly follow the dialogue, as Person B might be sarcastically expressing their feelings about the situation.\nFinal answer: <Answer>Yes</Answer>\n```\nWhen we analyze the error made by GPT-4 with CoT, we see that GPT-4 thought the sarcastic response was plausible. While this response could technically be possible, it might also violate our commonsense idea of what would be a reasonable response to a helpful query. Overall, we find other such examples where GPT-4 w/ CoT underperforms GPT-4 due to sarcastic or metaphorical elements in this dialogue.\n\n**In Table 4, there are 2 values per column. Are they Macro F1 and Situational Accuracy?**\n \nThank you for pointing it out, the results reported in Table 4 correspond to Macro-F1 and Situational Accuracy and we will fix the caption to reflect this. \n\n**Would wrong knowledge triplet be rejected in stage 1, CKV? Distinction between negative sample and wrong knowledge triplet?**\n\nFirst, we would like to clarify the distinction between the negative sample and negative (wrong) knowledge. Negative (wrong) knowledge is filtered out in our CKV stage. Negative samples do not refer to this as knowledge, but instead the constructed Winograd schema that are “semantically incorrect” based on a piece of commonsense knowledge. Second, the annotated commonsense knowledge is about both negative and positive samples, since this knowledge connects the context with the positive target and is used to generate the negative target. \n\n**Why did we use only negative samples for evaluating oracle knowledge with LLMs?**  \n\nDuring our annotation pipeline, commonsense dimensions are linked explicitly to the negative samples (which are constructed using a provided commonsense violation). For positive examples, multiple pieces of commonsense may be annotated (from the CKA stage). As a result, there is no 1-1 correspondence between positive samples and a piece of commonsense knowledge (i.e., no clear “oracle” knowledge”), so we opted to analyze the performance of these models on the commonsense knowledge dimension for negative samples only and report the results in Table 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The framework for constructing a commonsense reasoning dataset grounded in a real world NLP context seems like a fairly innovative, albeit minor, modification of existing techniques. It does seem to make the resulting dataset more nuanced to test the understanding of current and future models as the authors have shown a decent gap in performance between the humans and the LLMs.\n\nThe CRoW dataset is quite diverse and spans a wide variety of NLP tasks and different types of commonsense knowledge. The annotation guidelines are clear and the dataset quality verification is good. The evaluation of different LLMs on the dataset are interesting and it shows a decent gap between the human benchmark and the model performances.", "reasons_to_reject": "The human annotation size was limited to get a true estimate of human performance. Having only 2 annotators draws questions about reliability of this metric and tie break in case of no majority vote. Having discussion between the annotators to come to a conclusion makes the process not have independent annotators which means this was not robust.\n\nThe task selection part needed some justification on the tasks chosen to be included in the construction of CRoW. Same with the commonsense knowledge dimension selection.\n\nThe performance analysis of the LLMs on the different dimensions of commonsense knowledge was slightly lacking. This showed that all the different dimensions are nearly the same with not much distinction between them.\n\nThe analysis and reasoning for the performance difference between GPT4 and GPT4+CoT was relatively weak compared to the rest of the analysis.\n\nThe methodology of only using negative samples from the CKA stage for evaluating oracle knowledge with the LLMs may skew the results as only one part of the LLM prediction change will be captured.", "questions_to_authors": "Question A: In line 320-328, was there any ranking among the 25 potential tasks? How were the tasks chosen to be part of CRoW? Were intent detection, stance classification, and safety detection part of these 25?\n\nQuestion B: In line 261-268, to confirm, this conveys that other approaches ignore (3) and (4) while your technique does not?\n\nQuestion C: In Figure 4, what does the image convey?\nIn the first example, everything except GPT-4 fails? What about GPT-4 with CoT or oracle knowledge?\nIn the second example, everything except GPT-4 + CoT fails? While CoT generally improves the result, you have provided reasoning and analysis that GPT-4+CoT performs worse that base GPT-4.\n\nQuestion D: In Table 4, there are 2 values per column. Are they Macro F1 and Situational Accuracy? The oracle knowledge that is added is only for negative examples? Can you please expand more on footnote 4? The negative/wrong knowledge triplet would be rejected in stage 1, CKV?\n\nQuestion E: Can you explain why only 2 human annotators were chosen for the human benchmark? 2 is not enough to get general consensus and majority voting. The part about human discussion in case of no majority shows that it may not rigorous and proper.", "paper_topic_and_main_contributions": "Proposes a way to construct a commonsense reasoning dataset that is more grounded in 'real-world' scenarios. It crowdsources annotations of commonsense knowledge using a NLP task to provide examples of a context string and a target string. After commonsense knowledge validation, crowdsources the perturbation of the target to generate a Winograd schema.\n\nIt contributes CRoW, a commonsense reasoning dataset consisting of 6 NLP tasks (Machine translation, Open-domain Dialogue, Dialogue summarization, Intent detection, Stance classification, Safety detection) and 6 types of 'commonsense knowledge dimensions' (Attributional, Physical/Spatial, Temporal, Causal, Social and Comparative). It evaluates several LLMs on the constructed dataset to set benchmarks on the 6 NLP tasks as well as on the 6 commonsense knowledge dimensions.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggestion A: For lines 181-186, please rephrase as it was a bit confusing.\n\nSuggestion B: In figure 2, show more than 1 example in WSG of false intent (similar to CKA)\n\nSuggestion C: In Figure 3, category 'other' can not be seen clearly. Maybe change the colors or add a percentage in the graphs.\n\nSuggestion D: In Lines 261-268 give examples for each condition, 1 positive and 1 negative\n\nSuggestion E: Figure 4 caption to be expanded. Maybe add one failure and one success response at each subfigure. Also clearly mention what all failed and what succeeded.\n\nSuggestion F: Table 4 caption to be expanded. Maybe table 3&4 can be expanded to include more models. Add param numbers from table 2 to maintain consistency with table 3&4."}}, {"review_id": "COLf4hNKjf", "reviewer": "Reviewer_x28y", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments and suggestions. We are happy to see that the reviewer is excited about our work.\n\n**Discrepancy between intro Figure and the experimental setup in terms of task setting**\n\nWe propose to evaluate the commonsense reasoning skills in a binary classification setting and report macro-F1 and situation accuracy (for measuring robustness). In our intro Figure, we intended to demonstrate the contrast between different negative samples included in examples of our benchmark. We agree that this does give the false appearance of a multi-choice task and will fix this misunderstanding in the camera-ready version. \n\n**Why were there only 2 annotators chosen for the human evaluation?**  \n\nWe thank the reviewer for bringing up these important points about the reliability of our human evaluation results. First, our setup may only have two annotators per task, but these are expert annotators who are NLP researchers from our lab (importantly, they were not involved in the original data collection). As a result, the evaluators are more experienced, can clarify misunderstandings in the annotation guideline with us, and generally produce more careful annotations than crowd workers. Finally, following [1, 2], we intentionally allowed annotators to discuss in cases of disagreement to reduce variance and yield a robust upper bound for our task. In the camera-ready version, we will clarify these important points about our human study. \n\n[1] Amidei, Jacopo et al. 2018 Rethinking the Agreement in Human Evaluation Tasks, ACL\n \n[2] Oortwijn et al., 2021 Interrater Disagreement Resolution: A Systematic Procedure to Reach Consensus in Annotation Tasks, ACL", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel approach by combining commonsense reasoning with real-world NLP tasks, effectively bridging the gap between the commonsense reasoning domain and practical NLP applications. Unlike previous studies that mainly focused on probing commonsense reasoning ability in isolation, this work provides valuable insights into its practical implications.\n2. This paper comprehensively covers six real-world NLP tasks and six dimensions of commonsense reasoning types.\n3. The constructed method to generate the commonsense reasoning tasks is make sense. Meanwhile, the authors also consider the fine-grained dimensions for further analysis.\n4. The baseline models in experiments are sufficient.\n", "reasons_to_reject": "1. One major concern with the paper is the lack of clarity in its structure and statements, which makes it challenging to follow the content. For instance, Figures 1 and 2 depict the CRoW Evaluation as a multi-choice commonsense reasoning task, but the paper states that \"all tasks in CRoW are treated as binary classification tasks.\" Improving the clarity of these statements would significantly enhance the overall readability and understanding of the paper.\n2. The setup of only two annotators in the human evaluation raises concerns about the rigor and reliability of the evaluation process. In general, human evaluations should involve more than three annotators to ensure more robust and dependable results.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents CROW, a benchmark for evaluating commonsense reasoning in real-world NLP tasks. The authors propose a multi-stage data collection pipeline based on commonsense-violating perturbations following Winograd schemas. They apply this pipeline to construct CROW, comprising six real-world NLP tasks. The authors evaluate the performance of various language models on CROW and find that the models still fall far below human performance regarding commonsense reasoning. They also analyze the models' performance across different dimensions of commonsense knowledge.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. To make the comparison of model performance clearer and more insightful, consider using bold or underline to highlight the best-performing model for each task. This will help readers quickly identify the top-performing models. Additionally, it would be beneficial to include a new column in Table 2 to compute the average score across all tasks. This would provide a more holistic perspective on the overall performance of the models."}}, {"review_id": "mK5g0nzhmS", "reviewer": "Reviewer_H1md", "summary": "", "questions": "*Question A:* Did you try to use the LLMs in this study to generate valid responses for the language contexts? I wonder whether generating responses may somehow be easier or less confusing for LLMs than judging the validity of them (which may be subjective, especially in zero-shot settings).\n\n*Question B:* Are the model evaluations all zero-shot? I couldn’t find this detail. If so, is it possible that the LLM doesn’t quite pick up the format of this task (which is still a bit unusual despite being more relevant to real-world tasks) without some demonstrations?\n\n*Question C:* I was a bit confused by the metric choices in Tables 2 and 3. Why do we use macro-F1 in Table 2 but accuracy in Table 3? This makes it hard to compare the results across the two tables and understand the relationship there.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and helpful suggestions. We are happy that the reviewer finds our work quite exciting. We will take into account their suggestions in the camera-ready version of our paper.\n\n**Missing important details about LLM evaluation** \n\nWe thank the reviewer for pointing out several missing important details about LLM evaluation from the paper that we will add in the camera-ready version for better clarity. \n\n*Prompt templates*: We used the same prompt templates for LLM and human evaluation for fair evaluation. \n\n*Demonstrations of tasks*:  LLMs were evaluated in one-shot fashion for all tasks. We chose this setting for simplicity (we have several tasks and several models to evaluate) while making sure the models pick up the format of the task. \n\n*Randomness in Generation*: We haven’t observed any particular randomness in generating language with the evaluated models. We used the following hyperparameters which we found to result in good performance.\n\n(1) GPT-4 model: max_tokens=40, temperature=0.3, top_p=1\n\n(2) Other oper-sourced models: max_tokens=1, temperature=0.5, top_k=20, top_p =0.95\n\n**Why were there only 2 annotators chosen for the human evaluation and what is the statistical significance of evaluation results?**  \n\nWe thank the reviewer for bringing up these important points about the reliability of our human evaluation results. First, our setup may only have two annotators per task, but these are expert annotators who are NLP researchers from our lab (importantly, they were not involved in the original data collection). As a result, the evaluators are more experienced, can clarify misunderstandings in the annotation guideline with us, and generally produce more careful annotations than crowd workers. Finally, following [1, 2], we intentionally allowed annotators to discuss in cases of disagreement to reduce variance and yield a robust upper bound for our task. In the camera-ready version, we will clarify these important points about our human study. \n\nBased on the reviewer’s suggestion, we performed the binomial sign test on GPT-4 and human evaluation results as our sample size is small. We find that for Machine Translation (en-ru, en-de and en-fr), Intent detection and Safety detection tasks, results are statistically significant with a level of 0.01, for Dialogue and Summarization tasks with a level of 0.1. For Stance detection and zh-en machine translation task, we find the p-values slightly high (0.16 and 0.45 respectively), however, this is also aligned with our overall results with respect to the performance of GPT-4 on these tasks and the observed narrow gap. We will add these results to the camera-ready.\n\n[1] Amidei, Jacopo et al. 2018 Rethinking the Agreement in Human Evaluation Tasks, ACL \n\n[2] Oortwijn et al., 2021 Interrater Disagreement Resolution: A Systematic Procedure to Reach Consensus in Annotation Tasks, ACL \n\n**Generating valid responses using LLMs instead of judging their validity**\n\nWhile evaluating generated responses would be an interesting avenue, we decided to use classification-based evaluation of minimal pairs as a starting point. Generating responses is open-ended, meaning we may not always immediately uncover the differences along the commonsense dimensions we are interested to explore, which would not fully test the robustness of these models to semantic and pragmatic subtleties. However, the reviewer points out an interesting direction for future work, and one we are actively exploring at the moment.\n\n**Why did we report accuracy in Table 3?** \n\nWe report Accuracy on negative samples in Table 3. During our annotation pipeline, commonsense dimensions are linked explicitly to the negative samples (which are constructed using a provided commonsense violation). For positive examples, multiple pieces of commonsense may be annotated (from the CKA stage). As a result, there is no 1-1 correspondence between positive samples and a piece of commonsense knowledge, so we opted to analyze the performance of these models on the commonsense knowledge dimension for negative samples only and report the results in Table 3. \n\nFor the sake of completeness, we ran the new CK analysis with positive samples included (where each positive sample may have more than one piece of commonsense knowledge associated with it) and report the results below (Macro-F1 reported): \n\n|             |  **Attribution** | **Physical** | **Temporal** | **Causal** | **Social** | **Comparison** |\n| ----------- | ------------ | -------- | -------- | ------ | ------ | ---------- |\n| Flan-Alpaca | 69.9         | 71.6     | 54.3     | 70.2   | 71.8   | 73.5       |\n| Flan-T5     | 76.8         | 77.8     | 63.1     | 78.1   | 79.3   | 80.0       |\n| LLaMa       | 46.5         | 45.9     | 37.4     | 45.7   | 46.1   | 45.3       |\n| Stable-Vicuna | 54.9       | 56.2     | 44.2     | 53.8   | 57.8   | 55.1       |\n| BloomZ-7B   | 54.3         | 54.9     | 46.5     | 53.7   | 52.7   | 52.9       |\n| PaLM        | 55.4         | 53.6     | 50.1     | 53.9   | 57.6   | 53.9       |\n| GPT-3       | 64.5         | 65.2     | 57.2     | 62.8   | 65.9   | 69.2       |\n| GPT-4       | 75.6         | 73.9     | 64.9     | 74.3   | 74.6   | 75.5       |\n\nWe will include this result in the camera-ready version of our paper. As stated above, though, these results inflate the contribution of each positive sample to the final result (as they may be associated with multiple pieces of commonsense knowledge).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "*Strength A:* The paper is well motivated and timely, calling into question current benchmarking practices in NLP and demonstrating that they’re insufficient to evaluate systems for common practical settings. The message of the paper is well supported, and the community would benefit from it: the reasoning ability of NLP systems only really counts when it comes to their practical real-world application.\n\n*Strength B:* The data collection method and benchmark presented are a substantial and comprehensive effort, and can be beneficial to the community and used by others to more objectively evaluate and facilitate improvement of NLP systems on commonsense reasoning, thought of as one of the hardest challenges in AI. \n\n*Strength C:* The data collection and evaluation procedures presented in the paper are quite thorough. The benchmark covers a comprehensive set of practical NLP tasks selected from papers going back to 2000, and a wide range of commonsense knowledge sub-domains based on prior work. Data is validated in multiple ways and often by multiple humans after every step of the collection process. Evaluation metrics include not only accuracy on end tasks, but a deeper situational accuracy which judges the completeness of systems’ commonsense reasoning for each individual language context used in the benchmark. 13 diverse and relevant language models are evaluated, and discussion and analysis of results provides a variety of insights about the characteristics of models and the datasets.", "reasons_to_reject": "*Weakness A:* From what I can tell, the paper is missing important details about how LLMs are evaluated. While some prompt templates are given in the appendix, it’s actually not clear to me whether these were given to annotators or LLMs. It is unclear whether any demonstrations of tasks are given within prompts (i.e., for in-context learning), and if this is not the case, this raises questions about whether LLMs just have trouble picking up the format of the tasks rather than struggling with commonsense reasoning. It would also be important to know whether any randomness is involved in generating language, and how this is controlled.\n\n*Weakness B:* The human evaluation is a bit small-scale. While differences between human results and system results seem wide enough for this to be conclusive, I would be curious whether the differences are statistically significant, e.g., using a McNemar test.\n\nIf the authors provide sufficient detail to address these possible issues, I will consider modifying my soundness score.", "questions_to_authors": "*Question A:* Did you try to use the LLMs in this study to generate valid responses for the language contexts? I wonder whether generating responses may somehow be easier or less confusing for LLMs than judging the validity of them (which may be subjective, especially in zero-shot settings).\n\n*Question B:* Are the model evaluations all zero-shot? I couldn’t find this detail. If so, is it possible that the LLM doesn’t quite pick up the format of this task (which is still a bit unusual despite being more relevant to real-world tasks) without some demonstrations?\n\n*Question C:* I was a bit confused by the metric choices in Tables 2 and 3. Why do we use macro-F1 in Table 2 but accuracy in Table 3? This makes it hard to compare the results across the two tables and understand the relationship there.", "paper_topic_and_main_contributions": "Motivated by recent rapid advances on NLP benchmarks for commonsense reasoning made by large language models (LLMs), this work argues that the artificial scenarios provided in existing benchmarks for commonsense reasoning are not reflective of real-world NLP tasks. As such, they propose a new benchmark CRoW, which applies a novel data collection paradigm to generate Winograd schema-like rewrites of existing NLP task examples, where systems would be expected to judge whether a candidate response to one of these tasks is valid. \n\nTheir benchmark includes evaluations across 6 practical task settings (machine translation, open-domain dialogue, dialogue summarization, intent detection, stance classification, and safety detection, which cover 7 sub-domain categories of commonsense (temporal, causal, attribution, comparison, physical, social, and other). Upon evaluating 13 SOTA systems on CRoW, including GPT-4, the authors find that they fall short on real-world practical commonsense reasoning.\n\nThis paper presents a substantial data collection and evaluation effort, and the CRoW dataset is a useful and systematic tool that can be used by the community to more objectively evaluate NLP systems on commonsense reasoning. The methodology is remarkably thorough, and the paper is well written. However, some important details about model evaluation seem to be missing that would help me better understand the papers’ conclusions.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "ProPara (https://arxiv.org/abs/1805.06975) and TRIP (https://aclanthology.org/2021.findings-emnlp.422.pdf) may be relevant physical commonsense benchmarks to cover in Section 2.\n\nThere are some papers that that propose robustness metrics similar in spirit to your Situational Accuracy metric: “Group Consistency” in Ravichander et al. (2022; https://arxiv.org/pdf/2211.00295.pdf) and “Strict Coherence” in Storks and Chai (2021; https://aclanthology.org/2021.findings-emnlp.272.pdf) come to mind. It might not be necessary to cite them, but your work reminded me of them.", "Typos_Grammar_Style_and_Presentation_Improvements": "L078: Aren’t there 7 dimensions of commonsense annotated? I think you missed “social” here and also in the Limitations section. You may want to double check that you’re consistent about the categories used.\n\nL337: Small grammatical error in parentheses; can remove “the” to fix it.\n\nL891: the reference here has some extra numbers inserted into it."}}]

Author rebuttal:

Meta-review: he paper presents CROW, a benchmark for evaluating commonsense reasoning in real-world NLP tasks. All reviewers viewed the framework for constructing a commonsense reasoning dataset grounded in a real world NLP context as fairly innovative. However, the reviewers also raised some concerns about the soundness (details) of the construction procedure of the dataset.  Most of these concerns were at least partially addressed by the authors during the rebuttal phase. I recommend the authors to add more analysis of stableness.
Decision: Accept-Main