Abductive Commonsense Reasoning
Authors: Chandra Bhagavatula
Ronan Le Bras
Chaitanya Malaviya
Keisuke Sakaguchi
Ari Holtzman
Hannah Rashkin
Doug Downey
Wen-tau Yih
Yejin Choi
Venue:  (2020)
Keywords: Abductive Reasoning
Commonsense Reasoning
Natural Language Inference
Natural Language Generation

Abstract: Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and  caused  the  mess,  as  the  most  plausible  explanation.  While  abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation. We present the first study that investigates the viability of language-based abductive reasoning.  We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks – (i) Abductive NLI: a multiple-choice question answering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%.  On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform—despite their strong performance on the related but  more  narrowly  defined  task  of entailment NLI—pointing  to  interesting  avenues for future research.

Published as a conference paper at ICLR 2020
ABDUCTIVE COMMONSENSE REASONING
Chandra Bhagavatula Ronan Le Bras Chaitanya Malaviya Keisuke Sakaguchi Ari Holtzman 9, Hannah Rashkin Doug Downey Scott Wen-tau Yih Yejin Choi Allen Institute for AI, Seattle, WA, USA, Facebook AI, Seattle, WA, USA Paul G. Allen School of Computer Science & Engineering, WA, USA {chandrab, chaitaaan, kei keisukes) wawleeailllloooeeeee farih, hannahr, dougd] org [yejin] washi edu (scottyin]efb.. com"
[Cs.CL] 14 2020
ABSTRACT
Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. While abduction has long been considered to be at the core of how people interpret and read between the lines in natural language (Hobbs et al., 1988), there has been relatively little research in support of abductive natural language inference and generation.
We present the first study that investigates the viability of language- based abduc- tive reasoning. We introduce a challenge dataset, ART, that consists of over 20k commonsense narrative contexts and 200k explanations. Based on this dataset, we conceptualize two new tasks - (i) Abductive NIJ: a multiple-choice question an- swering task for choosing the more likely explanation, and (ii) Abductive NLG: a conditional generation task for explaining given observations in natural language. On Abductive NLI, the best model achieves 68.9% accuracy, well below human performance of 91.4%. On Abductive NLG, the current best language generators struggle even more, as they lack reasoning capabilities that are trivial for humans. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail to perform- despite their strong performance on the related but more narrowly defined task of entailment NLI 1--oonttin to interesting av- enues for future research.
INTRODUCTION
The brain is an abduction machine, continuously trying to prove abductively that the ob- servables in its environment constitute a coherent situation.
-Jerry Hobbs, ACI. 2013 Lifetime Achievement Award'
Abductive reasoning is inference to the most plausible explanation for incomplete observations (Peirce, 1965a). Figure illustrates an example. Given the incomplete observations about the world that O₁ "Jenny cleaned her house and went to work, leaving the window just crack open." and sometime later O2: "When Jenny returned home, she saw her house was a mess.", " we can hypothe- size different potential explanations and reason about which is the most likely. We can readily rule out H3 since it fails to justify the observation O2. While H1 and H2 are both plausible, the most likely explanation based on commonsense is H1 H2 is somewhat implausible given O₁.
One crucial observation Peirce makes about abductive reasoning is that abduction is "the only logical operation which introduces any new ideas". which contrasts with other types of inference such as entailment, that focuses on inferring only such information that is already provided in the premise.
Work done while at A12
'The full transcript of his award speech is available at https: / h/wp://wwww.m.mm tepressjourme osr doi htt/fuu1/1.0.0 12/ 1________0000___________________ 00171

---

Published as a conference paper at ICLR 2020
Jenny cleaned her house and work, leaving the window just crack open.
Jonny insecure opening to her house.
Somswwau Unilkeey.
Likey follw
the wi room was just crack open. large bird urilkaly to in?
11,
11,
11,
thief broke into the house by pulling open the window
a breesy day large bird flaw into house.
At work, she opened her window and the wind blew her papers everywhere.
The thief got house through the window rifled through Jonny's things, which mode
The inside
Although Jusiiy O. event happanad at Janny's workplace
trying to sacape, and made mess.
O2
When Janny returned home she that har house
Figure I: Example of Abductive Reasoning. Given observations O1 and O2, the aNLI task is to select the most plausible explanatory hypothesis. Since the number of hypotheses massive in any given situation, make simplifying assumption in dataett only choose between a pair of explanations.
Abductive reasoning has long been considered to be at the core of understanding narratives (Hobbs al., 1988), reading between the lines (Norvig, 1987: Charniak & Shimony, 1990), reasoning about everyday situations (Peirce, 1965b; Andersen, 1973), and counterfactual reasoning (Pearl, 2002; Pearl & Mackenzie, 2018). Despite the broad recognition of its importance, however, the study of abductive reasoning in narrative text has very rarely appeared in the NLP literature, in large part because most previous work on abductive reasoning has focused on formal logic, which has proven to be too rigid to generalize to the full complexity of natural language.
In this paper, we present the first study to investigate the viability of language- based abductive reasoning. This shift from logic-based to language-based reasoning draws inspirations from a sig- nificant body of work on language-based entailment (Bowman et al., 2015; Williams et al., 2018b), language-basee based logic (Lakoff, 1970; MacCartney & Manning. 2007), and language- based common- sense reasoning (Mostafazadch et al., 2016; Zellers et al., 2018). In particular, we investigate the use of natural language the representation medium, and probe deep neural models on language-based abductive reasoning.
More concretely, propose Abductive Natural Language Inference (aNLI) and Abductive Natural Language Generation (aNLG) as two novel reasoning tasks in narrative contexts.2 ? We formulate oNLI as multiple-choice task support easy and reliable automatic evaluation: given context, the task is to choose the more likely explanation from given pair of hypotheses choices. We also introduce new challenge dataset, ART, that consists of 20K narratives accompanied by over 200K explanatory hypothesis. 34 We then establish comprehensive baseline performance based on state-of-the-arr NLI and language models. The best baseline for oNLI based on BERT achieves 68.9% accuracy, with a considerable gap compared to human performance of 91.4%($55.) The best generative model, based on GPT2, performs well below human performance on the aNLG task ($5.2). Our analysis leads to insights into the types of reasoning that deep pre-trained language models fail to perform - despite their strong performance on the closely related but different task of entailment NLI pointing to future research directions.
TASK DEFINITION
Abductive Natural Language Inference We formulate aNLI as multiple choice problems con- sisting of pair of observations as context and pair of hypothesis choices. Each instance in ARTis defined as follows:
O₁: The observation at time tj.
"uNLI and aNLG pronounced alpha-NLI and alpha-NLG, respectively *990: Abductive Reasoning in narrative Text.
*Data available download at http:/ahhdectieeommoonnnse..y
2

---

Published as a conference paper at ICLR 2020
O2: The observation at time 12
f1.
ht: A plausible hypothesis that explains the two observations O1 and O2-
h": An implausible (or less plausible) hypothesis for observations O₃ and O2.
Given the observations and pair of hypotheses, the aNLI task is to select the most plausible expla- nation (hypothesis).
Abductive Natural Language Generation aNLG the task of generating a valid hypothesis h+ given the two observations O₁ and O2. Formally, the task requires to maximize P(h+ (Ott O2).
MODELS FOR ABDUCTIVE COMMONSENSE REASONING
3.1 ABDUCTIVE NATURAL LANGUAGE INFERENCE
A Probabilistic Framework for oNLI: A distinct feature of the oNLI task is that it requires jointly considering all available observations and their commonsense implications, to identify the correct hypothesis. Formally, the oNLI task is to select the hypothesis ht that is most probable given the observations.
arg max P(H 11 O1, O2)
h
Rewriting the objective using Bayes Rule conditioned on O₁, we have:
P(r O2) x P(O2)I",O,)PPiil) (.)
We formulate set of probabilistic models for oNLI that make various independence assumptions on Equation 2 starting from simple baseline that ignores the observations entirely, and building up to fully joint model. These models are depicted as Bayesian Networks in Figure 2.
a) Hypothesis-Oll
First Observation Only
e) Second Observation Only
d) Linear Chain
e) Fully Connected
O
O2
O
H
O2
Figure 2: Illustration of the graphical models described in the probabilistic framework. The "Fully Connected" model can, in theory, combine information from both available observations.
Our simplest model makes the strong assumption that the hypothesis is entirely independent of both observations, i.e. (H 1 O2, O2), in which case we simply to maximize the marginal P.H).
Hypothesis Only:
First (or Second) Observation Only: Our next two models make weaker assumptions: that the hypothesis depends on only one of the first O second O₂ observation.
Our next model uses both observations. but considers each observation's 's influ- ence on the hypothesis independently, i.c does not combine information across the observa- tions. Formally, the model assumes that the three variables (O,, H, O2) form linear Markov chain, where the second observation is conditionally independent the first, given the hypothesis (i.e. (O, 1 H)) Under this assumption, aim to maximize somewhat simpler objective than Equation 2:
Linear Chain:
= arg max P(01)))) where (O1 O2 he
Finally, most sophisticated model jointly models all three random variables as in Equation 2, and can in principle combine information across both observations to choose the correct hypothesis.
Fully Connected:

---

Published as a conference paper at ICLR 2020
Transformer
Kle CDMsT Embedorge
Wird Braiddegg
COMeT Commonsense Transformers
<01>w of wi </o1> <o?> wf ... </02>
<o1> oo
om </01>
<o2>
of
Wm </02>
Figure 3: Overview of oNLG model that integrates commonsense representations obtained from COMeT (Bosselut et al., 2019) with GPT2. Each observation is input to the COMETT model to obtain nine embeddings, each associated with commonsense inference type.
To help illustrate the subtle distinction between how the Linear Chain and Fully Connected models consider both observations, consider the following example. Let observation Oj: "Carl went to the store desperately searching for flour tortillas for a recipe." " and O2: "Carl left the store very frustrated.", Then consider two distinct hypotheses, an incorrect h²: "The cashier was rude" and the correct 2 "The store had corn tortillas, but not flour ones." For this example, Linear Chain model could arrive at the wrong answer, because it reasons about the observations separately- taking O₃ in isolation, both h' and h seem plausible next events, albeit each priori unlikely. And for Oz isolation-i.e. in the absence O₁, as for a randomly drawn shopper-the ht explanation of rude cashier seems a much more plausible explanation of Carl's frustration than are the details of the store's s tortilla selection. Combining these two separate factors leads the Linear Chain to select h² as the more plausible explanation. It is only by reasoning about Carl's goal in O1 jointly with his frustration in O2, as in the Fully Connected model, that we arrive at the correct answer h² as the more plausible explanation.
In our experiments, we encode the different independence assumptions in the best performing neural network model. For the hypothesis-only and single observation models, can enforce the inde- pendencies by simply restricting the inputs of the model to only the relevant variables. On the other hand, the Linear Chain model takes all three variables as input, but we restrict the form of the model to enforce the conditional independence. Specifically, we learn a discriminative classifier:
Piinear Chain Oh)O,, O2) M)+
where & and o' are neural networks that produce scalar values.
3.2 ABDUCTIVE NATURAL LANGUAGE GENERATION
Given h+= (wf ... w ........,,;
O1=(wpt wol and (2=(w ... ... w as sequences of tokens, the aNLG task can be modeled as 1( O₂) = II P(u / wpl ...wpl wo2 2 Option- ally, the model can also be conditioned on backgrount knowledge K. Parameterized models can then be trained to minimize the negative log-likelihood over instances in 2RT
log P(w) (w wit w w 02 ,,)
(4)
4 ART DATASET::: ABDUCTIVE REASONING IN NARRATIVE TEXT
ARTis the first large-scale benchmark dataset for studying abductive reasoning in narrative texts. consists of ~20K narrative contexts (pairs of observations (O,, O2)) with over 200K explanatory hypotheses. Table in the Appendix summarizes corpus-level statistics of the dataset Figure 4 shows some illustrative examples from ART (dev split). The best model based on BERT fails to correctly predict the first two dev examples.
'We will publicly release the SRT dataset upon acceptance.
4

---

Published as a conference paper at ICLR 2020
Given These Partial Observations
The More Plausible Hypothesis is
was very hot summer day.
He dacided to run in the hest
Hi
Ho felt much better!
He drank a glass cold
II
DEV
Chod loves Barry Bonds.
Chad got Barry online, chatting
ensured that took picture to remember event.
Chad waited after a game and met Barry.
11
DRV
O: Leslic went the mall for a purse to mutch her new dress
Leslic found beautiffll dress afler looking for hours.
X
O
took counter and paid right away, went home happy.
Lestie found one that was perfect
Hi
Figure 4: Examples from RRT((der split). The best model based on BERT fails to correctly predict the first two examples.
Collecting Observations: The pairs O1, O2 in ART are drawn from the ROCStories dataset (Mostafazadeh et al., 2016). ROCStories a large collection of short, manually curated five- sentence stories. It designed to have clear beginning and ending for each story, which naturally map to the first (O1) and second (O2) observations in ART.
Collecting Hypotheses Options: We crowdsourced the plausible and implausible hypotheses op- tions on Amazon Mechanical Turk (AMT) in two separate tasks
1. Plausible Hypothesis Options: We presented O₁ and O2 as narrative context to crowdworkers who were prompted to fill in "What happened in-between?" in natural language. The design of the task motivates the use of abductive reasoning to hypothesize likely explanations for the two given observations.
2. Implausible Hypothesis Options: In this task, we presented workers with observations O₁, O2 and one plausible hypothesis option h+ E 7 collected from the previous task. Crowdworkers were instructed make minimal edits (up words) given A+ to create implausible hypothesis variations for each plausible hypothesis.
significant challenge in creating datasets avoiding annotation artifacts - unintentional patterns in the data that leak information about the target label that several recent studies (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018) have reported on crowdsourced datascts To tackle this challenge, we collect multiple plausible and implausible hypotheses for each (O1, O2) pair (as described above) and then apply an adversarial filtering algorithm to retain one challenging pair of hypotheses that are hard distinguish between. We describe our algorithm in detail in Appendix A.5. While our final dataset uses BERT as the adversary, preliminary experiments that used GPT adversary resulted in similar drops in performance of all models, including all BERT variants. We compare the results of the two adversaries in Table 1.
5 EXPERIMENTS AND RESULTS
We now present our evaluation of finetuned state-of-ther pre-trained language models on the ART dataset, and several other baseline systems for both oNLI and aNLG. Since oNLI is framed binary classification problem, we choose accuracy our primary metric. For oNLG, we report performance on automated metrics such as BLEU (Papineni al., 2002), CIDEr (Vedantam al., 2015), METEOR (Banerjee & Lavie, 2005) and also report human evaluation results.
5.1 ABDUCTIVE NATURAL LANGUAGE INFERENCE
'Both crowdsourcing tasks are complex and require creative writing. Along with the ART dataset, we will publicly release templates and the full set of instructions for all crowdsourcing tasks to facilitate future data collection and research in this direction.
5

---

Published as a conference paper at ICLR 2020
Despite strong performance several other NLP benchmark datasets, the best baseline model based BERT achieves an accuracy of just 68.9% on ART com- pared human performance of 91.4%. The large gap between human perfor- mance and that of the best system provides significant scope for development of more sophisticated abductive reasoning models. Our experiments show that introducing the additional independence assumptions de- scribed in Section 3.1 over the fully con- nected model tends to degrade system per- formance (see Table 1) in general.
GPT AF
ART
Model
Acc. (%)
Acc. (%)
Random (2-way choice)
50.1
50.4
Majority (from dev set)
50.8
Infersent (Conneau et al., 2017)
50.9
50.8
ESIM+ELMo (Chen al., 2017)
58.2
58.8
Finetuning Pre-trained LMs
GPT-ft
52.6 (0.9)
63.1 (0:5)
BERT-fi [h' Only]
55.9 (0.7)
59.5 (0.2)
BERT-ft [O, Only]
63.9 (0.8)
63.5 (0.7)
BERT-ft [O2 Only]
68.1 (0.6)
66.6 (0.2)
BERT-ft [Linear Chain]
65.3 (1.4)
68.9 (0.5)
BERT-ft [Fully Connected]
72.0
68.6 (0.5)
Human Performance We compute hu- man performance using AMT. Each in- stance (two observations and two hypothe- sis choices) is shown to three workers who were prompted to choose the more plausi- ble hypothesis choice.? 7 We compute ma- jority vote the labels assigned which leads to a human accuracy of 91.4% on the Aestess set.
Human Performance
91.4
Table 1: Performance of baselines and finetuned-LM approaches on the test set of ΛRT. Test accuracy is re- ported as the mean of five models trained with random seeds, with the standard deviation in parenthesis.
Baselines We include baselines that rely on simple features to verify that ART not trivially solvable due noticeable annotation artifacts, observed in several crowdsourced datasets. The accuracies of all simple baselines are close to chance seaunc--erffrmance on the task indicating that the dataset is free of simple annotation artifacts.
model for the related but distinct task of entailment NLJ (e.g. SNLJ) forms a natural baseline for oNLI We re-train the ESIM+ELMo (Chen al., 2017; Peters al., 2018) model its performance on entailment NLI (88.9%) is close to state-of-the-arr models (excluding pre-trained language mod- els). This model only achieves an accuracy of 58.8% highlighting that performing well on 28T requires models to go far beyond the linguistic notion of entailment.
Pre-trained Language Models BERT (Devlin al., 2018) and GPT (Radford, 2018) have recently been shown to achieve state-of-the-art results on several NLP benchmarks (Wang et al., 2018). We fine- tune both BERT-Large and GPT sug-
Arcuuncy on Dav Sen
80
gested in previous work and we present each instance in their natural narrative or- der. BERT-ft (fully connected) is the best performing model achieving 68.9% accu- racy, compared to GPT's 63.1% Our AF approach was able to reduce BERT perfor- mance from over 88% by 20 points.
60
40
1000
10000
1000000
No. Training Exemples
Learning Curve and Dataset Size While there enough scope for consid- erably scaling up the dataset based on ROCStories, the learning curve in Figure 5 shows that the performance of the best model plateaus after 10, 000 instances. In addition, there is still a wide gap (~23%) between the performance of the best model and human performance.
Figure 5: BERT learning curve on the dev set of ART. For each point the x-axis, fine-tune BERT with five random seeds. Human performance is 91.4%.
Additiooal crowdsourcing details in the Appendix A.I
'The input format for the GPT model and BERT variants is deserihed in the Appendix A.4.

---

Published as a conference paper at ICLR 2020
Model
BLEU
METEOR
ROUGE
CIDEr
BERT-Score
Human
GP12-Fixed
0.0
9.29
9.99
3.34
36.69
O2-O2-Only
2.23
16.71
22.83
33.54
48.74
42.26
COMET-IX+4GF12
2.29
16.73
22.51
31.99
48.46
38.28
COMeI-Emb+GFTZ
3.03
17.66
22.93
32.00
48.52
44.56
Human-written Hypotheses
8.25
26.71
30.40
53.56
53.30
96.03
Table 2: Performance of generative models on the test set of ART All models except GPT2-Fixed are finetuned on RRT.
GPT Adversary Table 1 also includes results of our experiments where GPT was used as the adversary. Notably, in this case, adversarially filtering the dataset brings down GPT performance under 53%. On the other hand, the best BERT model, that encodes the fully connected bayesian net- work performs significantly better than the BERT model that encodes the linear chain assumptions 72% compared to 65%. Therefore, we use the BERT fully connected model as the adversary in UR.. The gap between the linear chain and fully connected BERT models diminishes when BERT used as an adversary - in spite of being a more powerful model which indicates that adversar- ial filtering disproportionately impacts the model used the adversary. However, the dataset also becomes more difficult for the other models that were not used as adversaries. For example, before any filtering, BERT scores 88% and OpenGPT gets 80%, which is much higher than either model achieves Table when the other model is used for filtering. This result is a reasonable indicator, albeit not a guarantee, that RRT will remain challenging for new models released in the future.
5.2 ABDUCTIVE NATURAL LANGUAGE GENERATION
Generative Language Models As described in Equation 4, we train GPT2 conditioned on the tokens of the two observations O, and O2. Both observations are enclosed with field-specific tags. ATOMIC (Sap et al., 2019), repository of inferential if-then knowledge is natural source of back- ground commonsense required to reason about narrative contexts in ART. Yet, there is no straightfor- ward way to include such knowledge into neural model ATOMIC's nodes are not canonicalized and are represented as short phrases of text. Thus, we rely on COMeT - a transformer model trained on ATOMIC that generates nine commonsense inferences of events in natural language." 9 Specifi- cally, we experiment with two ways of integrating information from COMeT in GPT2: (i) as textual phrases, and (ii) as embeddings.
Figure 3 shows how we integrate COMeT representations. Concretely, after the input tokens are em- bedded by the word embeddinn layer, we append eighteen (corresponding nine relations for each observation) embeddings to the sequence before passing through the layers of the Transformer ar- chitecture. This allows the model to learn each token's 's representation while attending to the COMeT embeddings effectively integrating background commonsense knowledge into language model. 10
Discussion Table 2 reports results on the oNLG task. Among automatic metrics, we report BLEU- (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE (Lin, 2004), CIDEr (Vedan- tam et al., 2015) and BERT-Score (Zhang et al., 2019) (with the bert-base-uncasee We establish human performance through crowdsoureing on AMT. Crowdworkers are shown pairs of observations and generated hypothesis and asked to label whether the hypothesis explains the given observations. The last column reports the human evaluation score. The last row reports the score of held-out human-written hypothesis and serves Human- hypotheses are found to be correct for 96% of instances, while our best genera- tive models, even when enhanced with background commonsense knowledge, only achieve 45% indicating that the aNLG generation task is especially challenging for current state-of-the-art text generators.
model).
ceiling for model performance.
Pleease see Appendix A.6 for a full list of the nine relations. We deseribe the format of input for each model in Appendix A.7.
7

---

Published as a conference paper at ICLR 2020
ANALYSIS
6.1
aNLI
Commonsense reasoning categories We in- vestigate the categories of commonsenss--asee abductive reasoning that are challenging for current systems and the ones where the best model over-performs. While there have been previous attempts to categorize commonsense knowledge required for entailment (LoBue & Yates, 201 I; Clark al., 2007), crowdsoure- ing this task at scale with high fidelity and high agreement across annotators remains challeng- ing. Instead, we aim to probe the model with soft categories identified by matching lists of category- keywords to the hypothesis choices.
Category
Human Accuracy
BERT Accuracy
A
All (1, 000)
91.4
68.8
22.6
Numerical (44)
88.6
56,8
21.8
Spatial (130)
91.5
65.4
26.1
Emotional (84)
86.9
72.6
14.1
Table BERT's performance and human evalua- tion on categories for 1,000 instances from the test set, based on commonsense reasoning domains (Numerical, Spatial, Emotional). The number in parenthesis indicates the size of the category.
Table 3 shows the accuracy of the best model (BERT-fi) across various categories of commonsense knowledge. BERT-ft significantly underperforms on instances involving Numerical (56.8%) and Spatial (65.4%) commonsense. These two categories include reasoning about numerical quantities and the spatial location agents and objects, and highlight some of the limitations of the language models. In contrast, it significantly overperforms the Emotional category (72.6%) where the hypotheses exhibit strong textual cues about emotions and sentiments.
Implausible transitions A model an in- stance of the ART dataset should discard im- plausible hypotheses in the context of the two given observations. In narrative contexts, there are three main reasons for an implausible hy- pothesis to be labeled such:
BERT-R Fully Connected Acc. (%)
Story Transition
of Dataset
Linear Chain Acc. (%)
OLAAT
32.5
73.6
71.6
A +A02
45.3
69.0
70.5
Plausible
22.2
62.5
58.5
1. O AIAh: h- is unlikely to follow after the first observation O,
(1,000)
100.0
Table 4: Fraction of dataset for which particular transition in the story broken the negative hypothesis, for 1,000 random instances from the test set.
2. h- 4400 h is plausible after O₁ but un-
likely to precede the second observation O2-
3. Plausible: (O1, h O2) is coherent narra- tive and forms a plausible alternative, but is less plausible than (O1, h+, O2)-
We analyze the prevalence of each of these reasons in ARI. We design crowdsoureing task in which we show the implausible option along with the narrative context (O,, O2) and get labels for which transition (01 h /02 or neither) in the narrative chain is broken. Table shows the proportion of each category from subset of 1, 000 instances from the test set. While h A02 accounts for almost half of the implausible transitions ARI three categories are substantially present in the dataset. BERT performance finds it particularly hard when the narrative created by the incorrect hypothesis is plausible, but less plausible than the correct hypothesis. On that subset of the test set, the fully connected model performs better than the linear chain model where it is important to consider both observations jointly to arrive the more likely hypothesis.
each of these categories indicates that the model
6.2 aNLG
Figure shows some examples of generations from the trained models compared to numan-writted generations. The example the left is example of instance that only humans could get correct, while for the one on the right, COMeT ROMeT-Emba Emb+GPT2also generates the correct explanation for the observations.
8

---

Published as a conference paper at ICLR 2020
Generated Hypotheses
Larryss yard was covered dead leaves.
year
X
aoddent, he spent your fring motherss busement,
actually,
X
wordened what fur could oo with Uei luovas.
01-02.Only
Junior mude jumo ropi.
X
X
deeidded pull
TXP+GPT2
brohens
her inturnal
X
Lary
EnbrGGT2
owiniiing the poii
aparri hours trying to clean the yard.
Human-riiten
Ve tonk Junier vet to chenk him
Larry decided give day inside.
Junior still going strong.
Figure 6: Examples of generated hypotheses from different models and human- written hypothesis for 2 instances from ART.
7 TRANSFER LEARNING FROM ART
ARTcontains a large number of questions for the novel abductive reasoning task. addition to serv- ing as a benchmark, we investigate if can be used as a resource to boost performance on other commonsense tasks. We apply transfer learning by first training model on A and subsequently training four target datasets WinoGrande Sakaguchi al. (2020), WSC Levesque et al. (201 I), DPR Rahman & Ng (2012) and HellaSwag Zellers et al. (2019). We show that compared to a model that is only trained the target dataset, model that is sequentially trained on ART first and then on the target dataset can perform better. In particular, pre- mre-trannnng on ART consistently improves performance related datasets when they have relatively few training examples.
On the other hand, for target datasets with large amounts of training data, pre-training A does not provide a significant improvement.
BERT-RI-
Dataset
BERT-R(D
BERT-D(D)
WineGrande Sakaguchi et (2020)
65.8%
67.2%
RELATED WORK
WSC
Levesque et al. (201
DPR
Rahman & Ng (2012)
Cloze-Style Task vs. Abductive Reasoning Since abduction is fundamentally concerned with plausible chains of cause-and-effect, our work draws inspiration from previous works that deal with narratives such as script learning (Schank & Abelson, 1975) and the narrative cloze test (Chambers & Jurafsky, 2009; Jans et al., 2012; Pichotta & Mooney, 2014; Rudinger et al., 2015). Rather than learning prototypical scripts or narrative chains, we instead reason about the most plausible events conditioned observations. We make use of the ROCStories dataset (Mostafazadeh et al., 2016), which was specifically designed for the narrative cloze task. But, instead of reasoning about plausible event sequences, our task requires reasoning about plausible explanations for narrative omissions.
Hellaswag
46.7%
46.1%
Zellers et al. (2019)
Table 5: Transfer Learning from ART
Entailment vs. Abductive Reasoning The formulation of oNLI is closely related to entailment NLI, but there are two critical distinctions that make abductive reasoning uniquely challenging. First, abduction requires reasoning about commonsense implications of observations (e.g., if we observe that the "grass is wet", a likely hypothesis that "it rained earlier") which go beyond the linguistic notion of entailment (also noted by Josephson (2000)). Second, abduction requires non-monotonic reasoning about a set of commonsense implications collectively, to check the potential contradictions against multiple observations and to compare the level of plausibility of different hypotheses. This makes abductive reasoning distinctly challenging compared to other forms of reasoning such as induction and deduction (Shank, 1998). Perhaps more importantly, abduction is closely related to the kind of reasoning humans perform in everyday situations, where information incomplete and definite inferences cannot be made.

---

Published as a conference paper at ICLR 2020
Generative Language Modeling Recent advancements in the development of large-scale pre- trained language models (Radford, 2018; Devlin et al., 2018; Radford et al., 2019) have improved the quality and coherence of generated language. Although these models have shown to gener- ate reasonably coherent text when condition on sequence of text, our experiments highlight the limitations of these models to 1) generate language non-monotonically and 2) adhere to common- sense knowledge. We attempt to overcome these limitations with the incorporation of generative commonsense model during hypothesis generation.
Related Datasets Our resource AUT complements ongoing efforts in building resources for natural language inference (Dagan et al., 2006; MacCartney & Manning, 2009; Bowman et al., 2015; Williams et al., 2018a; Camburu et al., 2018). Existing datasets have mostly focused on textual entailment in a deductive reasoning set-up (Bowman et al., 2015; Williams et al., 201 18a) and making inferences about plausible events (Maslan al., 2015; Zhang et al., 2017). In their typical setting, these datasets require system to deduce the logically entailed consequences of given premise. In contrast, the nature of abduction requires the use of commonsense reasoning capabilities, with less focus on lexical entailment. While abductive reasoning has been applied entailment datasets (Raina al., 2005), they have been applied in a logical theorem-proving framework as an intermediate step to perform textual entailment fundamentally different task than oNLI
CONCLUSION
We present the first study that investigates the viability language-based abductive reasoning. We conceptualize and introduce Abductive Natural Language Inference (aNLI) novel task focused on abductive reasoning narrative contexts. The task formulated as a multiple-choice question- answering problem. We also introduce Abductive Natural Language Generation (aNLG) novel task that requires machines to generate plausible hypotheses for given observations. To support these tasks, create and introduce new challenge dataset, IRT,, which consists of 20,000 commonsense narratives accompanied with over 200,000 explanatory hypotheses. our experiments, we establish comprchensive bascline performance this new task based on state-of-the-ar NLI and language models, which leads to 68.9% accuracy with a considerable gap with human performance (91 (9..4%). The aNLG task is significantly harder while humans the best generator models can only achieve 45%. Our analysis leads to new insights into the types of reasoning that deep pre-trained language models fail perform despite their strong performance on the closely related but different task of entailment NLI pointing to interesting avenues for future research. We hope that ART will serve as a challenging benchmark for future research language- based abductive reasoning and the oNLI and oNLG tasks will encourage representation learning that enables complex reasoning capabilities Al systems.
write valid explanation 96% of times,
ACKNOWLEDGMENNS
We thank the anonymous reviewers for their insightful feedback. This research was supported in part by NSF Grant No. DGE 1256082, DARPA CwC through ARO (W91 INF15-1 0543), DARPA MCS pro- gram through NIWC Pacific (N66001-1-22-400 beaker. org were supported in part by credits from Google Cloud.
the National Science Foundation Graduate Research Fellowship under
and the Allen Institute for AI. Computations on
REFERENCES
Henning Andersen. Abductive and deductive change. Language, pp. 765-793, 1973. URL https: / //www. jstorr forg/stabless http//ff/11200// pdf.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
10

---

Published as a conference paper at ICLR 2020
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv
preprint ar AKKYv:1906003317,, 2019.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015. URL https: wtw.fordsafoood..oomoo wdt/ saaids//nnnii paper pdl.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, pp. 9560-9572. 2018. URL https: nips cc cc/paper/ 81 63--- sil-e-snl-t - -aauuraa - pdf.
Nathanael Chambers and Dan Jurafsky. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACI. and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 602-610, Suntec, Singapore, August 2009, Association for Computational Linguisties. URL http: /www. ac aclweb. org/ httaaanthloly/////ooo /ttp://////////0005.22
Eugene Charniak and Solomon Eyal Shimony. Probabilistic semantics for cost based abduction. Brown University, Department of Computer Science, 1990. URLhttps: /www. aaai. org/
Papers/AAAI 1990 AAAI90 016.0.p........
Qian Chen, Xiao-Dan Zhu, Zhen- Ling, Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. ACL, 2017. URL https: http:://wwwwww http://wwwwwwww.c/ppwpt/p// aclweb. hntprg//nthoog//// P17-1155
Peter E. Clark, Philip Harrison, John A. Thompson, William R. Murray, Jerry R. Hobbs, and Chris- tiane Fellbaum. On the role of lexical and world knowledge rte3. In ACL-PASCALLACL, 2007. URL https: / www. aclweb. org/ anthology/w07- w
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 670- 680, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 00.18653/v1/D17--U0
URL https: / www. aclweb. org/ antthlogy/DD17 /D17-1070
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177-190. Springer, 2006. URL http:
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.0480) 2018.
URL hwtptttt////c /ttp:s://axxxv.orrrr www.siiiiorrrroooooooooo abs/1810 04805.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/1/NN18-2017 www.... . a e web. org httg//nhhology.. /N1 8 2017.
URL https::////
Jerry R. Hobbs, Mark Stickel, Paul Martin, and Douglas Edwards. Interpretation as abduction. In Proceedings of the 26th Annual Meeting of the Association for Computational Linguisties, pp. 95-103, Buffalo, New York, USA, June 1988. Association for Computational Linguistics. doi: 10.3115/98202399922 URL https: www ac web. org/ htt.//atooooyyyoo woa.llloyyyyy/8111111
Bram Jans, Steven Bethard, Ivan Vulić, and Marie-Francing Moens. Skip n- n-grams and ranking func- tions for predicting script events. In Proceedingy of the 13th Conference of the European Chapter
11

---

Published as a conference paper at ICLR 2020
of the Association for Computational Linguistics, pp. Avignon, France, April 2012. As-
sociation for Computational Linguistics. URL
www. http://www.........//.attattttoa
E12-1034.
Susan G. Josephson. Abductive inference: Computation, philosophy technology. 2000. URL
htttp:://phippapeee org/ azg/rec/JOSAIC.
George Lakoff. Linguisties and natural logic. Synthese, 22(1-2)1111271, 1970. URL https:
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, 2011.
Chin-Yer Lin. Rouge: package for automatic evaluation of summaries. Branches Out, 2004.
Text Summarization
Peter LoBue and Alexander Yates. Types of common-sense knowledge needed for recogniz- ing textual entailment. In ACL, 2011. URL P P11-2057.
Bill MacCartney and Christopher D. Manning. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 193-200, Prague, June 2007. Association for Computational Linguistics. URL https://ww//t aclwebs org/ anthology/W07-111
Bill MacCartney and Christopher D. Manning. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pp. 140-156, Tilburg, The Netherlands, January 2009, Association for Computational Linguistics. URL https: //www. aclweb. org/ antho logy ogy -
Nicole Maslan, Melissa Roemmele, and Andrew S. Gordon. One hundred challenge problems for logical formalizations of commonsense psychology. In AAAI Spring Symposia, 2015. URL http:// /ttp:://pppppl....m wetpople..c....... ict usc. wwtpd///orrooopuulltttttaataeeea /odo
weaetiinss/AAAASPPIAGOOOO PDF.. PDF.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Asvociation for Computational Linguistics: Human Language Technologies (NAACL), pp- 839-849. Association for Computational Linguisties, 2016. doi: 10.18653/v//N16- 1098. URL http: /ttp:://clweb.... org/ anthology /N16 6- 1098.
Peter Norvig. Inference text understanding. In AAAI, pp. 561-565, 1987. URL http://
norvig cwm/ aaai87 pdf.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Judea Pearl. Reasoning with cause and effect. Al Magazine, 23(1):95, 2002. URL https:// ftp. cs. www...ww.......ee... httpp/pbb/stttterr22333333 ai www.aagppp...
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, Inc., New York, NY. USA, 1st edition, 2018. ISBN 046509760X, 9780465097609, URL https: htt.//ll.aa acm. wtt.citationn..n.ooo/i/oo/// cfm?:d-32332300
Charles Sanders Peirce. Collected papers of Charles Sanders Peirce, volume 5. Harvard Uni- versity Press, 1965a. URL http:/ /wwww hup haavaadd.edu/cat..... phppishpp sbo=
9780674138001.
Charles Sanders Peirce. Pragmatism and pragmaticism, volume 5. Belknap Press of Harvard Uni- versity Press, 1965b. URL htttps://wwe http://ww.jtttttttooo http://ttorrrrg/ttaae//////////
12

---

Published as a conference paper at ICLR 2020
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1550000 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 10.3115/v1/114-111 URL https:: /www,ww.......o// a a web. org/ anthol ogy/ D14-1162.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, URL
June 2018. Association for Computational Linguistics. doi: 10.18653/11/N18-122 https: / www. wwwwweeeeee... org arrganntologyy///// 8
Karl Pichotta and Raymond Mooney. Statistical script learning with multi multi-argument events. In Pro- ceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 220-229, Gothenburg, Sweden, April 2014. Association for Computational Lin- guistics. URL http::htt////... /www ac webo org/ hntho/nttoooooooooo wwwhhholoyyyllll 40244
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10118863/v/SS 1/1/ //www. awweeb. org/ / ant
1/t///////1//10 8 2023. URL https:
2023.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The winograd schema challenge. In EMNLP-CoNLL, 2012.
Rajat Raina, Andrew Y Ng. and Christopher D Manning. Robust textual inference via learning and abductive reasoning. AAAI, pp. 1099-1105, 2005. URL https: stanford. edu/ "manninggpppprs////eeeeeeeee aatiss//aaaa0 05 www.naeductioon..oooooaoo ptd.
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pp. 1681-1686, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL http: //acl htt/w/lweebbb.... www.org///...e.................. anthol www.thoogg//.nnno-ooo ogy//15-111119999
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In AAAI, 2020.
Maarten Sap. Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah Smith, and Yejin Choi. Atomic: an atlas of machine com- monsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3027-3035 2019.
Roger C. Schank and Robert P. Abelson. Scripts, plans, and knowledge. In Proceedings of the 4th International Joint Conference on Artificial Intelligence Volume 1, IJCAI'75 pp. 151-157, San Francisco, CA, USA, 1975. Morgan Kaufmann Publishers Inc. URL http: //dl//l///// acm. org/ citation. cfm? i d= 62 626.626662 644
Gary Shank. The extraordinary ordinary powers of abductive reasoning. Theory Psychol- ogy, 86))841- 1998. URL https: //jornals sagepub.com. wwwm/ooo/o..... 0959354398086007.
Masatoshi Tsuchiya. Performance impact caused by hidden bias of training data for recognizing textual entailment. CoRR, abs/1804.08117,, 2018. URL http: /wwwww/w.. 1rec- conf. org/ proceedi http:////e////////////////1 .df.
13

---

Published as a conference paper at ICLR 2020
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guisties. URL https: / www. acc weabbborg/attoo ogy/ 84446.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverape challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pp. 1112-1122. Association for Computational Linguistics, 201 URL http::/ alw.bblw.bbb........ www.g/ganaooloor.o..ooo.oo /N1 B-1101
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018b. Association for Computational Linguistics. doi: 10.18653/11/118-11000 URL https: www. acl web. org/ hrgg/athology/Ni1 1101.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference Em- pirical Methods Natural Language Processing (EMNLP), 2018. URL https: //cp://///cwwbb... org/anthelogy /D18 1009.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In ACL, 2019.
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379-395, 2017. doi:
10.1162/tacl 00068. URL https: //tp://wwwww....... wwww www.eb. wttp://..........ooooooo/o////// 001000177111111
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: (rXiv:1904009675 2019.
14

---

Published as a conference paper at ICLR 2020
A APPENDICES
A.1 DATA COLLECTION DETAILS
We describe the crowdsourcing details of our data collection method.
Task 1 Plausible Hypothesis Options In this task, participants were presented an incomplete three-pan paree-patttoe story. which consisted of the first observation (O1) and the second observation (O2) of the story. They were then asked to complete the story by writing a probable middle sentence that explains why the second observation should follow after the first one. We instructed participants to make sure that the plausible middle sentence (1) is short (fewer than 10 words) and (2) simple as narrating to child, (3) avoids introducing any extraneous information, and (4) uses names instead of pronouns (e.g., he/she) wherever possible.
All participants were required to meet the following qualification requirements: (1) their location is in the US, (2) HIT approval rate is greater than (5%), and (3) Number of HITs approved is greater than 5,000. The reward of this task was set be $0.07 per question ($14/hour in average). and each HIT was assigned to five different workers (i.e., 5-way redundancy).
Task 2 Implausible Hypothesis Options In this task, participants were presented a three-part story, which consisted of the first observation (O1), a middle sentence (h*) collected in Task 1, and the second observation (O2) of the story. They were then asked to rewrite the middle sentence (h+) with minimal changes, so that the story becomes unlikely. implausible or inconsistent (h-). We asked participants to add or remove at most four words to h+t, while ensuring that the new middle sentence is grammatical. In addition, we asked them to stick to the context in the given story. For example, if the story talks about "doctors", they are welcome to talk about "health" or "diagnosis", but not mention "aliens". Finally, we also asked workers to verify if the given middle (h+) makes plausible story, in order to confirm the plausibility of h+ collected in Task 1.
With respect to this task's qualification, participants ments: (1) their location is the US or Canada, (2) HIT approval rate is greater than or equal to 99(% (%), and (3) number of HITs approved is greater than or equal 10, 000. Participants were paid $0.1 per question ($1 ($14/hour in average), and each HIT was assigned to three different participants (i.e., 3-way redundancy).
required fulfill the following require-
Task 3 - aNLI Human Performance Human performance was evaluated by asking participants answer the oNLI questions. Given a narrative context (O), O2) and two hypotheses, they were asked to choose the more plausible hypothesis. They were also allowed to choose "None of the above" when neither hypothesis was deemed plausible.
We asked each question to seven participants with the following qualification requirements: (1) their location either in the US, UK, or Canada, (2) HIT approval rate is greater than 98(%), (3) Number of HIT's approved is greater than 10, 000. The reward was set to $0.05 per HIT. We took the majority vote among the seven participants for every question to compute human performance.
A.2 ART DATA STATISTICS
Table 6 shows some statistics of the dataset.
A.3 FINE-TUNING BERT
We fine-tuned the BERT model using a grid search with the following set of pappmetrrametrs::
batch size: {3, 4, 8}
number of epochs: {3, 4, 10}
learning rate: (1e-5, 2c-5, 3c-5, 5e-5}
used for computing the loss. The best performance was obtained with a batch size of 4, learning rate of 5e-5, and number of epochs equal to 10. Table describes the input format for GPT and BERT (and its variants).
The warmup proportion set to 0.2, and cross-entropy
15

---

Published as a conference paper at ICLR 2020
Train
Dev
Test
Total unique occurrences
Contexts (O O2)
17,801
1,532
3,059
Plausible hyp. h+
72,046
1,532
3,059
Implausible hyp. hT
166,820
1,532
3,059
Avg. size per context
Plausible hyp. hut
4.05
1
Implausible hyp. h
9.37
Avg. word length
Plausible hyp. 4+
8.34
8.62
8.54
Implausible hyp. /
8.28
8.55
8.53
First observation O1
8.09
8.07
8.17
Second observation O2
9.29
9.3
9.31
Table 6: Some statistics summarizing the 2श dataset The train set includes all plausible and im- plausible hypotheses collected via crowdsoureing, while the dev and test sets include the hypotheses selected through the Adversarial Filtering algorithm.
A.4 BASELINES
The SVM classifier is trained on simple features like word length, overlap and sentiment features to select one of the two hypothesis choices. The bag-of-words bascline computes the average of GloVe (Pennington et al., 2014) embeddings for words in each sentence to form sentence embeddings. The sentence embeddings
story (two observations and hypothesis option) are concatenated and passed through fully-connected layers to produce a score for each hypothesis. The accuracies of both baselines are close to 50% (SVM: 50.6; BOW: 50.5).
Specifically, we train an SVM classifier and a bag-of-words model using L..Veeeemeddding Both models achieve accuracies close to 50%. An Infersent (Conneau et al., 2017) baseline that uses sentences embedded by max-pooling over Bi-LSTM token representations achieves only 50.8% accuracy.
Model
Input Format
GPT
[START] O h² [SEP] O2 [SEP]
BERT-ft [Hypothesis Only]
[CLS] h² [SRP]
BERT-ft
[First Observation Only]
[CLS] O [SEF] h² [SEP]
BERT-ft [Second Observation Only]
[CLS] A [SEP] Oz [SEP]
BERT-ft [Linear Chain]
[CLS] O [SEP] ] A² [SEP] : [CLS] A² [SEP] O2 (SEP)
BERT-ft [Fully Connected]
[CLS] O + O2 [SEP] h' [SEP]
Table 7: Input formats for GPT and BERT fine-tuning.
A.5 ADVERSARIAL FILTERING HYPOTHESES CHOICES
Given an observation pair and sets of plausible and implausible hypotheses (O,, Oz. H+ H our adversarial filtering algorithm selects one plausible and one implausible hypothesis (O,, Oz, ht h- ) such that h+ and h- are hard to distinguish between. We make three key improvements over the previously proposed Adversarial Filtering (AF) approach in Zellers et al. (2018). First, Instead of single positive sample, we exploit pool H of positive samples to choose from (i.e. plausible hypotheses). Second, Instead machine generated distractors, the pool 7. of negative samples (i.e. implausible hypotheses) is human-gent generated. Thus, the distractors share stylistic features of the positive samples as well as that of the context (i.e. observations O and O₂) making the negative samples harder to distinguish from positive samples. Finally, We use BERT (Devlin al., 2018) as
16

---

Published as a conference paper at ICLR 2020
the adversary and introduce temperature parameter that controls the maximum number of instances that can be modified in each iteration AF. In later iterations, fewer instances get modified resulting in smoother convergence of the AF algorithm (described in more detail below).
Algorithm provides a formal description of our approach. In each iteration i, we train an adver- sarial model M; random subset of the data and update the validation set V; to make it more challenging for M, For a pair ( h, of plausible and implausible hypotheses for an instance k we denote 5 = M, ( kg the difference in the model evaluation of hig and A1z- A positive value of 8 indicates that the model M, favors the plausible hypothesis hg over the implausible one hg- With probability 4,, we update instance & that M, gets correct with a pair (h+ h-) E H x H of hypotheses that reduces the value of 6, where H. (resp. H₂) is the pool of plausible (resp- implausible) hypotheses for instance k.
We ran AF for 50 iterations and the temperature 4; follows sigmoid function, parameterized by the iteration number, between 44 = 1.0 and Le 0.2. Our final dataset, 2RT, is generated using BERT as the adversary in Algorithm 1.
Algorithm 1: Dual Adversarial Filtering
input dataset Do, plausible & implausible hypothesis sets (H+ H 7),, number of iterations n, initial & final temperatures (ts, te)
output: dataset 'D,
for iteration 0..n do
1 0.36 4F)
3 Randomly partition D; into (Ti, Vi).
Train model M; on T7.
5 S. = 0. the selected hypotheses for V3.
for (h h5)) E Vi do
7
Pick uniformly at random in [0, 1].
ifr t, or M, (ht hī) then Add (ht hw to S
else
Pick (h+ h h) E Hi x Hy s.t. Oht, < AM. (M (ht hp) Add (h+ h to S..
11
12
end
end
15
Di+1 = T, U.
16
end
A.6 ATOMIC RELATIONS
ATOMIC (Sap et al., 2019) represents commonsense knowledge as a graph with events are nodes and the following nine relations edges:
1. xIntent: Why does X cause an event?
2. xNeed: What does X need to do before the event?
3. xAttr: How would X be described?
4. xEffect: What effects does the event have on X?
5. xWant: What would X likely want to do after the event?
6. xReaction: How does X feel after the event?
7. oReact: How do others' feel after the event?
8. oWant: What would others likely want to do after the event?
9. oRffect: What effects does the event have others?
17

---

Published as a conference paper at ICLR 2020
Model
Input Format
op12-Pixed
wi u wf n Because,
O1-O2-Only
(ol)u w (/ol) w (1)))(h))
COMeT-TXt+GP12
(1))T) To (oa) (Ff)Ty. (p)) ... w. (01)) ... wtt (211/22)1))
COMeT-Emb-GPT2
f...ch.e....coooly.... ..0o)/ ... w (o1) wa (1)
Table 8: Input format used to training and generated text from various GPT2 based models. refers to the COMeTembedding obtained using separate transformer model for relation i and observation Similarly, T` is the textual phrase for relation i, observation j. Where appropriate, field specific start and end-tags are added to the sequence of inputs.
A.7 GENERATION MODELS INPUT FORMAT
Table 8 describes the format of input to each variation of the generative model evaluated.
18


Reviews:
[{"review_id": "ByltNYc0tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new task/dataset for language-based abductive reasoning in narrative texts.\n\nPros: \n\n-\tThe proposed task is interesting and well motivated. The paper contributes a dataset (20,000 commonsense narratives and 200,000 explanatory hypotheses). The construction of the dataset was performed carefully (e.g., avoiding annotation artifacts).  \n\n-\tThe paper established many reasonable baselines.\n\n-\tThe paper conducted detailed analysis, which invites more research on this task: despite the strong performance of many existing systems on NLI/RTE, there are larger gaps between the performance of these models and human performance on the proposed task. The experiments well support the conclusions made in the paper.\n\n-\tThe paper is well structured and easy to follow. It is well written.\n\nCons/comments: \n\n-\tWhile this is a new and interesting task, the contribution (as discussed above in “pros” above) is somewhat limited. I also suggest the paper discusses e-SNLI a bit more. \n\n-\tThe paper has a specific form of formulation for abductive reasoning, where there are exactly two observations and one proceeds the other; the explanation happens in between. I can see this helps collect and annotate data, but also limit the form of abductive reasoning and how models should be developed. \n\n-\tShould the title of the paper specify the paper is about “language-based” abductive reasoning. \n\n-\tA minor one: “Table 7 reports results on the αNLI task.” Should it be “Table 2”?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklqO2_AKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces two new natural language tasks in the area of commonsense reasoning: natural language abductive  inference and natural language abductive  generation. The paper also introduces a new dataset, ART, to support training and evaluating models for the introduced tasks. The paper describes the new language abductive tasks, contrasting it to the related, and recently established, natural language inference (entailment) task.  They go on to describe the construction of baseline models for these tasks. These models were primarily constructed to diagnose potential unwanted biases in the dataset (e.g., are the tasks partially solved by looking at parts of the input, do existing NLI models far well on the dataset, etc.), demonstrating a significant gap with respect to human performance.\n\nThe paper, and the dataset specifically, represent an important contribution to the area of natural language commonsense reasoning. It convincingly demonstrates that the proposed tasks, while highly related to natural language entailment, are not trivially addressed by existing state-of-the-art models. I expect that teaching models to perform well on this task can lead to improvements in other tasks, although empirical evidence of this hypothesis is currently absent from the paper.\n\nBelow are a set of more specific observations about the paper. Some of these comments aim to improve the presentation or content of the paper.\n\n1. In Section “5.1 - Pre-trained Language Models” and attendant Table 1describe results of different baselines on the ART inference task. The results in the table confused me for quite some time, I’d appreciate some clarifications. With respect to the differences with columns 1 (GPT AF) and 2 (ART, also BERT AF) I would like the comparison to be made more clear. As far as I understand it, there are 2 parts of the dataset that can be varied: (1) the train+dev sets and (2) the test set. Furthermore, it seems that it makes sense to vary each of these at a time, if we are to compare results with variants. For example: we can fix the test set, and vary how we generate training and dev examples. If a model does better with the same test set, we can assume the train+dev examples were better for the model (for whatever reasons, closer distribution to test, harder or more \ninformative training examples, etc). We can also keep the train+set constant, and vary the test set. This allows us to evaluate which test set is harder with respect to the training examples. The caption of Table 1 implies that both columns are evaluations based on the “ART” test set. If that is correct, then the train+dev set generated from the GPT adversarial examples is of better quality, generating a BERT-ft (fully connected) model that is 3% better. But the overall analysis seems to indicate that this is not what was done in the experiment. Rather, it seems that *both* the train+dev _and_ the test sets were modified concurrently. If that is the case, I would emphasize that the text needs to make this distinction clear. Furthermore, I would say that varying both train and test sets concurrently is sub-optimal, and makes it a bit harder to draw the conclusion that BERT adversarial filtering leads to a stronger overall dataset.\n\n2. Along the lines of the argument in (1), above, I would urge the authors to publish the *entire* set of generated hypotheses (plausible and implausible) instead of only releasing the adversarially filtered pairs. Our group’s experience with training inference models is that it is often beneficial to train using “easy” examples, not only hard examples. I suspect the adversarially filtered set will focus on hard examples only. While this is fine to do in the test set, I think if the full set of annotated/generated hypotheses are released, model designers can experiment with combining pairs of hypothesis in different ways.\n\n3. Furthering the argument of (2): in certain few-shot classification tasks, one is typically asked to identify similarity between one test example and different class representatives. Experience shows that it is often beneficial to train the model on a larger number of distractor classes than what the model is eventually evaluated on (e.g., https://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning). In the alpha-NLI setting, have you experimented with training using multiple distractors, instead of only 1, during training (even if you end up evaluating over 2 hypotheses)?\n\n4. One potential argument for introducing a new natural language task is of transfer learning: learning to perform a complex natural language task should lead to better natural language models more generally, or, for some other related tasks. This paper does not really touch on this aspect of the work. But, potentially, one investigation that could be conducted is through a reversal of the paper’s existing NLI entailment experiment. The paper shows that NLI entailment models do not perform well on alpha-NLI. But it would be interesting to see if a model trained on alpha-NLI, and fine-tuned or multi-tasked on NLI entailment, does better on NLI entailment (i.e., is there transfer from alpha-NLI to entailment NLI?).\n\n5. Another option is to evaluate whether alpha-NLI helps with other commonsense tasks. One other example is Winograd Schema Challenge, which current systems also perform well below human performance. It also seems that the Winograd Schema Challenge questions are not too far from abductive inference.\n\n6. In the abstract of the paper, before the paper defines the abductive inference/generation task specifically, the claim that abductive reasoning has had much attention in research seemed awkward. Informally, most commonsense reasoning (including NLI entailment) could be cast as abductive reasoning.\n\n7. In at least one occasion, I found an acronym which was hard to find the definition for (“AF” used in Section “5.1 - Pre-trained Language Models”; I assumed it was “adversarial filtering”.)\n\n8. In Section “5.1 - Pre-trained Language Models” it seems that the text quotes an accuracy for BERT-ft (fully connected) of 68.9%, but Table 1 indicates 69.6%.\n\n9. In Section “5.1 - Learning Curve and Dataset Size”, there is a claim that the performance of the model plateaus at ~10,000 instances. This does not seem supported by Figure 5. There appears to be over 5-7% accuracy (absolute) improvements from 10k to 100k examples. Maybe the graph needs to be enhanced for legibility?\n\n10. It is great that the paper includes human numbers for both tasks, including all the metrics for generation.\n\n11. Period missing in footnote 8.\n\n12. The analysis section is interesting, it is useful to have in the paper. However, Table 3 is a bit disappointing in that ~26% of the sampled examples fit into one of the categories. It would be great if the authors could comment on the remaining ~74% of the sampled dataset.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlqUqW3tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the paper purposes a dataset of abductive language inference and generation. The dataset is generated by human, while the testing set is adversarially selected using BERT. The paper experiments the popular deep learning models on the dataset and observe shortcoming of deep learning on this task.\n\nComments: overall, the problem on abductive inference and abductive generation in language in very interesting and important. This dataset seems valuable. And the paper is simple and well-written.\n\nConcerns: I find the claim on deep networks kind of irresponsible. \n1. The dataset is adversarially filtered using BERT and GPT, which gives deep learning model a huge disadvantage. After all, the paper says BERT scores 88% before the dataset is attacked. \n2. The human score of 91.4% is based on majority vote, which should be compared with an ensemble of deep learning prediction. To compare the author should use the average score of human.\n3. The ground truth is selected by human.\n\nOn a high level, the main difficulty of abduction is to search in the exponentially large space of hypothesis. Formulating the abduction task as a (binary) classification problem is less interesting. The generative task is a better option.\n\nDecision: despite the seeming unfair comparison, this task is novel. I vote for weak accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a dataset, created using a combination of existing resources, crowdsourcing, and model-based filtering, that aims to tests models' understanding of typical progressions of events in everyday situations. The dataset represents a challenge for a range of state of the art models for NLP and commonsense reasoning, and also can be used productively as a training task in transfer learning.

After some discussion, reviewers came to a consensus that this represents an interesting contribution and a potentially valuable resource. There were some concerns—not fully resolved—about the implications of using model-based filtering during data creation, but these were not so serious as to invalidate the primary contributions of the paper.

While the thematic fit with ICLR is a bit weak—the primary contribution of the paper appears to be a dataset and task definition, rather than anything specific to representation learning—there are relevant secondary contributions, and I think that this work will be practically of interest to a reasonable fraction of the ICLR audience. 
Decision: Accept (Poster)