Intro
Reasoning is part of our every day routine: we hear Natural Language (NL) sentences, we participate in dialogues, we read books or legal documents. Successfully understanding, participating or communicating with others in these situations presupposes some form of reasoning: about individual sentences, whole paragraphs of legal documents, small or bigger pieces of dialogue and so on. The human reasoning performed in these different situations cannot be explained by a single rigid system of reasoning, plainly because reasoning is performed in different ways in each one of them. Consider the following example:
Three representatives are needed.
If a human reasoner with expert knowledge was to interpret the above utterance in a legal context, s/he will most probably judge that a situation where more than three references are provided could be compatible with the semantics of the utterance. To the contrary, if the same reasoner was to interpret the above as part of a casual, everyday conversation, then three would most likely be interpreted as exactly three, making the same situation incompatible with the utterance. In this paper, we want to focus on precise inference, inference which is either performed by experts or normal people after taking some time to consider the inferences that follow or not from a set of premises. The problem one encounters in testing systems of this type of reasoning is the fact that no large scale datasets of this sort of inference exist. The commonly used datasets for systems fit for this type of precise reasoning, i.e. logical systems based on some model of formal semantics for Natural Language (NL), are the FraCaS test suite and the Recognizing Textual Entailment (RTE) datasets.
The FraCaS test suite
The FraCaS test suite is an NLI data set consisting of 346 inference problems. Each problem contains one or more premises followed by one yes/no-question. There is a three way classification: YES, NO or UNK (unknown, see Figure 1 for an example from FraCaS). The FraCaS test suite was later on turned into machine-readable format by Bill McCartney
Expansions of FraCaS include: a) MultiFraCaS, in effect a multilingual FraCaS, and b) JSem, the Japanese counterpart to FraCaS, which expands the original FraCaS in a number of ways.
Even though the FraCaS test suite contains a rather small number of examples (346), it covers a lot of NLI cases and is, at least to some extent, multilingual. It is to some extent precise, even though there are test cases that do not involve a clear answer and thus are dubbed as undefined in Bill MacCartney's XML version. A further drawback of the FraCaS test stuite is that it involves constructed examples, rather than real text.
An UNK example from the FraCaS test suite.
A Scandinavian won the Nobel Prize.
Every Swede is Scandinavian.
Did a Swede win the Nobel prize?
A Swede won the Nobel prize.
UNK [FraCaS 065]
Recognizing Textual Entailment
The Recognizing Textual Entailment (RTE) challenges first appeared in 2004 as a means to test textual entailment, i.e. relations between a premise text and a hypothesis text ( "Recognizing Textual Entailment" ):
An entailment example from RTE1.
Budapest again became the focus of national political drama in the late 1980s, when Hungary led the reform movement in eastern Europe that broke the communist monopoly on political power and ushered in the possibility of multiparty politics.
In the late 1980s Budapest became the center of the reform movement.
Entailment [RTE702]
In contrast to the FraCaS test suite, the RTE challenges use naturally occurring data as premises. The hypothesis text is then constructed based on this premise text. There is either a binary or a tripartite classification of entailment — depending on the version of RTE. The first two RTE challenges follow the former scheme and make a binary classification of entailment (entailed or not entailed). Tripartite classification (entailment, negation of the hypothesis entailment or no entailment) is added in the later datasets, retaining two way classification versions as well. Seven RTE challenges have been created altogether.
The main advantages of the RTE challenges is their use of examples from natural text and the inclusion of cases that require presupposed information, mostly world knowledge. Indeed, the very definition of inference assumed in a number of the examples is problematic. As BIBREF1 have pointed out, RTE platforms suffer from cases of inference that should not be categorized as such. For these cases, a vast amount of world knowledge needs to be taken into consideration (that most importantly not every linguistic agent has). In this paper, and having the RTE as our starting point challenges, we claim that RTE is insufficiently precise to perform logical reasoning or precise reasoning tasks and we take up the task of validating our working hypothesis and proposing a method for doing proper collection of precise entailment pairs in the style of RTE. Of course, the creators of RTE had in mind a more loose definition of inference where both a precise and an imprecise definition of entailment would be at play. BIBREF2 mention that “our applied notion of textual entailment is also related, of course, to classical semantic entailment in the linguistics literature... a common definition of entailment specifies that a text t entails another text h (hypothesis, in our terminology) if h is true in every circumstance (possible world) in which t is true." This is close to what we want to capture in this paper. But, at the same time, BIBREF2 also mention that “however, our applied definition allows for cases in which the truth of the hypothesis is highly plausible, for most practical purposes, rather than certain". It is these cases we want to make more precise, in the sense of making the supporting hidden inferences that are at play in many of the RTE examples explicit. In a way, what we are aiming at is a methodology of constructing entailment datasets in the style of RTE, that will involve a more precise definition of entailment and will further record any missing/hidden premises are used in justifying or not an entailment pattern.
Method
We have randomly selected 150 problems out of the RTE corpus which were marked as “YES” (i.e. entailment holds). The problems were not further selected nor doctored by us. The problems were then re-rated by experts in logic and/or linguistics. For each problem, three experts were consulted, and each expert rated 30 problems. More precisely, the experts were instructed to re-consider each problem and be especially wary of missing hypotheses. If they considered the entailment to hold, we still gave the instruction to optionally mention any additional implicit hypothesis that they would be using. Similarly, if they considered that there was no entailment in the problem, they were given prompted to (optionally) give an argument for their judgement.
In order to facilitate data collection, the experts were chosen from the network of contacts of the author. Despite this method, the process of data collection took nearly six months. The authors themselves were put to contribution in the data-collection process (taking one set of 30 problems each) in order to complete the survey.
Additionally, collecting all the inputs received and using our best judgments, we have put together a test set of 150 problems comprised of the original problems, a new judgement (“yes” or “no”), and added missing hypotheses (if “yes” is a reasonable option).
Results
In the process, we have gather a total of 449 expert judgments (one expert failed to answer a given problem), 146 missing hypotheses and 47 explanations for negative judgments.
Despite being marked as “yes”, the problems with a reported missing hypothesis should really be classified as “no”, if one does not assume external knowledge. (See below for further discussion on the reported missing hypotheses.) Thus crunching the numbers, we see that more that half of the responses express some doubt about entailment. Remember that all problems were marked as “yes” by the creators of the RTE3 testsuite — we find here that one average, one expert in two is likely to cast a doubt over this “yes”.
However, each problem was classified by three experts. The histogram below shows the distribution of number of experts casting doubt on entailment, over all problems.
[ybar interval, ymax=0.5,ymin=0, minor y tick num = 0] coordinates (0, 0.26666666666666666) (1, 0.17333333333333334) (2, 0.3466666666666667) (3, 0.21333333333333335) (4,0) ;
Unfortunately we can only draw preliminary conclusions, due to the limited number of respondents for each problem. However, we can make the following observations:
We find this level of agreement indicative of a good level of reliability. Additionally, with three experts per problem, we are very likely to discover most missing hypotheses and incorrect entailments.
In our compilation of answers, we have marked 42 problems as straight “No”, 64 as “Yes” with missing implicit hypotheses and “44” as plain “Yes”. This means that, we expect, in our opinion, 28% of problems to be incorrectly labeled in RTE3 even assuming reasonable world knowledge. An additional 42% of problems require additional (yet reasonable to assume) hypotheses for entailment to hold formally, as prescribed by RTE3. This leaves only 30% of problems to acceptable as such. The reason that the amount of doubt is larger than in the average numbers quoted above is that, for many problems, certain missing hypotheses and/or error were not detected by a majority experts, but, after careful inspection, we judge that the minority report is justified.
We have additionally tagged each missing hypothesis according to the following classification:
“Yes if ...” vs “No because ...”?
The classification between “yes” with missing hypotheses and “no” is sometimes a tenuous one — which is why we elected to group those categories in our summaries above. Indeed, consider the following example:
Example: (Problem 672)
P: P: Philip Morris the US food and tobacco group that makes Marlboro, the world’s best-selling cigarette, shrugged off strong anti-smoking senti- ment in the US.
H: H: Philip Morris owns the Marlboro brand.
We got the following answers:
Yes, if making involves owning the brand
Yes, if making something implies owning the brand
No, because making the product does not imply owning the brand
It is clear for all experts that a premise is missing, but some will consider it acceptable to add, others will not.
Analysis of reported missing hypotheses and incorrect labeling in RTE3
While most errors in the original RTE classification can not be attributed
It appears that wrong conclusions are sometimes justified by appeal to pragmatic strengthening of the premises. (Many problems, at least 175, 50, 51, 454, 643, 722, 740, 278 in our sample). Indeed, in our experts judgments, we have found cases where problems were marked as “yes” in RTE, and seem to have been implicitly justified with a hypothesis which is, taken in isolation, false, but which could make sense in the context of the premises.
This can be problematic in the context of an entailment system. Indeed, the problem does not become “is there entailment”, but rather “does the questioner intend entailment”. To give an example, take a look at the following:
Example: (Problem 454)
P: On Aug. 6, 1945, an atomic bomb was exploded on Hiroshima with an estimated equivalent explosive force of 12,500 tons of TNT, followed three days later by a second, more powerful, bomb on Nagasaki.
H: In 1945, an atomic bomb was dropped on Hiroshima. (Bombs can explode without being dropped.)
The above example was marked as an entailment in the original RTE suite. However, one of our annotators marked as a non-entailment, providing the justification one sees in parentheses. The justification is of course correct. However, one can also claim that given the context of the text, and the fact that one can take dropping of the atomic bombs in Hiroshima and Nagashaki as a recoverable world-knoweldge premise, Yes is also an option. But not an option, if precise inference systems are to be trained.
The single one largest single specific source of incorrect labeling, found in 12 problems in our sample (750, 754, 756, 757, 51, 66, 178, 225, 294, 588,643,659) out of 42 errors, is mistaking claims for truth, as in the following example.
Example: (Problem 294)
P: Mental health problems in children and adolescents are on the rise, the British Medical Association has warned, and services are ill-equipped to cope.
H: Mental health problems increase in the young.
As it should be obvious with a instant's thought, the above should entail only if the word of the British Medical Association can be taken for fact. While it may be safe to behave as such in many situations in the real world, one cannot do so when reasoning precisely.
Another source of common mistakes is the confusion of intentions and facts, found in 8 problems in our sample (33,148,191,396,420,59,121,166).
Example: (Problem 191)
P: Though Wilkins and his family settled quickly in Italy, it wasn't a successful era for Milan, and Wilkins was allowed to leave in 1987 to join French outfit Paris Saint-Germain.
H: Wilkins departed Milan in 1987. (Even though Wilkins was allowed to leave, it does not mean he actually left.)
Rather obviously, events described in the past cannot be taken to hold currently. Yet this error is still found in 6 problems in our sample (255,230,118,308,454,175).
Example: (Problem 308)
P: On 29 June the Dutch right-wing coalition government collapsed. It was made up of the Christian-democrats (CDA) led by Prime Minister Jan Peter Balkenende, the right wing liberal party (VVD) and the so-called 'left-liberal' D66.
H: Three parties form a Dutch coalition government.
(The coalition may have collapsed at the time of solving the problem)
The final common source of errors that we identify is incorrect reasoning with lexical semantics. This error a bit more subtle than the others, but found only five occurrences in our sample (59,202,221,231,463). One explanation for its relatively low frequency is that RTE subject paid special attention to it. To give an example, consider the following:
Example: (Problem 463)
P: Catastrophic floods in Europe endanger lives and cause human tragedy as well as heavy economic losses
H: Flooding in Europe causes major economic losses.
In the above example, incorrect reasoning with adjectival semantics is made. Catastrophic floods forms a subset of all floods. Thus, the hypothesis does not follow, basically because not all floods are catastrophic.
Use of world-knowledge
We find that entailment problems which depend on any non-trivial amount of world knowledge are problematic from the point of view of training and testing systems for entailment. Indeed, in the presence of a large number of arbitrary facts, the conclusion can come solely from such knowledge, completely ignoring the premise. At best, the premise is serving as priming the memory of the reader. We find this issue to happen in RTE in a significant number of cases, including some of those listed above. For example, in problem 454, it is common knowledge that a bomb was indeed dropped on Hiroshima — yet we may consider that the entailment does not hold as such.
Conclusion and Future work
We find that our hypothesis is validated: RTE is not suitable as such to test a precise NLI system, because, for entailment to hold as tagged in RTE, much world-knowledge is required and many missing hypotheses are omitted.
By using a crowd of experts to repair the missing hypotheses, we have constructed a dataset of 150 precise entailment problems, based on text found in real-world corpora. Even though the dataset is on the small size, it is, to the best of our knowledge, the first of this kind.
An issue with the method that we used to construct our new dataset of entailment problems is that it is difficult to scale: it demands several minutes of scarce expert work per constructed problem. Our plan is to investigate the possibility to gamify the process, so that lots of people can participate in the construction of precise entailment problems, as a form of entertainment. We leave any detail to further work, but this would be an asymmetric game, where one one player tries construct watertight entailment problems, and the opposing player would try and refute such entailment problems (say, by giving counter-examples). Identifying the possible kind of mistakes, as we have done here, will help prompting the players about things to look for — in either of the possible roles.
Finally we find it striking that many mistakes committed in RTE are falling for classical fallacies (appealing to authority accounts for more than a quarter of errors). Thus, we believe that while the entailment problems are useful for the construction of NLI systems, the added hypotheses (or moves taken in our hypothetical game) could be of interest to the linguistic community at large.
Acknowledgments
We are grateful to the crowd of experts that performed the hard work of precisely annotating problems. Most of them chose to remain anonymous. The others were, in alphabetical order: Rasmus Blank, Robin Cooper, Matthew Gotham, Julian Hough and Aarne Talman.
Test suite
We join the complete compiled test suite below. Please note that the problem numbers are not consecutive, because we chose them to be their identifier in the RTE3 test suite.

Question: What was the inter-annotator agreement between the expert annotators?
Answer: Unanswerable

Unanswerable

Question: How were missing hypotheses discovered?
Answer: The problems were then re-rated by experts in logic and/or linguistics. , If they considered the entailment to hold, we still gave the instruction to optionally mention any additional implicit hypothesis that they would be using.

More precisely, the experts were instructed to re-consider each problem and be especially wary of missing hypotheses. If they considered the entailment to hold, we still gave the instruction to optionally mention any additional implicit hypothesis that they would be using.