Question: What is the difference between the FraCaS test suite and the RTE challenges?
Evidence: The Recognizing Textual Entailment (RTE) challenges first appeared in 2004 as a means to test textual entailment, i.e. relations between a premise text and a hypothesis text ( "Recognizing Textual Entailment" ):
An entailment example from RTE1.
Budapest again became the focus of national political drama in the late 1980s, when Hungary led the reform movement in eastern Europe that broke the communist monopoly on political power and ushered in the possibility of multiparty politics.
In the late 1980s Budapest became the center of the reform movement.
Entailment [RTE702]
In contrast to the FraCaS test suite, the RTE challenges use naturally occurring data as premises. The hypothesis text is then constructed based on this premise text. There is either a binary or a tripartite classification of entailment — depending on the version of RTE. The first two RTE challenges follow the former scheme and make a binary classification of entailment (entailed or not entailed). Tripartite classification (entailment, negation of the hypothesis entailment or no entailment) is added in the later datasets, retaining two way classification versions as well. Seven RTE challenges have been created altogether.
The main advantages of the RTE challenges is their use of examples from natural text and the inclusion of cases that require presupposed information, mostly world knowledge. Indeed, the very definition of inference assumed in a number of the examples is problematic. As BIBREF1 have pointed out, RTE platforms suffer from cases of inference that should not be categorized as such. For these cases, a vast amount of world knowledge needs to be taken into consideration (that most importantly not every linguistic agent has). In this paper, and having the RTE as our starting point challenges, we claim that RTE is insufficiently precise to perform logical reasoning or precise reasoning tasks and we take up the task of validating our working hypothesis and proposing a method for doing proper collection of precise entailment pairs in the style of RTE. Of course, the creators of RTE had in mind a more loose definition of inference where both a precise and an imprecise definition of entailment would be at play. BIBREF2 mention that “our applied notion of textual entailment is also related, of course, to classical semantic entailment in the linguistics literature... a common definition of entailment specifies that a text t entails another text h (hypothesis, in our terminology) if h is true in every circumstance (possible world) in which t is true." This is close to what we want to capture in this paper. But, at the same time, BIBREF2 also mention that “however, our applied definition allows for cases in which the truth of the hypothesis is highly plausible, for most practical purposes, rather than certain". It is these cases we want to make more precise, in the sense of making the supporting hidden inferences that are at play in many of the RTE examples explicit. In a way, what we are aiming at is a methodology of constructing entailment datasets in the style of RTE, that will involve a more precise definition of entailment and will further record any missing/hidden premises are used in justifying or not an entailment pattern.
In contrast to the FraCaS test suite, the RTE challenges use naturally occurring data as premises.
Answer: In contrast to the FraCaS test suite, the RTE challenges use naturally occurring data as premises.