Textual Entailment

Alicia Ageno
ageno@cs.upc.edu
Universitat Politècnica de Catalunya

Outline• Introduction• RTE in PASCAL and TAC• Techniques and general structure• Relevant Systems

Textual Entailment

2

Readings•

Textual Entailment Community:– The RTE Resource Pool can now be accessed from:http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool– The Textual Entailment Subzone can now be accessed from:http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Portal•

PASCAL Challenges
–
–
–

•
•

RTE-1
RTE-2
RTE-3

2005
2006
2007

TAC has been proposed as a generic task that captures major semantic
inference needs across many natural language processing applications.
TAC challenges––––

RTE-4 TAC 2008
RTE-5 TAC 2009
RTE-6 TAC 2010
RTE-7 TAC 2011

Textual Entailment

3

Readings•

Workshops– ACL 2005 Workshop on Empirical Modeling of Semantic Equivalence andEntailment, 2005
– First PASCAL Recognising Textual Entailment Challenge (RTE-1), 2005
– Second PASCAL Recognising Textual Entailment Challenge (RTE-2), 2006
– Third PASCAL Recognising Textual Entailment Challenge (RTE-3), 2007
– Answer Validation Exercise at CLEF 2006 (AVE 2006)– Answer Validation Exercise at CLEF 2007 (AVE 2007)– TAC workshops since 2008•

Tutorials– Horacio Rodríguez, Tutorial on TE at University of Sevilla, 2008
• http://www.lsi.upc.es/%7Ehoracio/varios/sevilla2008.zip– Horacio Rodríguez, talk on Paraphrasing at University of Basc Country,
2009• http://www.lsi.upc.es/%7Ehoracio/varios/paraphrasing.tar.gz•

Surveys– [Ghuge, Bhattacharya, 2013]

Textual Entailment

4

Readings•

Thesis– Oren Glickman (PHD, 2006)– Idan Szpecktor (MSC, 2005, PHD, 2009)– Milen Kouylekov (PHD, 2006)– Regina Barzilay (PHD, 2004)– Elena Cabrio (PHD, 2011)– Óscar Ferrández (PHD, 2009)– Prodromos Malakasiotis (PHD, 2011)– Annisa Ihsani (MSC, 2012)– Roy Bar Haim (PHD, 2010)– Shachar Mirkin (PHD, 2011)•

Other material– presentations of RTE 2 online• http://videolectures.net/pcw06_rus_trlsa/

Textual Entailment

5Textual Entailment
Motivation:• Text applications require semantic inference• A common framework for applied semantics is needed, but
still missing• Textual entailment may provide such framework
Meaning
Variability
Ambiguity

Language
Textual Entailment

6Textual Entailment
The Dow Jones Industrial Average closed up 255
Dow ends up

Dow gains 255 points

Dow climbs 255

Stock market hits arecord highModel variability of semantic expression as relation between
text expressions:• Equivalence:
text1
text2 (paraphrasing)• Entailment:
text1
text2 (the general case)
Textual Entailment

7Textual Entailment•

Textual entailment recognition is the task of deciding, given two text
fragments, whether the meaning of one text is entailed (can beinferred) from another text. This task captures generically a broad
range of inferences that are relevant for multiple applications.

•

For example, a QA system has to identify texts that entail the expected
answer:

Question

Expected answer form

Who killed Kennedy?

>>

The assassination of
Kennedy by Oswald

X killed Kennedy

entails

Oswald killed Kennedy

hypothesized answer
text• Other QA ex: “Who owns Overture?”“Yahoo acquired Overture” “Overture’s acquisition by Yahoo”, …
Textual Entailment

8Textual Entailment
•
•
•
•

•

In IE entailment holds between different text variants that express thesame target relation: (X kill Y X acquire Y)
In MDS a redundant sentence or expression, to be omitted from thesummary, should be entailed from other expressions in the summary.
In IR the concept denoted by a query expression should be entailed
from relevant retrieved documents.
In MT evaluation a correct translation should be semantically equivalent
to the gold standard translation, and thus both translations have to entail
each other.
Thus, in a similar spirit to WSD and NER which are recognized as
generic tasks, modeling textual entailment may consolidate and promotebroad research on applied semantic inference.

Textual Entailment

9Textual Entailment Applications

 Question-Answering Information Extraction
 Information Retrieval
 Multi-Document Summarization
 Named Entity Recognition
 Temporal and Spatial Normalization
 Semantic Parsing
 Natural Language Generation

Textual Entailment

10Textual Entailment Applications– Question-Answering• can often be reduced to a textual entailment problem
by rephrasing the question as a declarative statementthat is to be entailed by correct answers– Information Retrieval• for certain IR queries the combination of semantic
concepts and relations denoted by the query should beentailed from relevant retrieved documents– Information Extraction• entailment holds between different text variants that
express the same target relation– Multi-Document Summarization• Identify a redundant sentence, to be omitted

Textual Entailment

11Textual Entailment• Equivalence (Paraphrase): expr1  expr2• Entailment: expr1  expr2 – more general• Classical entailment definition:
A text t entails a hypothesis h if h is true in every
circumstance (possible world) in which t is true
• Strict entailment - doesn't account for some uncertainty
allowed in applications:t: The technological triumph known as GPS … was
incubated in the mind of Ivan Getting.
h: Ivan Getting invented the GPS.
Textual Entailment

12Textual Entailment• Directional relation between two text fragments: Text (t)and Hypothesis (h):t entails h (t  h) if, typically, a humanreading t would infer that h is most likely true”• Operational (applied) definition:– Human gold standard – as in NLP applications– Assuming common background knowledge – which is indeed
expected from applications

Textual Entailment

13Textual Entailment• Probabilistic interpretation:
t probabilistically entails h (t  h) if
P(h is true | t) > P(h is true)– t increases the likelihood of h being true
– ≡ Positive PMI – t provides information on h’s truth• P(h is true | t ): entailment confidence– The relevant entailment score for applications– In practice: “most likely” entailment expected

Textual Entailment

14Textual Entailment• The role of knowledge:– For textual entailment to hold we require:text AND knowledge  h– But knowledge should not entail h alone– Systems are not supposed to validate h’s truth regardless of t(e.g. by searching h on the web)• The knowledge sources available to the system are themost significant component of supporting TE

Textual Entailment

15

Textual Entailment ≈Human Reading Comprehension•

From a children’s English learning book (Sela and Greenberg):

Reference Text:“…The Bermuda Triangle lies in the Atlantic Ocean, off the coastof Florida. …”???

Hypothesis (True/False?):The Bermuda Triangle is near the United States••

Common method for testing human reading comprehension: test
the entailment capability
Difficulties:– Variability between question and text
– Knowledge needed - Florida in the US
Page 16

Textual Entailment examples

•

•

•

TEXT
Eyeing the huge market potential,
currently led by Google, Yahoo took
over search company Overture
Services Inc last year.
Microsoft's rival Sun Microsystems
Inc. bought Star Office last monthand plans to boost its development
as a Web-based device running over
the Net on personal computers andInternet appliances.
The
National
Institutefor
Psychobiology
inIsraelwasestablished in May 1971 as the
Israel Center for Psychobiology by
Prof. Joel.

HYPOTHESISYahoo bought
Overture.

•

ENTAILMENT
TRUE•

Microsoft boughtStar Office.

•

FALSE

•

Israel wasestablished in May
1971.

•

FALSE

•

Textual Entailment

17Textual Entailment examples
TEXT

HYPOTHESISENTAILMENT•

Since its formation in 1948,
Israel fought many wars withneighboring Arab countries.

•

Israel was established in
May 1948.

•

TRUE

•

Putting hoods over prisoners'
heads was also now banned, he
said.

•

Hoods will no longer beused to blindfold Iraqi
prisoners.

•

FALSE

•

The market value of u.s.
overseas assets exceeds theirbook value.

•

The market value of u.s.
overseas assets equals
their book value.

•

FALSE

Textual Entailment

18

Textual Entailment
PASCAL Recognizing Textual Entailment (RTE) Challenges(EU FP-6 Funded PASCAL Network of Excellence
2004-7)• 7 application settings in RTE-1, 4 in RTE-2/3
– QA
– IE
– “Semantic” IR– Comparable documents / multi-doc summarization– MT evaluation– Reading comprehension– Paraphrase acquisition• Most data created from actual applications output• RTE-2/3: 800 examples in development and test sets
• 50-50% YES/NO split
Textual Entailment

19

Analysis• Shallow techniques are promising when t-h pairsremain short, but their performance degrades as
longer and more syntactically complex entailment
pairs are considered• For the first time: deep methods (semantic/ syntactic/
logical) clearly outperform shallow methods(lexical/n-gram)• Still, most systems based on deep analysis did notscore significantly better than the lexical baseline
Textual Entailment

20

Why?
• System reports point at two directions:– Lack of knowledge (syntactic transformation rules, paraphrases,
lexical relations, etc.)– Lack of training data

• It seems that systems that coped better with theseissues performed best:– LCC (Hickl et al.) - acquisition of large entailment corpora fortraining– COGEX (Tatu et al.) – large knowledge bases (linguistic andworld knowledge)

Textual Entailment

21

Methods and Approaches•

Word overlap– lexical, syntactic, and semantic•

Logical approaches– Raina et al, 2005
– Bos et al, 2005, 2006
– Moldovan et al, 2003•

Graph matching approaches– Haghighi et al, 2005
– de Salvo et al, 2005
– de Marneffe et al, 2005, 2006•

Paraphrases and Entailment Rules– Moldovan and Rus, 2001
– Lin and Pantel, 2001 QA– Shinyama et al, 2002 IE

Textual Entailment

22

Methods and Approaches•

Measure similarity between t and h (coverage of h by t):
–
–
–
–
–
–
–

•

Lexical overlap (unigram, N-gram, subsequence)
Average Matched Word Displacement
Lexical substitution (WordNet, statistical)
Syntactic matching/transformations
Lexical-syntactic variations (“paraphrases”)
Semantic role labeling and matching
Global similarity parameters (e.g. negation, modality)

Sentence Alignment– Exhaustive Sentence Alignment– Web-based Sentence Alignment– Bigrams•

Cross-pair similarity– Similarity measure between textual entailment examples••

Detect mismatch (for non-entailment)Interpretation to logic representation + logic inference

Textual Entailment

23

Dominant approach: Supervised Learning

Classifiert,hFeatures:
Lexical, n-gram, syntactic
semantic, global

Feature vector•••

Features model both similarity and mismatch
Classifier determines relative weights of information sources
Train on development set and auxiliary t-h corpora

Textual Entailment

24

YES
NO

PASCAL RTE-3•

Analysis of the Knowledge Requirements– Clark et al, 2007
– Lexico-Syntactic Matching• Synonyms– 648.T "…go through … licencing procedures..."– 648.H "…go through the licencing processes."• Generalizations (Hypernyms)– 148.T "Beverly served...at WEDCOR"– 148.H "Beverly worked for WEDCOR."• Noun Redundancy– 607.T "single-run production process..."– 607.H "Single-run production..."• Noun-Verb Relations– 480 "Marquez is a winner..." → "Marquez won..."• Compound Nouns– 168 "Sirius CEO Karmazin" → "Karmazin is an executive ofSirius"– 583 "physicist Hawking" → "Hawking is a physicist"
Textual Entailment

25

PASCAL RTE-3•

Analysis of the Knowledge Requirements– Syntactic Matching• Definitions– 328 "sufferers of coeliac disease..." → "coeliacs..."• World Knowledge: General
–
–
–
–
–
–
–

273 "bears kill people" → "bears attack people"
499 "shot dead" → "murder"
705 "under a contract with" → "cooperates with"
721 "worked on the law" → "discussed the law"
731 "cut tracts of forest" → "cut trees in the forest"
732 "establishing tree farms" → "trees were planted"
639 "X's plans for reorganizing" → "X intends toreorganize"– 328 "the diets must be followed by <person>" → "thediets are for <person>"– 722 X and Y vote for Z → X and Y agree to Z.

Textual Entailment

26

PASCAL RTE-3•

Analysis of the Knowledge Requirements– World Knowledge• Core Theories (e.g., space, time, plans, goals)
–
–
–
–
–
–

6.T "Yunupingu is one of the clan of..."
6.H "Yunupingu is a member of..."491.T "...come from the high mountains of Nepal."
491.H "...come from Nepal."
178.T "...3 people in Saskatchewan succumbed to the storm."
178.H "...a storm in Saskatchewan."

Textual Entailment

27

PASCAL RTE-3•

Analysis of the Knowledge Requirements– World Knowledge• Frames and Scripts– Implicative Verbs» 668 "A survey shows that X..." → "X..."» 657 "...X was seen..." → "...X..."» 725 “...decided to X..." → "...X..."» 716 "...have been unable to X..." → "...do not X"– Metonymy/Transfer» 708.T "Revenue from stores funded..."» 708.H "stores fund..."– Idioms/Protocol/Slang» 12 "Drew served as Justice. Kennon returned to claim Drew's seat" →"Kennon served as Justice."» 486 "name, 1890-1970" → "name died in 1970"» 408 "takes the title of" → "is"» 688 "art finds its way back" → "art gets returned"

Textual Entailment

28

PASCAL RTE-3
Frequency

Textual Entailment

29

Textual Entailment: General View
Eyeing the huge market
potential, currently led by
Google, Yahoo took oversearch company
Overture Services Inc. lastyear

Entails
Subsumed by

Phrasal verb paraphrasingYahoo acquired Overture
Overture is a search company
Google is a search company……….

Entity matching
Alignment
How?

Semantic Role Labeling
Integration
Textual Entailment 30

A general Strategy for Textual Entailment
Given a sentence T

Re-represent T
Lexical Syntactic Semantic
Representation

Re-represent T
Re-represent T
Re-represent T
Re-represent T
Re-represent T
Re-represent
Re-represent T T

Given a sentence H

eRe-represent H
Lexical Syntactic Semantic

Knowledge Basesemantic; structural& pragmaticTransformations/rules

Decision
Find the set ofTransformations/Featuresof the new representation(or: use these to create acost function)that allows embedding of
H in T.
Textual Entailment 31

Details of the entailment strategy
1. Preprocessing––––

Multiple levels of lexical pre-processing
Syntactic Parsing
Shallow semantic parsing
Annotating semantic phenomena

2. Representation––

Bag of words, n-grams through tree/graph based representation
Logical representations

3. Knowledge sources––––

Syntactic mapping rules
Lexical resources
RTE specific knowledge sources
Additional Corpora/Web resources

4. Control Strategy & Decision Making––

Single pass/ iterative processing
Strict vs. Parameter based

Textual Entailment32

Preprocessing•

Syntactic Processing:– Syntactic Parsing (Collins; Charniak; CCG)
– Dependency Parsing (+types)•

Lexical Processing– Tokenization; lemmatization– For each word in Hypothesis, Text
– Phrasal verbs– Idiom processing– Named Entities + Normalization– Date/Time arguments + Normalization•

Only a few systems

Semantic Processing– Semantic Role Labeling– Nominalization– Modality/Polarity/Factive– Co-reference}}

often used only during
decision making

often used only during
decision making
Page 33

Representation
Meaning
Representation

Inference

Logical Forms
Semantic
Representation

Representation

Syntactic Parse
Local Lexical

Raw Text
Textual Entailment•

Most approaches augment the basic structure defined by theprocessing level with additional annotation and make use of atree/graph/frame-based system.

Page 34

Knowledge Sources• The knowledge sources available to the system are themost significant component of supporting TE.
• Different systems draw differently the line between
preprocessing capabilities and knowledge resources.
• The way resources are handled is also different acrossdifferent approaches.

Page 35

Control Strategy and Decision Making• Single Iteration– Strict Logical approaches are, in principle, a single stage
computation.
– The pair is processed and transformed into the logic form.
– Existing Theorem Provers act on the pair along with the KB.

• Multiple iterations– Graph based algorithms are typically iterative.
– Transformations are applied and entailment test is done after eachtransformation is applied.
– Transformations can be chained, but sometimes the order makes adifference. The algorithm can be a greedy algorithm or can be moreexhaustive, and search for the best path found [Braz et. al’05;BarHaim et.al 07]– Possible transformations: phrasal verb rewriter, nominalization
promoter, apposition rewriter, use of Wordnet, ...
Page 36

Notable TE Systems• TEASE• COGEX• LCC Hickl

Textual Entailment

37

Notable Systems
TEASE and improvements•

Idan Szpektor (2005) Scaling Web-based Acquisition of Entailment
Relations (Ms. thesis)•

Idan Szpektor et al (2004)•

Idan Szpektor and Ido Dagan (2007)•

Lorenza Romano et al (2007)•

Ido Dagan et al (2008)

Textual Entailment

38

TEASE
Input Verb•

Learned Templates
X near Y
X stride to Y
X pass a note to Y

X approach Y

X go to Y
X step to Y
X walk to Y

X defeat Y

Y lose to X
X beat Y
X victory over Y

X destroy Y
X win YX conquer Y

X host Y

bring Y to X
Y is held in X
Y come to X

Y is played in X
X venue of Y
X play host to Y

X preclude Y

X prevent Y
X bar YX prohibit Y

X exclude Y
X deny Y
X forbid Y

X plant Y

X grow Y
X produce Y
X cultivate Y

X raise Y
X sow Y
X farm Y

First direct algorithm for extracting rules
Textual Entailment

39

TEASE• This system proposes a principled semantic inference mechanism
over parse-based representations. That is, a proof system that
operates over syntactic trees.
• Control Strategy: it is an iterative system. The prover applies
entailment rules that aim to transform T into H through a sequenceof intermediate parse trees• Uses MINIPAR– Broad-coverage dependency parser for English– It generates dependency trees, each link between 2 words in adependency tree represents a direct semantic relationship

Textual Entailment

40

TEASE• Scalable unsupervised web-based method for extracting candidate
entailment relations• Web as main textual resource• Acquired template structures as general as possible• Feature-Vector Similarity Approach– The algorithm constructs for each possible template two feature vectors,
representing its co-occurrence statistics with the two anchors. Two
templates with similar vectors are suggested as paraphrases (termed
inference rule)• Unsupervised manner– Input is a verb lexicon (verb-based expressions), each element
describing a specific relation– For each verb, it searches the Web for related syntactic entailment
templates
Textual Entailment

41

TEASE• A template, Ti , is a connected parse graph fragment (or dependency
parse-tree fragment) with optional variables at some nodes.
– example
subj

obj

X  prevent  Y• A pair of templates T1 and T2 is denoted as <T1 ,T2>.
• A pair of templates is called an entailment relation if T1 and T2
contain the same variables and the meaning of T2 can be inferredfrom the meaning of T1, or vice versa, in some contexts, under thesame variable instantiation.

Textual Entailment

42

TEASE
subj•

exampleobj

X  prevent  Y

subjentailsobj

X  reduce  Y risk

“aspirin reduces heart attacks risk” can be inferred from “aspirin prevents
heart attacks”•

An entailment relation does not need to hold under all possible variableinstantiations, i.e. the correctness/validity of an entailment relation depends
on specific variable instantiations•

A pivot P is a lexical phrase, such as a verb, a phrasal verb or a nounphrase (typically a nominalization) that expresses a semantic relation.
Ex: aquire, fall to, prevent, victory over, near, ...

•A pivot template, denoted {P, TP}, is a pivot P with its syntactic template TP
in the form of a parse graph fragment including at least two variable slots(nodes).
Textual Entailment

43

TEASE

Textual Entailment

44

TEASEFirst step: create a complete template TP for the input pivot P•

Variable slots are added for the major types of syntactic relations thatinteract with P, based on its syntactic type
subj

obj

X V Y•

input template for a transitive verb V•

phrasal verb consisting of a verb V and a preposition
subj

prep

mod

X  V  Preposition  Y•

The output of the TEASE algorithm is a ranked list of templates {Ti} <TP, Ti>
is an entailment relation candidate

Textual Entailment

45

TEASE•An anchor-set is a set of words (or terms). Each member of an anchor-set istermed an anchor•

An anchor in an anchor-set that is designated to be an instantiation of atemplate variable in a sentence is termed a slot anchor•

An anchor in an anchor-set that is not designated to be an instantiation of atemplate variable in a sentence is termed a context anchor.

•

A matching of a template T in a sentence s is the embedding of the parsegraph of T as a sub-graph in the parse-graph of s•

An instantiation of a template T by an anchor-set AS in a sentence s is amatching of T in s where each variable of T is instantiated with thecorresponding value of a slot anchor in AS, and all the values of the context
anchors in AS appear elsewhere in s

“Antibiotics in pregnancy prevent miscarriage”
Slot anchors:subjX  prevent  Y

Context anchor: pregnancy
Textual Entailment

obj

46

TEASEA good AS should satisfy a proper balance between specificity and generality•

A minimal anchor-set is an anchor-set that contains only slot anchors•

AS is termed a characteristic anchor-set of a template T if for every template
Ti that is instantiated by AS in some sentence, the entailment relation <T,Ti> holds between T and Ti•

A diverse anchor-set is an anchor-set that instantiates more than one
template in sentences•

An anchor-set that is both characteristic of a template T and diverse istermed a productive anchor-set for a template T•

Contex anchors provide for specificity

Textual Entailment

47

TEASE• Algorithm– For each input pivot template TP:• Extract productive anchor-sets for the pivot template (ASE phase)– Construct a sample corpus for the pivot template by retrieving sentences
containing the pivot template from the Web.
– Extract candidate anchor-sets from the sentences in the sample corpus.
– Filter out candidate anchor-sets that fail certain criteria• Extract templates (TE phase)– Construct a sample corpus by retrieving sentences containing the anchorsets extracted in phase 1 from the Web.
– Extract repeated sub-structures in the sample corpus to be templatecandidates Ti.

• Rank each extracted template Ti according to the confidence level inthe correctness of the entailment relation <TP, Ti>.

Textual Entailment

48

TEASE
Input pivot template:X sub j-accuse-obj Y

WEB

Sample corpus for input template:Paula Jones accused Clinton…
Sanhedrin accused St.Paul…
…

TEASE
Anchor Set Extraction(ASE)

Anchor sets:{X: Paula Jones; Y: Clinton}{X: Sanhedrin; Y: St.Paul}…

Sample corpus for anchor sets:
Paula Jones called Clinton indictable…
St.Paul defended before the Sanhedrin…

Template Extraction(TE)

Templates:
X call Y indictableY defend before X…
Figure 3: An example of the TEASE algorithm flow for the input pivot template 'X accuse Y'.

Textual Entailment

49

TEASE•ASE Algorithm– For each input pivot template TP:• Construct a sample corpus that consists of sentences containing TP.
– Retrieve sentences the Web using a query containing the template's words– Retrieve more sentences from the Web using refined queries, based on the sentences retrieved atstep• Extract productive anchor-set candidates from the constructed corpus.
– Extract one minimal anchor-set, containing only the slot anchors, from each sentence in the sample
corpus.
– Extract one more anchor-set from each sentence, containing one context anchor in addition to theslot anchors, if possible.

• Filter out candidates that fail certain filtering criteria:– Applying thresholds over individual anchor-set statistics.
– Filtering anchor-sets that are redundant or inconsistent relative to other anchor-sets.

Textual Entailment

50

TEASE – ASE phase
Pivot Template

X establish Y

X write Y

X calculate Y

Learned Anchor Sets
X = epa, Y = national emission standard, C1 = asbestos
X = canada agricultural products act, Y = review tribunal
X = school district, Y = breakfast program
X = federal government, Y = conservation corps
X = erisa, Y = minimum standards
X = constantine, Y = new rome
X = laurie, Y = numerous songs
X = lewis carrol, Y = alice's adventures
X = plato, Y = detailed account, C1 = atlantis
X = mendelssohn, Y = incidental music
X = shakespeare, Y = great tragedies
X = thomas malthus, Y = essay
X = katz equation, Y = membrane potential
X = eratosthenes, Y = circumference
X = nernst equation, Y = equilibrium potential
X = language model, Y = probabilities
X = following table, Y = annual cost
X = acos, Y = arc cosine

Table 4: Examples of anchor-sets learned for several pivot templates. X and Y are the slot anchors. C1 isa context anchor.

Textual Entailment

51

TEASE – TE phase
Pivot Template

X establish Y

X write Y

X calculate Y

X set YX develop Y
X create Y
X found Y
X enforce Y
X form Y
X offer Y
X release Y

Learned Templates
X promulgate Y
X issue Y
X implement Y
X provide Y
X make Y
X launch Y
X institute Y
X for the establishment of Y

X who write Y
X publish Y
X compose Y
read Y by X
Y attributed to X
perform Y by X
X writer of Y
selected Y of X

X produce Y
X pen YX create Y
X’s Y
X complete Y
X book of Y
X say in Y
X work include Y

X determine Y
X compute Y
X give estimate of Y
X return Y
X assess Y
X generate Y
X recalculate Y
X work out Y

X measure Y
X calculation of Y
X yield Y
X get YX produce Y
Y according to X
Y obtained from X
X evaluate Y

Table 5: Examples of templates learned for several pivot templates.

Textual Entailment

52

TEASE

Pivot template
subj

obj

subj

obj

X  write  Y
X  write  Y
subj

obj

X  produce  Y

Learned template
subj

obj

X  present  Y

ReasonPresenting a cheque is synonymous
to writing a cheque.

subj

obj

Issuing a report (e.g. by a policeman)is synonymous to writing a report.

subj

obj

A telescope producing images issynonymous to a telescope obtaining
images.

X  issue  Y
X  obtain  Y

Table 6: Examples of learned templates that participate in non-trivial entailment relations with therelated pivot template.

Textual Entailment

53

TEASETEASE and improvements•

Learning Canonical Forms of Entailment Rules– Idan Szpektor and Ido Dagan (2007)– Morpho-Syntactic Template Variations• there is still no sound solution for addressing it at learning time.

– A Modular Approach for Entailment Rule Learning• generic morphological and syntactic regularities.
• transform lexical-syntactic template variations that occur in a text into their canonicalform

Textual Entailment

54

TEASE’s improvements•

Morpho-Syntactic Canonization Module– Syntactic-based Rules– Nominalization Rules– Chaining of Canonization Rules

Textual Entailment

55

TEASE’s improvements•

Passive to active substitution rule

Textual Entailment

56

TEASE’s improvements

Textual Entailment

57

TEASE’s improvements

examples ofcanonization
rules

Textual Entailment

58

COGEXCOGEX, Tatu 2006, 2007

• Combination of LEX, COGEXd, COGEXc• Textual entailment as a logical implication between meanings:
1. Transforms T and H into three-layered semantically-rich logic form
representations
2. Generates an abundant set of lexical, syntactic, semantic and world
knowledge axioms
3. Iteratively searches for a proof for the entailment between T and (a possibly
relaxed version of ) H
4. A pair T-H is labeled as positive if the score of the found proof is above athreshold learned on the training data

Textual Entailment

59

COGEX

Textual Entailment

60

COGEX
The system is a natural language prover that performs proof by
refutation: if a refutation is found then the proof is complete•

Usable list: axioms used by the system to generate inferences•

Set of support: list of clauses used to initiate the search for inferences. It
is initially loaded with the negated form of H and the predicates thatmake up the logic form of T.

•

Backoff: relaxation of predicate arguments that is carried out if a
refutation cannot be found. If argument relaxation fails to produce a
refutation, predicates are dropped from the negated H until a refutation isfound.

•

Scoring: once a proof found, a score for it is calculated by starting withan initial perfect score and deducting points for axioms that are used in
the proof, arguments that are relaxed, and predicates that are dropped

Textual Entailment

61

COGEX•

EXtended WordNet Knowledge Base (XWN-KB)– XWN Lexical Chains: for each relation in the best lexical chain found between
one of T’s constituents and one of H’s constituents, an axiom is created– Captures and stores the rich world knowledge encoded in WN’s glosses into
a KB: glosses are transformed into a set of semantic relations using asemantic parser•

NLP Axioms– Break down complex syntactic structures (complex nominals, coordinating
conjunctions)– Links a NE to its set of aliases• World Knowledge– For ex., it allows to infer “nearly half a million dollars” from “$480.000”
Textual Entailment

62

COGEX•

Named Entity Check– In the proof scoring phase, deducts points for each pair whose H contains atleast one named entity not-derivable from T•

TARSQI Toolkit– Temporal Awareness and Reasoning System for Question Interpretation– Modular system which detects, resolves and normalizes time expressions•

Logic Representation of Events– Logic representation of the event’s describing concept augmented with a
special predicate event_EV(e1)•Negation–•

Use polarity info attached to the identified events (TARSQUI output) and negate theevent’s predicate

Coreference Resolution–

Dedicated pronominal coreference resolution module to cope with long text
pairs

Textual Entailment

63

Textual Entailment: what are we missing?
•

It is completely clear that the key resource missing is knowledge.
– Better resources translate immediately to better results.
– At this point existing resources seem to be lacking in coverage andaccuracy.
– Not enough high quality public resources; no quantification.

•

Some Examples– Lexical Knowledge: Some cases are difficult to acquire systematically.
• A bought Y  A has/owns Y• Many of the current lexical resources are very noisy.

– Numbers, quantitative reasoning– Time and Date; Temporal Reasoning.
– Robust event based reasoning and information integration

Page 64