KitAi: Textual Entailment Recognition System for NTCIR-10 RITE2
Kazutaka Shimada, Yasuto Seto, Mai Omura and Kohei Kurihara
Kyushu Institute of Technology (Department of Artificial Intelligence)
NTCIR-10 RITE2 Poster ID - RITE05
Task and basic idea
BC subtask: Given a text pair <t1, t2>, identify whether t1 entails a hypothesis t2 or not
ExamBC subtask: Multiple-choice questions of real university entrance exams
The basic idea: Classification based on word correspondence and syntactic & semantic information

Features

Synonym1
Synonym2....

Synonym1
Synonym2....

Correspondence- Word overlap and edit distance between t1 and t2- Expansion with ontologies; Japanese WordNet andNihongo-Goi-Taikei- All word, content word and simplified word lists

Synonym1
Synonym2....

WordA2

WordA1

Synonym1
Synonym2....

Word-Synonym
relation

WordA5

WordA4

WordA3

Synonym1
Synonym2....

Word alignment
WordB2

WordB1
Synonym1
Synonym2....

Syntactic and semantic information- Hierarchical Directed Acyclic Graph (HDAG)

WordB4

WordB3

Synonym1
Synonym2....

Synonym1
Synonym2....

Ontology
WordXX

Synonym1
Synonym2....

Synonym

Synonym

Synonym

Dependency relation
WordA1

WordA3

WordA2Others- # of negation words and sentence length

WordA5Main verb

WordA4

Simplified

WordA2

WordA5

WordA4

Classifier
BC task- SVM (MethodBC1), Logistic regression (MethodBC2),
Rotation forest (MethodBC3)

Bunsetsu
noun

particleperson namelocation nameα×SVM＋β×Logistic＋γ×Rotation Forest
Score =3

Results for development data
BC taskFeaturesAccuracy MacroF1
WCall
65.6
58.8
WCcont77.1
76.7
WCsimp59.1
47.0
WCall+WCcont
78.1
77.8
WCall+WCcont+WCsimp77.9
77.5
WC+HDAG
77.9
77.5
WC+OTHER
79.5
79.3
ALL79.4
79.2
ALL without Ontologies
78.4
78.2
ExamBC task
Features
WCall
WCcontWCsimpWCall+WCcont
WCall+WCcont+WCsimp
WC+HDAG
WC+OTHERALL
ALL without Ontologies

Accuracy MacroF1
58.6
43.8
65.9
64.6
58.8
43.6
65.3
64.2
65.4
64.6
66.7
65.8
65.5
64.4
66.5
65.5
64.1
63.3

Results for formal run
BC task
MethodHDAG MacroF1
MethodBC1 With
77.1
Out
76.9
MethodBC2 With72.4
Out75.5
MethodBC3 With
76.2
Out76.0
ExamBC task
MethodHDAG MacroF1
MethodEX1 With59.8
Out61.3
MethodEX2 With57.2
Out57.4
MethodEX3 With59.1
Out60.8
The correct answer ratioon the formal run
MethodMacroF1 CorrectAR
MethodEX1 59.8
36.1
MethodEX2 57.2
39.8
MethodEX3 59.1
45.4

particlehanouncopulaauthorpoet
noYukiguni(Snow Country)

desu(is)sakusha(writer)Matched: high similarity value

Bunsetsu
noun

Bunsetsu

particlelocation namestation nameTokyo

Experimental result

noun

Bunsetsu

publication
territory

Kawabata

ExamBC task
- AdaBoost with SVM (MethodEX1), Logistic regression(MethodEX2), Weighted Voting (MethodEx3)

Bunsetsu

noun

Bunsetsu
particlenationuniversity
haNihon(Japan)

noun

copulacity
noShuto(capital)

desu
(is)

An incorrect instancewith HDAGt1. ストックホルムオリンピックは，1912 年 5 月 5 日から 7 月 27 日まで，スウェーデンのストックホルムで行われた夏季オリンピックである．
Stockholm Olympic is the Olympic Summer Games in Stockholm,
Sweden from May 5, 1912 to July 27.
Word Correspondence of t2 to t1 is high.

conflict!

Word relation:
t1.「ストックホルムオリンピック（Stockholm Olympic）」→「夏季オリンピック（Olympic Summer Games）」
t2.「ストックホルム（Stockholm）」→「都市（city）」

t2. ストックホルムはスウェーデンの都市である．
Stockholm is a city in Sweden.

Discussion and Future Work
- Ontologies are important- Confidence value is important
for the ExamBC task
- Optimal parameters for thevoting method- Structure of HDAG