KitAi: Textual Entailment Recognition System for
NTCIR-10 RITE2
Kazutaka Shimada Yasuto Seto Mai Omura Kohei Kurihara
Department of Artificial Intelligence
Kyushu Institute of Technology{shimada, y seto, m omura, k kurihara}@pluto.ai.kyutech.ac.jp

ABSTRACTThis paper describes Japanese textual entailment recognition systems for NTCIR-10 RITE2. The tasks that weparticipated in are the Japanese BC subtask and the ExamBC subtask. Our methods are based on some machinelearning techniques with surface level, syntax and semantic features. We use two ontologies, the Japanese WordNet
and Nihongo-Goi-Taikei, and Hierarchical Directed Acyclic
Graph (HDAG) structure as the syntax and semantic information. For the ExamBC task, the conﬁdence value froma classiﬁer is important to judge the correctness as the entrance exams. To predict a suitable conﬁdence value, weapply a weighting method of each output from several classiﬁers. In formal runs, the best accuracy rates in the methods for the BC and the ExamBC tasks were 77.11 points and59.84 on the macro F1 measure, respectively. Although themethod based on SVMs was better than others in terms ofthe macro F1 measure, the weighted scoring method produced the best performance for the correct answer ratio(45.4%).

apply a surface-based alignment process with the semantic
information to our textual entailment recognition systems.
However, the surface-based alignment process can not handle the structural information of each sentence, such as dependency relations. Therefore, we introduce a structure feature with semantic and grammatical information. We use
the Hierarchical Directed Acyclic Graph (HDAG) structure[7] to compute a similarity between t1 and t2 . The HDAG
structure can handle the structural information of a sentence
and semantic information of each word in the sentence.
For the ExamBC task, a conﬁdence value of each output is
one of the most important factors because of the correctness
as the answer of an examination question. Therefore we
introduce a weighted scoring method to estimate a suitable
conﬁdence value for the selection process.
In the next section, we describe features and methods
about our textual entailment recognition systems. Next, we
discuss our experimental results on the development data
and formal run in Section 3. Finally, we conclude our methods in Section 4.

Team Name

2.

KitAi / Kyushu Institute of Technology (Department of Artiﬁcial Intelligence)First, we explain features for our methods. Then, we describe classiﬁers for the BC and ExamBC tasks.

Subtasks
BC and ExamBC tasks (Japanese)

Keywords
Correspondence, Edit Distance, Ontologies, Hierarchical Directed Acyclic Graph, Weighted Scoring

SYSTEM DESCRIPTION

2.1

2.1.1

1. INTRODUCTION
This paper describes Japanese textual entailment recognition systems for NTCIR-10 RITE2 (the Japanese BC subtask and the ExamBC subtask) [8]. Our methods, KitAi1 ,are based on some machine learning techniques such as SVM.
The basic features are based on surface-based alignment.
However, a simple bag of words feature is generally insuﬃcient. Therefore, we introduce semantic information. As thesemantic information, we use two ontologies; the Japnaese
WordNet [1] and Nihongo-Goi-Taikei [2]. In other words, we

Features

The feature set for our methods consists of three types of
linguistic information; (1) word correspondence, (2) HDAG
and (3) others. For the feature extraction process, we use
JUMAN2 as a morphological analyzer and KNP as a dependency parser3 . Finally, we obtain 36 features for our textual
entailment recognition systems.

Word Correspondence

The basic features in our method are based on correspondence between t1 and t2 in surface level. We compute therates of words of t1 containing in t2 and words in t2 containing in t1 , respectively. We also compute the WER (worderror rate) score based on the edit distance.
For these features, we use three types of input lists fort1 and t2 : the all word list, the content word list and thesimpliﬁed word list. The all word list contains all words in
t1 and t2 . The content word list contains nouns, adjectives,
verbs in t1 and t2 . The simpliﬁed word list contains words
linking directly to the main predicate verb in each sentence.

1

Short of Kyushu Institute of T echnology (Department of
Artiﬁcial Intelligence). The English meaning is “expectation.”

23

http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN
http://nlp.ist.i.kyoto-u.ac.jp/index.php?KNP

Dependency relation
WordA1

Bunsetsu

WordA2

WordA3

WordA5Main verb

WordA4noun

WordA4

particleperson namelocation nameSimplified

WordA2

Bunsetsu

Kawabata

WordA5noun

Bunsetsu
particle

publication
territoryha

Yukiguni(Snow Country)

noun

copulaauthorpoetnosakusha(writer)

desu
(is)

Figure 1: The simpliﬁcation process.
Figure 3: An example of HDAG structures.
Synonym1
Synonym2....

Synonym1
Synonym2....

Synonym1
Synonym2....

Synonym1
Synonym2....

Synonym1
Synonym2....

WordA1

WordA2

WordA3

WordA4

WordA5

WordB1

WordB2

WordB3

WordB4

Synonym1
Synonym2....

Synonym1
Synonym2....

Synonym1
Synonym2....

Synonym1
Synonym2....

Word-Synonym
relation
Word alignment
Ontology
WordXX

Synonym

Synonym

Synonym

Figure 2: The word correspondence with ontologies.

Figure 1 shows the simpliﬁcation process of a sentence. By
using the simpliﬁed word list, we can compute a similarity
of compressed meanings between sentences.
Furthermore, we extend each word by using two ontologies: the Japnaese WordNet [1] and Nihongo-Goi-Taikei [2].
For the Japnaese WordNet, we use the synonyms database,which is created by synsets and manually annotated. For
Nihongo-Goi-Taikei, we use words that belong to the same
semantic class for each word. Figure 2 shows the processof calculation of word correspondence degrees. We computethe edit distance by using the DP matching.

2.1.2 HDAG
To compute a similarity between two sentences, bag-ofwords representation is the most general way to express features of them. However, it is insuﬃcient to represent thefeatures of each sentence because of lack of relations between
words.
To solve the problems, Suzuki et al. [7] have reported
a new graph-based approach, called Hierarchical Directed
Acyclic Graph kernels (HDAG). The method can handle
many linguistic features in a sentence and includes characteristics of sequence and tree kernels. The HDAG is a hierarchized graph-in-graph structure. It represents semantic
and grammatical information in a sentence. We have usedthe HDAG for a sentiment sentence extraction task [6].
In this task, we introduce this structure for the similarity
calculation. We apply three layers to the structure: poslayer, semantic layer and word layer. Figure 3 shows anexample of the structure with the three layers. Our method
computes the similarity of the HDAGs of two sentences.

2.1.3 Others
We add some features to our method. First, we use thenumber of negation words in each sentence. The inconsistency of the number of negation words between two sentences has the potential of the disagreement of entailment

between them.
Next, we compute the length values of the all word list
and content word list. If the length values of two sentences
are extremely-diﬀerent, it indicates that the two sentences
disagree with high probability.

2.2

Methods

For the BC task, we use three classiﬁers with the open
source software Weka4 . They are SMO, Logistic and Rotation Forest. SMO (MethodBC1) is a support vector classiﬁer with John Platt’s sequential minimal optimization algorithm [3]. Logistic (MethodBC2) is a multinomial logistic
regression model with a ridge estimator. Rotation Forest(MethodBC3) is a regression model with a base learner [5].
The base learner is the C4.5 algorithm [4]. These methods
are determined heuristically.
For the ExamBC task, we use AdaBoost with SMO (MethodEX1) and Logistic (MethodEX2) in a similar way. We
focus on a weighted scoring approach as the 3rd method(MethodEX3) for the ExamBC task. The reason that weapply the weighted score to the ExamBC task is that thetask evaluates the correct answer ratio for entrance exams.
In the ExamBC task, the output value of each instance isused as a conﬁdence score for tie-breaking multiple Y labels
on series of pairs on a certain topic. Therefore, estimationof the output of the method is one of the most importantfactors for the correct answer ratio.
The MethodEX3 computes a conﬁdence from three classiﬁers; AdaBoost with SMO, Logistic and Rotation Forest.
First, the method obtains three output values from the classiﬁers. Then, it computes a weighted score by
α × SM O + β × Logistic + γ × RotationF orest
3(1)
where α = 0.5, β = 1.0 and γ = 0.5. These values are
determined heuristically from the development data set.
Score =

3.

EXPERIMENTS

In this section, we describe the results of our methods onthe development data ﬁrst. Next, we discuss the formal run
results.

3.1

Development data

First, we evaluated our method with the development data
set by using the leave-one-out cross validation. In this experiment, we focused on the eﬀectiveness of each feature,such as word correspondence and HDAG. The methods in
4

http://www.cs.waikato.ac.nz/ml/weka/

Table 1: The experimental result for BC task.
FeaturesWCbaseWCcontWCsum
WCbase +WCcont
WCbase +WCcont +WCsum
WCall +HDAG
WCall +OTHER
ALL
ALL without Ontologies

Accuracy
65.6
77.1
59.1
78.1
77.9
77.9
79.5
79.4
78.4

MacroF1
58.8
76.7
47.0
77.8
77.5
77.5
79.3
79.2
78.2

t1. ࢫࢺࢵࢡ࣒࣍ࣝ࢜ࣜࣥࣆࢵࢡࡣ㸪1912 ᖺ 5 ᭶ 5 ᪥࠿ࡽ 7 ᭶ 27 ᪥ࡲ࡛㸪ࢫ࢙࣮࢘ࢹࣥࡢࢫࢺࢵࢡ࣒࡛࣍ࣝ⾜ࢃࢀࡓኟᏘ࢜ࣜࣥࣆࢵࢡ࡛࠶ࡿ㸬
Stockholm Olympic is the Olympic Summer Games in Stockholm,
Sweden from May 5, 1912 to July 27.

Word Correspondence of t2 to t1 is high.

conflict!

Word relation:
t1.ࠕࢫࢺࢵࢡ࣒࣍ࣝ࢜ࣜࣥࣆࢵࢡ㸦Stockholm Olympic㸧ࠖࠉࠉ→ࠕኟᏘ࢜ࣜࣥࣆࢵࢡ㸦Olympic Summer Games㸧ࠖt2.ࠕࢫࢺࢵࢡ࣒࣍ࣝ㸦Stockholm㸧ࠖ→ࠕ㒔ᕷ㸦city㸧ࠖ

t2. ࢫࢺࢵࢡ࣒࣍ࣝࡣࢫ࢙࣮࢘ࢹࣥࡢ㒔ᕷ࡛࠶ࡿ㸬
Stockholm is a city in Sweden.

Figure 4: An incorrect instance.
Table 2: The experimental result for ExamBC task.
FeaturesWCbaseWCcontWCsum
WCbase +WCcont
WCbase +WCcont +WCsum
WCall +HDAG
WCall +OTHER
ALL
ALL without Ontologies

Accuracy
58.6
65.9
58.8
65.3
65.4
66.7
65.5
66.5
64.1

MacroF1
43.8
64.6
43.6
64.2
64.6
65.8
64.4
65.5
63.3

Table 3: The results on the formal run.
Data

MethodBC1
BC

MethodBC2
MethodBC3
MethodEX1

EX
the experiment were MethodBC1 for the BC task and MethodEX1 for the ExamBC task, respectively. In other words,
the methods were based on SVMs.
Table 1 and Table 2 show the experimental results for
the BC task and the ExamBC task, respectively. In thetables, WC denotes word correspondence features. WCbase ,WCcont and WCsum denote the features using all word, content word and simpliﬁed word lists, respectively. “+” denotes the combination of features. WCall denotes all wordcorrespondence features, namely WCbase +WCcont +WCsum .
HDAG and OTHER denote the features described in Section
2.1.2 and Section 2.1.3. “ALL without Ontologies” denotes
the features that combined WCall without two ontologies
and HDAG+OTHER.
The WCall +OTHER feature set for the BC task and theWCall +HDAG feature set for the ExamBC task produced
the best performance. The features about the content word
list were eﬀective for both the tasks. For the ExamBC task,
the method with HDAG generated a slight positive eﬀect
on the both criteria. Although the method with HDAG
correctly classiﬁed some instances that were incorrect by
the method without HDAG, some instances were classiﬁed
incorrectly by using HDAG, namely structure information.
Figure 4 shows an example of the mistake. In this instance,some word correspondence features for t2 to t1 generated
high values. As a result, the method without HDAG classiﬁed this instance correctly. However, the relations between
the subject word and the main predicate verb, [Stockholm
Olympic → Olympic Summer Games] and [Stockholm →city], mismatched. Therefore, the method with HDAG predicted the incorrect label for this instance.
The method with ontologies outperformed that without

MethodMethodEX2
MethodEX3

HDAG
With
Out
With
Out
With
Out
With
Out
With
Out
With
Out

MacroF1
77.1
76.9
72.4
75.5
76.2
76.0
59.8
61.3
57.2
57.4
59.1
60.8

ontologies for both the tasks (79.2 vs. 78.2 on MacroF1 for
the BC and 65.3 vs. 63.3 on MacroF1 for the ExamBC).
This result shows the eﬀectiveness of expansions using ontologies, such as synonyms of each word.

3.2

Formal run

Next, we discuss the formal run results. Table 3 shows theresults. In the table, “With” and “Out” denote a methodwith HDAG and a method without HDAG, respectively. All
submitted runs included HDAG information as the features,i.e., the method with “With” in the table are our formal run
results for the BC and ExamBC tasks.
As in the case of the formal runs, incorporating HDAG
features to our method did not lead to the improvementof the macro F1 measure. In some situations, such as the
MethodBC2 and the MethodEx1, the HDAG feature decreased the macro F1 measure. One reason that the accuracy decreased is that the layer of the HDAG structure.
Since we used the POS layer as the upper layer of the semantic layer, the similarity based on the HDAG was sensitive
to structure information of sentences rather than semantic
relations. Figure 5 shows an example of the problem. In
this example, the HDAG generates a high similarity value
because the POS layer is completely matched although themeanings of them are entirely-diﬀerent. We need to consider the layers of the HDAG for computing more suitablesimilarity values.

Bunsetsu
noun

Bunsetsu
noun

particleperson namelocation nameBunsetsu
particle

publication
territory

Kawabatahanoun

copulaauthorpoet

Yukiguni(Snow Country)nosakusha(writer)

desu
(is)

Matched: high similarity value

Bunsetsu
noun

Bunsetsu

location namestation name
Tokyo

noun

particle

Bunsetsu
particlenationuniversity
ha

Nihon
(Japan)noun

copula

city
no

Shuto
(capital)

desu
(is)

the macro F1 measure (10th among 42 methods). The best
accuracy rate for the ExamBC task was 59.84 on the macro
F1 measure (13th among 32 methods)5 . All our methods
outperformed the baseline system [8]. For the correct answer
ratio on the ExamBC task, the weighted scoring method predicted the suitable conﬁdence value even though the textual
entailment accuracy was lower than other methods.
In the development data and formal run, there was no
signiﬁcant diﬀerence between the methods with HDAG andwithout HDAG. To improve the accuracy, we need to consider the layers in the HDAG. For the correct answer ratio on
the ExamBC task, estimation of the conﬁdence score fromthe method is one of the most important points. To improvethe criterion, we need to add more suitable classiﬁers to theweighting method. In addition, to determine appropriate
parameters, namely weights in Equation (1), is important
future work.

5.
Figure 5: An example of the layer problem.

Table 4: The correct answer ratio on the formal run.
MethodMethodEX1
MethodEX2
MethodEX3

MacroF1
59.8
57.2
59.1

CorrectAR
36.1
39.8
45.4

Then, we compared the correct answer ratio for the ExamBC task. For the ExamBC, important evaluation criteria are not only the classiﬁcation accuracy of textual entailment, namely Yes or No, but also the accuracy on theentrance exam. Table 4 shows the correct answer ratios of
three methods. The mirco F1 values are the same as Table
3. The CorrectAR in the table denotes the correct answer
ratio.
On the F1 measure, MethodEX1 (AdaBoost with SMO)
produced the best performance in our methods. However,the MethodEX3 (weighted scoring) outperformed the two
methods on the correct answer ratio even though the MacroF1of MethodEX1 was slightly better than thet of MethodEx3.
For the selection of the correct answer, the conﬁdence value
from each method is important. Therefore, the MethodEX3
based on a weighting function with several classiﬁers’ outputs was suitable for the answer selection on the entrance
exam task. Parameter tuning on the weighting approachand integration of other classiﬁers are our important future
work.

REFERENCES[1] F. Bond, H. Isahara, S. Fujita, K. Uchimoto,T. Kuribayashi, and K. Kanzaki. Enhancing thejapanese wordnet. In The 7th Workshop on Asian
Language Resources, in conjunction with ACL-IJCNLP
2009, 2009.
[2] S. Ikehara, M. Miyazaki, S. Shirai, A. Yokoo,
H. Nakaiwa, K. Ogura, Y. Ooyama, and Y. Hayashi.
Goi-Taikei - A Japanese Lexicon. Iwanami Shoten,
1999.
[3] J. Platt. Fast Training of Support Vector Machines
using Sequential Minimal Optimization. Advances in
Kernel Methods - Support Vector Learning, 1998.
[4] R. Quinlan. C4.5: Programs for Machine Learning.
Morgan Kaufmann Publishers, 1993.
[5] J. J. Rodriguez, L. I. Kuncheva, and C. J. Alonso.
Rotation forest: A new classiﬁer ensemble method.
IEEE Transactions on Pattern Analysis and MachineIntelligence, 28(10):1619–1630, 2006.
[6] K. Shimada, D. Hashimoto, and T. Endo. Sentiment
sentence extraction using a hierarchical directed acyclic
graph structure and a bootstrap approach. In
Proceedings of the 22nd Paciﬁc Asia Conference on
Language, Information and Computation (PACLIC22),
pages 341–349, 2008.
[7] J. Suzuki, Y. Sasaki, and E. Maeda. Hierarchicaldirected acyclic graph kernel. Systems and Computers
in Japan, 37(10):58–68, 2006.
[8] Y. Watanabe, Y. Miyao, J. Mizuno, T. Shibata,H. Kanayama, C.-W. Lee, C.-J. Lin, S. Shi,
T. Mitamura, N. Kando, H. Shima, and K. Takeda.
Overview of the recognizing inference in text (RITE-2)
at NTCIR-10. In Proceedings of the 10th NTCIR
Conference, 2013.

4. CONCLUSIONSThis paper described a Japanese textual entailment recognition system, which is named KitAi. We used machinelearning techniques, such as SVM, and a weighted scoring
method for the BC and ExamBC tasks. We applied not
only simple bag of words features but also semantic andgrammatical information, namely ontologies and the HDAG
structure.
The best accuracy rate for the BC task was 77.11 points on

5
In terms of the correct answer ratio, the best accuracy was
45.37 and 11th among 32 methods.