Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Overview of the NTCIR-8 ACLIA Tasks:
Advanced Cross-Lingual Information Access
Teruko Mitamura* Hideki Shima* Tetsuya Sakai** Noriko Kando#
Tatsunori Mori† Koichi Takeda‡ Chin-Yew Lin** Ruihua Song**
Chuan-Jie Lin+ Cheng-Wei Lee@
*Carnegie Mellon University **Microsoft Research Asia #National Institute of Informatics†Yokohama National University ‡IBM Research - Tokyo+National Taiwan Ocean University @Academia Sinica
teruko@cs.cmu.edu

ABSTRACTThis paper presents an overview of the ACLIA (Advanced
Cross-Lingual Information Access) task cluster at NTCIR-8.
The task overview includes: a definition of and motivation for
the evaluation; a description of the complex and factoid
question types evaluated; the document sources and exchange
formats selected and/or defined; the official metrics used in
evaluating participant runs; the tools and process used todevelop the official evaluation topics; summary data regarding
the runs submitted; and the results of evaluating the submitted
runs with the official metrics.

a closely related QA task. During the evaluation, the question
text and QA system question analysis results were provided as
input to the IR4QA task, which produced retrieval results thatwere subsequently fed back into the end-to-end QA systems.
The modular design and XML interchange format supported by
the ACLIA architecture make it possible to perform such
embedded evaluations in a straightforward manner. Moredetails regarding the XML interchange schemes and so on canbe found on the ACLIA wiki [6].

Keywords Evaluation Metrics, Automatic Evaluation,
Evaluation Tools

1. INTRODUCTION
Current research in QA is moving beyond factoid questions,so there is significant motivation to evaluate more complex
questions in order to move the research forward. In the
Advanced Cross-Lingual Information Access (ACLIA) task at
NTCIR-7, we have evaluated cross-lingual and monolingual
QA on complex questions (i.e. events, biographies/definitions,and relationships). Our goal in ACLIA for NTCIR-8 is todevelop effective CCLQA 1 evaluations for complex questions
as well as factoid questions. We will evaluate end-to-end
systems and conduct module-based evaluations for question
type analysis, document retrieval and answer extraction.
As a central problem in question answering evaluation, the lackof standardization has been pointed out [1], which makes it
difficult to compare systems under a certain condition. In NLP
research, system design is moving away from monolithic, blackbox architectures and more towards modular architectural
approaches that include an algorithm-independent formulationof the system’s data structures and data flows, so that multiple
algorithms implementing a particular function can be evaluatedon the same task. Following this analogy, the ACLIA data flow
includes a pre-defined schema for representing the inputs andoutputs of the document retrieval step, as illustrated in Figure 1.
This novel standardization effort made it possible to evaluatecross-lingual information retrieval (CLIR) task called IR4QA(Information Retrieval for Question Answering) in a context of1

Although the task name CCLQA stands for Complex Crosslingual Question Answering, the task includes evaluation of
simple (in terms of information need) factoid questions, andmonolingual questions.

Figure 1. Data flow in ACLIA task cluster showing how
interchangeable data model made inter-system and intertask collaboration possible.

The modular design of this evaluation data flow ismotivated by the following goals: a) to make it possible fororganizations to contribute component algorithms to an
evaluation, even if they cannot field an end-to-end system; b) tomake it possible to conduct evaluations on a per-module basis,in order to target metrics and error analysis on importantbottlenecks in the end-to-end system; and c) to determine which
combination of algorithms works best by combining the resultsfrom various modules built by different teams. In order to― 15 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanevaluate many different combinations of systems effectively,
human evaluation must be complemented by development of
automatic evaluation metrics that correlate well with humanjudgment. Therefore, we have developed mechanisms toevaluate the CCLQA results using automatic evaluation
methods.
The CCLQA task included topics developed initially for theevaluation of monolingual systems, and then translated toEnglish. Since translation is considered an important challengein CCLQA, monolingual QA results on the same topics andcorpora provide an important baseline for comparison.
The remainder of this paper is structured as follows. Section
2 describes the specific tasks, topics and corpora that were usedin the ACLIA evaluation, along with a list of the datainterchange formats that were specified to support the exchangeof system inputs and outputs. Section 3 provides the details of
the CCLQA task definition, and Section 4 provides a detailed
explanation of the evaluation metrics and measures (including
both human measurement and automatic evaluation
techniques). Section 5 provides a brief overview of the support
tools that were built to support topic creation and nugget
evaluation. Section 6 presents the evaluation results for theparticipant runs that were submitted. The paper concludes in
Section 7.

2. ACLIA TASKS & RUNSACLIA consisted of two tasks named IR4QA and CCLQA,
which evaluated Information Retrieval and Question Answering,
respectively. The same set of topics was shared between the
two tasks. For details on the IR4QA task evaluation, we refer
the reader to the IR4QA overview paper [2]. This paper
focuses on the CCLQA task, which evaluated QA systems on
both complex and factoid questions.
For both tasks, we evaluated monolingual and cross-lingual
topics. Topic languages included English (EN), Simplified
Chinese (CS), Traditional Chinese (CT), and Japanese (JA);target (corpus) languages included CS, CT and JA. An active
participant (i.e. a participant who submitted at least one run;hereafter called simply a participant) submitted a system result
for at least one of six pairings of source language (i.e. language
for questions) and target language (i.e. language for documents),
listed below:•
Cross-lingual: EN-CS, EN-CT, EN-JA•
Monolingual: CS-CS, CT-CT, JA-JA
For each track, a participant submitted up to three runs. For
each run, we evaluated the top 30 system responses for eachquestion. All official runs were evaluated by independent
assessors.

2.1 Evaluation Topics

Table 1. Example Questions
Type#

Example Question

RelatedPast NTCIR
Task

DEFINITION

10 What is the Human ACLIA
Genome Project?

BIOGRAPHY

10 Who is Howard Dean?

RELATIONSHIP

20 What is the relationship ACLIA
between
SaddamHussein and Jacques
Chirac?

EVENT

20 What are the major ACLIA
conflicts between India
and China on borderissues?

WHY

20 Why doesn't U.S. ratify QAC-4
the Kyoto Protocol?

PERSON

5 Who is the Finland's first QAC
1-3,
woman president?
CLQA 1,2

ORGANIZATION

1-3,
5 What is the name of the QAC
company that produced CLQA 1,2
the first Fairtrade coffee?

LOCATION

1-3,
5 What is the name of the QAC
river
that
separates CLQA 1,2
North Korea from China?

DATE

5 When did Queen Victoria QAC
1-3,
die?
CLQA 1,2

ACLIA

A topic developer created a topic by first generating aquestion and a narrative-style information need in the target
language, which were subsequently translated into English.
This approach supported a comparison between monolingualand cross-lingual QA using the same set of topics and corpora.
A group of volunteers from the participant group created a setof pilot training topics so that details of the task definitions
could be refined and finalized.
For the formal evaluation, an independent third-party
organization created 100 topics for each target language. Someof the topics are shared topics which contain a question
originally created for another target language.

2.2 Corpus
The target corpus consists of digital newswire articles (seeTable 2). We select newswire articles in the same time span
(ranging from 2002 through 2005) in order to support theevaluation of shared topics.
Table2. Corpora used in ACLIA2.

We focused on the evaluation of five types of complex
questions: DEFINITION, BIOGRAPHY, RELATIONSHIP,
EVENT, and WHY; four types of factoid questions: PERSON,ORGANIZATION, LOCATION, and DATE. Examples areshown below.

Language

Corpus Name

Time Span# document

CS
CT
JA

XinhuaUDN
Mainichi

2002-2005
2002-2005
2002-2005

308,845
1,663,517
377,941

2.3 Input/Output Format
In order to combine a CLIR module with a CLQA system
for module-based evaluation, we defined five types of XML― 16 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

schema to support exchange of results among participants andsubmission of results to be evaluated:

Table 4. Number of CCLQA runs submitted, followed bynumber of Question Analysis submissions in parenthesis.
Team Name

•
•

•
•
•

Topic format: The organizer distributes topics in thisformat for formal run input to IR4QA and CCLQA
systems.
Question Analysis format: CCLQA participants whochose to share Question Analysis results submit theirdata in this format. IR4QA participants can accept task
input in this format.
IR4QA submission format: IR4QA participants submit
results in this format.
CCLQA submission format: CCLQA participants
submit results in this format.
Gold Standard Format: Organizer distributes CCLQA
gold standard data in this format.

1 (1)

1 (2)

Total by lang pair

6 (3)

1 (2)

3

Question Analysis Track: Question Analysis results
contain key terms and answer types extracted from the
input question. These data are submitted by CCLQA
participants and released to IR4QA participants.
CCLQA Main Track: For each topic, a system
returned a list of system responses (i.e. answers to thequestion), and human assessors evaluated them.
Participants submitted a maximum of three runs foreach language pair.
IR4QA+CCLQA Collaboration Track: Using
possibly relevant documents retrieved by the IR4QA
participants, a CCLQA system generated QA results inthe same format used in the main track. Since weencouraged participants to compare multiple IR4QA
results, we did not restrict the maximum number ofcollaboration runs submitted, and used automatic
measures to evaluate the results. Participants were able
to use the full list of IR4QA results available for eachquestion (up to 1000 documents per topic).

3 (1)

3 (1)

3 (1)

3 (1)

3

3

7 (5)

Team Name
APQA
DLUT
KECIR
LTI
NTOUA
WHUQA
Total by lang pair

3
6

6 (2)

CS-CS EN-CS CT-CT EN-CT JA-JA EN-JA

5
3
0

Total by target langParticipants in the CCLQA task submitted results for thefollowing four tracks:

•

2 (1)
1
2 (1)

Table 5. Number of IR4QA+CCLQA Collaboration runs
submitted

3. CCLQA TASK

•

APQA
DLUT
KECIR
LTI
NTOUA
WHUQA
Total by target lang

For more details regarding each interchange format, see thecorresponding examples on the ACLIA wiki [6].

•CS-CS EN-CS CT-CT EN-CT JA-JA EN-JA

0
0

4

9

3

9
12

5

4
9

3.1 Answer Key Creation
In order to build an answer key for evaluation, third party
assessors created a set of weighted nuggets for each topic. A"nugget" is defined as the minimum unit of correct informationthat satisfies the information need. In the rest of this section, we
will describe steps taken to create the answer key data.

3.1.1 Answer-bearing Sentence Extraction
A nugget creator searches for documents that may satisfy the
information need, using a search engine. During this process, adeveloper tries different queries that are not necessarily basedon the key terms in the question text. Whenever a developer
finds an answer-bearing sentence or paragraph, it is saved withthe corresponding document ID.

3.1.2 Nugget Extraction
In the CCLQA task, there were six participating teams (seeTable 3). The number of submitted runs is shown in Table 4 for
the CCLQA main and Question Analysis tracks, and in Table 5
for the IR4QA+CCLQA collaboration tracks.
Table 3. CCLQA Task Participants.
Team Name
APQA
DLUT
KECIR
LTI
NTOUA
WHUQA

A nugget creator extracts nuggets from a set of answerbearing sentences. In some cases, multiple answer-bearing
sentences map to one nugget because they represent the same
meaning, even though the surface text is different. In othercases, multiple nuggets are extracted from a single answerbearing sentence.

Organization
The University of Tokushima
Dalian University of Technology
Shenyang Institute of AeronauticalEngineering
Language Technologies Institute, Carnegie
Mellon University
National Taiwan Ocean University
Wuhan University

3.1.3 Nugget Voting
After nuggets are extracted, we wish to assign weights
ranging from 0 to 1 to each nugget in order to model its
importance in answering the information need.
In earlier TREC evaluations, assessors made binary
decisions as to whether a nugget is vital (contains information― 17 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanto satisfy the information need) or ok. More recently,
TREC/TAC adopts a pyramid nugget evaluation inspired byresearch in text summarization. In a pyramid evaluation,
multiple assessors make a vital/ok decision for each nugget, andweights are assigned according to the proportion of vital scoresassigned [3].
We adapted the pyramid nugget voting method for the
ACLIA evaluation. For each language, there were multiple
independent assessors who voted on answer nuggets. Interassessor agreement was measured via Fleiss’ Kappa statistic, as
shown in Table 6. The observed measurements suggest that it
would be risky to rely on votes from a single assessor; in thisevaluation, each nugget was assessed by all assessors.

Table 6. Inter-assessor agreement on vital/non-vital
judgments on nuggets, measured by Fleiss’ Kappa.
Language# ofnuggets# ofassessors

Kappa (Interassessor
agreement)

423
1375
919

3
9
3

0.458
0.179
0.163

CS
CT
JA

Lang

Type
DEF
BIO

CS

CT

JA

Avg #

Avg Char
LengthAvg
Weight

4.1
6.5

24.9
18.3

0.50
0.34

REL

4.7

30.6

0.42

EVE

6.0

24.1

0.44

WHY

4.0

29.8

0.48

PER
LOC

1.0
1.0

3.4
9.6

0.93
0.87

ORG

1.2

9.6

0.97

DAT
DEF

1.4
25.8

4.7
19.3

0.87
0.78

BIO

13.4

14.4

0.67

REL

9.7

21.3

0.80

EVE
WHY

26.8
11.9

29.5
19.9

0.82
0.76

PER

1.2

8.4

0.92

LOC

1.2

6.3

0.94

ORG

1.4

14.0

0.91

DAT

2.2

20.9

0.87

DEF
BIO
REL
EVE
WHY
PER
LOC
ORG
DAT

11.1
21.0
8.4
13.0
7.3
1.0
1.4
1.4
1.4

14.1
13.4
17.4
14.4
16.0
9.2
3.6
9.0
5.7

0.74
0.57
0.73
0.73
0.68
1.00
0.96
1.00
1.00

In this section, we present the evaluation framework used in
ACLIA, which is based on weighted nuggets. To avoid thepotential ambiguity of the word “answer” (i.e. as in “systemanswer” and “correct answer”), we use the term system
responses or SRs to denote the output from a CCLQA systemgiven a topic. The term gold standard denotes a piece of
information that satisfies the information need.
Both human-in-the-loop evaluation and automatic
evaluation were conducted using the same topics and metrics.
The primary difference is in the step where nuggets in system
responses are matched with gold standard nuggets. During
human assessment, this step is performed manually by humanassessors, who judge whether each system response nugget
matches a gold standard nugget. In automatic evaluation, thisdecision is made automatically. The subsections that follow, we
detail the differences between these two styles of evaluation.

4.1 Human-in-the-loop Evaluation Metrics

We also compared the total number of nuggets and theiraverage character length and weight over the set of topics (see0).
Table 7. Macro-average nugget statistics over topics.

4. EVALUATION METRICS

In CCLQA, we evaluate how good a QA system is atreturning answers that satisfy information needs on average,
given a set of natural language questions.
In an earlier related task, NTCIR-6 QAC-4 [10], eachsystem response was assigned to one of four levels ofcorrectness (i.e. A, B, C, D); in practice, it was difficult forassessors to reliably assign system responses to four different
levels of correctness. For CCLQA, we adopt the nugget
pyramid evaluation method [3] for evaluating CCLQA results,
which requires only that human assessors make a binary
decision whether a system response matches a gold standard
vital or ok nugget. This method was used in the TREC 2005 QA
track for evaluating definition questions, and in the TREC
2006-2007 QA tracks for evaluating "other" questions.
A set of system responses to a question will be assigned an
F-score calculated as shown in Figure 2. We evaluate eachsubmitted run by calculating the macro-average F-score over allquestions in the formal run dataset.
In the TREC evaluations, a character allowance parameter C
is set to 100 non-whitespace characters for English [4]. In the
NTCIR-7 ACLIA, we adjusted the C value for each language
based on the character length statistics of the nuggets in theformal run dataset. In the NTCIR-8 ACLIA, we varied C value
depending on answer type (see Table 7, column Avg Char
Length). In this way, we can accommodate questions withlonger answers (e.g. C=24.9 in CS DEF) and questions withshort answers (e.g. C=3.4 in CS PER) in the same evaluation
framework.

― 18 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, JapanAs an example calculation of an F3 score, consider aquestion with 5 gold standard answer nuggets assigned weights{1.0, 0.4, 0.2, 0.5, 0.7}. In response to the question, a system
returns a list of SRs which is 200 characters in total. A humanevaluator finds a conceptual match between the 2nd nugget andone of SRs, and between the 5th nugget and one of SRs. Then,Letrsum of weights over matched nuggetsR

sum of weights over all nuggets

a HUMAN# of nuggets matched in SRs by
human

L

total character-length of SRs

Ccharacter allowance per match

allowancea HUMAN × C

recall =precision =Thenrecallprecision

F (β )F ( β = 3) =

r
=
R
1
⎧
⎪
= ⎨ allowance
⎪
⎩
L

=

0.4 + 0.7
= 0.391.0 + 0.4 + 0.2 + 0.5 + 0.7

if L < allowance2 × 24= 0.24200

10 × 0.24 × 0.39
= 0.37
9 × 0.24 + 0.39

The evaluation result for this particular question is therefore
0.37.

otherwise( β 2 + 1) × precision × recall
β 2 × precision + recall

4.2 Automatic Evaluation Metrics

Figure 2. Official per-topic F-score definition based onnugget pyramid method.
Note that precision is an approximation, imposing a simple
length penalty on the SR. This is due to Voorhees’ observationthat "nugget precision is much more difficult to compute since
there is no effective way of enumerating all the concepts in a
response" [5]. The precision is a length-based approximation
with a value of 1 as long as the total system response length perquestion is less than the allowance, i.e. C times the number ofnuggets defined for a topic. If the total length exceeds theallowance, the score is penalized. Therefore, although there isno limit on the number of SRs submitted for a question, a long
list of SRs harms the final F score.
The F ( β = 3) or simply F3 score has emphasizes recall

ACLIA also utilized automatic evaluation metrics forevaluating the large number of IR4QA+CCLQA Collaboration
track runs. Automatic evaluation is also useful during
developing, where it provides rapid feedback on algorithmic
variations under test. The main goal of research in automatic
evaluation is to devise an automatic metric for scoring that
correlates well with human judgment. The key technicalrequirement for automatic evaluation of complex QA is a realvalued matching function that provides a high score to system
responses that match a gold standard answer nugget, with ahigh degree of correlation with human judgments on the sametask.
The simplest nugget matching procedure is exact match of
the nugget text within the text of the system response. Formally,
the assessor a HUMAN in Figure 2 is replaced by a EXACTMATCH as
follows:

over precision, with the β value of 3 indicating that recall isweighted three times as much as precision. Historically, a β ofa EXACTMATCH =5 was suggested by a pilot study on definitional QA evaluation[4]. In the more recent TREC QA tasks, the value has been to 3.
Figure 3 visualizes the distribution of F3 scores versus recall
and precision.

∑ max I( n, s )(1)⎧1 : s contains n in surface text level
I EXACTMATCH ( n, s ) = ⎨⎩0 : otherwise(2)n∈Nuggetss∈SRs

EXACTMATCH

Although exact string match (or matching with simple
regular expressions) works well for automatic evaluation offactoid QA, this model does not work well for complex QA,
since nuggets are not exact texts extracted from the corpus text;
the matching between nuggets and system responses requires adegree of understanding that cannot be approximated by astring or regular expression match for all acceptable system
responses, even for a single corpus.
For the evaluation of complex questions in the TREC QA
track, Lin and Demner-Fushman [8] devised an automatic
evaluation metric called POURPRE by replacing a HUMAN with

Figure 3. F3 score distribution parameterized by recall andprecision.

an automatically generated value based on nugget recall:― 19 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japana SOFTMATCH =∑ max NuggetRecalln∈Nuggets(n, s)(3)

the pyramid method. These three activities are described in thesubsections that follow.

| tokenize(n) ∩ tokenize(s) || tokenize(n) |(4)

5.1.1 Topic Developments∈SRs

NuggetRecall token (n, s) =token

Since the TREC target language was English, the evaluation
procedure simply tokenized answer texts into individual words
as the smallest units of meaning for token matching. In contrast,
the ACLIA evaluation metric tokenized Japanese and Chinese
texts into character unigrams. We did not extract word-based
unigrams since automatic segmentation of CS, CT and JA texts
is non-trivial; these languages lack white space and there are no
general rules for comprehensive word segmentation. Since asingle character in these languages can bear a distinct unit ofmeaning, we chose to segment texts into character unigrams, astrategy that has been followed for other NLP tasks in Asian
languages (e.g. Named Entity Recognition [9]).
One of disadvantages of POUPRE is that it gives a partial
score to a system response if it has at least one common token
with any one of the nuggets. To avoid over-estimating the score
via aggregation of many such partial scores, we devised a novel
metric by mapping the POURPRE soft match score values into
binary values:a BINARIZED =∑ max I (n, s)n∈Nuggets

s∈SRs

θ

⎧1 : NuggetReca ll token (n, s ) > θI θ (n, s) = ⎨⎩0 : otherwise

(5)

1.

2.

3.

4.

(6)

We set the threshold θ to be somewhere in between no
match and an exact match, i.e. 0.5, and we used this
BINARIZED metric as our automatic evaluation metric for
ACLIA.

5. EVALUTION TOOLS
To support the creation of test and evaluation topics, as wellas the sharing of system and module I/O using XML
interchange formats, we developed SEPIA (Standard
Evaluation Package for Information Access) 2 which is an
improved version the EPAN system used for NTCIR-7. The
SEPIA toolkit contains a web interface, a set of utilities and abackend database for persistent storage of evaluation topics,
gold standard nuggets, submitted runs, and evaluation results
for training and formal run datasets.

5.1 Topic Creation Tools
The SEPIA topic creation tools consist of interfaces for
topic development, nugget extraction and nugget voting using

2

Figure 4 shows the topic development interface. The left
side is the topic creation form, and the right side is an interface
to the Lemur/Indri search engine [7], which is used by the topic
developer to search for documents relevant to each topic. Topic
developers follow these steps:
If the developer wishes to modify an existing topic, they
can select a topic title from a pull-down list. Topics
marked [x] are completed topics. If the developer wishes
to start creating a new topic, they can type in thecorresponding data and click the “Add” button.
Once the developer has created a topic, then they canprovide additional information related to the topic: anassociated question, a question type, a scenario describing
the information need, and a memo containing any extra
notes about the topic.
In order to search for documents relevant to the topic
being created, the developer may directly enter an Indriquery, or enter key terms and use the “Generate Query”
button to generate an Indri query automatically. When theuse is satisfied with the query, it is sent to the Indri
retrieval engine.
A ranked list of retrieved documents is displayed. The
developer can click on a rank number to browse thecorresponding full document. When the developer selects
a passage which satisfies the information need, thecorresponding information is automatically copied into
the “Answer Text” and “Doc ID” fields in the Answer
data section. The characteristics of the answer-bearing
sentences extracted during the ACLIA evaluation aresummarized in Section 3.1.1.

5.1.2 Nugget Extraction from Answer Text
Figure 5 shows the nugget extraction interface, which isused to extract nuggets from answer-bearing sentences. (Seedetails in Section 3.1.2)
The user selects a topic title from a list of previously
completed topics in the Topic Development task. The user
examines the topic data for the selected topic and the answer
texts for the selected topic. The users type in the corresponding
answer nugget and click “Add” to save the update.

5.1.3 Nugget Voting for Pyramid Method
Figure 6 shows the nugget voting interface, which is used to
identify vital nuggets from among the set of nuggets extractedusing the nugget extraction tool. (See details in Section 3.1.3).
The user first selects a topic title from a list of previously
completed titles in the Topic Development task. The user
examines the topic data for the selected topic, and toggles thecheck boxes next to nuggets which they judge to be vital.

http://sourceforge.net/projects/opensepia― 20 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Table 9. CS-CS official human evaluation.

5.2 Download and Submission

CS-CS Runs
APQA-CS-CS-01-T㻌
APQA-CS-CS-01-D㻌
DLUT-CS-CS-01-T㻌
KECIR-CS-CS-01-T㻌
KECIR-CS-CS-02-T㻌
WHUQA-CS-CS-01-T

The SEPIA is used by each participant to upload theirsubmission file for each run submitted. The SEPIA is also used
to download intermediate results submitted by otherparticipants, as part of an embedded evaluation, For example,
ACLIA participants were able to download the results from
Question Analysis and IR4QA in order to conduct an embedded
CLIR evaluation.

ALL
0.1388㻌
0.1422㻌
0.1954㻌
0.3450㻌
0.3354㻌
0.2566

Table 10. EN-CT official human evaluation.
EN-CT Runs

5.3 Evaluation

NTOUA-EN-CT-01-T㻌
NTOUA-EN-CT-02-T㻌
NTOUA-EN-CT-03-T㻌

The SEPIA provides interfaces for supporting the corehuman-in-the-loop part of evaluation: relevance judgment for
IR4QA and nugget matching for CCLQA. In each task, items tobe evaluated belong to a pool created by aggregating the system
responses from all systems, based on run priority. For the threeruns submitted by each team in each ACLIA task, we createdthree pools of system responses. For the CCLQA task, the firstpool (corresponding to run 1) was evaluated by independent
third-party assessors hired by NII. The second and third pools(corresponding to runs 2 and 3) were evaluated by volunteers
including members of the participant teams. Details of the
CCLQA results are provided in Section 6.1. For the embedded
IR4QA collaboration track, the system responses were
evaluated automatically; details are provided in Section 6.2.

ALL
0.1096㻌
0.1362㻌
0.1159㻌

Table 11. CT-CT official human evaluation.
CT-CT RunsNTOUA-CT-CT-01-T㻌
NTOUA-CT-CT-02-T㻌
NTOUA-CT-CT-03-T㻌

ALL
0.1815㻌
0.1988㻌
0.1461㻌Table 12. EN-JA official human evaluation.
EN-JA Runs
LTI-EN-JA-01-T㻌
LTI-EN-JA-02-T㻌
LTI-EN-JA-03-T㻌

ALL
0.1074㻌
0.1130㻌
0.1045㻌Table 13. JA-JA official human evaluation.

6. EVALUATION RESULTS

JA-JA Runs
LTI-JA-JA-01-T㻌
LTI-JA-JA-02-T㻌
LTI-JA-JA-03-T

In this section, we will present evaluation results for the
CCLQA main track and IR4QA collaboration track.

ALL
0.1069㻌
0.1443㻌
0.14386.1.2 Automatic Evaluation results

6.1 CCLQA Main Track
The human evaluation results for CCLQA are shown foreach language pair in Tables 8 through 13. Assessors evaluatedall (up to 30) system responses per run per question.
Unfortunately, an intermittent but serious bug in the evaluation
system for CCLQA caused some system responses to betruncated prematurely. Since the bug was detected quite close to
the publication deadline for the proceedings, we were unable tocorrect the problem and re-evaluate the affected systems in time
for this paper. A preliminary analysis indicates that thecorrected results may be a few percentage points lower in somecases. We plan to repeat the human evaluation for the
problematic system responses and release the official results assoon as possible. On a positive note, we were able to fix thetruncation problem and re-run the automatic evaluation in time
for this publication. The results are shown in Section 6.1.2.

Table 14. EN-CS automatic evaluation.
EN-CS RunsALL

WHUQA-EN-CS-01-T㻌

0.2895

Table 15. CS-CS automatic evaluation.
CS-CS Runs
APQA-CS-CS-01-T㻌
APQA-CS-CS-01-D㻌
DLUT-CS-CS-01-T㻌
KECIR-CS-CS-01-T㻌
KECIR-CS-CS-02-T㻌
WHUQA-CS-CS-01-T

Table 16. EN-CT automatic evaluation.
EN-CT Runs

6.1.1 Official Runs (Preliminary Results)

NTOUA-EN-CT-01-T㻌
NTOUA-EN-CT-02-T㻌
NTOUA-EN-CT-03-T㻌

Table 8. EN-CS official human evaluation.
EN-CS RunsALL

WHUQA-EN-CS-01-T㻌

0.1841㻌ALL
0.3117
0.26680.30800.46140.45020.3513ALL
0.18130.20350.1717

Table 17. CT-CT automatic evaluation.
CT-CT RunsNTOUA-CT-CT-01-T㻌
NTOUA-CT-CT-02-T㻌
NTOUA-CT-CT-03-T㻌

― 21 ―

ALL
0.28230.25790.1885

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Table 18. EN-JA automatic evaluation.
EN-JA Runs

ALL

CCLQA

0.21070.20290.2040

CYUT1
IR4QA

LTI-EN-JA-01-T㻌
LTI-EN-JA-02-T㻌
LTI-EN-JA-03-T㻌

Table 23. CT-CT Collaboration Track: F3 score based on
automatic evaluation.

KDEG-CT-CT-05-T

0.2973

CYUT2

CYUT3

0.27970.2649

Table 19. JA-JA automatic evaluation.
JA-JA Runs
LTI-JA-JA-01-T㻌
LTI-JA-JA-02-T㻌
LTI-JA-JA-03-T

ALL

7. CONCLUSION

0.20240.22590.2252

6.2 IR4QA+CCLQA Collaboration Track
Tables 20 through 23 show the results from the
IR4QA+CCLQA Collaboration track (see Section 3), evaluatedusing the automatic evaluation method described in Section 4.2.

Table 20. EN-JA Collaboration Track: F3 score based on
automatic evaluation.
CCLQA

IR4QA

0.3169CYUT-EN-JA-02-T

0.3252

CYUT-EN-JA-03-D

0.3358CYUT-EN-JA-04-DN

0.3349REFERENCES

Table 21. JA-JA Collaboration Track: F3 score based on
automatic evaluation.
CCLQA

IR4QA

LTI
BRKLY-JA-JA-01-DN

0.2934BRKLY-JA-JA-02-T
BRKLY-JA-JA-03-DN

0.26860.2074BRKLY-JA-JA-04-DN

0.3000BRKLY-JA-JA-05-T

0.2746

Table 22. EN-CT Collaboration Track: F3 score based on
automatic evaluation.
CCLQA

IR4QA

CYUT1

CYUT2

CYUT3

CYUT-EN-CT-02-T

0.15200.16790.1608

KDEG-EN-CT-02-DN

0.17260.17900.1654QUTIS-EN-CT-04-T

0.12870.14790.1466ACKNOWLEDGEMENTS
We greatly appreciate the efforts of all the NTCIR-8 ACLIA
participants, organizers and advisors.

LTI
CYUT-EN-JA-01-T

This paper presented an overview of the ACLIA (Advanced
Cross-Lingual Information Access) task cluster at NTCIR-8,
with a specific focus on the CCLQA evaluations. We described
the official metrics used in evaluating participant runs; the tools
and process used to develop the official evaluation topics;
summary data regarding the runs submitted; and the preliminary
results of evaluating the submitted runs with the official metric.
We also utilized automatic evaluation metrics for both CCLQA
evaluation and the IR4QA+CCLQA collaboration tasks.

[1] Lita, L.V. Instance-Based Question Answering. Doctoral
Dissertation, Computer Science Department, Carnegie
Mellon University. 2006.
[2] Sakai, T., H. Shima, N. Kando, R. Song, C.-J. Lin, T.
Mitamura, M. Sugimoto. Overview of NTCIR-8 ACLIAIR4QA, Proceedings of NTCIR-8, 2010.
[3] Lin, J., and D. Demner-Fushman. Will pyramids built ofnuggets topple over?, Proceedings of the main conferenceon Human Language Technology Conference of the North
American Chapter of the Association of Computational
Linguistics. 2006.
[4] Voorhees, E. M. Overview of the TREC 2003 Question
Answering Track, Proceedings of TREC 2003, 2004.
[5] Voorhees, E. M. Overview of the TREC 2004 Question
Answering Track, Proceedings of TREC 2004, 2005.
[6] The ACLIA wiki, http://aclia.lti.cs.cmu.edu/ntcir8[7] Indri search engine: http://www.lemurproject.org[8] Lin, J., and D. Demner-Fushman. Methods for
Automatically Evaluating Answers to Complex Questions,
Information Retrieval, 9(5):565-587, 2006.
[9] Asahara, M., Y. Matsumoto. Japanese Named Entity
Extraction with Redundant Morphological Analysis,
Proceedings of NAACL/HLT 2003, 2003.
[10] Fukumoto, J., T. Kato, F. Masui, T. Mori. An Overview of
the 4th Question Answering Challenge (QAC-4) at NTCIR
Workshop 6, Proceedings of NTCIR-6, 2007.

― 22 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Figure 4. SEPIA topic development interface.

― 23 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Figure 5. SEPIA nugget extraction interface.

Figure 6. SEPIA nugget voting interface.

― 24 ―