Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Overview of the NTCIR-7 ACLIA Tasks:
Advanced Cross-Lingual Information Access
Teruko Mitamura* Eric Nyberg* Hideki Shima* Tsuneaki Kato†
Tatsunori Mori‡ Chin-Yew Lin# Ruihua Song# Chuan-Jie Lin+
Tetsuya Sakai@ Donghong Ji◊ Noriko Kando***Carnegie Mellon University †Tokyo University ‡Yokohama National University#Microsoft Research Asia +National Taiwan Ocean University @NewsWatch, Inc.
◊Wuhan University **National Institute of Informatics
teruko@cs.cmu.edu

Abstract
This paper presents an overview of the ACLIA(Advanced Cross-Lingual Information Access) taskcluster. The task overview includes: a definition of andmotivation for the evaluation; a description of thecomplex question types evaluated; the document sourcesand exchange formats selected and/or defined; theofficial metrics used in evaluating participant runs; thetools and process used to develop the official evaluation
topics; summary data regarding the runs submitted; andthe results of evaluating the submitted runs with theofficial metrics.

input to the IR4QA task, which produced retrieval
results that were subsequently fed back into the end-toend QA systems. The modular design and XML
interchange format supported by the ACLIA architecture
make it possible to perform such embedded evaluations
in a straightforward manner. More details regarding the
XML interchange schemes and so on can be found on
the ACLIA wiki [6].

1. Introduction
Current research in QA is moving beyond factoid
questions, so there is significant motivation to evaluate
more complex questions in order to move the researchforward. The Advanced Cross-Lingual Information
Access (ACLIA) task cluster is novel in that it evaluatescomplex cross-lingual question answering (CCLQA)
systems (i.e. events, biographies/definitions, andrelationships) for the first time. Although the QAC4 task
in NTCIR-6 evaluated monolingual QA on complex
questions, no formal evaluation has been conducted incross-lingual QA on complex questions in Asian
languages until now.
As a central problem in question answering
evaluation, the lack of standardization has been pointedout [1], which makes it difficult to compare systems
under a certain condition. In NLP research, systemdesign is moving away from monolithic, black box
architectures and more towards modular architectural
approaches that include an algorithm-independent
formulation of the system’s data structures and dataflows, so that multiple algorithms implementing aparticular function can be evaluated on the same task.
Following this analogy, the ACLIA data flow includes a
pre-defined schema for representing the inputs andoutputs of the document retrieval step, as illustrated in
Figure 1. This novel standardization effort made it
possible to evaluate cross-lingual information retrieval(CLIR) task called IR4QA (Information Retrieval for
Question Answering) in a context of a closely related
QA task. During the evaluation, the question text andQA system question analysis results were provided as

Figure 1. Data flow in ACLIA task cluster
showing how interchangeable data model
made inter-system and inter-task
collaboration possible.
The modular design of this evaluation data flow ismotivated by the following goals: a) to make it possible
for organizations to contribute component algorithms toan evaluation, even if they cannot field an end-to-end
system; b) to make it possible to conduct evaluations on
a per-module basis, in order to target metrics and error
analysis on important bottlenecks in the end-to-end
system; and c) to determine which combination ofalgorithms works best by combining the results fromvarious modules built by different teams. In order toevaluate many different combinations of systems
effectively, human evaluation must be complemented bydevelopment of automatic evaluation metrics that- 11 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japancorrelate well with human judgment. Therefore, wehave developed mechanisms to evaluate the CCLQA
results using automatic evaluation methods.
The CCLQA task included topics developed initiallyfor the evaluation of monolingual systems, and thentranslated to English. Since translation is considered an
important challenge in CCLQA, monolingual QA resultson the same topics and corpora provide an important
baseline for comparison.
The remainder of this paper is structured as follows.
Section 2 describes the specific tasks, topics and corporathat were used in the ACLIA evaluation, along with alist of the data interchange formats that were specified tosupport the exchange of system inputs and outputs.
Section 3 provides the details of the CCLQA task
definition, and Section 4 provides a detailed explanationof the evaluation metrics and measures (including bothhuman measurement and automatic evaluation
techniques). Section 5 provides a brief overview of the
support tools that were built to support topic creationand nugget evaluation. Section 6 presents the evaluation
results for the participant runs that were submitted.
Section 7 provides further analysis of the evaluation
results; the paper concludes in Section 8.

2. ACLIA Tasks & Runs
ACLIA consisted of two tasks named IR4QA andCCLQA, which evaluated Information Retrieval andQuestion Answering, respectively. The same set of
topics was shared between the two tasks. For details on
the IR4QA task evaluation, we refer the reader to the
IR4QA overview paper [2]. This paper focuses on the
CCLQA task, which evaluated QA systems on complex
questions.
For both tasks, we evaluated monolingual and crosslingual topics. Topic languages included English (EN),
Simplified Chinese (CS), Traditional Chinese (CT), andJapanese (JA); target (corpus) languages included CS,
CT and JA. An active participant (i.e. a participant who
submitted at least one run; hereafter called simply aparticipant) submitted a system result for at least one ofsix pairings of source language (i.e. language forquestions) and target language (i.e. language fordocuments), listed below:• Cross-lingual: EN-CS, EN-CT, EN-JA• Monolingual: CS-CS, CT-CT, JA-JA
For each track, a participant submitted up to three runs.
For each run, we evaluated the top 50 system responses
for each question. The official run, Run 1, was evaluatedby independent assessors. Unofficial runs 2 and 3 were
evaluated by volunteer assessors, including assessors
from participant teams.

2.1 Evaluation Topics• DEFINITIONo What is the Human Genome Project?
o What are stem cells?
o What is ASEAN?
o What is the Three Gorges project?
o What is Falun Gong?
• BIOGRAPHY
o Who is Kim Jong-Il?
o Who is Alberto Fujimori?
o Who is Lee Kuan Yew?
o Who is Howard Dean?
• EVENTo List the major events related to controversies
regarding the new Japanese history textbooks.

o List major events in Saddam Hussein's life.
o List major events in formation of European Union.
o List the major conflicts between India and China on
border issues.

• RELATIONSHIPo What is the relationship between Saddam Hussein
and Jacques Chirac?

o Does Iraq possess uranium, and if so, where did itcome from?

A topic developer created a topic by first generating aquestion and a narrative-style information need in thetarget language, which were subsequently translated into
English. This approach supported a comparison between
monolingual and cross-lingual QA using the same set of
topics and corpora. A group of volunteers from theparticipant group created a set of pilot training topics sothat details of the task definitions could be refined andfinalized. The total number of topics in the training
dataset was 88, 84 and 101 for CS, CT, and JA
respectively.
For the formal evaluation, an independent third-party
organization created 100 topics (20 DEFINITION, 20
BIOGRAPHY, 30 RELATIONSHIP and 30 EVENT)
for each target language. Some of the topics are shared
topics which contain a question originally created foranother target language. An analysis of shared topics ispresented later in Section 7.3.

2.2 Corpus
The target corpus consists of digital newswire articles(see Table 1). We select newswire articles in the sametime span (ranging from 1998 through 2001) in order tosupport the evaluation of shared topics.
Table 1. Corpora used in ACLIA.
Language
CS
CT
JA

Corpus Name

Time Span# document

XinhuaLianhe Zaobao
cirb20cirb40Mainichi

1998-2001
1998-2001
1998-1999
2000-2001
1998-2001

295,875
249,287249,508
901,446
419,759

2.3 Input/Output Format

We focused on the evaluation of four types ofquestions:
DEFINITION,
BIOGRAPHY,RELATIONSHIP, and EVENT; examples are shown
below.

In order to combine a CLIR module with a CLQA
system for module-based evaluation, we defined five
types of XML schema to support exchange of results- 12 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japanamong participants and submission of results to beevaluated:

Table 2. CCLQA Task Participants.

• Topic format: The organizer distributes topics inthis format for formal run input to IR4QA andCCLQA systems.
• Question Analysis format: CCLQA participants
who chose to share Question Analysis results
submit their data in this format. IR4QA
participants can accept task input in this format.
• IR4QA submission format: IR4QA participants
submit results in this format.
• CCLQA
submission
format:
CCLQAparticipants submit results in this format.
• Gold Standard Format: Organizer distributes
CCLQA gold standard data in this format.

ATR/NiCT

National Institute of Information andCommunication Technology
Beijing University of Posts & Telecoms
Language Technologies Institute, Carnegie
Mellon University
School of Computer Science, Wuhan
University
Yokohama National University
Institute of Information Science, Academia
SinicaShenyang Institute of AeronauticalEngineering
NTT Communication Science LabsACLIA CCLQA Organizer

CSWHU
Forst
IASL
KECIR
NTCQA
Organizer(baseline)

Table 3. Number of CCLQA runs submitted,
followed by number of Question Analysis
submissions in parenthesis.

3. CCLQA Task

In the CCLQA task, there were eight participatingteams (see Table 2), supplemented by an Organizer team
who submitted simple runs for baseline comparison. The
number of submitted runs is shown in Table 3 for the
CCLQA main and Question Analysis tracks, and in
Table 4 for the IR4QA+CCLQA collaboration tracks.

Organization

Apath
CMUJAVFor more details regarding each interchange format,
see the corresponding examples on the ACLIA wiki [6].

Participants in the CCLQA task submitted results forthe following four tracks:• Question Analysis Track: Question Analysis
results contain key terms and answer types
extracted from the input question. These data aresubmitted by CCLQA participants and released to
IR4QA participants.
• CCLQA Main Track: For each topic, a system
returned a list of system responses (i.e. answers
to the question), and human assessors evaluatedthem. Participants submitted a maximum of threeruns for each language pair.
• IR4QA+CCLQA
CollaborationTrack(obligatory): Using possibly relevant documents
retrieved by the IR4QA participants, a CCLQA
system generated QA results in the same format
used in the main track. Since we encouraged
participants to compare multiple IR4QA results,
we did not restrict the maximum number ofcollaboration runs submitted, and used automatic
measures to evaluate the results. In the obligatory
collaboration track, only the top 50 documents
returned by each IR4QA system for eachquestion were utilized.
• IR4QA+CCLQA
Collaboration
Track(optional): This collaboration track was identicalto the obligatory collaboration track, except thatparticipants were able to use the full list of
IR4QA results available for each question (up to
1000 documents per topic).

Team NameTeam Name

CS-CS

EN-CS

ATR/NiCTApath
CMUJAV
CSWHU
Forst
IASL
KECIR
NTCQA
Organizer (baseline)
Total by lang pair
Total by target lang

3
2 (1)
3 (1)
2 (3)

3
1 (1)
3 (1)

2
1 (1)

CT-CT

JA-JA

EN-JA

3 (1)

3 (1)

1

1

3
2

1
1
14 (6)
10 (2)
24 (8)

3
3

2
1
1
1
7 (1)
6 (1)
13 (2)

Table 4. Number of IR4QA+CCLQA
Collaboration runs submitted for obligatory
runs followed by optional runs inparenthesis.
Team Name
ATR/NiCTApath
CMUJAV
Forst
KECIR
NTCQA
Total by lang pair
Total by target lang

CS-CS

EN-CS

CT-CT

JA-JA

EN-JA

14 (14)11 (11)11

6
2 (2)
20 (20)14 (14)

(20)

(18)

22 (42) 20 (32)42 (74)00(14)
14 (28) 22 (11)36 (39)3.1. Answer Key Creation
In order to build an answer key for evaluation, third
party assessors created a set of weighted nuggets foreach topic. A "nugget" is defined as the minimum unitof correct information that satisfies the information need.
In the rest of this section, we will describe steps taken to
create the answer key data.

3.1.1 . Answer-bearing Sentence Extraction
A nugget creator searches for documents that may
satisfy the information need, using a search engine.
During this process, a developer tries different queries
that are not necessarily based on the key terms in the- 13 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japanquestion text. Whenever a developer finds an answerbearing sentence or paragraph, it is saved with thecorresponding document ID.

Among the four topic types, nuggets for BIOGRAPHY
topics have the shortest length on average for all target
languages. Average nugget weight is much lower for JA(0.57) than for CS (0.85) and CT (0.86).

3.1.2 . Nugget Extraction

Table 7. Macro-average nugget statistics overtopics.

A nugget creator extracts nuggets from a set ofanswer-bearing sentences. In some cases, multiple
answer-bearing sentences map to one nugget because
they represent the same meaning, even though thesurface text is different. In other cases, multiple nuggets
are extracted from a single answer-bearing sentence.
A comparison of character length is shown in Table 5,
which compares the average length for all answerbearing sentences and nuggets in the formal dataset. The
average value for nugget length is incorporated as aparameter in the evaluation model described in Section 4.

Lang

CS

CT

Table 5. Micro-average character length statistics.
Language

Answer-bearing
Sentence

CS
CT
JA

JA

Nugget

46.0
51.4
72.7

18.3
26.8
24.2

After nuggets are extracted, we wish to assign
weights ranging from 0 to 1 to each nugget in order tomodel its importance in answering the information need.
In earlier TREC evaluations, assessors made binary
decisions as to whether a nugget is vital (contains
information to satisfy the information need) or ok. Morerecently, TREC introduced a pyramid nugget evaluation
inspired by research in text summarization. In a pyramidevaluation, multiple assessors make a vital/ok decision
for each nugget, and weights are assigned according to
the proportion of vital scores assigned [3].
We adapted the pyramid nugget voting method for
the ACLIA evaluation. For each language, there were
three independent assessors who voted on answer
nuggets. Inter-assessor agreement was measured via
Cohen’s Kappa statistic, as shown in Table 6. The
observed measurements suggest that it would be risky torely on votes from a single assessor; in this evaluation,
each nugget was assessed by all three assessors.
Table 6. Inter-assessor agreement on vital/nonvital judgments on nuggets, measured by
Cohen’s Kappa.

CS
CT
JA

Avg #

Avg Char
LengthAvg
Weight

DEF
BIO
REL
EVE
Overall
DEF
BIO
REL
EVE
Overall
DEF
BIO
REL
EVE
Overall

4.3
6.0
6.6
11.9
7.6
8.3
18.1
6.0
14.4
11.4
10.4
15.5
10.8
14.4
12.8

26.4
8.3
15.6
21.4
18.0
27.9
16.5
23.5
36.8
27.0
18.9
15.5
24.6
32.3
23.9

0.91
0.87
0.84
0.82
0.85
0.80
0.87
0.91
0.85
0.86
0.59
0.54
0.53
0.61
0.57

4. Evaluation Metrics

3.1.3 . Nugget Voting

Language

Answer
TypeInter-assessor
agreement
0.537
0.491
0.529

We also compared the total number of nuggets andtheir average character length and weight over the set of
topics (see Table 7). Nuggets in JA topics have (12.87.6)/7.6 = 70% more nuggets on average than CS topics.

In this section, we present the evaluation frameworkused in ACLIA, which is based on weighted nuggets. To
avoid the potential ambiguity of the word “answer” (i.e.
as in “system answer” and “correct answer”), we use theterm system responses or SRs to denote the output froma CCLQA system given a topic. The term gold standard
denotes a piece of information that satisfies theinformation need.
Both human-in-the-loop evaluation and automatic
evaluation were conducted using the same topics andmetrics. The primary difference is in the step where
nuggets in system responses are matched with gold
standard nuggets. During human assessment, this step isperformed manually by human assessors, who judge
whether each system response nugget matches a gold
standard nugget. In automatic evaluation, this decision ismade automatically. The subsections that follow, we
detail the differences between these two styles of
evaluation.

4.1. Human-in-the-loop Evaluation Metrics
In CCLQA, we evaluate how good a QA system is atreturning answers that satisfy information needs on
average, given a set of natural language questions.
In an earlier related task, NTCIR-6 QAC-4 [10], eachsystem response was assigned to one of four levels ofcorrectness (i.e. A, B, C, D); in practice, it was difficult
for assessors to reliably assign system responses to four
different levels of correctness. For CCLQA, we adopt
the nugget pyramid evaluation method [3] for evaluating
CCLQA results, which requires only that humanassessors make a binary decision whether a system
response matches a gold standard vital or ok nugget.
This method was used in the TREC 2005 QA track for- 14 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japanevaluating definition questions, and in the TREC 20062007 QA tracks for evaluating "other" questions.
A set of system responses to a question will beassigned an F-score calculated as shown in Figure 2. We
evaluate each submitted run by calculating the macroaverage F-score over all questions in the formal run
dataset.
In the TREC evaluations, a character allowance
parameter C is set to 100 non-whitespace characters forEnglish [4]. We adjusted the C value to fit our dataset
and languages. Based on the micro-average character
length of the nuggets in the formal run dataset (seeTable 5), we derived settings of C=18 for CS, C=27 forCT and C=24 for JA.
Let
rRa HUMANL
C
allowancesum of weights over matched nuggets
sum of weights over all nuggets# of nuggets matched in SRs by
humantotal character-length of SRs
character allowance per match

Figure 3. F3 score distribution parameterized
by recall and precision.
As an example calculation of an F3 score, consider aquestion with 5 gold standard answer nuggets assigned
weights {1.0, 0.4, 0.2, 0.5, 0.7}. In response to thequestion, a system returns a list of SRs which is 200
characters in total. A human evaluator finds a conceptual
match between the 2nd nugget and one of SRs, andbetween the 5th nugget and one of SRs. Then,a HUMAN × C

0.4 + 0.7
= 0.391.0 + 0.4 + 0.2 + 0.5 + 0.7
2 × 24= 0.24
precision =200
10 × 0.24 × 0.39
F ( β = 3) == 0.37
9 × 0.24 + 0.39

recall =Then

recallprecision
F (β )

=

r
R

1


=  allowanceL=

if L < allowanceotherwise( β 2 + 1) × precision × recall
β 2 × precision + recall

The evaluation result for this particular question istherefore 0.37.

Figure 2. Official per-topic F-score definition
based on nugget pyramid method.

4.2. Automatic Evaluation Metrics

Note that precision is an approximation, imposing asimple length penalty on the SR. This is due to
Voorhees’ observation that "nugget precision is much
more difficult to compute since there is no effective wayof enumerating all the concepts in a response" [5]. The
precision is a length-based approximation with a valueof 1 as long as the total system response length perquestion is less than the allowance, i.e. C times thenumber of nuggets defined for a topic. If the total length
exceeds the allowance, the score is penalized. Therefore,
although there is no limit on the number of SRs
submitted for a question, a long list of SRs harms the
final F score.
The F ( β = 3) or simply F3 score has emphasizes
recall over precision, with the β value of 3 indicatingthat recall is weighted three times as much as precision.
Historically, a β of 5 was suggested by a pilot study on
definitional QA evaluation [4]. In the more recent TREC
QA tasks, the value has been to 3. Figure 3 visualizes
the distribution of F3 scores versus recall and precision.

ACLIA also utilized automatic evaluation metrics forevaluating the large number of IR4QA+CCLQA
Collaboration track runs. Automatic evaluation is also
useful during developing, where it provides rapid
feedback on algorithmic variations under test. The main
goal of research in automatic evaluation is to devise an
automatic metric for scoring that correlates well withhuman judgment. The key technical requirement for
automatic evaluation of complex QA is a real-valued
matching function that provides a high score to system
responses that match a gold standard answer nugget,with a high degree of correlation with human judgmentson the same task.
The simplest nugget matching procedure is exact
match of the nugget text within the text of the system
response. Formally, the assessor a HUMAN in Figure 2 isreplaced by a EXACTMATCH as follows:- 15 -a EXACTMATCH =∑ max I( n, s )(1)

1 : s contains n in surface text level
I EXACTMATCH ( n, s ) = 
0 : otherwise(2)n∈Nuggetss∈SRs

EXACTMATCH

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

5. Evaluation Tools
Although exact string match (or matching withsimple regular expressions) works well for automatic
evaluation of factoid QA, this model does not work wellfor complex QA, since nuggets are not exact texts
extracted from the corpus text; the matching between
nuggets and system responses requires a degree ofunderstanding that cannot be approximated by a stringor regular expression match for all acceptable system
responses, even for a single corpus.
For the evaluation of complex questions in the TRECQA track, Lin and Demner-Fushman [8] devised an
automatic evaluation metric called POURPRE byreplacing a HUMAN with an automatically generated value
based on nugget recall:aSOFTMATCH =∑ max NuggetRecalln∈Nuggetss∈SRs

NuggetRecall token (n, s) =token

(n, s)| tokenize(n) ∩ tokenize( s) || tokenize(n) |∑ max I (n, s)n∈Nuggets

s∈SRs

θ

1 : NuggetReca ll token (n, s ) > θI θ ( n, s ) = 
0 : otherwise

5.1. Topic Creation Tools
The EPAN topic creation tools consist of interfaces
for topic development, nugget extraction and nugget
voting using the pyramid method. These three activities
are described in the subsections that follow.

(3)

5.1.1 . Topic Development(4)

Figure 14 shows the topic development interface.
The left side is the topic creation form, and the right side
is an interface to the Lemur/Indri search engine [7],which is used by the topic developer to search for
documents relevant to each topic. Topic developers
follow these steps:

Since the TREC target language was English, theevaluation procedure simply tokenized answer texts into
individual words as the smallest units of meaning for
token matching. In contrast, the ACLIA evaluation
metric tokenized Japanese and Chinese texts into
character unigrams. We did not extract word-based
unigrams since automatic segmentation of CS, CT and
JA texts is non-trivial; these languages lack white space
and there are no general rules for comprehensive word
segmentation. Since a single character in theselanguages can bear a distinct unit of meaning, we choseto segment texts into character unigrams, a strategy thathas been followed for other NLP tasks in Asian
languages (e.g. Named Entity Recognition [9]).
One of disadvantages of POUPRE is that it gives apartial score to a system response if it has at least one
common token with any one of the nuggets. To avoid
over-estimating the score via aggregation of many suchpartial scores, we devised a novel metric by mapping the
POURPRE soft match score values into binary values:a BINARIZED =To support the creation of test and evaluation topics,
as well as the sharing of system and module I/O using
XML interchange formats, we created the Evaluation
Package for ACLIA and NTCIR (EPAN). The EPAN
toolkit contains a web interface, a set of utilities and abackend database for persistent storage of evaluation
topics, gold standard nuggets, submitted runs, andevaluation results for training and formal run datasets.

1.

2.

3.

4.

(5)

(6)

We set the threshold θ to be somewhere in between
no match and an exact match, i.e. 0.5, and we used this
BINARIZED metric as our official automatic evaluation
metric for ACLIA. In Section 7.1, we provide further
comparison of automatic evaluation scores with humanassessor scores, for the three nugget matchingalgorithms introduced in this section.

If the developer wishes to modify an existing topic,
they can select a topic title from a pull-down list.
Topics marked [x] are completed topics. If thedeveloper wishes to start creating a new topic, they
can type in the corresponding data and click the“Add” button.
Once the developer has created a topic, then they
can provide additional information related to the
topic: an associated question, a question type, ascenario describing the information need, and amemo containing any extra notes about the topic.
In order to search for documents relevant to the
topic being created, the developer may directly
enter an Indri query, or enter key terms and use the“Generate Query” button to generate an Indri query
automatically. When the use is satisfied with thequery, it is sent to the Indri retrieval engine.
A ranked list of retrieved documents is displayed.
The developer can click on a rank number tobrowse the corresponding full document. When thedeveloper selects a passage which satisfies theinformation need, the corresponding information is
automatically copied into the “Answer Text” and“Doc ID” fields in the Answer data section. The
characteristics of the answer-bearing sentences
extracted during the ACLIA evaluation aresummarized in Section 3.1.1.

5.1.2 . Nugget Extraction from Answer Text
Figure 15 shows the nugget extraction interface,which is used to extract nuggets from answer-bearing
sentences. (See details in Section 3.1.2)
The user selects a topic title from a list of previously
completed topics in the Topic Development task. The
user examines the topic data for the selected topic and- 16 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japanthe answer texts for the selected topic. The users type in
the corresponding answer nugget and click “Add” tosave the update.

5.1.3 . Nugget Voting for Pyramid Method
Figure 16 shows the nugget voting interface, which isused to identify vital nuggets from among the set ofnuggets extracted using the nugget extraction tool. (Seedetails in Section 3.1.3).
The user first selects a topic title from a list ofpreviously completed titles in the Topic Developmenttask. The user examines the topic data for the selected
topic, and toggles the check boxes next to nuggets which
they judge to be vital.

5.2. Download and Submission
EPAN is used by each participant to upload theirsubmission file for each run submitted. EPAN is also
used to download intermediate results submitted byother participants, as part of an embedded evaluation,
For example, ACLIA participants were able todownload the results from Question Analysis and
IR4QA in order to conduct an embedded CLIR
evaluation.

6.1. CCLQA Main Track
The official human evaluation results for CCLQA are
shown in Table 8 through Table 12 for each language
pair. Runs in Tables 13 through 17 were judged byvolunteers including members of participant teams. We
evaluated up to 50 system responses per run perquestion.
Organizer runs are generated from a sentence
extraction baseline system, sharing the same architecture
as CMUJAV but with a minimally implemented
algorithm that does not take into account answer types.
The run has been motivated by the SENT-BASE
algorithm introduced in TREC 2003 definition subtask
as a baseline [4] that worked surprisingly well, i.e.
ranked 2nd out of 16 runs. In the question analysis stage,
the system translates the entire question string with
Google Translate for crosslingual runs. Then, the system
extracts all noun phrases as key terms. Subsequently inthe retrieval stage, the system retrieves documents withIndri’s simplest query form, “#combine()”. Finally, inthe extraction phrase, starting from the highest ranked
document, the baseline system selects sentences that
contain one of the key terms, until a maximum of 50
system responses have been gathered.

6.1.1 Official Runs

5.3. Evaluation

Table 8. EN-CS official human evaluation.

EPAN provides interfaces for supporting the corehuman-in-the-loop part of evaluation: relevancejudgment for IR4QA and nugget matching for CCLQA.
In each task, items to be evaluated belong to a poolcreated by aggregating the system responses from all
systems, based on run priority. For the three runs
submitted by each team in each ACLIA task, we createdthree pools of system responses. For the CCLQA task,
the first pool (corresponding to run 1) was evaluated byindependent third-party assessors hired by NII. The
second and third pools (corresponding to runs 2 and 3)
were evaluated by volunteers including members of theparticipant teams. Details of the CCLQA results areprovided in Section 6.1. For the embedded IR4QA
collaboration track, the system responses were evaluatedautomatically; details are provided in Section 6.2.

EN-CS Runs

DEF

BIO

REL

EVE

ALL

ATR/NiCT-EN-CS-01-T
CMUJAV-EN-CS-01-T
KECIR-EN-CS-01-T
Apath-EN-CS-01-T
Organizer-EN-CS-01-T

0.22160.21290.24930.16940.13580.31580.26780.25630.11650.14170.23350.18840.15840.11880.10520.14540.13460.13640.07060.07930.22110.19300.18950.11400.1108

Table 9. CS-CS official human evaluation.
CS-CS Runs

DEF

BIO

REL

EVE

ALL

CSWHU-CS-CS-01-T
ATR/NiCT-CS-CS-01-T
IASL-CS-CS-01-T
CMUJAV-CS-CS-01-T
Apath-CS-CS-01-T
Organizer-CS-CS-01-T

0.47520.24150.15360.23260.18000.13600.60120.33760.32450.24980.16620.12480.45920.24290.25480.23210.20670.11010.26620.14300.10430.12190.12980.06400.43290.23160.20340.20270.17020.1044

Table 10. CT-CT official human evaluation.
CT-CT Runs
IASL-CT-CT-01-T

DEF

BIO

REL

EVE

0.3020 0.4075 0.2509 0.1650

ALL
0.2666

Table 11. EN-JA official human evaluation.

6. Evaluation Results
In this section, we will present official evaluation
results for the CCLQA main track, IR4QA collaboration
track, and Question Analysis track.

- 17 -

EN-JA Runs

DEF

BIO

REL

EVE

ALL

CMUJAV-EN-JA-01-T
Organizer-EN-JA-01-T
Forst-EN-JA-01-T
NTCQA-EN-JA-01-T

0.37720.19380.17850.16990.12500.11870.1403
0.09320.16410.12530.11030.0476

0.04330.04390.05160.00230.16270.11330.11230.0676

Table 12. JA-JA official human evaluation.
JA-JA Runs

DEF

BIO

REL

EVE

ALL

CMUJAV-JA-JA-01-T
NTCQA-JA-JA-01-T
Organizer-JA-JA-01-T
Forst-JA-JA-01-T

0.42010.28880.25370.23130.19000.17880.15270.15980.23320.22090.14580.1161

0.0937
0.09150.09160.07860.22010.18730.15250.1366

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Table 18. EN-CS Collaboration Task : F3 scores

6.1.2 . Non-official Runs

based on automatic evaluation
CCLQA

Table 13. EN-CS non-official human evaluation.
DEF

BIO

REL

EVE

ALL

ATR/NiCT-EN-CS-02-T
CMUJAV-EN-CS-03-T
KECIR-EN-CS-03-T
Apath-EN-CS-02-T
ATR/NiCT-EN-CS-03-T
CMUJAV-EN-CS-02-T

0.23370.2192
0.21860.17340.12570.13090.30270.23240.27520.15670.20770.12590.22800.21450.06920.13360.12100.10320.14080.13450.07320.08500.08090.06720.21790.19500.14150.13160.12730.1025

IR4QA

EN-CS Runs

ATR/NiCTCMUJAV
CYUT
HIT
MITEL
RALI

BIO

REL

EVE

ALL

CSWHU-CS-CS-02-D
ATR/NiCT-CS-CS-02-T
CMUJAV-CS-CS-03-T
IASL-CS-CS-02-T
Apath-CS-CS-02-T
ATR/NiCT-CS-CS-03-T
KECIR-CS-CS-02-DN
CMUJAV-CS-CS-02-T

0.47520.25650.23050.17530.18180.16180.23450.12550.60120.32530.20660.15430.17410.1806
0.15320.08970.4592
0.2376
0.26820.27270.19340.16500.11060.13530.26620.14070.15270.10250.13170.10860.1026
0.07390.43290.22980.21370.17850.16870.15060.14150.1058

BIO

REL

EVE

0.3348 0.3833 0.2568 0.1580
0.2946 0.3854 0.2509 0.1658

CMUJAV
KECIR
NLPAI
OT
RALI
WHUCC

DEF

BIO

REL

EVE

ALL

0.13880.10830.16670.12100.05100.03950.16710.1440CMUJAVCYUTTACMUJAV

0.28730.26940.04000.17390.09940.0102

based on automatic evaluation
CCLQA

BIO

REL

EVE

ALL

CMUJAV-JA-JA-02-T
CMUJAV-JA-JA-03-T

0.41700.41840.20590.19580.22690.20330.08300.08470.21760.2092IR4QA

Table 17. JA-JA non-official human evaluation.
DEF(0.1684)(0.0667)(0.0075)

Table 21. JA-JA Collaboration Task : F3 scores
NTCQA

JA-JA Runs

(0.1857)(0.1865)(0.1939)(0.1830)(0.1814)(0.1905)

CCLQA
Forst

Table 16. EN-JA non-official human evaluation.
0.37010.37120.22310.20370.20450.20080.2059KECIR(0.2433)(0.2161)(0.2344)(0.2226)(0.2303)based on automatic evaluation

0.26810.2610EN-JA Runs

CMUJAV

0.2695 (0.2653)
0.2592 (0.2456)-

Table 20. EN-JA Collaboration Task : F3 scoresALL

CMUJAV-EN-JA-02-T
CMUJAV-EN-JA-03-T(0.1684)(0.1651)(0.1816)(0.1829)(0.1631)

CCLQA

IR4QA

IASL-CT-CT-02-T
IASL-CT-CT-03-T

DEF

0.1746
0.19300.1982

APath

Table 15. CT-CT non-official human evaluation.
CT-CT Runs

KECIR

(0.1997)(0.2032)(0.2168)-

based on automatic evaluation

IR4QA

DEF

CMUJAV

Table 19. CS-CS Collaboration Task : F3 scores

Table 14. CS-CS non-official human evaluation.
CS-CS Runs

0.27630.2750-BRKLY
CMUJAVOTCMUJAV(0.2287)(0.2162)(0.2050)

0.26110.24810.2566(0.2477)(0.2394)(0.2333)

6.3. Question Analysis Track
6.2. IR4QA+CCLQA Collaboration Track
Tables 18 through 21 show the results from the
IR4QA+CCLQA Collaboration track (see Section 3),
evaluated using automatic evaluation method describedin Section 4.2. In each table, rows represent IR systems
and columns represent QA systems. Scores are frommandatory runs followed by optional runs in parenthesis.
The best performing runs for all four language pairs
were combinations of different IR4QA and CCLQA
teams, where scores are shown in bold face.

For each Question Analysis runs submitted, we
calculated the percentage of correctly predicted answer
types, shown in Table 22.
Table 22. Question Analysis: Answer Type
Precision
Lang

CS

JA

- 18 -Runs
KECIR-CS-CS-01-T
CSWHU-CS-CS-01-T
CSWHU-CS-CS-03-DN
CSWHU-CS-CS-02-D
CMUJAV-CS-CS-01-T
Apath-EN-CS-01-T
Apath-CS-CS-01-T
CMUJAV-JA-JA-01-T
CMUJAV-EN-JA-01-T

By answer type
DEF

BIO

REL

EVE

1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.90

1.00
0.95
0.95
0.95
1.00
1.00
1.00
0.90
0.55

1.00
0.97
0.97
0.97
1.00
0.97
1.00
0.83
0.87

0.97
0.93
0.93
0.93
0.60
0.50
0.47
0.33
0.17

All
0.99
0.96
0.96
0.96
0.88
0.84
0.84
0.73
0.60

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

EXACTMATCH

7. Further Analysis
In this section, we present further analysis that was
performed following the human and automatic
evaluations of the run data.

7.1. Reliability of Automatic Evaluation
BINARIZEDThis subsection discusses the correlation between theautomatic and human evaluation metrics presented in
Section 4.2, for three different nugget matchingalgorithms (see Table 23).

SOFTMATCH

Table 23. Per-run and per-topic correlation
between automatic nugget matching andhuman judgment

Official F3
Binarized

ORGANIZER-CS-CS-01-T

Runs

ORGANIZER-EN-CS-01-T

Apath-EN-CS-01-T

KECIR-EN-CS-01-T

Apath-CS-CS-01-T

CMUJAV-EN-CS-01-T

IASL-CS-CS-01-T

0.18
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
CMUJAV-CS-CS-01-T

0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
ATR/NiCT-EN-CS-01-T

Figure 5.Human-in-the-loop vs three automatic
metric scores on CS CCLQA Main track data
Softmatch
Hardmatch

NTCQA-EN-JA-01-T

Forst-EN-JA-01-T

ORGANIZER-EN-JA-01-T

0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0
Forst-JA-JA-01-T

0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0

Automatic Metric Scroes

Official F3
Binarized

ORGANIZER-JA-JA-01-T

We compared per-run (# of data points = # of humanevaluated runs for all languages) and per-topic (# of data
points = # of human evaluated runs for all languages
times # of topics) correlation between scores fromhuman-in-the-loop evaluation and automatic evaluation.
The Pearson measure indicates the correlation between
individual scores, while the Kendall measure indicatesthe rank correlation between sets of data points.
The results show that our novel nugget matchingalgorithm BINARIZED outperformed SOFTMATCH
for both correlation measures, and we chose
BINARIZED as the official automatic evaluation metricfor the CCLQA task.
The plots in Figure 4 compare the per-run human vs.
automatic scores, measured using F3 with the different
nugget matching algorithms. We can roughly observethat the distribution of points in the BINARIZED plot
falls
roughly
betweenthe
distributionsof
EXACTMATCH and SOFTMATCH.
The plots in Figures 5 and 6 compare the official
human-in-the-loop and automatic metric scores using
three different nugget matching algorithms. Scores
generated by the human-in-the-loop and automatic
metrics appear to correspond, although there are some
exceptions such as can be seen in runs ATR/NiCT-CSCS-01-T, Apath-CS-CS-01-T, Forst-EN-JA-01-T and
Forst-JA-JA-01-T. We plan to analyze the possible
causes for these outlying data points.

Softmatch
Hardmatch

Automatic Metric Scores

0.5272 0.4054
0.6383 0.4230
0.6758 0.5228

CMUJAV-EN-JA-01-T

0.23640.34790.4506

Figure 4. Per topic plot for Official (human-in-theloop) vs. three kinds of automatic metric scores.

Kendall

NTCQA-JA-JA-01-T

0.44900.63000.7382

Pearson

CSWHU-CS-CS-01-T

EXACTMATCH CHARSOFTMATCH CHAR
BINARIZEDCHAR

Kendall

ATR/NiCT-CS-CS-01-T

Pearson

Per topic( N = 40 × 100 )CMUJAV-JA-JA-01-T

Per run( N = 40 )Official (Human-in-the-loop) Scores

Token

Official (Human-in-the-loop) Scores

Algorithm

Runs

Figure 6.Human-in-the-loop vs three automatic

metric scores (JA CCLQA Main track data)- 19 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Table 25. Scores of crosslingual main trackruns by overlapping topics.

7.2. Comparing IR4QA Relevant
Documents and CCLQA Answerbearing Documents

Overlapping topics

① CS
② CT
③ JA
④ CS & CT⑤ CS & JA⑥ CT & JA⑦ CS & CT & JA

Although we did not utilize document IDs directly as
a form of answer justification during CCLQA evaluation,
we did use them to perform a post-hoc analysis of thecorrelation between documents judged relevant in the
IR4QA task and answer-bearing documents discoveredin the CCLQA task. Table 24 shows the distribution of
relevance levels in IR4QA given CCLQA answerbearing document IDs for each language and answer
type. Unjudged shows the ratio of CCLQA answerbearing documents which belong to IR4QA pools notyet judged.
From the table, we can observe that the majority ofanswer-bearing documents belong to relevant (L2) andpartially relevant (L1) levels and not many were judged
irrelevant (L0).
Table 24. Distribution of IR4QA relevancelevels given CCLQA answer-bearing
document IDs.
Lang

CS

CT

JA

Answer
Type

L2

L1

DEF

77.2%

12.5%

3.7%

6.6%

BIO

27.9%

43.3%

14.6%

14.2%

REL

60.9%

19.5%

8.2%

11.3%

EVE

65.0%

7.3%

12.5%

15.2%

DEF

47.2%

26.4%

3.2%

23.2%

BIO

33.8%

29.3%

1.9%

35.0%

REL

52.6%

12.2%

9.7%

25.5%

EVE

47.7%

20.0%

5.1%

27.2%

DEF

51.8%

25.4%

9.2%

13.6%

BIO

68.0%

14.1%

13.9%

4.0%

REL

57.3%

26.7%

14.2%

1.8%

EVE

58.1%

26.8%

5.7%

9.4%

L0

We also compared system performance on topics thatwere evaluated across languages. Figure 7, combinedwith Table 25 and Table 26 show the average human-inthe-loop scores for 7 different sets of shared topics incrosslingual and monolingal track runs.

⑤

④
⑦
②
CT

⑥

EN-JA
(6 runs)(77 topics)

0.1592-(80 topics)-

0.0000-(72 topics)--

0.1061(5 topics)

0.07250.0000-(13 topics)

0.1781-0.1746(10 topics)-

0.00000.2128(5 topics)

0.18350.00000.1488

Table 26. Scores of monolingual main trackruns by overlapping topics.
Overlapping topics

① CS
② CT
③ JA
④ CS & CT⑤ CS & JA⑥ CT & JA⑦ CS & CT & JA

(77 topics)(80 topics)(72 topics)(5 topics)(13 topics)(10 topics)(5 topics)

CS-CS(14 runs)
0.20760.13800.25300.2445CT-CT(3 run)0.26280.23620.28880.2862JA-JA(6 runs)0.18390.16020.25810.16397.4. Effect of System Response Size

7.3. Shared Topics Across Languages

①

EN-CT
(0 run)In general, we observe that the average scores forshared topics are lower for JA related runs as compared
to CS and CT.

UnjudgedCS

EN-CS(11 runs)

In Figure 8 and Figure 9, we compare F3, recall,precision, and the average number of SRs returned per
topic. In some cases, especially with respect to theJapanese Organizer’s baseline run, we observe that the
number of SRs seems to have an effect on the score.
Given this observation, we performed a post-hoc
analysis using a threshold N, such that only the top N
SRs are evaluated for each topic. We varied the value ofN from 0 to 25 to observe the effect of SR size on
official score1. Figure 10 shows official scores vs. N for
official CS runs. We can see that the score curves reacha plateau and some tail off, suggesting that somesystems could have been optimized to return a shorter
list of system responses and that such optimization
might have improved the overall system rank. In
contrast, the curves for the JA dataset shown in Figure
11 indicate that ranks among runs are relatively
consistent over different values of N. Flat part of curves
can be seen because average number of SRs is 15 at
most except organizer’s runs, as also read from Figure 9.
These monotonically increasing trends mean theplateaus have not been reached and the number of SRs
for JA runs could have been increased for higher scores2.

③
JA

Figure 7. Overlapping topics sharing the same
English question.

1

As mentioned in Section 6.1, N=50 in official evaluation.
We admit it is not obvious to estimate the best number of SRs toreturn before the task though.

2- 20 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Official F3
Official Precision

Official Recall
Avg # of SRs

0.7

60

0.6

50
40

0.4
30
0.3
20

0.2

Runs

ORGANIZER-CS-CS-01-T

Apath-EN-CS-01-T

ORGANIZER-EN-CS-01-T

Apath-CS-CS-01-T

KECIR-EN-CS-01-T

CMUJAV-CS-CS-01-T

CMUJAV-EN-CS-01-T

IASL-CS-CS-01-T

0
ATR/NiCT-CS-CS-01-T

0
ATR/NiCT-EN-CS-01-T

10

CSWHU-CS-CS-01-T

0.1

Avg # of SRs

Scores0.5

Figure 8. Official scores and Avg. # System
Response for EN-CS and CS-CS runs.
Official F3
Official Precision

0.25
0.2
0.15
0.1
0.05

We thank Jim Rankin (CMU) for helping with thedevelopment and maintenance of the EPAN tool set.
We also thank Xinhua, Zaobao, CIRB and Mainichi forproviding the corpora that were used for the evaluations.
We also thank Fred Gey (UC Berkeley) Kui-Lam Kwok
(Queens College) for their valuable advice. We also
greatly appreciate the efforts of all the ACLIA
participants.

References

NTCQA-EN-JA-01-T

Forst-EN-JA-01-T

ORGANIZER-EN-JA-01-T

Forst-JA-JA-01-T

ORGANIZER-JA-JA-01-T

CMUJAV-EN-JA-01-T

NTCQA-JA-JA-01-T

CMUJAV-JA-JA-01-T

0

Avg # of SRs

50
45
40
35
30
25
20
15
10
5
0

0.3

Scores

Acknowledgements

Official Recall
Avg # of SRs

0.35

used to develop the official evaluation topics; summary
data regarding the runs submitted; and the results ofevaluating the submitted runs with the official metric.
Novel aspects of the evaluation included cross-lingual,complex QA evaluation for Chinese and Japanese
corpora, and an embedded evaluation of information
retrieval technologies used for QA.
We utilized
automatic evaluation metrics for the embedded
evaluation, and analyzed both per-topic and per-run
correlation between human-in-the-loop and automatic
evaluation. We also analyzed performance on topics
shared across the different language tracks, and foundthat for some topics, crosslingual QA performance wasbetter than monolingual performance.
We hope that the results of the NTCIR-7 ACLIA task
will contribute to continued rapid progress inInformation Retrieval and Question Answering.

Runs

Figure 9. Official scores and Avg. # System
Responses for EN-JA and JA-JA runs.

7.5. Crosslingual vs Monolingual Runs
To compare the distribution of scores between
crosslingual and monolingual, we plotted official scoresfrom all officially evaluated runs in Figure 12 andFigure 13 for CS and JA respectively. Interestingly,
crosslingual (crosses) outperforms monolingual (circles)
for many topics in CS, but the same is not true for JA.

8. Conclusion
This paper presented an overview of the ACLIA(Advanced Cross-Lingual Information Access) taskcluster at NTCIR-7, with a specific focus on the
CCLQA evaluations. We described the official metrics
used in evaluating participant runs; the tools and process[1] Lita, L.V. Instance-Based Question Answering. Doctoral
Dissertation, Computer Science Department, Carnegie
Mellon University. 2006.
[2] Sakai, T., N. Kando, C.-J. Lin, T. Mitamura, H. Shima, D.
Ji, K.-H. Chen, E. Nyberg. Overview of the NTCIR-7
ACLIA IR4QA Task, Proceedings of NTCIR-7, 2008.
[3] Lin, J., and D. Demner-Fushman. Will pyramids built ofnuggets topple over?, Proceedings of the main conferenceon Human Language Technology Conference of the North
American Chapter of the Association of Computational
Linguistics. 2006.
[4] Voorhees, E. M. Overview of the TREC 2003 Question
Answering Track, Proceedings of TREC 2003, 2004.
[5] Voorhees, E. M. Overview of the TREC 2004 Question
Answering Track, Proceedings of TREC 2004, 2005.
[6] The ACLIA wiki, http://aclia.lti.cs.cmu.edu[7] Indri search engine: http://www.lemurproject.org[8] Lin, J., and D. Demner-Fushman. Methods for
Automatically Evaluating Answers to Complex Questions,
Information Retrieval, 9(5):565-587, 2006.
[9] Asahara, M., Y. Matsumoto. Japanese Named Entity
Extraction with Redundant Morphological Analysis,
Proceedings of NAACL/HLT 2003, 2003.
[10]Fukumoto, J., T. Kato, F. Masui, T. Mori. An Overview of
the 4th Question Answering Challenge (QAC-4) at NTCIR
Workshop 6, Proceedings of NTCIR-6, 2007.

- 21 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

0.45

0.4

0.35

Apath-EN-CS-01-T
Apath-CS-CS-01-T

0.3

ATR/NiCT-EN-CS-01-T
ATR/NiCT-CS-CS-01-T

0.25

ScoresCMUJAV-EN-CS-01-T
CMUJAV-CS-CS-01-T
0.2

CSWHU-CS-CS-01-T
IASL-CS-CS-01-T

0.15

KECIR-EN-CS-01-T
ORGANIZER-EN-CS-01-T
ORGANIZER-CS-CS-01-T

0.1

0.05

0
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
System Response Threshold

Figure 10. Official CS scores using only top-N System Responses per topic

0.25

CMUJAV-EN-JA-01-T
0.2

CMUJAV-JA-JA-01-T
Forst-EN-JA-01-T
0.15

ScoresForst-JA-JA-01-T
NTCQA-EN-JA-01-T

0.1

NTCQA-JA-JA-01-T
ORGANIZER-EN-JA-01-T

0.05

ORGANIZER-JA-JA-01-T

0
0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
System Response Threshold

Figure 11.Official JA scores using top-N System Responses per topic- 22 -

61
54
1 6 12

1 6 12

19

19

26

26

33

33

40

47

Topic

54
47
40

Topic

61

68

68

75

75

82

82

89

89

96

96

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Official Score

Official Score

Figure 12. Official CS scores plotted for each
topic where circles are for monolingual andcrosses are for crosslingual runs.

Figure 13. Official JA scores plotted for each
topic where circles are for monolingual andcrosses are for crosslingual runs.

- 23 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Figure 14. EPAN topic development interface.

- 24 -

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Figure 15. EPAN nugget extraction interface.

Figure 16. EPAN nugget voting interface.

- 25 -