The Chinese Text Retrieval Tasks of NTCIR Workshop 2
Kuang-hua Chen+ and Hsin-Hsi Chen*+

Department of Library and Information Science
National Taiwan University
1, Sec. 4, Roosevelt Road
Taipei 10617, Taiwan
khchen@ccms.ntu.edu.tw*

Department of Computer Science and Information Engineering
National Taiwan University
1, Sec. 4, Roosevelt Road
Taipei 10617, Taiwan
hh_chen@csie.ntu.edu.tw

Abstract
This paper is a report of Chinese Text Retrieval(CHTR) tasks in NTCIR Workshop 2. CHTR tasks
fall into two categories: Chinese-Chinese IR (CHIR)
and English-Chinese IR (ECIR). The definitions,schedules, test collection (CIRB010), search results,
evaluation, and initial analyses of search results of
CHIR and ECIR are discussed in this paper.

1. Introduction
This paper describes the Chinese Text Retrieval tasksof NTCIR workshop 2 and introduces the Chinese
Information Retrieval Benchmark version 1(CIRB010) using in the tasks of NTCIR workshop 2.
NTCIR-1 is the first evaluation workshop designed
to enhance research in Japanese text retrieval [1]. We
have negotiated this kind of joint efforts in evaluating
Eastern-Asia text retrieval with Dr. Kando for a longtime. NTCIR-2 is the result of the attempt and is thefirst evaluation workshop designed to enhanceresearch in Japanese and Chinese text retrieval. The
CHTR tasks fall into two categories: Chinese queries
against Chinese documents (CHIR, a monolingual IRtask) and English queries against Chinese documents(ECIR, a cross-language IR task). Both CHIR andECIR are ad hoc IR tasks, i.e., the document set isfixed for various topics.
The goals of CHIR and ECIR tasks in NTCIR are
shown as follows.
llll

Promote the Chinese IR researches
Investigate effective techniques for Chinese IR
Construct a mechanism for Chinese IR evaluation
Provide a forum to present research results andexchange research ideas

The test collection used in CHTR tasks of
NTCIR-2 is called CIRB010. It contains 132,173
documents [2]. These documents are all news stories

downloaded from web sites of Chinatimes [3],Chinatimes Commercial [4], Chinatimes Express [5],Central Daily News [6], and China Daily News [7]during the period of May 1998 to May 1999. We areauthorized to use these news stories for evaluation in
NTCIR. The advantages of using news wire aremanifold . The newswire is usually quite novel,
quickly-updated and with multiple subjects incontents. In addition, the evaluation results would be
more reliable through the common and popular data
available from real environment.
Each participant could conduct ECIR task, CHIR
task or both tasks. Sixteen groups from sevencountries or areas had enrolled CHTR tasks. Amongthem, 14 groups enrolled CHIR task and 13 groups
enrolled ECIR task. However, not all enrolled groups
submit search results. Table 1 shows the distributionof groups enrolling CHIR and ECIR tasks and groups
submitting search results at final. The search resultsof 115 runs were submitted from 11 groups. 98 runs
from 10 groups are for CHIR task; 17 runs from 7
groups are for ECIR task. Table 2 shows the detailed
statistics.
Table 1. Distribution of Participants
Enrolled
Submitted
CHIR
ECIR
CHIR
ECIR
Canada
1
1
0
0
China
2
1
2
1
Hong Kong
2
1
1
0
Japan
3
2
3
1
Taiwan
2
2
2
2
UK
1
1
0
0
USA
3
5
2
3
Table 2. Participants of CHTR Tasks
CHIR
ECIR
Total
# of groups enrolled14
13
16# of groups submitted
10
711# of submitted runs
98
17
115

The rest of this report will focus on the testcollection (CIRB010) and the CHIR task and ECIR
task. Section 2 will introduce the task of CHTR in
NTCIR workshop 2. Section 3 will describe the testcollection used in the CHTR tasks. Section 4 will
give a picture of the evaluation mechanism. Section 5
will analyze the search results in a broad view.
Section 6 will give a conclusion.

2. CHIR Task and ECIR Task
Two kinds of IR tasks have been arranged for
NTCIR-2 Chinese Text Retrieval. The first is
Chinese IR Task (a monolingual IR task) and thesecond is English-Chinese IR Task (a cross-language
IR task). Both tasks are ad-hoc-based tasks, that is tosay, the document set is fixed against the different
topics.
2.1 Schedule
By
Submit an application. Online
2000-07-15: application is available at:
http://www.rd.nacsis.ac.jp/~ntcadm/wo
rkshop/application2/app2-en.html
2000-08-31: CIRB-1-CH CD (132,172 documents
and 50 Chinese topics) will be
distributed to the participants ofChinese IR Task, and CIRB-1-EN CD(132,172 documents and 50 English
topics) will be distributed to theparticipants of English-Chinese IR
Task.
2000-09-30: Search results and system description
forms submission.
2001-01-10: Results of Relevance Assessments will
be distributed to the participants.
2001-02-12: Papers for the working-note
proceedings submission.
2001-03-07/ Workshop meeting at NII, Tokyo,
2001-03-09: Japan.
2001-03-16: Camera-ready copies for theproceedings.
2.2 Task Typel Chinese IR Task (“CHIR”)
The Chinese IR Task is to assess the capability ofparticipating systems in retrieving Chinese
documents using Chinese queries. Chinese texts,
which are comp osed of characters without
explicit word boundary, make the retrieval task
more challengeable than English ones. The
participating systems can employ any approaches.
Either word-based or character-based systems are
acceptable. The organizer will not provide any
segmentation tools and Chinese dictionaries.
l English-Chinese IR Task (“ECIR”)
The English-Chinese IR Task is to assess thecapability of participating systems in retrieving

Chinese documents using English queries. The
organizer will not provide any segmentation tools
and English-Chinese dictionaries.
2.3 Query TypeWe distinguish each run according to the length ofquery. Three different types of run are defined as thefollows.
l Long query (“LO”): Any query uses the narrativeof the topics.
l Short query (“SO”): Any query uses no narrativeof the topics.
l Very short query (“VS”): Any query uses neither
narrative nor question of the topics.
l Title query (“TI”): Any query uses the title of the
topics only.
The participating group could use any type of query
to carry out the IR tasks.

3. The Test Collection: CIRB010
3.1 Document Set
In order to facilitate the process of identification andanalysis of the contents, documents are supposed tobe consistent in their format. Therefore, we edit thehtml documents downloaded from web and delete thenois es. In addition, we add tags to mark thedesignated fields, which are document id-number,news reporting date, title, paragraphs, etc.
Consequently, every document has the same format
and tags. The documents are encoded in BIG5 with
XML-style tags. We add tags to documents to marktheir specifications and sections. The meaning of eachtag is described below:
l <doc> </doc>: Denote the beginning and theending of a document.
l <id> </id>: Denote the identification code of the
document, which is composed of the source, thesubject category, and the serial number of
document. The code can also be seen as the full
file path of each document in our data CD. The
document is assigned a 7-digits serial number ineach category, so we can identify each document
and recognize its source by this unique id code.
l <date> </date>: Denote the date of the news using
ISO8601 format. It is presented in the formatof “year (in 4 digits)-month (in 2 digits)-date (in2 digits)”.
l <title> </title>: Denote the title of news.
l <text> </text>: Denote the text of news.
l <p> </p>: Denote the paragraphs of news.
Figure 1 shows a sample document. We only modify
documents’ format and do not change their contents.
Table 3 shows the statistics of CIRB document

<doc><id>cts_foc_0005657</id><date>1999-05-07</date><title>解決高鐵融資 尋求第三管道</title><text><p>【記者羅兩莎台北報導】據負責台灣高速鐵路聯合貸款的主辦銀行表示，高鐵融資問題目前仍卡在銀行
團、交通部高鐵局以及台灣高鐵公司「三方合約」內
容的訂定。在銀行團和交通部一直未能就相關歧見達成共識之下，三大主辦銀行原則決定，將尋求行政院
經建會等第三管道與交通部協調，以儘早解決銀行團
和交通部之間對融資問題的歧見。</p><p>
高鐵案將向國內銀行融資二千八百多億元，這項聯貸
案確定由交銀、台銀和中國國際商業銀行共同主辦。
不過，由於高鐵是國內首宗ＢＯＴ案，潛在風險究竟
有多高，銀行無從評估。</p><p>
據主辦銀行主管表示，銀行當然希望債權確保不會有問題，譬如，在三方合約中訂定，由政府出面保證萬一將來台灣高鐵公司蓋不下去時，政府可以出面買下，負責把工程完成等。</p></text></doc>

Figure 1. Example of CIRB Document Tagging
Table 3. Document Set
News Agency# of Document Percentage
Chinatimes
38,16328.8%
Chinatimes Commercial
25,812
19.5%
Chinatimes Express
5,747
4.4%
Central Daily News
27,77021.0%
China Daily News34,72826.3%
Total
132,173(200MB)
3.2 Topic
Three main procedures constructing topics of
CIRB010 are shown as follows:(1) Collecting information request
In order to increase the similarity between ourbenchmark and real environment, we build the topics
using real users’ information requests. We collected
405 requests through questionnaire on web. There areboth closed and open-ended questions about thetypes and subject of requests, narratives of requests,
and other related information. The basic assumptionof the method is that users may state their specificinformation request distinctly and exhaustively.
(2) Selecting information request
The responses of questionnaires gained from Internet
was not entirely so qualitative, complete andexhaustive, and the type and subject of the request
provided by user is not necessarily suitable for
evaluation purpose in our IR benchmark. Therefore,
we pic k out 50 best requests from 405 collected
requests according to some important criteria in threephases explained as follows. In the first phase we
examine the statements and narratives, and thendelete too simple, short, ambiguous, and subjective
ones. Besides, we also exclude the requests with thefollowing criteria: (a) the coverage of subject is too
broad; (b) request shows great gap in document set;(c) the answer to request change rapidly from time totime. In the second phase, a full-text information
retrieval system is used to test the number of possible
relevant documents. The motive of this method is todetermine if the request is too comprehensive or too
limited in their subject coverage. In the final phase,
we select 50 requests best fitting in the requirements.
The main consideration lies in the similarity between
requests and degree of the explicitness anddistinctness of the request. The results of request
selection are shown as Table 4.
Table 4. The Selection of Information Request
Selection
methodNumber of
topics deleted
Number of
topics reserved

1st selection 2nd selection 3rd selection
Judged by Supported Judged byresearcher by full-text researcher
IR software
163

173

19

242

69

50

(3) Constructing Topics
The main task of this phase is to establish the topics
in accordance with the 50 final requests. We use four
fields: title, question, narrative, and concepts torepresent topics in accordance with the TREC’s
convention. The meaning, content, syntax andresources of each field are shown in Table 5. The“title” field has the widest coverage in its content
with comparison to the other three fields. The
coverage of “question” field is the second to “title”
field. The “narrative” field is the most specificbecause of its detailed description. The keywords in“concepts” field touch on the contents of above threefields. The relationship between the four fields canbe illustrated in Figure 2. The 50 topics can beroughly classified into 9 categories. The average
number of words in a topic is 169. No significant
differences show in the corresponding fields among
different topics with comparison to other testcollections. The statistics is shown in Table 6.
Content coverage

set. CIRB010 contains 132,173 documents with thesize of 200MB. The subjects of documents arevarious, such as politics, finance, social, life, sports,
entertainment, international issue, and information
technology and so on.

<narrative><question><concepts><title>Figure 2. The Relation between Topic Fields

Table 5. Fields of CIRB Topics
Fields<title><question><narrative><concepts>

Content
SyntaxResources
Concise representation of information request Noun or Noun Phrase The subject ofsubject.
information request
The whole informa tion
Brief descriptions of content of information request. One or two sentences
requestNarratives of information request such as thefurther interpretation to request and proper nouns,
The whole information
the list of relevant or irrelevant information, and the Several sentences
requestspecific requirements or limitations of relevant
documents.
Relevant or irrelevant
Keywords relevant to the whole topic.
One or more keywords keywords about
information request<topic><number> CIRB010TopicZH011</number><title>金融機構合併。</title><question>
查詢我國政府單位鼓勵金融機構合併之各項措施。
</question><narrative>
財政部等相關單位為健全金融市場、改善金融體質，推動了一連串鼓勵銀行、證券商及保險公司等金融機構合併的措施。相關文件內容包括各項具體的獎勵優惠辦法、施行細節、法令中明定之規範條文、以及各界對相關政策的討論與評估。若文件中只陳述金融機構合併之個案，視為不相關。
</narrative><concepts>
金融機構、合併、銀行合併、租稅優惠、租稅減免、稅前盈餘、低利融資、促進產業升級條例、財政部、經濟部、
央行、中央銀行、增值稅、印花稅、證交稅。
</concepts></topic>

Figure 3. Example of CIRB Topic
Table 6. Document Length of CIRB and TREC

CIRB

TREC
Chinese Topic
1-54
TREC-6 Topic
301-350(Chinese translation)

Fields<title><question><narrative><concepts>
Total<title><desc><narr>
Total<title><desc><narr>
Total

Minimum
3
12
57
26
103
4
6
31
53
3
7
26
64

Maximum
13
37
141
74
244
29
35
174
204
13
87
217
237

4. Evaluation
This is our first attempt to organize Chinese IR
evaluation workshop. We follow the method used in
TREC and NTCIR-1. The TREC’s evaluation
program is used to score the research results. It
provides the interpolated recall and precision at 11
points, average precision (non-interpolated) over al
relevant documents and precision at 5, 10, 15, 20, 30,
100, 200, 500, and 1000 documents. Each

Average
6.52
23.64
93.90
44.68
168.74
12.30
17.48
81.54
111.32
6.80
30.14
94.56
131.5

Standard Deviation
2.23
5.92
20.43
11.58
27.77
5.58
7.40
30.28
31.36
2.28
16.88
42.15
42.03

Standard Deviation/Average
0.34
0.25
0.22
0.26
0.16
0.45
0.43
0.37
0.28
0.34
0.56
0.45
0.32

participating group has to submit its search results inthe designated format. The result file is a list oftuples in the following form:
qid
iter docid
rank
sim
runid
giving CIRB010 document “docid” (a string
extracted from the <id> </id> field, e.g. <id>cts_cec_1999111514 </ id>) retrieved by query “qid”(an integer extracted from the last 3 digits in<number> </number> field of topic, e.g., <number>
CIRB010TopicEN002 </number>, the “qid” is 002)

with similarity sim (a float). The result file isassumed to be sorted numerically by “qid”. “Sim” isassumed to be higher for the documents to be
retrieved first. The “iter” and “rank” could be
regarded as the dummy filed in tuples. In addition,
each field in tuples is separated by inserting ‘TAB’(\x0A, \t) character.
A sample file is shown as follows.
001
001
001
001
001
001
001
001
001
001
001
001

0
0
0
0
0
0
0
0
0
0
0
0

CTS_POL_0003207 0 0.992917
CTS_POL_0003217 1 0.992338
CDN_FOC_00057312 0.988612
CDN_FOC_00041853 0.987523
CHD_POL_00058924 0.985803
CDN_FOC_00056435 0.984673
CTS_POL_0002947 6 0.978978
CTS_POL_0003249 7 0.973372
CDN_FOC_00041968 0.966231
CTS_POL_0003216 9 0.931390
CHD_POL_000251210 0.867546
CDN_FOC_000695111 0.857641

RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID
RUN-ID

A list of relevance between each topic anddocuments in a benchmark is needed to facilitate thecomparison and evaluation of IR systemeffectiveness. This is the so-called “relevancejudgment.” The relevance judgment is undertaken bypooling method. Due to the fact that each run submits
about 1,000 documents, the following criteria areused to form the pool.
l For each participating group, only one run foreach query type run is selected, i.e., at most 4
runs are selected for each participating group.
l A document qualifying as a member of pool has
to be retrieved by more than one run.
On average, the size of pool for each topic is about
900 documents. Table 7 shows the total number of
documents in pool.
Table 7. The Size of Pool for each Topic
topic 001
topic 002
topic 003
topic 004
topic 005
topic 006
topic 007
topic 008
topic 009
topic 010
topic 011
topic 012
topic 013
topic 014
topic 015
topic 016
topic 017

1035
922
1016
956
1015
703
1175
1177
663
1145
995
1086
748
928
882
926
889

topic 018
topic 019
topic 020
topic 021
topic 022
topic 023
topic 024
topic 025
topic 026
topic 027
topic 028
topic 029
topic 030
topic 031
topic 032
topic 033
topic 034

716
896
880
987
1271
845
905
895
1158
812
807
830
923
704
736
710
776

topic 035 1008
topic 036
898
topic 037
918
topic 038
540
topic 039
720
topic 040 1134
topic 041
935
topic 042
912
topic 043
765
topic 044897
topic 045
881
topic 046
764
topic 047
720
topic 048 1017
topic 049
989
topic 050
754
average
898.48

While performing relevance judgments, every
judge should read and understand the meaning of the
topic carefully and assign each of them to the mostappropriate category (mentioned below) from their

viewpoint mainly according to “question” field of
topic. In order to keep judges’ criterion consistent,
the judges must complete the judgments for a topic ina period of time. Each topic is judged by 3 judges. In
total, 23 judges spend 799 hours in relevancejudgment.
The “subject relevance” concept is adopted in
relevance judgment. That is to say, we pay more
attention to the concrete meaning, which can beperceived from the text. Based on this concept, thejudges should make an objective link between
document and topic. This will increase theconsistency and reliability of judgments performedby different judges. As for measurement granularity,
it is supposed that some distinct definitions of
relevance degree should be identified to keep
judgment objective. 4 categories of relevance are
identified: “ Very Relevant”, “Relevant”, “Partiallyrelevant”, and “Irrelevant.” Each kind of relevance isassigned a relevance score. “Very relevant” is 3,“Relevant” is 2, “Partially relevant” is 1, and“Irrelevant” is 0.
Since one unified relevance score have to be
produced for final relevance judgment using TREC’sscoring program, we combine judgment results ofthree judges, and then decide how to interpret themeaning of the score and how can it be applied t o IR
evaluation. Based on the following philosophy, we
devise a method to integrate 3 relevance scores toform one relevance score.
l Each judge has equal contribution to final
relevance score.
l Each judgment is independent.
The following formula is used to combine 3
judges’ relevance score,(XA + XB + XC ) 3
R=3
where Ｘ means the relevance category assigned byeach judge, and A, B, C represent the three different
judges. The value of R will be between 0 and 1. The
closer the score to 1, the more relevant the document
to the topic.
As mentioned above, TREC scoring program isused to calculate recall and precision. Since it uses
binary relevance judgment, we have to decide thethreshold. Two thresholds are decided: one is 0.6667;
the other is 0.3333. The so-called rigid relevancemeans the final relevance score should between
0.6667 and 1. That is to say, it is equivalent that each
person assigns “relevant (2)” to the document.
[(2+2+2)/3/3=0.6667]
The so-called relaxed relevance means the final
relevance score should between 0.3333 and 1. That is
to say, it is equivalent that each person assigns“partially relevant (1)” to the document.
[(1+1+1)/3/3=0.3333]

5. Search Results
We will report the search results in a broad view andanalyze some of runs using different query types inthis section. The different techniques which eachparticipating group took could be referred to each
paper in workshop proceedings.

query type and the technical details of eachparticipating group should be referred to thecorresponding papers in conference proceedings.
Table 8. List of CHIR Participating Groups
1

Communications Research Laboratory

2

University of California at Berkeley

5.1 CHIR Task

3

Queens College, CUNY

The search results of CHIR task are submitted from
10 participating groups listed in Table 8. Somegroups are “Full Participation” and some are“Anonymous Participation”. The number ofsubmitted runs by the query types is shown in Table
9. The query types have been mentioned in Section 2.

4

Chinese research group of Lab Furugori, theUniversity of Electro-Communications

5

Basically, most of the participating groups use
tf/idf approach in a little different form. This showsthat the long-history tf/idf approach still play animportant role in information retrieval.
Both CRL group and PIRCS group adopt
probabilistic model and they are the leading groupsin CHIR task. This implies that the probabilistic
model shows good performance in CHIR task of
NTCIR workshop 2.
In general, the performance of “Very Short
Query” is the best; the performance of “Short Query”
is better than that of “Long Query”. The performanceof “Title Query” is worst. It seems that the longquery conveys many noises. On the contrary, the title
query conveys little information. Observing thesearch results, we conclude that the short query isappropriate for CHIR task. It seems that the name of“Very Short Query” will mislead us to draw a directconclusion that query should be short. In fact, the“Very Short Query” means that the participants could
use the concepts identified in the topic. In the case of
CIRB010, the <concepts> field contains many
significant keywords. As a result, the runs applying“Very Short Query” perform well.
Since the leading groups show good performancein all runs, we will not expla in the details in eachHong Kong Polytechnic University

6

Umemura Lab. Department of Information andComputer Sciences, Toyohashi University ofTechnology

7

NTHU NLP Lab and Knowledge Express
Technology Inc

8

Institute of Software, Chinese Academy of
Sciences

9

Trans-EZ Information Technology Inc.

(1) All Runs
The recall/precision graphs of top runs of CHIR task
are showed in Figure 4 (relaxed relevance) and
Figure 5 (rigid relevance). The techniques used inthese top runs are showed in Table 10 and Table 11.
From Table 10 and Table 11, it is easy to find out that
all runs use query expansion techniques except
Brkly-CHIR-LO-01. CRL group performs well in all
query types, since it uses automatic feedback to carryout query expansion. We also find that stop-word list
seems a good resource for Chinese information
retrieval. Brkly group applies logistic regression
technique to tune the various parameters. The unique
technique shows good performance in CHIR task.

Department of Computing

10 Fujitsu R&D Center
Table 9. Number of Submitted Runs of CHIR
# of Groups

10# of Total RUN

98# of "LO" RUN

30# of "SO" RUN

12# of "VS" RUN

27# of "TI" RUN

29(2) “Long Query” Runs
The recall/precision graphs of top runs of CHIR“Long Query” are showed in Figure 6 (relaxed
relevance) and Figure 7 (rigid relevance). Since the
top runs measured in relaxed relevance and in rigid
relevance are the same, we only use one table (Table
12) to describe the techniques used in these top runs.
(3) “Short Query” Runs
The recall/precision graphs of top runs of CHIR
“Short Query” are showed in Figure 8 (relaxed
relevance) and Figure 9 (rigid relevance). Since the
top runs measured in relaxed relevance and in rigid
relevance are the same, we also use one table (Table
13) to describe the techniques used in these top runs.
(4) “Title Query” Runs
The recall/precision graphs of top runs of CHIR
“Title Query” are showed in Figure 10 (relaxed
relevance) and Figure 11 (rigid relevance). The
techniques used in these top runs are showed in Table
14 and Table 15.

(5) “Very Short Query” Runs
The recall/precision graphs of top runs of CHIR
“Very Short Query” are showed in Figure 12 (relaxed
relevance) and Figure 13 (rigid relevance). The
techniques used in these top runs are showed in Table
16 and Table 17.
5.2 ECIR Task
The search results of ECIR task are submitted from 7
participating groups listed in Table 18. All groups are“Full Participation”. The number of submitted runsby the query types is shown in Table 19.
Table 18. List of ECIR Participating Groups
University of California at Berkeley
University of Maryland
Queens College, CUNY
Umemura Lab. Department of Information andComputer Sciences,
Toyohashi University of Technology
5 NTHU NLP Lab and Knowledge Express
Technology Inc
6 Institute of Software, Chinese Academy of
Sciences
7 Trans-EZ Information Technology Inc.
1
2
3
4

Table 19. Number of Submitted Runs of ECIR
# of Groups
7# of Total RUN
17# of "LO" RUN
8# of "SO" RUN
2# of "VS" RUN
6# of "TI" RUN
1

Observing the index unit, we find thatword-based approaches are much better than otherapproaches in ECIR task. In addition, PIRCS groupcombines
word-based
andcharacter-based
approaches to construct index file.
Since only PIRCS group submits runs of all querytypes in ECIR task, we will compare theperformances of each query type based on the search
results of PIRCS. The “Title query” is the worst
among all query types. The difference among “Long
Query”, “Short Query”, and “Very Short Query” islittle. However, the “Short Query” is better thanothers. As mentioned before, the “Very Short Query”
in CIRB010 conveys many important keywords, so
the performance is good. This phenomenon isdifferent from the observation pointed out in
NTCIR-1 Japanese IR task [1].
Since the leading groups show good performancein all runs, we will not explain the details in eachquery type. Again, the interested readers have to referto the corresponding papers in the conference
proceedings for technical details.
(2) “Long Query” Runs
The recall/precision graphs of top runs of CHIR“Long Query” are showed in Figure 16 (relaxed
relevance) and Figure 17 (rigid relevance).
(3) “Short Query” Runs
The recall/precision graphs of top runs of CHIR
“Short Query” are showed in Figure 18 (relaxed
relevance) and Figure 19 (rigid relevance).

(1) All Runs

(4) “Title Query” Runs

The recall/precision graphs of top runs of ECIR task
are showed in Figure 14 (relaxed relevance) and
Figure 15 (rigid relevance). Since there are only 17
ECIR runs, we use one table (Table 20) to list thetechniques which participating groups used in ECIR
task. Observing Figure 14, we find that PIRCS groupoutperforms other groups using relaxed relevancemetric. Further investigating Table 20, we have an
idea that PIRCS uses MT software to carry outtranslation commission. On the contrary, most groups
use dictionaries with select-all, select-top-1,
select-top-n, or select-all approaches. Among
select-X approach, select-all is better thanselect-top-3; select-top-3 is better than select-top-2;
select-top-2 is better than select-top-1. We could notconclude directly that select-all is the best among all
select-X approaches, since some groups also apply
corpus-based approach at the same time. However,no enough information shows how participatinggroups utilize corpus. Did they calculate mutual
information? Did they calculate the bilingual mutual
information? The detailed information should bereferred to the corresponding papers.

Only one group submits ECIR “Title Query” run.
The recall/precision graphs of top runs of CHIR
“Title Query” are showed in Figure 20 (relaxed
relevance) and Figure 21 (rigid relevance).
(5) “Very Short Query” Runs
The recall/precision graphs of top runs of CHIR
“Very Short Query” are showed in Figure 22 (relaxed
relevance) and Figure 23 (rigid relevance).

6. Conclusions and Future Works
The NTCIR Workshop 2 is the first international
joint effort in providing an evaluation mechanism forJapanese and Chinese Text Retrieval. We hope thismechanism could encourage the IR researches in
Eastern Asia, promote the concept of IR evaluation,
provide an opportunity to share the research ideas
and results, investigate the useful techniques for IR
researches, develop the effective method inconstruction of test collection, and enhance theeffectiveness of IR systems.

Through the initial analyses on the submitted runs,some findings are shown as follows. The further anddetailed analyses on particular systems could bereferred to each system report in this workshop.
l Most participating groups apply inverted fileapproach for index structure.
l Many participating groups adopt tf/idf-based
approaches.
l “Short Query” and “Very Short Query” showmuch better performance.
l Query expansion is a good method to increasesystem performance.
l In general, the probabilistic model shows better
performance.
l For CHIR task, stop-word list is a good
resource for enhancing system performance.
l For ECIR task, select-all approach seems to bebetter than other select-X approaches, if nofurther techniques are adopted.
l For ECIR task, MT approach is much better
than dictionary-based approach.
l For ECIR task, word-based indexing approachis better.
We would like to say again that these findings are
drawn from the submitted runs using the “CIRB010”
test collection. They cannot directly apply to othertest collection, since each test collection has its own
characteristics and each language also has its own
characteristics. We have to carry out more detailed
analyses using other test collections.
As mentioned previously, the keywords inconcepts filed of topic provide the crucial
information and make the performance higher thanother IR evaluation forum. We would like to explain
the procedure for keywords preparation. We haveexecuted a pre-test for CIRB010 test collection. As aresult, the positive documents and negative
documents for each topic have been constructed. Wethen analyze these documents and extract the good
keywords for each topic. According our analysis and
Brkly’s experiment, using concepts field as the querywill produce the best performance with comparison
to the other fields. We could say that the concepts
field makes the Chinese IR tasks much easier.
Therefore, we are considering the role of concepts
field.
Since the limitation in staff and the lack ofexperience, we make some mistakes in CHTR tasks.
We all work in university and the teaching load isalways an issue. The time is crucial for us. As aresult, we will plan to cooperate with NII in a more
tight way next year. We would like to apologize formaking any inconvenience this year.
Some important issues are worthy of presenting
in this report.

l We have planed to execute a CLIR task nextyear, which is cross-language in document
collection level. We have discussed this issue
with Dr. Noriko Kando in Taipei.
l The copyright issue of continuing uses
CIRB010 in researches is under negotiation.
The original agreement is that we could use it in
the tasks of NTCIR workshop. We are about tosolve this problem.
l According to experience of TREC, the fields of
topic are changeable year by year. Do we haveto proceed the same experiment?
l Since the high quality of keywords in concepts
field of topic, we will reserve them for
relevance judgment only next year and thenmake Chinese IR tasks more difficult.

Acknowledgments
We would like to thank Ch inatimes, Chinatimes
Commercial, Chinatimes Express, Central Daily
News, China Daily News, and United Daily News fortheir kindly providing test materials. We are gratefulto all pioneers in the area of IR evaluation for theirefforts in paving a smooth way for followers. We
would like to thank the participants for theircontributions and the relevance judges for their hardworking. Special thanks are due to Dr. Noriko Kando
for her substantial help.

References[1] Kando, N. et al. Overview of IR Tasks at the FirstNTCIR Workshop. In Proceedings of the FirstNTCIR Workshop on Research in Japanese Text
Retrieval and Term Recognition, pages 1-46,
Tokyo, 1999.
[2] Chiang, Y. -T. A Study on Design and
Implementationfor
ChineseInformation
Retrieval Benchmark, Master Thesis, Departmentof Library and Information Science, National
Taiwan University, 1999.
[3] China Times. http://news.chinatimes.com.tw/[4] Chinatimes Commercial. http://news.chinatimes.
com.tw/[5] Chinatimes Express. http://news.chinatimes.
com.tw/[6] Central Daily News. http://www.cdn.com.tw/[7] China Daily News. http://www.cdns.com.tw/

CHIR Task (Relaxed Relevance) 11-Point Precision
Top Run1.0
0.9

Brkly-CHIR-LO-01
CRL-CHIR-LO-06
CRL-CHIR-LO-14
CRL-CHIR-SO-02
CRL-CHIR-SO-03
PIRCS-CHIR-SO
CRL-CHIR-VS-01
CRL-CHIR-VS-03
PIRCS-CHIR-VS

0.8

Precision

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 4. CHIR Task 11-Point Precision (Relaxed Relevance)

CHIR Task (Rigid Relevance) 11-Point Precision
Top Run1.0

Precision

0.9

Brkly-CHIR-LO-01

0.8

CRL-CHIR-LO-06

0.7

CRL-CHIR-LO-14

0.6

CRL-CHIR-SO-02

0.5

CRL-CHIR-SO-03
PIRCS-CHIR-SO

0.4

CRL-CHIR-VS-02

0.3

CRL-CHIR-VS-03
0.2

PIRCS-CHIR-VS

0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 5. CHIR Task 11-Point Precision (Rigid Relevance)

CHIR Long Query (Relaxed Relevance) 11-Point Precision(Top Run)
1.0

Precision

0.9
0.8

Brkly-CHIR-LO-01

0.7

CRL-CHIR-LO-06
CRL-CHIR-LO-14

0.6

IOS-CHIR-LO-01

0.5

c

IOS-CHIR-LO-02

0.4

NTHU-CHIR-LO-01
NTHU-CHIR-LO-02

0.3

PIRCS-CHIR-LO

0.2

SSTUT-CHIR-LO-03

0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 6. CHIR Long Query 11-Point Precision (Relaxed Relevance)

CHIR Long Query (Rigid Relevance) 11-Point Precision(Top Run)
1.0
0.9
Brkly-CHIR-LO-01
0.8
CRL-CHIR-LO-06
0.7

Precision

CRL-CHIR-LO-14
0.6
IOS-CHIR-LO-01
0.5
IOS-CHIR-LO-02
0.4

NTHU-CHIR-LO-01

0.3

NTHU-CHIR-LO-02

0.2

PIRCS-CHIR-LO

0.1

SSTUT-CHIR-LO-03

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 7. CHIR Long Query 11-Point Precision (Rigid Relevance)

CHIR IR Short Query (Relaxed Relevance) 11-Point Precision(Top Run)

Presicion

1.0
0.9

CRL-CHIR-SO-02

0.8

CRL-CHIR-SO-03

0.7

FRDC-CHIR-SO-01

0.6

IOS-CHIR-SO-01

0.5

IOS-CHIR-SO-02

0.4

PIRCS-CHIR-SO

0.3

SSTUT-CHIR-SO-01

0.2

SSTUT-CHIR-SO-02

0.1

TRANS-CHIR-SO

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 8. CHIR Short Query 11-Point Precision (Relaxed Relevance)

Precision

CHIR IR Short Query (Rigid Relevance) 11-Point Precision(Top Run)
1.0

CRL-CHIR-SO-02

0.9

CRL-CHIR-SO-03

0.8

FRDC-CHIR-SO-01

0.7

IOS-CHIR-SO-01

0.6

IOS-CHIR-SO-02

0.5

PIRCS-CHIR-SO

0.4

SSTUT-CHIR-SO-01

0.3

SSTUT-CHIR-SO-02

0.2

TRANS-CHIR-SO

0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 9. CHIR Short Query 11-Point Precision (Rigid Relevance)

CHIR IR Title Query (Relaxed Relevance) 11-Point Precision (Top Run)

1.0
0.9

0.8

BRKLY-CHIR-TI-01
0.7

CRL-CHIR-TI-02

Precision

0.6

CRL-CHIR-TI-03

0.5

HKPU1-CHIR-TI-01

0.4

HKPU1-CHIR-TI-02

0.3

PIRCS-CHIR-TI

0.2

IOS-CHIR-TI-01

0.1

IOS-CHIR-TI-02

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 10. CHIR Title Query 11-Point Precision (Relaxed Relevance)

CHIR IR Title Query (Rigid Relevance) 11 Point Precision(Top Run)

Precision

1.0
0.9

BRKLY-CHIR-TI-01

0.8

CRL-CHIR-TI-02

0.7

CRL-CHIR-TI-03

0.6

HKPU1-CHIR-TI-02
HKPU1-CHIR-TI-11

0.5

PIRCS-CHIR-TI
0.4
IOS-CHIR-TI-01
0.3
IOS-CHIR-TI-02
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 11. CHIR Title Query 11-Point Precision (Rigid Relevance)

CHIR Very Short Query (Relaxed Relevance) 11-Point Precision(Top Run)
1.0
0.9
CRL-CHIR-VS-01

0.8

CRL-CHIR-VS-02

Precision

0.7

CRL-CHIR-VS-03

0.6

HKPU1-CHIR-VS-03

0.5
0.4

HKPU1-CHIR-VS-21

0.3

IOS-CHIR-VS-01

0.2

IOS-CHIR-VS-02

0.1

PIRCS-CHIR-VS

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 12. CHIR Very Short Query 11-Point Precision (Relaxed Relevance)

CHIR Very Short Query (Rigid Relevance) 11-Point Precision(Top Run)
1.0
0.9
0.8

Precision

0.7

CRL-CHIR-VS-02

0.6

CRL-CHIR-VS-03

0.5
IOS-CHIR-VS-01
0.4
IOS-CHIR-VS-02

0.3

PIRCS-CHIR-VS

0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 13. CHIR Very Short Query 11-Point Precision (Rigid Relevance)

ECIR Task (Relaxed Relevance)
Top Run
1.0
0.9
0.8

Precision

0.7

SSTUT-ECIR-LO-01

0.6

SSTUT-ECIR-LO-02

0.5

pircs-ECIR-LO
0.4

pircs-ECIR-SO

0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 14. ECIR All Task 11-Point Precision (Relaxed Relevance)

ECIR Task (Rigid Relevance)
Top Run1.0
0.9

Presicion

0.8
0.7

pircs-ECIR-LO

0.6

pircs-ECIR-SO

0.5

SSTUT-ECIR-LO-01

0.4

SSTUT-ECIR-LO-02

0.3

UMD-ECIR-LO-03

0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 15. ECIR Task 11-Point Precision (Rigid Relevance)

ECIR Long Query (Relax Relevance) 11-Point Precision
Top Run1.2

1.0
pircs-ECIR-LO

Precision

0.8
SSTUT-ECIR-LO-01
0.6

UMD-ECIR-LO-03

0.4

0.2

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 16. ECIR Long Query 11-Point Precision (Relaxed Relevance)

ECIR Long Query (Rigid Relevance) 11-Point Precision
Top Run
1.0
0.9
0.8
0.7

Precision

pircs-ECIR-LO
0.6
0.5

SSTUT-ECIR-LO-01

0.4

UMD-ECIR-LO-03

0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 17. ECIR Long Query 11-Point Precision (Rigid Relevance)

ECIR Short Query (Relaxed Relevance)
Top Run
1.0
0.9
0.8

Precision

0.7
pircs-ECIR-SO

0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 18. ECIR Short Query 11-Point Precision (Relaxed Relevance)

ECIR Short Query (Rigid Relevance)
Top Run
1.0
0.9
0.8

Precision

0.7
0.6

pircs-ECIR-SO
0.5
0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 19. ECIR Short Query 11-Point Precision (Rigid Relevance)

ECIR Title Query (Relaxed Relevance)
only one run
1.0
0.9
0.8

Precision

0.7
0.6
0.5

pircs-ECIR-TI

0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 20. ECIR Title Query 11-Point Precision (Relaxed Relevance)

ECIR Title Query (Rigid Relevance)
only one run
1.0
0.9
0.8

Precision

0.7
0.6
pircs-ECIR-TI

0.5
0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 21. ECIR Title Query 11-Point Precision (Rigid Relevance)

ECIR Very Short Query (Relaxed Relevance)
Top Run
1.0
0.9
0.8

Precision

0.7
0.6
0.5

pircs-ECIR-VS

0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 22. ECIR Very Short Query 11-Point Precision (Relaxed Relevance)

ECIR Very Short Query (Rigid Relevance)
1.0
0.9
0.8

Precision

0.7
0.6
0.5

pircs-ECIR-VS

0.4
0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RecallFigure 23. ECIR Very Short Query 11-Point Precision (Rigid Relevance)Table 10. CHIR Task (Relaxed Relevance) Top Runs
RunID
IndexUnit
Brkly-CHIR-LO-01 bi-character
CRL-CHIR-LO-06 mostly
bi-character +charact er
CRL-CHIR-LO-14 mostly
bi-character +character
CRL-CHIR-SO-02 mostly
bi-character +character
CRL-CHIR-SO-03 mostly
bi-character +character
PIRCS-CHIR-SO word+char

PIRCS-CHIR-VS

word+char

CRL-CHIR-VS-01 mostly
bi-character +character
CRL-CHIR-VS-03 mostly
bi-character +characterIndexTechstopwordIndexStru QueryUnit
Inverted bi-character
filleusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +character
dictionary, Zipf-thrhld Inverted word+char
file,
network
dictionary, Zipf-thrhld Inverted word+char
file,
network
using all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +character

IRModellogistic
regression
probabilistic
model (okapi)

Ranking
QueryExpan
tf/idf/dl/ql/cl/cf NO
okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model +
spread-activ
probabilistic
model +
spread-activ
probabilistic
model (okapi)

tf/ ictf

top40doc+
100term

tf/ ictf

top40doc+
100term

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

Table 11. CHIR Task (Rigid Relevance) Top Runs
RunID
IndexUnit
Brkly-CHIR-LO-01 bi-character
CRL-CHIR-LO-06 mostly
bi-character +character
CRL-CHIR-LO-14 mostly
bi-character +character
CRL-CHIR-SO-02 mostly
bi-character +character
CRL-CHIR-SO-03 mostly
bi-character +character
PIRCS-CHIR-SO word+char
PIRCS-CHIR-VS

word+char

CRL-CHIR-VS-02 mostly
bi-character +character
CRL-CHIR-VS-03 mostly
bi-character +characterIndexTechstopwordIndexStru QueryUnit
Inverted bi-character
fileusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +character
dictionary, Zipf-thrhld Inverted word+char
file,
network
dictionary, Zipf-thrhld Inverted word+char
file,
network
using all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +character

IRModellogistic
regression
probabilistic
model (okapi)

Ranking
QueryExpan
tf/idf/dl/ql/cl/cf NO
okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedback

probabilistic
model +
spread-activ
probabilistic
model +
spread-activ
probabilistic
model (okapi)

tf/ ictf

top40doc+
100term

tf/ ictf

top40doc+
100term

okapi weightautomatic
feedback

probabilistic
model (okapi)

okapi weightautomatic
feedbackTable 12. CHIR Long Query (Relaxed Relevance and Rigid Relevance) Top Runs
RunID
IndexUnit
Brkly-CHIR-LO-01 bi-character

IndexTechstopwordIndexStru
Inverted file

CRL-CHIR-LO-06

mostlyusing allinverted filebi-character + characters andcharacterbi-characters
CRL-CHIR-LO-14 mostly
using allinverted filebi-character + characters andcharacterbi-characters
IOS-CHIR-LO-01
2-gram2-gramInverted file
IOS-CHIR-LO-02
word
Stopword+dictio Inverted filenary
NTHU_CHIR_LO_01 bi-wordmorphology
Inverted file

QueryUnitbi-character

IRModelRankinglogistic
tf/idf/dl/ql/cl/cf
regression
mostlyprobabilistic okapi weightbi-character + model (okapi)
charactermostlyprobabilistic okapi weightbi-character + model (okapi)
character2-gram
Vector space tf/idf
word
Vector space tf/idf

QueryExpan
NO

word

tf/idfNo

tf/idfNo

tf/ ictf

top40doc+
100term

NTHU_CHIR_LO_02 bi-wordmorphology

Inverted file

word

PIRCS-CHIR-LO

dictionary,Zipf-thrhld

Inverted
file, network

word+char

as issuffix arraybigramword+char

SSTUT-CHIR-LO-03 bigram

vector spacemodulevector spacemoduleprobabilistic
model +
spread-activ
probabilistic
model

automatic
feedback
automatic
feedbackNoNotf, idf,
Yesburstiness,expansio
n frequency

Table 13. CHIR Short Query (Relaxed Relevance and Rigid Relevance) Top Runs
RunID
CRL-CHIR-SO-02

IndexUnit
IndexTechmostlyusing all characters
bi-character +and bi-characterscharacter
CRL-CHIR-SO-03 mostly
using all characters
bi-character +and bi-characterscharacter
FRDC-CHIR-SO-01 character+bi-gram dictionary
characters

QueryExpan
automatic
feedback

IOS-CHIR-SO-01

2-gramNo

IOS-CHIR-SO-02

word

PIRCS-CHIR-SO

word+char

SSTUT-CHIR-SO-01 bigram
SSTUT-CHIR-SO-02 bigram

Trans-CHIR-SO

bi-wordIndexStru QueryUnit
IRModelRanking
inverted mostly
probabilistic model okapi weightfile
bi-character + (okapi)
characterinverted mostly
probabilistic model okapi weightfile
bi-character + (okapi)
characterInverted word
Vector space Mode tf/idf, word
file
association,
document
length2-gramInverted 2-gram
Vector space
tf/idffile
Stopword+dictionary Inverted word
Vector space
tf/idffile
dictionary,
Inverted word+char
probabilistic model tf/ ictf
Zipf-thrhld
file,+ spread-activ
network
as issuffixbigramprobabilistic model tf, idf,arrayburstiness
as issuffixbigramprobabilistic model tf, idf,arraywith dynamic
burstinessprogrammingNo
inverted phrase
vector space model t f
file

automatic
feedback
QE basedon term
contributeNo
top40doc+
100term
No
No

noTable 14. CHIR Title Query (Relaxed Relevance) Top Runs
RunID
IndexUnit
Brkly-CHIR-T I-01 bi-character

IndexTechstopword

CRL-CHIR-T I-02

mostlybi-character +character
CRL-CHIR-T I-03 mostly
bi-character +character
HKPU1-CHIR-TI-01 Hybrid (Word +Bigram)HKPU1-CHIR-TI-02 Hybrid (Word +Bigram)PIRCS-CHIR-T I
word+charusing all characters inverted fileand bi-characters

Stop word +dictionary
Stop word +dictionary
dictionary,Zipf-thrhld

Inverted File

IOS-CHIR-T I-01
IOS-CHIR-T I-02

2-gramStopword +dictionary

Inverted file
Inverted file

2-gramword

IndexStru
QueryUnit
Inverted file bi-character

using all characters inverted fileand bi-characters

Inverted File
Inverted
file, network

IRModelRanking
QueryExpan
Logistic
tf/idf/dl/ql/cl/cf No
regression
probabilistic okapi weight automatic
model (okapi)
feedback

mostlybi-character +charactermostlyprobabilistic okapi weight automaticbi-character + model (okapi)
feedbackcharacter
Hybrid (Word + Vector Space Tf/idf
No
Bigram)Hybrid (Word + Vector Space Tf/idf + Term No
Bigram)Lengthword+charprobabilistic tf/ ictf
top40doc+
model +
100term
spread-activ
2-gram
Vector space tf/idf
No
word
Vector space tf/idf
NoTable 15. CHIR Title Query (Rigid Relevance) Top Runs
RunID
IndexUnit
Brkly-CHIR-T I-01 bi-character

IndexTechstopwordIndexStru QueryUnit
IRModelInverted bi-character
Logistic
file
regression
CRL-CHIR-T I-02 mostly
using all characters inverted mostly
probabilisticbi-character +and bi-characters
filebi-character + model (okapi)
charactercharacter
CRL-CHIR-T I-03 mostly
using all characters inverted mostly
probabilisticbi-character +and bi-characters
filebi-character + model (okapi)
charactercharacter
HKPU1-CHIR-TI-02 Hybrid (Word + Stop word +Inverted Hybrid (Word + Vector Space
Bigram)dictionary
File
Bigram)HKPU1-CHIR-TI-11 Word
Stop word +Inverted Word
Vector Space
dictionary
File
PIRCS-CHIR-T I
word+chardictionary,
Inverted word+char
probabilistic
Zipf-thrhld
file,model +
networkspread-activ
IOS-CHIR-T I-01
2-gram2-gramInverted 2-gram
Vector space
file
IOS-CHIR-T I-02
word
Stopword+dictionary Inverted word
Vector space
file

Ranking
QueryExpan
tf/idf/dl/ql/cl/cf No
okapi weightautomatic
feedbackokapi weightautomatic
feedback

Tf/idf + Term No
Length
Tf/idfNo
tf/ ictf

top40doc+
100term

tf/idfNo

tf/idfNo

Table 16. CHIR Very Short Query (Relaxed Relevance) Top Runs
RunID
CRL-CHIR-VS-01

IndexUnit
mostlybi-character +character
CRL-CHIR-VS-02
mostlybi-character +character
CRL-CHIR-VS-03
mostlybi-character +character
HKPU1-CHIR-VS-03 Hybrid (Word +Bigram)HKPU1-CHIR-VS-21 Bigram
IOS-CHIR-VS-01

2-gramIOS-CHIR-VS-02

word

PIRCS-CHIR-VS

word+char

RunID
CRL-CHIR-VS-02

IOS-CHIR-VS-01

IndexUnit
mostlybi-character +charactermostlybi-character +character2-gramIOS-CHIR-VS-02

word

PIRCS-CHIR-VS

word+charIndexTech
IndexStru QueryUnit
IRModelusing all characters inverted mostly
probabilistic
and bi-characters
filebi-character + model (okapi)
characterusing all characters inverted mostly
probabilistic
and bi-characters
filebi-character + model (okapi)
characterusing all characters inverted mostly
probabilistic
and bi-characters
filebi-character + model (okapi)
character
Stop word +Inverted Hybrid (Word + Vector Space
dictionary
File
Bigram)Stop word +Inverted Bigram
Vector Space
dictionaryFile
2-gramInverted 2-gram
Vector space
file
Stopword+dictionary Inverted word
Vector space
file
dictionary,
Inverted word+char
probabilistic
Zipf-thrhld
file,model +
networkspread-activ

Ranking
QueryExpan
okapi weight automatic
feedbackokapi weight automatic
feedbackokapi weight automatic
feedback
Tf/idf + Term No
Length
Tf/idfNo
tf/idfNo

tf/idfNo

tf/ ictf

top40doc+
100term

Table 17. CHIR Very Short Query (Rigid Relevance) Top Runs

CRL-CHIR-VS-03

IndexTech
IndexStru QueryUnit
using all characters inverted mostly
and bi-characters
filebi-character +characterusing all characters inverted mostly
and bi-characters
filebi-character +character2-gramInverted 2-gram
file
Stopword+dictionary Inverted word
file
dictionary,
Inverted word+char
Zipf-thrhld
file,network

IRModelprobabilistic
model (okapi)

Ranking
QueryExpan
okapi weight automatic
feedback

probabilistic
model (okapi)

okapi weight automatic
feedback

Vector space

tf/idfNo

Vector space

tf/idfNo

probabilistic
model +
spread-activ

tf/ ictf

top40doc+
100termTable 20. ECIR All Runs
RunID
IndexUnit IndexTech IndexStru QueryUnit
Brkly-ECIR-LO-01 word
stopword+ Inverted word
dictionary file
IOS-ECIR-*
N/A
N/A
N/A
N/A
NTHU-ECIR-LO-01 bi-wordmorphology invertedfile word

IRModel Ranking QueryExpan
TransTechlogistic
tf/idf/dl/
NO
Dictionary-based,
regression ql/cl/cf
select two
N/A
N/A
N/A
N/A
vector space tf/idf
No
dictionary-based,
modulecorpus-based
PIRCS-ECIR-*
word+char dictionary, Inverted word+char probabilistic tf/ ictf
top40doc+ bi-word list + MT
Zipf-thrhld file,model +
100term
softwarenet work
spread-activ+ retrieval
combination
SSTUT-ECIR-LO-01 all n-grams as issuffix array word
probabilistic tf, idf,
No
dictionary-basedmodel
burstinesshandmade
SSTUT -ECIR-LO-02 all n-grams as issuffix array word
probabilistic tf, idf,
No
dictionary-basedmodel with burstinesshandmadedynamic
programming
Trans-ECIR-SO
bi-wordNo
inverted file word
vector space t f
noDictionary-based andmodel
corpus-based,select-top-1
UMD-ECIR-LO-01 overlapping hexicificati inverted file within word probabilistic tf/idf
nodictionary-based,character on of
overlapping model
select-all
bigramChinesecharactercharacters
bigram
UMD-ECIR-LO-02 overlapping Chinese
inverted file within word probabilistic tf/idf
nodictionary-based,character character
overlapping model
select-top-3
bigramhexifying
characterbigram
UMD-ECIR-LO-03 word
Chineseinverted file word
probabilistic tf/idf
nodictionary-based,charactermodel
select-top-3
hexifying