Proceedings of the Third NTCIR Workshop

Asian Language Parsing Evaluated by
Hummingbird SearchServerTM at NTCIR-3
Stephen Tomlinson
Hummingbird
Ottawa, Ontario, Canada
stephen.tomlinson@hummingbird.com
November 24, 2002

Abstract
Hummingbird submitted ranked result sets for the
Chinese, Japanese and Korean Single Language Information Retrieval tracks of the Cross-Language Retrieval Task of the 3rd NII-NACSIS Test Collection for
IR Systems Workshop (NTCIR-3). SearchServer 5.3’s
segmenter for Asian text, compared to an overlapping n-gram approach, was found to modestly increase precision scores for Japanese, to have a neutral impact for Chinese, and to be detrimental for Korean. SearchServer’s option to case normalize Hiragana and Katakana n-grams increased precision substantially for one Japanese query and was of neutral
impact for the others. Newline suppression was foundto be of only minor benefit for n-gram parsing. Normalizing Han characters to Hangul had almost no effect on the Korean test collection. Keywords: Hummingbird, SearchServer, NTCIR, Chinese, Japanese,
Korean, Asian, parsing, n-grams, segmenting.

2

Setup

The experiments described in this paper were conducted in August, 2002 with an internal developmentbuild of SearchServer 5.3 (5.3.500.279).

1 Introduction
Hummingbird SearchServer1 is an indexing, searchand retrieval engine for embedding in Windows andUNIX information applications. SearchServer, originally a product of Fulcrum Technologies, was acquiredby Hummingbird in 1999. Founded in 1983 in Ottawa, Canada, Fulcrum produced the first commercial
application program interface (API) for writing inforR Ful/TextTM .
mation retrieval applications, FulcrumThe SearchServer kernel is embedded in many Hummingbird products, including SearchServer, an application toolkit used for knowledge-intensive applications that require fast access to unstructured information.
1 Fulcrum
R

SearchServer supports a variation of the Structured
Query Language (SQL), called SearchSQLTM , which
has extensions for text retrieval. SearchServer conforms to subsets of the Open Database Connectivity(ODBC) interface for C programming language applications and the Java Database Connectivity (JDBC) interface for Java applications. Almost 200 document
formats are supported, such as Word, WordPerfect,
Excel, PowerPoint, PDF and HTML.
SearchServer works in Unicode internally [2] andsupports most of the world’s major character sets andlanguages. The major conferences in text retrieval
evaluation (NTCIR [4], CLEF [1] and TREC [7]) haveprovided opportunities to objectively evaluate SearchServer’s support for a dozen languages. This paper
focuses on evaluating SearchServer’s parsing options
for Chinese, Japanese and Korean, using the NTCIR-3
Formal Test Collections.

SearchServerTM ,is a registered trademark, andSearchSQLTM , Intuitive SearchingTM and Ful/TextTM are trademarks of Hummingbird Ltd. All other copyrights, trademarks andtradenames are the property of their respective owners.

© 2003 National Institute of Informatics

2.1

Data

The document sets of the NTCIR-3 Formal Test
Collections (Cross-Language Retrieval task) consistof tagged (SGML-formatted) news articles from the1990’s in Chinese, Japanese and Korean. Table 1 gives
their sizes. For more information, see the NTCIR web
site [4].

2.2

Indexing

A separate SearchServer table was created for eachlanguage and parsing option with a SearchSQL statement such as the following:

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Table 1. Sizes of NTCIR-3 Document Sets
Language
Text Size#Documents
ChineseJapanese
Korean555,285,156 bytes
301,019,356 bytes
77,510,533 bytes

381,681236,664
66,146n-grams (usually bigrams). SearchServer’s unicode
parser includes Asian specific enhancements such as
normalizing old forms of Kanji to new at index andsearch-time (this normalization is not done at fetch
time so that the original form can be viewed). Asian
language options of SearchServer’s unicode parser include the following:• Case normalization of Hiragana and Katakana:
By default (/c=0), SearchServer treats big and little Hiragana and Katakana characters as different.
Specifying /c=1 treats them the same (like upper and lower case Latin characters are normallytreated the same). This option is likely to make adifference only for Japanese text.

CREATE SCHEMA NTC3J
CREATE TABLE NTC3J(DOCNO VARCHAR(256) 128)
STOPFILE ’japancase.stp’
CONSTRUCT_STEMS ’FALSE’PERIODICBASEPATH ’e:\data’;• Normalizing Han characters to Hangul: By default (/k=0), SearchServer does not map Han
characters to Hangul. Specifying /k=1 enables
this mapping. This option is only recommended
for Korean.

The STOPFILE parameter names a file which specifies any non-default parsing options (described below) to apply at index and search-time; the file may
also contain stop words, but none were used any forthese tasks. The CONSTRUCT STEMS parameter
was set to ’FALSE’ to disable stemming of any Latin
terms that might have appeared in the text.
Into each table, one row was inserted, specifying
the top directory of the library files for the language,
and the text reader list. For example:
INSERT INTO NTC3J(FT_SFNAME, FT_FLIST) VALUES(’NTCIR3\Japanese’,’cTREC/E/d=128:s!
nti/t=EUC_JP_UCS2:cTREC/p/@:s’);The text reader list included the nti (translation)text reader with the appropriate /t option to translate the documents to Unicode format: BIG5 UCS2
for Chinese, EUC JP UCS2 for Japanese, and
KSC 5601 1992 UCS2 for Korean. The custom text
reader called cTREC, originally written for handling
TREC document sets [8], handled expansion of the library files of the NTCIR document sets. The cTREC/p option passed through all of the document content
unaltered (including the SGML tags, which preferably
wouldn’t be indexed, but it seemed unlikely to matterfor this task).
To index each table, a Validate Index statement suchas the following was run:
VALIDATE INDEX NTC3JVALIDATE TABLE;

2.3 Parsing Options
SearchServer 5.3 supports two approaches to indexing Asian text: overlapping n-grams and segmentation
into words.
By default, SearchServer uses a parser named “unicode” which indexes Asian text using overlapping• Newline suppression: By default (/n=0), SearchServer deletes a newline character that follows
an Asian character during indexing, and blanks
following a deleted new line character are alsodeleted. Specifying /n=1 disables this feature.
SearchServer 5.3 also includes a parser named“ixasian” which segments Chinese, Japanese and Korean text into words, using Inxight LinguistX Platform
3.3.1. It has a /l option to specify the language forwhich to optimize.
As mentioned earlier, the parser and options arecontrolled by the PARSER line in the stopfile. For example, the line
PARSER="unicode/c=1"
specifies the unicode parser with case normalizationof Hiragana and Katakana. Table 2 lists the parsing
options evaluated in this paper.

3

Search Techniques

The NTCIR organizers created several “topics”: 50
for Chinese and Japanese (which were translations ofeach other) and 30 for Korean (the Korean news articles covered a different time period than the Chineseand Japanese articles). Each topic contained a “Title”(subject of the topic), “Description” (a specification of
the information need, typically one sentence), “Narrative” (more detailed guidelines for what a relevant
document should or should not contain, often several
sentences), and “Concepts” (typically a list of nouns
or noun phrases separated by commas). The participants were asked to use just the Description field forat least one automatic submission per track to facilitatecomparison of results.

Proceedings of the Third NTCIR Workshop

An ODBC application called QueryToRankings.c
was created (based on the example stsample.c program included with SearchServer) to parse the NTCIR topics files, construct and execute corresponding
SearchSQL queries, fetch the top 1000 rows, and writeout the rows in the results in the requested submission format. SELECT statements were issued with
the SQLExecDirect api call. Fetches were done with
SQLFetch (typically 1000 SQLFetch calls per query).

3.1 Intuitive Searching
For all runs, SearchServer’s Intuitive Searching wasused, i.e. the IS ABOUT predicate of SearchSQL,
which accepts unstructured text. For example, if the
Title for a topic was “textbook issues in Japan”, then acorresponding SearchSQL query would be:
SELECT RELEVANCE(’V2:3’) AS REL,DOCNOFROM NTC3J
WHERE FT_TEXT IS_ABOUT ’textbook
issues in Japan’
ORDER BY REL DESC;
This query would create a working table with the 2columns named in the SELECT clause, a REL column
containing the relevance value of the row for the query,and a DOCNO column containing the document’s
identifier. The ORDER BY clause specifies that themost relevant rows should be listed first. The statement “SET MAX SEARCH ROWS 1000” was previously executed so that the working table would containat most 1000 rows. Also, the statement “SET VECTOR GENERATOR ’’” was previously executed todisable inflection of any Latin text that might have appeared in the query.
Of course, the actual topics (for the Single Language tracks) were in Chinese, Japanese or Korean,
not English. So before running the SearchSQL statements, the appropriate “SET CHARACTER SET” option was specified (’BIG5’ for Chinese, ’EUC JP’ forJapanese, ’KSC 5601 1992’ for Korean), so that thequery text inside the IS ABOUT predicate would be
converted to Unicode correctly.
The same parser and options are applied to thequery text as are applied to the text at index-time,
though at search-time some optimizations may bemade. For example, in a CONTAINS search (exact
phrase matching), when n-grams are used, a minimal
set of n-grams may be used for more efficient searching. Before version 5.3, SearchServer also used a minimal set of n-grams for the IS ABOUT predicate, but
as of version 5.3, overlapping n-grams are used with
the IS ABOUT predicate. The official submissions,even though they were made before SearchServer 5.3
was released, included this change. The impact of thischange on the NTCIR-3 collections is shown below.

3.2

Statistical Relevance Ranking

SearchServer calculates a relevance value for a rowof a table with respect to a vector of words or n-grams
based on several statistics. The inverse document frequency of the word or n-gram is estimated from information in the dictionary. The term frequency (number of occurrences of the word or n-gram in the row)is determined from the reference file. The length of
the row (based on the number of indexed charactersin all columns of the row, which is typically dominated by the external document), is optionally incorporated. The count of the word or n-gram in the vector is also used. To synthesize this information into a
relevance value, SearchServer dampens the term frequency and adjusts for document length in a manner
similar to Okapi [5] and dampens the inverse document frequency in a manner similar to [6]. SearchServer’s relevance values are always an integer in therange 0 to 1000.
SearchServer’s RELEVANCE METHOD setting
can be used to optionally square the importance of
the inverse document frequency (by choosing a RELEVANCE METHOD of ’V2:4’ instead of ’V2:3’). The
importance of document length to the ranking is controlled by SearchServer’s RELEVANCE DLEN IMP
setting (scale of 0 to 1000). For the experiments described in this paper, RELEVANCE METHOD was
set to ’V2:4’ and RELEVANCE DLEN IMP was set
to 500. Different settings were used for some of theofficial submissions, as described later.

4

Results

The Single Language Information Retrieval tracksof the Cross-Language Retrieval Task were to run thegiven Chinese, Japanese and Korean queries against
document collections in the same language and submit a list of the top-1000 ranked documents to NII for
judging (in February, 2002). NII produced a list of relevance assessments for each track: a list of documents
judged to be highly relevant, relevant, partially relevant or not relevant for each topic. NII also produced 2“qrels” files for each track in a format compatible with
Chris Buckley’s trec eval program, from which theevaluation measures can be calculated. One qrels filejust counted highly relevant and relevant documents asrelevant (referred to by the organizers as “rigid” mode)and the other qrels file additionally counted partiallyrelevant documents as relevant (“relaxed” mode).
The evaluation measures are expected to be explained in an appendix of this volume. Briefly: “Precision” is the percentage of retrieved documents which
are relevant. “Precision@n” is the precision aftern documents have been retrieved. “Average precision” for a topic is the average of the precision after each relevant document is retrieved (using zero as

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

the precision for relevant documents which are not retrieved). “Recall” is the percentage of relevant documents which have been retrieved. For a set of topics, the measure is the average of the measure for each
topic (i.e. all topics are weighted equally).
For topics and languages in which fewer than 3 documents were judged relevant or highly relevant, the organizers discarded the topic for that language. This
critera caused 8 of the 50 topics to be discarded for
Chinese and Japanese, hence the resulting Formal Test
Collections for those languages just contain 42 topics.
No topics were discarded for Korean, which remains a
30-topic collection.

4.1 Impact of Parsing Options
Table 2 shows the SearchServer 5.3 parsing options
evaluated in this paper and the label assigned to each.
The first letter of each label indicates the language(C=Chinese, J=Japanese, K=Korean), and is followedby an indicator of the experiment:• “Base” indicates the baseline run for each language. The unicode parser (n-grams) is used
for the baseline runs. For Chinese, the baseline
uses the default options. For Japanese, the baseline specifies /c=1 (case normalization of Hiragana and Katakana). For Korean, the baseline
specifies /k=1 (normalization of Han characters
to Hangul).
• “Line” indicates that newline suppression is disabled (/n=1).
• “Case” (Japanese only) indicates that case normalization of Hiragana and Katakana is disabled(i.e. /c=1 is not specified).
• “Han” (Korean only) indicates that normalizationof Han characters to Hangul is disabled (i.e. /k=1
is not specified).
• “Seg” indicates that the ixasian parser (which
segments Asian text into words) is used.
3 runs were done for each parsing option, distinguished by the 3rd part of each label:• “T” indicates just the Title field of each topic wasused• “D” indicates just the Description field was used
• “C” indicates just the Concepts field was used.
The 3 runs let us check if the parsing options havedifferent impacts on short queries, descriptive queries,
and keyword lists.
For each run, scores are shown for both qrels files.
The 4th part of the label is “p” when “partially” relevant documents are counted as relevant.

Table 2. Labels for Parsing Options
Label
Parser and Options
C-BaseC-LineC-Seg

unicodeunicode/n=1
ixasian/l=traditional-chinese

J-BaseJ-CaseJ-LineJ-Seg

unicode/c=1
unicodeunicode/c=1/n=1
ixasian/l=japanese

K-Base
K-Han
K-LineK-Seg

unicode/k=1
unicodeunicode/k=1/n=1
ixasian/l=korean

Table 3 shows the precision scores for the baseline
runs (“AvgP” for Average Precision, “P@10” for Precision at 10 documents).
For the other parsing options, instead of showing
their absolute scores, the difference in their scoresfrom the baseline run in the same language is shown.
The “AvgDiff” column is the difference in the precision score; a positive difference means that the run
had a higher score than the baseline by the difference
shown. The “vs. Base” column shows the number of
topics on which the precision score was higher, lower
and tied with the baseline’s score, respectively (thesenumbers should always add to 42 for the Chinese andJapanese runs, and add to 30 for the Korean runs).
Table 4 shows the impact of disabling newline suppression on the average precision measure, ordered bydecreasing difference in average precision. On thesetest collections at least, the difference is always minor,
though it appears if anything that disabling this feature
has a slight negative impact (i.e. it’s better to stick withthe default and not specify /n=1).
Table 5 shows the impact of disabling case normalization of Hiragana and Katakana (Japanese only) on
the average precision measure, ordered by decreasing difference in average precision. The “vs. Base”
column shows that for the majority of topics there
is no difference, which might mean that the majority
of topics do not contain Hiragana or Katakana. For
the remaining topics, the differences on average arenegative, indicating that case normalization of Hiragana and Katakana may be modestly beneficial on average. This is mostly the result of one topic (topic
14, “computer virus”), for which case normalization
helped substantially. However, no topic was substantially hurt by case normalization.
While case normalization should increase recall(more documents match), it could decrease early precision (if the original case was meaningful). Table 6
shows the impact of disabling case normalization ofProceedings of the Third NTCIR Workshop

Table 3. Precision of Baseline Runs
Run
AvgP
P@10
C-Base-T
C-Base-T-p
C-Base-D
C-Base-D-p
C-Base-C
C-Base-C-p
J-Base-T
J-Base-T-p
J-Base-D
J-Base-D-p
J-Base-C
J-Base-C-p

0.1986
0.23930.18600.22570.23920.27840.29770.36000.28450.32640.31070.35130.26430.36430.27620.38330.3286
0.43330.35000.46900.34520.45710.36670.4738

K-Base-T
K-Base-T-p
K-Base-D
K-Base-D-p
K-Base-C
K-Base-C-p

0.28630.31760.20620.24370.27770.33130.33330.43670.27330.38670.32000.4733

Table 4. Impact of Disabling Newline Suppression on Average Precision
Experiment
AvgDiff
vs. Base
J-Line-T-p
J-Line-T
C-Line-C-p
K-Line-T-p
J-Line-D
J-Line-D-p
J-Line-C
J-Line-C-p
C-Line-T-p
K-Line-T
C-Line-C
C-Line-T
K-Line-C
C-Line-D-p
K-Line-C-p
C-Line-D
K-Line-D-p
K-Line-D

0.00070.00060.00030.0003−0.0000−0.0000−0.0000−0.0001−0.0002−0.0004−0.0005−0.0007−0.0007−0.0011−0.0013−0.0017−0.0018−0.00194-1-373-1-3818-18-6
11-16-3
3-4-35
7-3-325-4-334-5-33
16-19-7
14-11-5
19-15-8
20-13-9
16-10-4
21-19-2
14-15-1
18-19-5
11-16-3
12-14-4

Table 5. Impact of Disabling Case Normalization of Hiragana and Katakana on
Average Precision
Experiment
AvgDiff
vs. Base
J-Case-C-p
J-Case-T
J-Case-T-p
J-Case-C
J-Case-D-p
J-Case-D−0.0057−0.0068−0.0070−0.0092−0.0142−0.0143

8-7-274-1-374-1-378-6-284-5-332-6-34

Table 6. Impact of Disabling Case Normalization of Hiragana and Katakana on
Precision@10
Experiment
AvgDiff
vs. Base
J-Case-C-p
J-Case-C
J-Case-T
J-Case-T-p
J-Case-D
J-Case-D-p

0.0000−0.0048−0.0095−0.0095−0.0190−0.02142-1-39
1-1-400-1-410-1-410-3-39
1-3-38Hiragana and Katakana on the Precision@10 measure.
It turns out Precision@10 is also negatively impacted
by disabling this option (again, primarily because of
topic 14).
On the whole, while it usually makes little difference, it appears there may be an occasional substantial
benefit to enabling the /c=1 option for Japanese. However, the above results for average precision and Precision@10 are not statistically significant (at the 5%
level by the two-sided Wilcoxon signed rank test), soa larger experiment (more topics) would be needed tobe more confident of this result.
Table 7 shows the impact of disabling normalization of Han characters to Hangul (Korean only). The“vs. Base” column shows that the scores for all of the
topics are the same (except one, the Concepts field of
topic 20, and even the difference for it was slight).
This result probably means the Korean documents andtopics primarily use Hangul characters and seldom use
Han characters.
Table 8 shows the impact of using the segmenterfor each language (instead of n-grams) on the average
precision measure. The segmenter appears to be modestly beneficial for Japanese, neutral for Chinese, anddetrimental for Korean.
Table 9 shows the impact of the segmenter on thePrecision@10 measure. Compared to the result for
average precision, the beneficial impact for Japanese
seems a little higher, the impact for Chinese seems

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Table 9. Impact of Segmenting on Precision@10
Experiment
AvgDiff
vs. Base
Table 7. Impact of Disabling Normalization of Han Characters to Hangul on Average Precision
Experiment
AvgDiff
vs. Base
K-Han-T
K-Han-T-p
K-Han-D
K-Han-D-p
K-Han-C
K-Han-C-p

0.00000.00000.00000.00000.00000.00000-0-300-0-300-0-300-0-301-0-29
1-0-29

Table 8. Impact of Segmenting on Average Precision
Experiment
AvgDiff
vs. Base
J-Seg-D-p
J-Seg-CJ-Seg-C-p
J-Seg-T
J-Seg-T-p
J-Seg-D

0.02850.02030.01750.01750.01040.010228-14-0
26-15-1
27-15-0
23-18-1
23-19-0
25-17-0

C-Seg-T-p
C-Seg-D-p
C-Seg-T
C-Seg-D
C-Seg-C-p
C-Seg-CK-Seg-D
K-Seg-D-p
K-Seg-CK-Seg-T-p
K-Seg-C-p
K-Seg-T

0.01590.0037−0.0026−0.0107−0.0291−0.0390−0.0444−0.0603−0.0935−0.1087−0.1149−0.1162

23-19-0
21-21-0
20-22-0
18-24-0
19-23-0
14-28-0
8-22-07-23-05-24-1
5-25-03-27-05-25-0J-Seg-C-p
J-Seg-D-p
J-Seg-DJ-Seg-CJ-Seg-T-p
J-Seg-T
C-Seg-T-p
C-Seg-T
C-Seg-D-p
C-Seg-D
C-Seg-C-p
C-Seg-C

0.05240.05240.04520.04050.03810.02620.0143−0.0000−0.0024−0.0238−0.0524−0.0857

16-9-17
15-10-17
18-5-19
15-10-17
14-9-19
13-8-21
14-13-15
11-10-21
14-18-10
14-17-11
15-17-10
9-22-11

K-Seg-T-p
K-Seg-D-p
K-Seg-D
K-Seg-C-p
K-Seg-CK-Seg-T

0.0000−0.0100−0.0433−0.0467−0.0567−0.0733

10-13-7
13-13-4
8-11-118-11-118-12-104-14-12more detrimental, and the impact for Korean less detrimental.
Table 10 shows estimators2 of the impact of segmenting on the average precision measure and 95%
confidence intervals3 for the estimators. For Japaneseand Chinese, 5 of the 6 results are not found to be statistically significant at the 5% level, and even in theremaining case, one end of the confidence interval isvery close to zero. For Korean, all 6 results are statistically significant at the 5% level (in fact, 5 of the 6 are
statistically significant at the 1% level), and in mostcases, the end of the interval closest to zero would still
represent a noticeable impact.

4.2

Impact of SearchServer 5.3 Change

As mentioned earlier, for the IS ABOUT predicate,
as of version 5.3 the unicode parser produces overlapping n-grams; previous versions, including version
5.0, used a minimal set of n-grams. For example, if thequery was ABCD (where the letters represent Asian
2 The estimator is the Walsh average [3] which, when subtracted
from the scores of the segmenter run, maximizes the significancelevel by the two-sided Wilcoxon signed rank test (for all of our experiments, there was just one such value). This estimator is usually
the same as the Hodges-Lehmann Estimator [3] and is less sensitive
to outliers than the average difference.
3 The listed 95% confidence intervals are derived from the rangeof values for which, when subtracted from the scores of the segmenter run, the significance level by the two-sided Wilcoxon signed
rank test is at least 5%. The listed boundary points might not need to
be in the interval; the values are rounded so as to enlarge the interval
when necessary to ensure the listed interval covers the minimal one.

Proceedings of the Third NTCIR Workshop

Table 10. Confidence Intervals for Impact
of Segmenting on Average Precision
Experiment
EstDiff
95% Confidence Int.
J-Seg-D-p
J-Seg-C-p
J-Seg-DJ-Seg-CJ-Seg-T
J-Seg-T-p
C-Seg-T-p
C-Seg-D-p
C-Seg-T
C-Seg-D
C-Seg-C-p
C-Seg-C

0.03880.02560.02540.01960.01490.01440.00550.00440.0024−0.0084−0.0180−0.0280[ 0.0058, 0.0658 ][−0.0005, 0.0495 ][−0.0069, 0.0514 ][−0.0079, 0.0483 ][−0.0049, 0.0438 ][−0.0083, 0.0411 ][−0.0129, 0.0494 ][−0.0222, 0.0346 ][−0.0156, 0.0275 ][−0.0384, 0.0160 ][−0.0519, 0.0067 ][−0.0606,−0.0047 ]K-Seg-D
K-Seg-D-p
K-Seg-CK-Seg-T
K-Seg-T-p
K-Seg-C-p−0.0416−0.0612−0.0919−0.0958−0.1093−0.1111[−0.0781,−0.0073 ][−0.1146,−0.0145 ][−0.1390,−0.0322 ][−0.1536,−0.0285 ][−0.1658,−0.0303 ][−0.1736,−0.0460 ]characters), previous versions would likely have justcontained AB and CD in the internal query vector,
whereas SearchServer 5.3 would likely contain AB,
BC and CD. This change substantially increased thescores in internal experiments on the NTCIR-2 Chinese test collection, with little impact on the scoreson the NTCIR-1 Japanese test collection. How thischange affects the scores on the NTCIR-3 collections
is now checked.
Tables 11 and 12 show the impact of going backto minimal n-grams (SearchServer 5.0) on the average
precision and Precision@10 measures, respectively.
Negative differences would suggest that the change
made for SearchServer 5.3 was beneficial. The tables
suggest that the change was modestly beneficial for the
Description-only queries in Chinese and Korean, but
was modestly detrimental in most other cases, thoughagain less so for Description-only queries in Japanesethan the other query types.
Table 13 shows estimators of the impact of goingback to minimal n-grams (SearchServer 5.0) on the average precision measure and 95% confidence intervals
for the estimators (based on the two-sided Wilcoxon
signed rank test as described in the earlier footnotes).
Generally speaking, the results are consistent with theaverage differences shown earlier. While some of theresults are statistically significant at the 5% level (theconfidence interval does not contain zero), none of theestimators are large (at most a couple points).

Table 11. Impact of Minimal N-grams on
Average Precision
Experiment
AvgDiff
vs. Base
J-SS5-CJ-SS5-T
J-SS5-C-p
J-SS5-T-p
J-SS5-D-p
J-SS5-D
K-SS5-C
K-SS5-C-p
K-SS5-T-p
K-SS5-T
K-SS5-D
K-SS5-D-p

0.02560.01900.01800.01730.00720.00080.02730.02300.01850.0120−0.0176−0.041723-12-7
14-10-18
24-11-7
14-9-19
29-11-2
27-13-2
18-6-6
20-5-5
11-6-13
10-7-13
9-18-3
9-18-3

C-SS5-T-p
C-SS5-T
C-SS5-C-p
C-SS5-C
C-SS5-D
C-SS5-D-p

0.01910.01280.0052−0.0040−0.0276−0.0277

23-17-2
22-18-2
20-18-4
18-21-3
12-30-0
15-27-0

Table 12. Impact of Minimal N-grams on
Precision@10
Experiment
AvgDiff
vs. Base
J-SS5-C-p
J-SS5-CJ-SS5-T
J-SS5-T-p
J-SS5-D-p
J-SS5-D
K-SS5-T-p
K-SS5-C
K-SS5-C-p
K-SS5-T
K-SS5-D
K-SS5-D-p

0.03570.03100.02140.02140.02140.01670.04000.03670.03330.0133−0.0467−0.0567

10-6-26
8-4-30
7-6-29
6-7-2912-7-23
12-7-23
4-4-229-2-199-5-16
4-3-23
5-9-16
7-9-14

C-SS5-C-p
C-SS5-T
C-SS5-T-p
C-SS5-C
C-SS5-D
C-SS5-D-p

0.00710.0048−0.0024−0.0071−0.0310−0.0452

15-10-17
9-9-248-12-2210-10-22
8-17-17
7-17-18

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Table 13. Confidence Intervals for Impactof Minimal N-grams on Average Precision
Experiment
EstDiff
95% Confidence Int.
J-SS5-C-p
J-SS5-D-p
J-SS5-CJ-SS5-DJ-SS5-T
J-SS5-T-p
K-SS5-C
K-SS5-C-p
K-SS5-T-p
K-SS5-T
K-SS5-D
K-SS5-D-p

0.01500.01360.01130.00990.00030.00030.01710.01600.00070.0001−0.0030−0.0189[−0.0003, 0.0338 ][ 0.0013, 0.0338 ][ 0.0000, 0.0333 ][−0.0050, 0.0253 ][−0.0001, 0.0213 ][−0.0020, 0.0216 ][ 0.0086, 0.0315 ][ 0.0044, 0.0283 ][−0.0002, 0.0080 ][−0.0005, 0.0161 ][−0.0233, 0.0052 ][−0.0591,−0.0012 ]C-SS5-T-p
C-SS5-T
C-SS5-C-p
C-SS5-C
C-SS5-D-p
C-SS5-D

0.00490.00340.0020−0.0014−0.0208−0.0216[−0.0086, 0.0189 ][−0.0080, 0.0194 ][−0.0084, 0.0150 ][−0.0095, 0.0088 ][−0.0481,−0.0019 ][−0.0434,−0.0036 ]4.3 Official Submissions
The official submissions in February, 2002 used anolder, experimental version of SearchServer, which
for Asian languages would give similar rankings as
the subsequent commercial release version of SearchServer 5.3. All of the official submissions used n-gramparsing.
For the 3 Description-only submissions, the options used were the same as for the baseline Donly runs of Table 3, including the same parser
options (“unicode” for Chinese, “unicode/c=1” forJapanese, “unicode/k=1” for Korean), same RELEVANCE METHOD (’V2:4’) and same RELEVANCE DLEN IMP (500). The official scores were
almost the same as those listed in the table.
For the 3 Title+Concepts submissions, RELEVANCE METHOD was set to ’V2:3’ and RELEVANCE DLEN IMP was set to 0. Also, for theJapanese run, /c=1 (case normalization of Hiragana
and Katakana) was not specified.
For the 3 full topic submissions, RELEVANCE METHOD was set to ’V2:3’ and RELEVANCE DLEN IMP was set to 1000. Again, for theJapanese run, /c=1 (case normalization of Hiragana
and Katakana) was not specified.
No query expansion techniques (such as blind feedback) were applied for any of the submissions (or anyof the other experiments described in this paper). Allof the query text was used; no attempt was made to

identify and discard Asian text corresponding to common instruction words (e.g. “Find relevant documents
about”), which usually increases the scores a little.
3 monolingual English runs were donated for the
benefit of the cross-language judging pools. Generally, the same topic fields and parameters were used
as for the Asian runs. However, the “unicode” parser
indexes Latin text using words, not n-grams. English
stemming was not enabled (an oversight). Unlike for
the Asian runs, stop words (e.g. “the”, “by”) were notindexed, and common instruction words based on past
TREC topics (e.g. “find”, “relevant”, “documents”)
were discarded. The English runs are ignored elsewhere in this paper.
The scores for the official submissions are expected
to be listed in an appendix of the proceedings.

References[1] Cross-Language Evaluation Forum (CLEF) web
site. http://www.clef-campaign.org/[2] Andrew Hodgson. Converting the Fulcrum SearchEngine to Unicode. In Sixteenth International Unicode Conference, Amsterdam, The Netherlands,
March 2000.
[3] Myles Hollander and Douglas A. Wolfe. Nonparametric Statistical Methods. Second Edition, 1999.
John Wiley & Sons.
[4] NTCIR(NII-NACSIS
Test
CollectionforIR
Systems)
Home
Page.
http://research.nii.ac.jp/∼ntcadm/index-en.html[5] S. E. Robertson, S. Walker, S. Jones, M. M.
Hancock-Beaulieu, M. Gatford. (City University.) Okapi at TREC-3. In D. K. Harman, editor, Overview of the Third Text REtrieval Conference (TREC-3). NIST Special Publication 500-226.
http://trec.nist.gov/pubs/trec3/t3 proceedings.html[6] Amit Singhal, John Choi, Donald Hindle, David
Lewis and Fernando Pereira. AT&T at TREC-7. In
E. M. Voorhees and D. K. Harman, editors, Proceedings of the Seventh Text REtrieval Conference (TREC-7). NIST Special Publication 500-242.
http://trec.nist.gov/pubs/trec7/t7 proceedings.html[7] Text REtrieval Conference (TREC) Home Page.
http://trec.nist.gov/[8] Stephen Tomlinson and Tom Blackwell. Hummingbird’s Fulcrum SearchServer at TREC-9. In
E. M. Voorhees and D. K. Harman, editors, Proceedings of the Ninth Text REtrieval Conference (TREC-9). NIST Special Publication 500-249.
http://trec.nist.gov/pubs/trec9/t9 proceedings.html