NTCIR-2 Chinese, Cross Language Retrieval Experiments Using PIRCS
K. L. Kwok
Computer Science Department, Queens College,
City University of New York, Flushing, NY 11367, USAemail: kwok@ir.cs.qc.edu

Abstract
We participated in the monolingual Chinese andEnglish-Chinese cross language retrieval track using
our PIRCS retrieval system. Employing the querytranslation approach for crosslingual IR, two
methods of translation were tried: MT software, anddictionary lookup followed with disambiguation
techniques. Retrieval lists from the two methods
were combined to form the final result. Pseudorelevance feedback was used, but no pre-translation
expansion or collection enrichment was employed.
Runs for all query lengths were submitted. Shortword with character representation was used for both
documents and queries. Using the ‘rigid’ criteria forevaluation, both VS (very short) and SO (short)queries gave monolingual mean average precision atover 0.6. LO (long) query type surprisingly wasworse by about 5% at over 0.57. TI queries (title onlyof a few words) returned a good 0.46 mean average
precision. Cross language retrievals perform at
between 77 to 83% of monolingual for the longer
query types, and only at 55% for the TI queries.
A prominent factor in cross language retrieval
failure is unknown word (such as proper noun)
translation. This is particularly acute with TI queriesof a few words. We show that it may be overcome tosome extent by using longer queries that can provide
more redundancy and better context for translations
to hedge for errors. Crosslingual IR can also be
efficiently performed, often with improved results, by
mixting both translations as one single query.
Keywords: Query Translation and Disambiguation;
Bilingual Wordlists; MT software; CLIR

1 Introduction
Cross language information retrieval (CLIR) is
important because it provides the ability for people of
one culture to filter and retrieve stored information ofanother. This is rendered even more useful becauseof the ease and convenience of access and delivery of
foreign documents provided by Internet logistics.
Accessing Chinese documents via English is a major
sub-problem within CLIR because so many people in
the world use these two languages, the fact that

Chinese is so different from English, and that the
Chinese language is perceived to be quite difficult forforeigners to learn. This paper describes the methods
and procedures we took in participating in thisNTCIR-2 Chinese and cross language retrieval track.
Our PIRCS system was used for retrieval, and we
employed the query translation approach for CLIR.
Results for all query sizes were submitted, viz.: LO(long - all sections in a topic were used), SO (short title, query and concepts sections), VS (very short title and concepts) and TI (title section only). Section
2 describes our procedures for translating the English
queries into Chinese with disambiguation; Section 3summarizes our PIRCS system and retrieval
methodology; Section 4 discusses our submitted
results; Section 5 describes some additional
experiments with other query types and methods, andSection 6 contains our conclusions.

2 Query Translation and Disambiguation
Although several approaches are available totackle CLIR [1,2] we believe, as many others do, thatquery translation is often efficient and easier toimplement, and may be just as or more effective.
Within the query translation approach, one could use
more IR-oriented techniques such as [3] (whichemphasizes on frequency and term-weighting eventhough translation model and HMM approach for
retrieval are introduced), or more linguistics-oriented
techniques such as [4] (where parsing and phrase
identification and translation seems to play a largerrole). We investigated a combination approach
where our highly effective PIRCS IR system is
employed with two methods of translating English
queries into Chinese, viz.: MT software and bilingual
wordlist [5]. Even though we did not implement the
MT software, we assume that a certain amount of
linguistics sophistication has been employed.
Translation is inherently ambiguous; we apply acommon-sense observation that ‘two heads are better
than one’. The 'Transperfect' package (now known
as 'TransWiz', produced by a Taiwanese company
Otek http://www.otek.com.tw) seems ideal since itshould produce Chinese translations that would agree
well dialectically with the collection, which also has
its source from Taiwan. We had experience with thispackage before [6]. It can translate via a u-mode(meaning that every English word/phrase would
produce a unique Chinese term), or an m-mode which
can output at most three Chinese terms to hedge fortranslation errors. We employed only the u-mode
because we intend to rely on our second method,
dictionary translation, for hedging purposes.
For dictionary approach, we used the Chinese-toEnglish bilingual wordlist from LDC (Linguistic
Data Consortium http://www.morph.ldc.edu/Projects/Chinese, which we call ldc2ce) as our maintranslation dictionary. It has about 120K entries. In
addition, about 6000 pairs of word/phrase translations
were obtained from the Hong Kong Law parallel text
collection that is also available from LDC. Afterdictionary look-up, an English word usually has
many translations. We employed four successive
procedures to try to disambiguate and isolate themore correct outputs. These four methods werediscussed in [7]:
•
•

•

•

Dictionary structure-based: ldc2ce format is
employed to select the more correct ones among
word translations;Phrase-based: ld2ce can be regarded as a phrase
dictionary by matching query strings withEnglish explanations of Chinese terms, givingmuch more accurate phrase translations;
Corpus frequency-based: select translation words
with higher occurrence frequency in the target
collection, which usually have higher probabilityof being correct; andWeight-based: a Chinese term set translated for
one English term can be weighted as a synonym
set that is more effective for retrieval [8].

Thus, MT software and dictionary translation
procedures provide us with two independent sets ofqueries. These were used separately for retrieval. An
example of translation is query #50 of the SO type
(topic, question, concepts) shown below.
The
Chinese word set enclosed between two ^’s indicate
the mappings for each English word and the numerics
indicate the number of terms in a synonym set.
Query #50:
Original English
Hot springs resorts in Taiwan .
****the related information about the hot springs
resorts in Taiwan.
spa, hot springs, travel, spa site, hot springs resorts,soak in hot springs, spa therapy, spa soaking, springsoaking, spa pool, water quality, healing effect, skin,facility, consume, special product (of a place), inn,
hotel, accommodation, transportation, itinerary plan.

 15Hë×****¶Â1ÂÍ 1÷5Hë×1~
Query #50:

Translation by TransperfectüH 5H ³é üHñ¬ 5Hë×.55H üHí üHæ ãw1ÝüHÃ ½ í1¯¨ <{Âã[ý´( ±pm)[ t.¨ A³éµàd.
5HÄ aWñê ¶ÂÓ Qê ¶Â$ã ~ àJ 
5HÄ aWñ

5Hîñ
5HÄ
³
³é µ P
5Hîñ

¬
;?
5HÄ
aWñ
æ  ä
5HÄ
5Hîñ
í
5Hîñ
3Ø
 Ý
H
3Ø
5Hîñ
½Ã ÃÀ Ã 
ç
½ ó
ih ´
ý 
Ð
¯à ¯¨
4 «¯
<Å
A{Â
Á{
3{
ã
ý
ýØ
ýs
14
´ y´ "

³t
<
{Â
Js
¨

²Ó
à½ ud ±Â {àQuery #50:
Translation by Wordlist^1.0^ ^1.0^ IN ^1.0^.
**** THE
^0.25
0.25
0.25
0.25
^^0.25
0.25
0.25
0.25
^
ABOUT THE^1.0
^ ^1.0^ IN ^1.0^.
^1.00^ ^1.0^^0.25
0.25
0.25
0.25
^^1.00^^0.330.33
0.33^ ^1.0^ ^1.0^^0.33
0.33
0.33
^ IN
^1.0
^^1.00^ ^1.0^^1.00^ ^1.0^^0.330.33
0.33^ ^1.0^^1.00^^0.25
0.25
0.25
0.25
^^0.330.33
0.33^^0.25
0.25
0.25
0.25
^ ^1.0^^0.25
0.25
0.25
0.25
^^0.500.50^^0.330.33
0.33^ ^1.0^^0.25
0.25
0.25
0.25
^^0.330.33
0.33^
OF PLACE^1.0^ ^1.0^^0.25
0.25
0.25
0.25
^ ^1.0^ ^1.0^^0.25
0.25
0.25
0.25
^.

Table 2.1 records the number of unique index
terms averaged over 50 queries (after stopword
removal) for different query types in our system. The
Chinese representations (under heading ‘Orig.
Chinese’) are longer than the English (under ‘Orig.
English) because of the use of single characters withshort words.
‘Dictionary’ translation introduces
multiple mappings for each English term andtherefore produces even longer queries. TI, TQ,
TQN denote queries using ‘title’, plus ‘question’, andadditionally the ‘narrative’ sections of a topic. The
latter two types are discussed in Section 5. VS, SO
Before Translation
After Translation
Type English Chinese MT Dictionary Mix Both
3.9
5.1
6.0
12.2
14.6
TI
7.5
13.1
13.0
26.9
32.9
TQ
40.3
37.3
81.3
100.5
TQN 20.4
23.8
27.3
35.0
79.6
94.0
VS
25.9
33.5
40.1
83.0
100.7
SO
35.1
55.2
58.3
124.2
151.3
LO
Table 2.1: Sizes of Different Query Types

LO types correspond to TI, TQ and TQN queries
respectively but include the ‘concepts’ section of a
topic as well. This section introduces between 15 to
20 new English terms compared to not using it. The‘narrative’ section used in the TQN and LO types
also adds ten or more unique terms.

3 PIRCS Retrieval System and Collection
Processing
Current approaches to text retrieval usually assign
a retrieval status value (RSV) to each document d ina collection based on the properties of a given query
q and those of the document and collection.
Different systems employ different retrieval models
to come up with useful RSV’s. After this is done, thedocuments are ranked (i.e. sorted according to the
RSV’s) and presented to the user in an orderedfashion. Our PIRCS, acronym for Probabilistic
Indexing and Retrieval – Components – System,
employs a combination of two retrieval algorithms
producing a document-focused RSVd and a queryfocused RSVq for each document, and combinedwith a mixing parameter α. It is an extension of theprobabilistic retrieval model and viewed as anactivation spreading process in a network withlearning (see [9] for greater details). Thus:
RSV(q,d) = α*RSVd + (1- α)*RSVq(1)
with
RSVd = Σk S(qtf k/Lq)* wdk(2a)wdk = log [tfk/(Ld- tfk)*(Nw-Ld-Fk+tfk)/(Fk-tfk)](2b)
RSVq = Σk S(tf k/Ld)* w qk
(3a)
wqk = log [qtfk/(Lq-qtfk)*(Nw-Fk)/Fk](3b)where tf k, qtf k are the frequency of term k in d and q
respectively, Ld = Σk tf k , Ld = Σk qtf k are the lengthsof d and q, S is a sigmoid-like function, Fk= Σall doc tf k
is the collection frequency of term k, and Nw = Σk Fk
is the number of tokens used in the collection.
Our approach considers every term of a document(or query) as a conceptual component self-relevant to
the document (query) itself, and we work in auniverse consisting of document components rather
than documents. Because of the self-relevanceassumption, every query (document) therefore has a
relevant and irrelevant set even when no relevant
judgment has been made, and we are able tobootstrap and provide probabilistic weights to ourterms at the initial retrieval stage. Because we workwith conceptual components, repeat term usage andquery/document lengths are accounted for, enabling
us to remove the binary assumption restriction in
traditional probabilistic retrieval model [10]. The
weight formula of Eqn. 3b is the familiar
probabilistic query term weights but using
components instead of whole document. Eqn. 2b isfor document-focused retrieval and the form of theweighting, after taking the approximation Nw >> allother frequencies, turns out to be very similar to
those used by [11] via a language model approach,but with a different smoothing coefficient.
Thus, our PIRCS system may also be viewed as a
combination of the probabilistic retrieval model and asimple language model. It has been employed to dolarge scale IR experiments such as those run by
TREC (see e.g. [12]) with consistently superior
results. After translating queries from English to
Chinese, crosslingual retrieval becomes monolingual
Chinese retrieval if we do not need to worry abouttranslating retrieved documents into English. This istrue in these experiments.
Previous experience has shown that several
methodologies are available to enhance CLIR such as
pre-translation query expansion, two-stage retrieval
with post-translation query expansion and collectionenrichment. Pre-translation query expansion means
expanding a given English query with highly
associated terms based on the top retrieved
documents from an appropriate English collection.
We did not use this technique because the English
collections that we have (from TREC) may not beappropriate with this new target collection andqueries. Post-translation query expansion means
performing a pseudo-relevance feedback using twostage retrieval with the target Chinese collection andthe translated Chinese queries. The first stage
retrieval defines the top documents and best terms to
be used to expand the Chinese queries. This is done
as a default in our PIRCS system. The parameters
used were n=40 top documents and m=100 best
terms.
Collection enrichment entails using anappropriate external Chinese collection during firststage retrieval. The idea is to improve the chance ofgetting more good documents into the top retrieved,
and this might lead to better terms in those chosen as
best. Again, we did not use this technique because
the target collection is new and unfamiliar, and weare afraid that the available TREC Chinesecollections may not be appropriate for enrichment
purposes.
Chinese text needs to be segmented for retrieval
purposes. We used short-words with characters as
our default document and query representation
method, since we have experience that it is both
effective and efficient [13]. However, we differedfrom our previous work by using the translation
dictionary (discussed in Section 2) as oursegmentation dictionary.
In that wordlist, weessentially keep all the Chinese terms that are four orless characters in length.
After text processing and indexing of thecollection, the resultant dictionary generated has asize of 86,848. After setting Zipf thresholds of 3 and20,000 to define statistical stopwords, the uniqueindexing terms remaining were 55,550.

A default mode of our system is to break long
documents into approximately 550 character subdocuments ending on a paragraph boundary. We
ended up with 176,924 sub-documents. We did notprocess the last batch of ‘chi’ documents that were
not available when the task began, because we werepressed for time. We believe this batch would not
influence results materially since its statistics is
insignificant compared to the rest of the collection.
Also it turns out there is only 1 relevant document inthis batch for query #23 using relax assessment.
Monolingual retrieval was done as previously
described. Crosslingual retrieval makes use of both
the MT-software translated queries as well as thedictionary-mapped queries, each providing its own
retrieval list. The RSV’s of these two lists are
combined using a ratio of 6:4 in favor of MTsoftware retrieval to produce the final result. This isdone for all query types.

but it is not true here. Perhaps the concept sections
have such precise and rich descriptions of theinformation needs that additional wordings only
serve to add noise to the retrieval. The relevants
retrieved (RR) at 1000 documents of 652 is 100% of
those judged relevant using the 'rigid' assessment
protocol. The performance of the four official
monolingual runs are also displayed as precision
recall curves in Fig. 4.1
For comparison, the overall best submittedmonolingual MAP values are respectively: 0.4683,
0.6596, 0.6529 and 0.6486 for the TI, VS, SO andLO query types using ‘rigid’ assessment. Except for
the LO type, we are within 8% of these best values.
For the LO type, our MAP value is off the best value
by nearly 12%.
Monolingual: LO, SO, VS, TI -- rigid
1

0.9

4

Results and Discussion

0.8

4.1 Monolingual Retrieval
We first look at our monolingual runs, which will
form the basis from which crosslingual results will bemeasured. Table 4.1a shows our submitted results
using the ‘rigid’ assessment method. The values inthe ‘%’ columns are percentage increases calculatedusing the TI values as basis. It is seen that even the
short ‘title only’ queries provide very respectable
results at a mean average precision (MAP) of 0.4653.
The query types with the ‘concepts’ and othersections added: VS, SO LO all give much better
MAP values of ~0.6. Interestingly, these MAP
values are at similar levels to those done for the
TREC 5&6 Chinese experiments [13].
TI
VS
SO
LO

MAP
.4653
.6139
.6037
.5726

%
+32
+30
+23

RR
651
652
652
652

%
+0
+0
+0

P@10
.4680
.5740
.5660
.5420

%
+23
+21
+16

P@20
.3420
.4100
.4170
.4000

%

P@20
.5930
.6780
.6790
.6640

%

+20
+22
+20

a) Rigid Assessment
TI
VS
SO
LO

MAP
.5860
.6998
.6936
.6806

%
+19
+18
+16

RR
1561
1568
1568
1567

%
+0
+0
+0

P@10
.6820
.7940
.7940
.7860

%
+16
+16
+15

mean av. precision

0.7

0.6
chLO.rig 652/.5726
chSO.rig 652/.6037
chVS.rig 652/.6139
chTI.rig 651/.4653

0.5

0.4

0.3

0.2

0.1

0
-

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

recallFigure 4.1: Recall-Precison Curve
Monolingual ---- Rigid Assessment

for

Table 4.1b shows the same runs using the ‘relax’assessment. It is even better in all measures except
the relevants retrieved which vary between 99% toclose to 100% of the 1571 judged relevant. We
believe the ‘rigid’ assessment is probably more
realistic. It turns out that the pseudo-relevance
feedback parameters for these runs were not set well.
Improved monolingual results with better parameters
are discussed in Section 5.

4.2 Crosslingual Retrieval
+14
+15
+12

b) Relax Assessment
Table 4.1: Monolingual Retrieval Results (%increase is calculated using ‘TI’ as basis)
Best result is achieved with the VS version, which
we defined as using ‘title and concepts’ sections only.
For example, within 10 top retrieved documents one
can expect on average over 5.7 relevant documents,
or over 8.2 within the top 20. Usually, longer queries(such as the SO or LO versions) lead to better results,

Our automatic crosslingual results are tabulated inTable 4.2. The percentage columns are calculatedwith respect to the corresponding monolingual valuesof Table 4.1 for evaluation purposes. Except for the‘title only’ queries, the submitted runs are able toachieve ~80% in MAP values, ~95% of relevants
retrieved and > 80% of precision at ten and twenty
documents when compared to monolingual
effectiveness using ‘rigid’ assessment. ‘Title only’
queries give precision values between 55 to 60% ofmonolingual, and 88%of relevants retrieved. Similarvalues of ‘relax’ assessment are shown in Table 4.2b.

MAP
.2554
.4724
.4774
.4733

TI
VS
SO
LO

%
55
77
79
83

RR
572
627
626
628

%
88
96
96
96

P@10
.2680
.4580
.4580
.4660

%
57
80
81
86

P@20
.2050
.3450
.3490
.3440

%
60
84
84
86

%
50
70
70
70

P@20
.3380
.5570
.5580
.5490

%
57
82
82
83

a) Rigid Assessment
MAP
.3164
.5483
.5522
.5499

TI
VS
SO
LO

%
54
78
80
81

RR
1329
1478
1475
1459

%
85
94
94
93

P@10
.3380
.5570
.5580
.5490

b) Relax Assessment
Table 4.2: Crosslingual Retrieval Results (%are calculated using Table 4.1 corresponding
monolingual values as basis)
Mono vs Xlingual: SO & TItle - rigid
1

0.9

0.8

mean av precision

0.7

the ‘question’ section in the SO type that provides
extra redundancy to hedge for errors during querytranslation. A MAP of ~0.47 for the longer querytypes are quite reasonable for this difficult task. The
precision at 10 and 20 top documents retrieved showthat one can expect on average close to 4.6 and 7
relevant documents respectively. These numbers are
somewhat less than those for monolingual, but stillquite respectable from a utility point of view. A
comparison between monolingual and crosslingual
precision-recall curves are shown in Fig.4.2. The twosolid curves are for SO and the dotted curves for TI
queries. It is observed that crosslingual SO queries
achieve much closer performance to its monolingual
counterpart than TI queries. We again attribute thisto more redundancy and better context for translation
with longer query types. LO and VS queries produce
precision recall curves almost indistinguishable fromthis SO curve. Fig.4.3 shows the TI performancebroken up into MT software and dictionary
contribution. In general, MT software is somewhat
better in these experiments.

0.6
chso.rig 652/.6037
ecso.rig 626/.4774
chti.rig 651/.4653
ecti.rig 572/.2554

0.5

4.3 Failure Analysis for ‘Title’ Queries

0.4

0.3

0.2

0.1

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recallFigure 4.2: Monolingual vs Crosslingual: ‘SO’and ‘TI’ ---- Rigid Assessment

The results for the ‘title only’ queries (MAP
0.2554) are not good, achieving only about 55% ofmonolingual MAP value. These average to less thanfour English content terms per query (Table 2.1).
Unfortunately, it is well known that most users
generally prefer to issue short queries (in the web, for
Mono vs Xlingual: Title - Rigid
1

Mono vs Xlingual: TI Various -- rigid
0.9

0.8

0.8
0.7

0.7

0.5
chti.rig 651/.4653
ecti.rig 572/.2554
ectitp.rig 517/.2274
ecti416.rig 505/.1912

0.4

Mean Av Precision

mean av precision

0.6

0.6

0.5

0.4

0.3

0.3

0.2

0.2

0.1

0.1

Figure 4.3: MT, Dictionary and Combination
P-R curves for ‘TI’ Queries ---- Rigid
Assessment
The percentage of monolingual effectiveness issimilar to the ‘rigid’ assessment, except for the
precision at 10 documents where the percentage is
70% vs 80%. These crosslingual MAP are state-ofthe-art values.
The best absolute MAP value of 0.4774 is attainedwith the SO query type, while for monolingual it is
the VS type. We conjecture that this is because of

49

47

45

43

41

39

37

35

33

31

9

7

29

1

27

0.9

25

0.8

23

0.7

21

0.6

19

0.5
recall

17

0.4

15

0.3

13

0.2

11

0.1

5

1
0

3

0

0

Query #

Figure 4.4: Monolingual vs Crosslingual per
query: ‘TI’ ---- Rigid Assessment
example, searching with one or two words iscommon). These results demonstrate that users will
be disappointed issuing short queries for CLIR.
‘Relax’ assessment provides better absoluteeffectiveness (MAP 0.3164), but still in the 54%
level compared to monolingual. (In a sense, evenhumans may misunderstand intentions of a few
words, not to mention machines.) Fig.4.4 shows
individual TI query performance compared with

monolingual. There are 6 topics (#1, 6, 24, 28, 32,50) better than monolingual, but 44 worse. Two
topics (#7, 10) have zero MAP, and 15 have MAP
values <0.1 while their monolingual values are >0.1.
These are topics #2, 4, 5, 12, 13, 19, 20, 21, 22, 30,
35, 42, 44, 46 and 48. We assume that these queries
fail because of translation problems. What follows
is an attempt to analyze why they perform badly.
Query 7 failed because ‘Carter’ was translated
into:‘the worker who carts’, whilequery 10 needs a special Chinese termfor‘Treasury stock’ and is lacking in our translation
approaches. #12 (Michael Jordan), 30 (El Nino), 44(Hua-shan), 46 (Ma Yo-yo) are related to unknown
proper nouns or transliteration. #21 has IC in Englishthat was not present in the original Chinese topic. IC
was translated as ‘
RUIntegrated Circuits)’. This constitutes noise. #2 and5 have reasonable translations but performed badly.
The rest have genuine bad or no translations for somewords such as: #4 ‘commercials’ has no or badtranslation, 35 ‘diet product’ mapped to ‘(‘diet’ in the Japanese Diet sense),
RU’, #42 ‘millennium’ has a special slangthat was missed These 17 queries accountfor about 56% of the difference between ‘title only’
monolingual and crosslingual MAP result.
There are other queries like #3 (White Horror), 6(Kosovar), 15 (Bai Xiao-yan), 27 (Meinung), 28(Chilan), 33 (Bai-feng), 34 (Viagra), 38 (Chunghwa,
ROCSTAT), 47 (Jin Yong) that involve un-translated
proper nouns or transliterations. However, their
MAP values are >0.1 and not dismal. In fact, #6 and28 performs better than monolingual! Queries #23(Disneyland), 39 (Leonid), 45 (Cloud Gate), andseveral queries contain (Taiwan).
These were
properly translated. Two more queries #13 (NBA)and #43 (CIH) have abbreviations that also failedtranslation. However, the two original Chinese
queries also did not use them. Thus, these 50 topics
seem to have a realistic proportion of proper nouns.
Generally, these terms lead to good results formonolingual retrieval because they are unambiguous,
but their translation failures cause the corresponding
crosslingual queries to behave poorly -- at only
~55%. However, their longer counterparts VS, SO or
LO query types have sufficient redundant
descriptions and manage to restore the performanceto ~80%. This points to the importance of using
longer query descriptions for CLIR.

¥üA;

h¤v

Ä( Ä

å Ä
Òï|

oC´|
|I5

Additional Experiments

5.1 Improving Monolingual Results
As discussed in Section 4.1, our monolingual
blind experiments, though good, are less than the top
runs by about 8% and more for the LO queries. After

results were known, we discovered that theparameters that we set for pseudo-relevance feedback(prf) of 40 top documents and 100 best terms aregood for crosslingual but sub-optimal formonolingual.
After adjusting to using 10 top
documents 100 terms, the result improved as shownin Table 5.1.
prf: 10d100t
TI
TQ
TQN
VS
SO
LO

MAP
.4853
.5179
.5450
.6384
.6271
.6214

%
+4
+5
+7
+4
+4
+9

P@10
.4500
.5020
.4840
.6020
.5720
.6000

prf: 40d100t
P@20
.3450
.3760
.3780
.4230
.4240
.4170

MAP
.4653
.4943
.5092
.6139
.6037
.5726

P@10
.4680
.4860
.4760
.5740
.5660
.5420

P@20
.3420
.3750
.3710
.4100
.4170
.4000

Table 5.1: Comparing Monolingual Results –New and Old Parameters (Rigid Assessment)
The values based on new parameters are shown at
the left with the MAP percentage improvements over
the old values shown on the right. These new values
for the official query types are within about +/- 4% of
the overall best monolingual results submitted. The
VS query type still performs best, but the long LO
type improves substantially over our submitted
values. The RR values remain as before.
Table 5.1 also shows two new runs for the TQ and
TQN queries. As pointed out before, TI, TQ and
TQN correspond to VS, SO and LO types minus the‘concepts’ section.
The use of ‘question’ and‘narrative’ sections in TQ and TQN improves resultsover using ‘title’ only, but still far from theeffectiveness returned by incorporating the ‘concepts’
section.

5.2 Cross Language IR with‘Question’ and ‘Narrative’ Sections‘Title’,The ‘concepts’ section in a topic contains manyrich and precise phrases and wordings concerning theinformation needs. This is very good for bothmonolingual and crosslingual IR results but isunrealistic in real-life situations. It is difficult fornormal users to supply these terms. On the other
hand, ‘title’ only queries perform poorly.
We
therefore investigated whether adding the ‘question’and ‘narrative’ part of a topic to the ‘title’ section(forming TQ and TQN queries) can lead to better
performance.
The ‘question’ part is a natural
language statement of a user’s needs, while the‘narrative’ can be regarded as further exposition infree text. They should be easier for a user tocompose than the ‘concepts’. We translate as before
and perform retrieval with them. These are shown in
Table 5.2.
It is seen that he TQ queries get between 8 to 20%
better precision values than ‘title’ only, while TQN

TI
TQ
TQN

MAP
.2554
.2968
.4105

%
+16
+61

RR
572
598
620

%
+5
+8

P@10
.2680
.2900
.3960

%
+8
+48

P@20
.2050
.2470
.3080

%
+20
+50

Table 5.2: Comparing Crosslingual TI with
TQ, and TQN Queries – Rigid Assessment
queries lead to between 48 to 61% improvements.
The latter has a MAP value of 0.41 and nearly 4
documents out of the top 10 retrieved are relevant.
These are good performance.
Thus, just adding
more related text can bring us much closer to the 0.47
MAP values for queries with ‘concepts’ section(Table 4.1). Number of relevants retrieved, RR, also
improves. This illustrates again that longer
descriptions of needs should be encouraged for
CLIR. The wordings need not be high precision
conceptual terms.

Comparing with the improved monolingual resultsof Table 5.1, ‘title’ only queries still operate at adisappointing 52% of monolingual. Precisions at 10,
20 documents however are better at ~60%. On
average >2.5 documents among the top 10, or 4
among top 20 retrieved are relevant. Lengthening thequeries with the ‘question’ section (TQ) improvesmonolingual comparison to 60%, and to 70% when‘narrative’ section is also used (TQN). There would
be on average over 3 and nearly 5 relevant documents
among the top 10 and 20 retrieved respectively if one
employs the ‘title’ and ‘question’ sections as queries(TQ). Queries with ‘concepts’ operate at 76 to 80%
of monolingual. Plots of crosslingual performance at
different (English) query sizes are shown in Fig. 5.1.
Crosslingual MAP& % Variation with Monolingual for Different English Query Siz es% of mono

xling.map

0.9

0.8

5.3 Mixing Two Translations as One Query

0.7

TI
TQ
TQN
VS
SO
LO

MAP
.2544
.3107
.3814
.4872
.4999
.4835

%
52
60
70
76
80
79

RR
578
602
617
646
645
642

%
89
92
95
99
99
98

P@10
.2580
.3160
.3820
.4720
.4920
.4920

%
57
63
79
78
86
82

P@20
.2060
.2490
.2850
.3610
.3700
.3560

%
60
66
75
85
87
85

Table 5.3: CLIR Results using Mixing ofTranslations in One Query ---- Rigid
Assessment (% calculated using Table 5.1
corresponding monolingual values as basis)me an av. Prec / % of Monolingua l<------ has concepts (VS, SO, LO)------>

In our submitted experiments, the two translation
outputs – MT-software and bilingual wordlist lookup
with disambiguation – were employed individuallyfor retrieval, and then their retrieval lists are
combined. An alternative is to combine both outputs
into one single longer query, and perform retrieval
with it. This makes sense in that if one translation
method is faulty (missing or wrong), the othermethod may have the correct mappings and thus
remedy the query to a certain extent. If both methods
confirm the same translation for an English term, theChinese output will attain double weight compared toother translations that do not agree. Moreover, one
needs to do a single retrieval only and is more
efficient. We call this process ‘mixing translations’into one query. The result is shown in Table 5.3.
Except for TI and TQN types, mixing translations canlead to 2-5% better results for all other query types
when compared to Tables 4.2a and 5.2 wherecombination of retrieval lists is employed. TI has
MAP value decrease slightly by half a percent, but
TQN decreases substantially by 7%. It needs further
investigation to see why mixing translations for TQN
queries is not good.

0.6

0.5
<-----------

no conce pts (TI, TQ, TQN)-- ---------

0.4

0.3

0.2

0.1

0
0

5

10

15

20

25

30

35

40

# of Unique English Query Te rms

Figure 5.1. Crosslingual Performance at
Different English Query sizes
Essentially the plots summarize what we
observed: when queries contain the ‘concepts’
section (right side of Fig. 5.1), adding the ‘question’section (SO) helps, but further adding the ‘narrative’section (LO) leads to worse performance. For
queries without ‘concepts’ (left side), performance
improves monotonically with more free texts (TQ
and TQN).

6

Conclusion and Future Work

This NTCIR-2 Chinese and cross language
experiments make use of ~200MB of texts and 50
topics. All topics have at least four relevant answer
documents in the target collection. A reasonable
number of the topics also include proper nouns,
transliterations that are difficult to translate. The
monolingual mean average precision effectivenesswas high: at over 0.6 for long queries, and at about
0.47 even for queries involving only a few words ofthe title section of a topic.
Cross language IR leads to about 0.52-0.55% ofmonolingual MAP value for ‘title’ only queries. We
attribute this low effectiveness to queries involving

proper nouns and transliterations. These accentuate
the gap because their un-ambiguity gives goodmonolingual results while their translation failure
leads to poor crosslingual values. Lengthening thequeries with various amount of related free texts canenhance precision to 0.38-0.4 or 70% ofmonolingual. Adding highly precise and related
concept terms improve crosslingual MAP results
further to nearly 0.5 or 76 to 80% of monolingual.
Long query descriptions are recommended forEnglish-Chinese CLIR.
MT-software and dictionary lookup translation
with disambiguation techniques seem to be able tocomplement each other. Our approach of linearly
combining their retrieval lists appears to work well.
A more efficient strategy is to mix translations into
one query. It is shown that except for the TQN query
type, it can lead to about equal or improved resultscompared to linear combination of retrieval lists.
One cannot do crosslingual retrieval without
properly translating the queries (or the documents).
Proper noun and transliteration translation are
therefore crucial for CLIR. Expanding coverage of
the LDC bilingual wordlist or improving theweighting of translation terms may also lead to better
CLIR performance. These are some of the topics weintend to investigate in the future.

Acknowledgments
This work was partially supported by the Space
and Naval Warfare Systems Center San Diego, under
grant No. N66001-1-8912. Norbert Dinstl helped
formatting the queries and collections.

References[1] G. Grefenstette. Cross language Information Retrieval.
Kluwer, 1998.
[2] D. Oard & A. Diekema. Cross-language information
retrieval. In: Annual Review of Information Science andTechnology, 33: 223-256, 1998.

[3] J. Xu & R. Weischedel. TREC-9 cross-lingual retrieval
at BBN. Preliminary paper at TREC-9 Conference,
Gaithersburg, MD, Nov, 2000.
[4] J. Gao, J-Y. Nie, J. Zhang, E. Xun, Y. Su, M. Zhou &C. Huang. TREC-9 CLIR experiments at MSRCN.
Preliminary paper at TREC-9 Conference, Gaithersburg,
MD, Nov, 2000.
[5] K.L. Kwok, L. Grunfeld, N. Dinstl & M. Chan. TREC9 cross-lingual, web and question-answering track
experiments using PIRCS (Draft). Preliminary paper at
TREC-9 Conference, Gaithersburg, MD, Nov, 2000.
[6] K.L. Kwok. English-Chinese cross language retrieval
based on a translation package. In: MT Summit VII
Workshop: Machine Translation for Cross Language IR.,
8-13, 1999.
[7] K.L. Kwok. Exploiting a Chinese-English biligual
wordlist for English-Chinese cross language information
retrieval. Proc. of 5th Intl.Workshop on Information
Retrieval with Asian Languages IRAL2000, 173-179,
2000.
[8] A. Pirkola, H. Keskustalo & K. Jarvelin. The effects ofconjunction, facet structure and dictionary combinations
concept-based cross-language retrieval. Information
Retrieval, 1(3): 217-250, 1999.
[9] K.L. Kwok. Improving English and Chinese ad-hoc
retrieval: a Tipster Text Phase 3 project report. Information
Retrieval, 3(4): 313-338, 2000.
[10] S.E. Robertson & K. Sparck Jones. Relevanceweighting of search terms. Journal of the American Society
for Information Science, 27: 129-146, 1976.
[11] D. Hiemstra & W. Kraaj. Twenty-One at TREC-7:adhoc and cross language track. In: Information Technology:
The Seventh Text Retrieval Conference (TREC-7).
E.M.Voorhees & D.K. Harman, (eds.), NIST Special
Publication 500-242, GPO: Washington, D.C, 227-238,
1999[12] E.M. Voorhees, & D.K. Harman. Overview of theEigth Text REtrieval Conference (TREC-8). In:
Information Technology: The Eigth Text REtrieval
Conference (TREC-8). E.M.Voorhees & D.K. Harman,(eds.), NIST SP 500-240, pp.1-24. GPO: Washington, D.C.
[13] K.L. Kwok. Employing multiple representations for
Chinese information retrieval. Journal of the American
Society for Information Science, 50(8): 709-723, 1999.