NTCIR-2 ECIR Experiments at Maryland:Comparing Pirkola’s Structured Queries and Balanced Translation
Douglas W. OARD and Jianqiang WANG
College of Information Studies and Institute for Advanced Computer Studies
University of Maryland, College Park, MD 20742 USA(oard,wangjq)@glue.umd.edu

Abstract

week summer workshop at the Johns Hopkins University, where the Mandarin-English Information (MEI)
team found that so-called balanced translation compared favorably with Pirkola’s structured query formulation method and identified post-translation resegmentation as a potentially important issue in CrossLanguage Information Retrieval (CLIR).
Both MEI and ECIR used English queries to retrieve Chinese documents, so ECIR provided an excellent opportunity to apply what we learned at MEI
to a different (and far larger) test collection. Interestingly, we obtained results that are somewhat different
from what we saw at the MEI workshop. In this paper
we provide some background about the two key issues
that we explored, review what was learned about thesequestions at the MEI workshop, present both our official ECIR results and some post hoc experiments that
we have scored locally, and then summarize the differences between the MEI workshop and the ECIR evaluation that might explain the differences in the results
we obtained.

Pirkola’s word-based structured queries have beenshown to perform well for word-based cross-language
information retrieval in European languages. Monolingual Chinese retrieval experiments, by contrast often find that character bigrams perform as well as(and sometimes better than) automatically segmented
words. During the Mandarin-English Information(MEI) project at the Johns Hopkins Summer 2000
Workshop, Pirkola’s structured queries were compared with an alternative technique known as balanced translation. The results suggested that balanced
translation coupled with post-translation character
bigram resegmentation might outperform Pirkola’s
word-based technique. The NTCIR-2 English/Chinese
Information Retrieval (ECIR) evaluation provided theopportunity to replicate this experiment on a farlarger collection. The results showed that on the
ECIR collection, (1) the best word-based and overlapping character bigram-based techniques we tried
performed comparably; (2) with word-based retrieval,Pirkola’s structured queries outperformed balanced
translation; (3) with overlapping character bigrambased retrieval, a technique analogous to Pirkola’s
method performed comparably with balanced translation; and (4) using all known translation alternatives significantly outperformed using the three mostfrequent translation alternatives.

2 Background
Oard and Diekema have identified three basic approaches to CLIR: query translation, document translation, and interlingual techniques [6]. English exhibits less segmentation ambiguity than Chinese, andour initial experiments with English/Chinese CLIR
indicated that pre-translation segmentation ambiguity
can adversely affect retrieval effectiveness [7]. Since
the ECIR queries are in English, we chose a querytranslation approach. Dictionary-based CLIR has been
the focus of much of our resent work, so we chose tofocus on Dictionary-based Query Translation (DQT).
DQT raises four key issues:

1 Introduction
The University of Maryland participated in the English/Chinese Information Retrieval (ECIR) track at
the second NII Test Collection Information Retrieval(NTCIR-2) evaluation. Our experiments focused on
two key issues: (1) comparison of two query formulation techniques that are designed to mitigate the effect of translation ambiguity, and (2) investigating theeffect of post-translation resegmentation. These questions were motivated by intriguing results from a six-The MEI team included Helen Meng and Wai-Kit Lo (Chinese
University of Hong Kong), Berlin Chen (National Taiwan University), Erika Grams (Advanced Analytic Tools), Sanjeev Khudanpur(Johns Hopkins University), Gina Levow (University of Maryland),Patrick Schone (U.S. Department of Defense), Karen Tang (Princeton University), Hsin-Min Wang (Academia Sinica, Taiwan), andthe authors of this paper.

1Pre-translation term selection. Selecting the unitsof meaning (which we call “terms”) that are to
be translated.

Englishterm E Termtranslation

Dictionary coverage. Obtaining a dictionary with
sufficient coverage to assure that that correcttranslations of the selected terms are known.

Chineseterm C1

TF1
DF1
LCompute WC1weightChineseterm C2

TF2
DF2
LCompute WC2weightAverage

WE

Figure 1. Estimating query term weights
using balanced translation.

Translation selection. Choosing appropriate translation(s) for each selected term.
Query formulation. Construction of a query that accommodates any unresolvable translation orsegmentation ambiguity.

so-called bag-of-terms information retrieval systems,
term weights are computed from three sources of evidence:

We chose to focus our ECIR experiments on the last
question, so we adopted a simple approach to English term selection (translating each word in the query
separately), reused an existing English/Chinese bilingual dictionary, and (except for some contrastive experiments) used all known translations. In this section, we describe three word-based query formulation
techniques and then introduce the question of posttranslation resegmentation.

Term frequency () The number of times term
appears in document (a property of a term in adocument).



 

) The number of docuDocument frequency ( ments term appears in (a property of a term).
Document length (  ) The number of terms document contains (a property of a document).
Retrieval systems typically compute term weights asa nonlinear function ofIn these three parameters.

Pirkola’s technique,,, and  for the querylanguage are estimated as:

2.1 Query Formulation
In early work on DQT for CLIR, queries were typically formed by including all translations for all ofthe query terms. When used with retrieval systems inwhich all translations contribute equally (e.g., vector
space methods), this approach gives more weight toquery terms that have many translations than to thosethat have few. This is generally an undesirable trait fora retrieval system, since terms with fewer translations
are usually more specific (and hence more useful for
retrieval) than terms for which many different translations are possible. This unbalanced query formulation
technique is still often used as a baseline in CLIR experiments, but better techniques are now known.
An obvious improvement is to rebalance the contribution of each term in some way. This insight was
simultaneously introduced at the 1999 Topic Detection
and Tracking evaluation by two teams [3, 4]. The key
idea, which Levow and Oard called balanced translation, is that the weight associated with each translationof a query term can be averaged in some way to compute a weight for that query term. Balanced queries
formulated in this way can be thought of as estimating
the weights for query-language terms (as if the documents had been written in the query language) andthen performing retrieval using those weights.
Remarkably, the best known alternative to balanced
translation was also simultaneously reported, in thiscase at SIGIR 98 [1, 8]. Lacking a better title for thetechnique, we refer to it simply as “Pirkola’s method,”
since Pirkola wrote more extensively on the issue. In 

     




 

 

where
is the number of times  translation  for 
is used toterm appears in document and  indicate the document frequency that would be computed for the union of the sets of documents in which
the translations for term are found. The weight foreach query language term is then computed directlyfrom these estimates.
Englishterm E Termtranslation

Chineseterm C1

TF1
DF1

Chineseterm C2

L
TF2
DF2

Sum

TF
Computeweight

Union

WE

DF

L

Figure 2. Pirkola’s method for estimating
query term weights.

Balanced translation and Pirkola’s method both estimate query term weights from the same evidence, but
nonlinearities in the term weight computation resultin different estimates. Figures 1 and 2 illustrate thetwo approaches. As Sperer and Oard have observed,
Pirkola’s technique tends to be conservative, estimating a high document frequency (which results in a low

Pirkola called the technique a “structured” query, but balanced
translation also produces queries with structure.

2

term weight) if any translation of a term has a highdocument frequency [9]. Balanced translation, by contrast, allows rare translations to contribute their relatively high term weights to the query term on a moreequal basis. We are not aware of any careful comparisons between balanced translation and Pirkola’s
method, so one goal of our ECIR experiments was toperform such a comparison.

that we investigated in the MEI project, which is described in the next section.

2.3 The MEI Project
The MEI project team worked together for six
weeks in July and August of 2000 at the Johns Hopkins University Center for Language and Speech Processing [5]. The principal focus of the project was development of techniques for cross-language speech retrieval. The MEI project reused two test collections
that were originally developed for the Topic Detection and Tracking (TDT) evaluation. Both the TDT-2
and TDT-3 collections contain English newswire articles from the New York Times and the AssociatedPress, Mandarin Chinese radio broadcast stories fromthe Voice of America (with known story boundaries),and event-based relevance judgments for multiple topics. Machine-produced (errorful) Chinese transcriptsof the Voice of America broadcasts are also available.
The MEI task was to perform query-by-example on thecollection of Mandarin Chinese audio stories, using asingle English newswire story as the example document. Since this was a retrospective retrieval task, avariant of mean average precision was used as the principal measure of effectiveness.
Initial experiments using the TDT-2 collection (17topics, 2,265 Mandarin Chinese audio stories) suggested that balanced translation and Pirkola’s method
performed about equally well. Since post-translation
character bigram resegmentation seemed to help balanced translation more than it helped Pirkola’s methodin these initial exploratory experiments, balanced
translation was adopted for the remainder of the MEI
project. Ultimately, post-translation resegmentation
into overlapping character bigrams was found to produce a statistically significant 11% relative improvement over the use of words when balanced translation
was used with the TDT-2 collection. We did all of ourdevelopment work with the TDT-2 collection, holding
out the entire TDT-3 collection (56 topics, 3,371 Mandarin Chinese audio stories) for a formal evaluation at
the end of the project. Surprisingly, no improvement
over word-based retrieval was observed when bigram
resegmentation was used with balanced translation on
the TDT-3 collection. The MEI project thus framed
the questions well, but left for future work the careful comparison of balanced translation with Pirkola’s
method and the detailed study of the interaction between those techniques and post-translation query resegmentation.

2.2 Post-Translation Resegmentation
Retrieval of Chinese documents brings into sharp
focus an issue that is present to some degree in any
language: the terms that result from translation might
not be the best terms to use for retrieval [5]. Specificity is a desirable characteristic of terms to be translated, since specific terms naturally exhibit little translation ambiguity. For this reason, translation of multiword expressions typically improves CLIR effectiveness when compared to word-by-word translation [2].
Two competing effects must be considered when selecting terms for retrieval, however. The use of very
specific terms tends to increase precision, while theuse of less specific terms tends to benefit recall. Manyexperiments with English retrieval have shown that it
is generally better to use the constituent words of amultiword expression as if they were separate terms. 
Documents that contain the entire expression will still
accumulate more weight than documents that contain
only a portion of it, but documents with only a portion of the words also become retrievable. This suggests that it might be beneficial to resegment multiword translations into individual words prior to retrieval.
Chinese adds a new twist to this issue: word boundaries are generally not marked, so the proper degreeof granularity for post-translation resegmentation is
unclear. One simple expedient, finding the smallest
components of a translation that could possibly bewords, would usually result in indexing single characters, since almost every Chinese character can be usedalone as a word. Indexing overlapping character bigrams is known to result in far better retrieval effectiveness than indexing single characters [10], and ourexperience in the MEI project (described below) suggests that this is also a reasonable approach to posttranslation resegmentation for queries that have beentranslated into Chinese.
It is not immediately clear how post-translation
resegmentation and query formulation should interact. Balanced translation and Pirkola’s method areboth reasonable approaches to combination of evidence from alternate translations, but how should theevidence from each bigram of a multi-character translation be combined? This was one of the key questions

3 ECIR Experiment DesignFigure 3 provides an overview of the processing
stages in our ECIR experiments. English queries were

If properly weighted, it can be even better to index multiword
expressions and their constituent terms.

3formulated by using every word in the title, description and narrative fields of the topic description. The
average query length was 115 words, about 23% of
the number of words found in an average MEI query.
Three alternative query translation algorithms were
implemented: Pirkola’s method, balanced translation,
and a baseline unbalanced “bag of translations” approach. Consistent segmentation was used for bothquery formulation and indexing. For word-based segmentation, we used freely available software from the
Linguistic Data Consortium (LDC). As an alternative, we used locally-developed software to form overlapping character bigrams. Term boundaries were
known after query translation from English to Chinese,so only within-term bigrams were generated. Termboundaries were not known in the Chinese documents,
so all possible bigrams were generated. When only
overlapping bigrams were indexed, single-character
Chinese translations of query terms were effectively
ignored.
Englishquery

Pre-translation
Term selection

Englishquery

ily overcome by using the hexadecimal representation of each term. For example, the GB code for thetwo-character Chinese word pei2chang2 (compensate)
would be represented as “0xC5E2B3A5.” For each
topic, Inquery produces a ranked list of documents,
for which retrieval effectiveness measures were computed using NTCIR-2 ECIR relevance judgments andthe freely available trec eval software. In this paper wereport mean uninterpolated average precision over 50
topics, and treat differences as statistically significant
if a two-tailed paired % -test results in &('*),+ ).- .
We focused our experiments on three questions:/ Is Pirkola’s structured query method effective
for Chinese?

/ Can post-translation resegmentation into character bigrams improve over word-based techniques?

/ Can limiting the number of translation alternatives that must be considered improve retrieval
effectiveness?
As originally designed, Pirkola’s method is a wordbased technique. The Chinese implementation is thus
quite straightforward when words found using the
LDC segmenter are indexed. The design space is
far larger in the second case, since both Pirkola’s
structured query method and Levow and Oard’s balanced translation technique are silent on the questionof which Inquery operator (if any) should be used togroup the component bigrams of a translation that contains more than two Chinese characters. The simplest approach is to treat multiple bigrams from thesame translation in the same way as multiple translations from the same English term. The MEI project
reported that balancing the contribution of each termusing Inquery’s #sum operator could be helpful whenusing balanced translation, so we tried that condition
as well. Nesting a #sum inside a #syn is not possible because #sum produces belief values while #synoperates on term frequency and document frequency
statistics. Accordingly, to construct a technique analogous to Pirkola’s structured query method we used the#OD0 (ordered distance) operator to group the component bigrams of a translation. 1 That operator computes
term frequency and document frequency statistics forthe specified ordered sequence of bigrams. This is essentially a “back door” way of approximating wordbased translation, but with the matching based on the
known translations rather than the LDC term list.

Term translation

Chinesequery

Retrieval

Chinesequery

Post-translation
query formulation

Figure 3. System design.
Our English/Chinese bilingual term list was represented in the GB code that is commonly used on the
Chinese mainland, but the document collection was
represented in the Big 5 code that is commonly used in
Taiwan and Hong Kong. Conversion from Big 5 to GB
is straightforward, since the mapping in that direction
is is generally many-to-one, so we chose to standardize on GB and used freely available software to convert
the documents into that representation. ! .
The ECIR collection contains 132,173 Mandarin
Chinese news articles from five news agencies in Taiwan, 50 topic descriptions, and relevance judgments
developed using a pooled assessment methodology
with seven participating systems. We used version
3.1p1 of the Inquery text retrieval system, which does
not include native support for the multibyte character
representation used in GB. This limitation was eas-"

The LDC segmenter can generate only terms that are containedin its term list. We made no adjustment to the term list to align itwith our translation lexicon. The LDC segmenter and the term list
are # available at http://morph.ldc.upenn.edu/Projects/Chinese/.
Some document bigrams contained punctuation or white space,but such bigrams would never match query bigrams and hence did
not $ affect retrieval results.
ftp://ftp.cuhk.hk/pub/chinese/ifcss/software/unix/convert/

4 Results and Analysis
We submitted three experiment runs for official
judgment, and scored an additional eight runs locally2

The value of 3 was set separately for each translation at one
fewer than the number of bigrams.

4

Official Run
UMD-ECIR-LO-01
UMD-ECIR-LO-02
UMD-ECIR-LO-03

Condition
char all syn odchar 3 syn od
word 3 syn

Word
Bigrams

#syn
0.36
0.30

#sum
0.19
0.25

#none
0.19
0.24

Table 2. Comparison of mean uninterpolated average precision for words andcharacter bigrams (no cross-bigram operator, all translations).

Table 1. Official runs.
using the ECIR relevance judgments. We have adopted
a four-field nomenclature to indicate the experimentconditions for each run:techniques that we tried. 4 . Table 2 compares the mean
uninterpolated average precision for runs under comparable conditions. For words, the Pirkola:Balanced
and Pirkola:Unbalanced differences are statisticallysignificant. The Balanced:Unbalanced difference issmall, and not statistically significant. For character
bigrams, although our technique that is analogous to
Pirkola’s method appears to consistently achieve bettermean uninterpolated average precision than balanced
translation, that result is statistically significant only
when no cross-bigram operator is used. Stated anotherway, the technique analogous to Pirkola’s method(with #OD0 across bigrams) is statistically indistinguishable from balanced translation (with #sum acrossbigrams). Finally, the difference between unbalanced
and balanced translation is not statistically significant.

Indexed unit. “word” for automatically segmented
words, ”char” for overlapping character bigrams.
Number of translations. The maximum number oftranslation alternatives that would be considered. In our experiments, this is either ”all” or”3”.
Translation grouping operator. The Inquery operator used to group the alternate Chinese translations of a single English query term. We used”syn” for Pirkola’s method, ”sum” for balanced
translation, or ”none” for unbalanced queries.
Bigram grouping operator. The Inquery operator
used to group the constituent bigrams of a single Chinese term. We used ”od” to enforce anordered distance constraint (adjacent and in order), “sum” to use average bigram weight, or”none” (effectively treating bigrams as if they
were alternate translations). This field was omitted for word-based retrieval.
For example, the best run for character bigram-based
retrieval is ”char all syn od”, which means we indexed character bigrams, used all of the translation alternatives that were found in the dictionary for eachquery term, grouped alternate translations with Inquery’s #syn operator, and grouped the constituent bigrams of any translation that contained more than two
characters using the #OD0 operator with an appropriate value of 0 . Similarly, for the best word-based
retrieval result, ”word all syn” indicates that we indexed automatically segmented Chinese words, used
all known translation alternatives, and grouped the alternate translations for each term using Inquery’s #synoperator. Table 1 shows the correspondence between
our official runs and this nomenclature.

Figure 4. Word-based techniques.

4.2 Analysis
Our analysis of these results has produced the following observations:

4.1 Results

5

After the workshop, we found an error in processing documentswith overlapping character bigrams. We corrected the error, re-ran
all the experiments in which character bigrams were used, and updated our analysis. The results described in this paper are based on
those corrected runs.

Figure 4 shows the recall-precision curves for theword-based retrieval techniques that we tried, and Figure 5 shows curves for the character bigram-based
5

Max Transall3

word syn *
0.36
0.30

word sum *
0.19
0.22

char syn * od
0.32
0.24

Table 4. Effect on mean uninterpolated
average precision of limiting the numberof translation alternatives.

ful.

/ Limiting the number of translation alternatives
in the way that we tried does not appear to be
helpful. Table 4 shows a contrastive condition inwhich only the three translations with the highest frequency in a monolingual Chinese corpus
were used. We used a corpus frequency list provided by LDC for this purpose. 6 . This resulted
in a statistically significant decrease in uninterpolated mean average precision for Pirkola’s
word-based method. No statistically significant
effect was observed for balanced translation. Finally, limiting the number of translation alternatives had a statistically significant adverse effecton the one post-translation resegmentation configuration that we tried.

Figure 5. Character bigram-based techniques.

none
#sum
#OD0

#syn
0.30

#sum
0.25
0.29

0.32

Table 3. Effect on mean uninterpolated
average precision of cross-bigram operators (vertical) for two cross-translation
operators (horizontal).

/ Finally, comparison of word-based retrieval andoverlapping character bigram-based retrieval
under similar conditions shows no statisticallysignificant difference. This is consistent withresults that have been previously reported formonolingual Chinese retrieval.

/ It appears that we achieved the best results from
Pirkola’s word-based method. Among wordbased methods, Pirkola’s method clearly outperformed the other two techniques that we tried.
Among bigram-based methods, char all syn od
and char all syn none did the best and were
statistically indistinguishable. Pirkola’s wordbased method seems to lead to better mean uninterpolated average precision than either of these,
but the differences are not statistically significant.

4.3 Comparison with MEI Results
In the MEI project, we had some indication thatbalanced translation and Pirkola’s method performed
about equally well. With the ECIR collection, however, we observed that Pirkola’s method achieveda statistically significant improvement over balanced
translation (with automatically segmented words). On
a related point, in the MEI project we found that
post-translation resegmentation into character bigrams
could be helpful (with balanced translation and nocross-bigram operator). In the ECIR evaluation, we
found that although this was true for balanced translation, the situation was reversed for Pirkola’s method.
Several factors might explain these differences:/ We found that #sum is an effective cross-bigram
operator. As Table 3 shows, use of #sum
as cross-bigram operator (with balanced translation) improves mean uninterpolated average
precision from 0.25 to 0.29 and this differenceis statistically significant. By contrast, thatuse of #OD0 as a cross-bigram operator (withthe method analogous to Pirkola’s) appeared
to improve mean uninterpolated average precision from 0.30 to 0.32, but that difference is
not statistically significant. Overall, these experiment results indicate that properly handlingpost-translation resegmentation could be help-/ The comparison between Pirkola’s method and7
6balanced translation that was done in the MEI
project was based on a preliminary system configuration, and time constraints precluded replication of that experiment using the final MEI
Available at http://morph.ldc.upenn.edu/Projects/Chinese/configuration. Our conclusion at MEI that those
two techniques performed about equally wellmust therefore be regarded as tentative.

for use in Chinese. We also found that it is better
to use all translations with Pirkola’s method rather
than limiting consideration to the three most common
ones. We were not able to clearly determine whether
post-translation resegmentation is helpful, in part because we have not yet explored the full range of possible cross-bigram operators. A diverse set of English/Chinese CLIR test collections are now available,
and we are interested in exploiting those resources tocontinue our exploration of the ideas introduced in thispaper.

/ Multiword expressions were translated in our
MEI experiments whenever the expression
could be found in our dictionary. Because oftime constraints, in ECIR we used word-byword translation instead. This almost certainly
resulted in a lower baseline and fewer multiword
translations. With fewer long translations, multiple bigrams may have been less common.

/ The TDT-2 and TDT-3 test collections are far

Acknowledgments

smaller than the ECIR test collection, and the
MEI queries were considerably longer. Together, these effects would seem to make the
ECIR a more challenging evaluation environment.

The authors would like to thank Fred Jelinek and hiscolleagues at Johns Hopkins for sponsoring the MEI
workshop that stimulated this work, the MEI teamfor developing the query translation software that wehave used in these experiments and for their pioneering work on post-translation resegmentation, KuangHua Chen for organizing the ECIR track at NTCIR2, and Gina Levow for her assistance with the corrected experiment runs. This work has been supported
in part by DARPA contract N6600197C8540 and and
DARPA cooperative agreement N660010028910, and
NSF grant IIS-00712125.

/ The test collections used in MEI included
speech recognition errors. This could tend to favor shorter indexing units such as character bigrams.

/ We attempted to translate every query term for
ECIR, but for MEI we performed pre-translation
stopword removal. This may tend to favor
Pirkola’s method at ECIR, since at least one
translation of an English stopword is likely to
be common.

References/ Exhaustive relevance judgment was done for the[1] L. Ballesteros and W. B. Croft. Resolving ambiguity for cross-language retrieval. In C. V. R.
W. Bruce Croft, Alistair Moffat, editor, Proceedingsof the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 64–71. ACM Press, Aug. 1998.
[2] D. A. Hull and G. Grefenstette. Querying acrosslanguages: A dictionary-based approach to multilingual information retrieval. In Proceedings of the
19th Annual International ACM SIGIR Conference
on Research and Development in Information Retrieval, 1996. http://www.xerox.fr/people/grenoble/
hull/papers/sigir96.ps.
[3] T. Leek, H. Jin, S. Sista, and R. Schwartz. The BBN
crosslingual topic detection and tracking system. In
Working Notes of the Third Topic Detection and Tracking Workshop, Feb. 2000. http://www.itl.nist.gov/iaui/
894.01/tests/tdt/tdt99/papers/index.htm.
[4] G.-A. Levow and D. W. Oard. Translingual topic
tracking with PRISE. In Working Notes of the Third
Topic Detection and Tracking Workshop, Feb. 2000.
http://www.glue.umd.edu/8 oard/research.html.
[5] H. Meng, B. Chen, E. Grams, W.-K. Lo, G.-A.
Levow, D. Oard, P. Schone, K. Tang, and J. Q.
Wang. Mandarin-English information (MEI): Investigating translingual speech retrieval. Technical report,
Johns Hopkins University, Baltimore, MD, Oct. 2000.
http://www.clsp.jhu.edu/ws2000/groups/mei/.
[6] D. W. Oard and A. R. Diekema. Cross-language information retrieval. In Annual Review of Information

TDT collections, but a pooled relevance assessment methodology was used for ECIR. Relevance judgments in TDT and ECIR were also
based on different criteria. A TDT audio story
was judged as relevant if it resulted from thesame event as the example story. A ECIR document was judged to be relevant if the subject ofthe document was the same as the subject specified in the topic description. Overall, we suspectthat TDT topics are likely to be somewhat finergrained than ECIR topics, but a careful comparison would be needed to substantiate this conjecture.

/ We used segmentation software from New Mexico State University (NMSU) for MEI. For
ECIR, we found that the LDC segmenter was
better able to handle the large collection. We
prefer to use the NMSU segmenter when possible because it includes specific provisions forproper segmentation of common proper names.

5 Conclusion
Our experiments indicate that Pirkola’s method forthe formulation of structured queries is well suited7

Science and Technology, volume 33. American Society for Information Science, 1998.
[7] D. W. Oard and J. Wang. Effects of term segmentation on Chinese/English cross-language information
retrieval. In Proceedings of the Symposium on String
Processing and Information Retrieval, Sept. 1999.
http://www.glue.umd.edu/8 oard/research.html.
[8] A. Pirkola. The effects of query structure and dictionary setups in dictionary-based cross-language information retrieval. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, pages 55–63,
Aug. 1998.
[9] R. Sperer and D. W. Oard. Structured translation
for cross-language IR. In N. J. Belkin, P. Ingwersen, and M.-K. Leong, editors, Proceedings of the
23rd Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval,
pages 120–127. ACM Press, July 1998.
[10] R. Wilkinson. Chinese document retrieval at TREC6.
In D. K. Harman, editor, The Sixth Text
REtrieval Conference (TREC-6). NIST, Nov. 1997.
http://trec.nist.gov/.

8