ICT-Crossn: The System of Cross-lingual Information Retrieval of ICTin NTCIR-7
Weihua Luo1,2 Tian Xia1,2 Ji Guo2,3 Qun Liu2
1 Graduate University of Chinese Academy of Sciences, Beijing 100190, China
2 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology (ICT),
Chinese Academy of Sciences, Beijing 100190, China
3 School of Software and Microelectronics, Peking University, Beijing 100871, China{luoweihua, xiatian, guoji, liuqun}@ict.ac.cn

Abstract
IR4QA is a new task in NTCIR-7, which intends toevaluate which IR techniques are more helpful to a
QA system. This paper describes in detail theimplementation of our IR4QA system, ICT-Crossn.
The system consists of a query translation
component that integrates the methods of phrase
based statistical machine translation and OOVtranslation methods based on search engine, and adocument retrieval component which combines
outputs of multiple IR models with a linear model.
We tune the parameters on the development set
constructed on the dry run set. The official
evaluation results show our method achieves agood performance.
Keywords cross-lingual information retrieval,question answering, statistical machine translation,model combination

1 Introduction
In 2008, Multilingual Interactive Technology
Laboratory of Institute of Computing Technology(MITEL as the team name) takes part in the NTCIR-7
evaluation for the first time. Compared with theprevious sessions, the setting of evaluation tasks in
NTCIR-7 has changed greatly. Since the organizers of
NTCIR think the research in CLIR (Cross-Lingual
Information Retrieval) has matured and wonder how IR
techniques help the QA research, CLIR is no longer
taken as an individual task to be evaluated. Instead,
IR4QA (Information Retrieval for Question Answering)
replaces the traditional CLIR evaluation and becomes a
subtask of ACLIA (Advanced Cross-Lingual
Information Access) task cluster. IR4QA evaluates theranked document lists generated by IR systems using
the traditional metrics such Mean Average Precision(AP), but the retrieval task is treated as a component of
the cross-lingual QA system[1]. That is, the test set forevaluation consists of various kinds of questions instead1of ad hoc queries, and the evaluation metrics are devisedin favor of the effectiveness of IR systems as a module
in QA systems.
We implement a CLIR system titled as “ICT-Crossn”.
Because of time limit, we have not developed a QA
system yet and only participate in the EN-CS (EnglishSimp. Chinese) and CT-CT (Trad. Chinese-Trad.
Chinese) subtasks of the IR4QA task.
ICT-Crossn consists of a query translation component
and a document retrieval component. When performinga CLIR task, e.g. reading the English queries andsearching in the Chinese corpus, ICT-Crossn firstly
translates the English queries into Chinese, and thensearches for the relevant documents in the Chinese
corpus. In the translation process, we use the output ofphrase-based statistical machine translation (SMT) andaccomplish the translation of OOV (out-of-vocabulary)words with search engines[2,3,4]; while in the retrieval
process, we combine the results generated by multiple
IR models. To tune the parameters of the system, weconstruct manually the development set based on thedry run set. The official evaluation results show ourmethod achieves good performance.
The remainder of the paper is organized as follows.
Section 2 describes the overall architecture of thesystem. Section 3 and 4 explain respectively our method
in query translation and document retrieval. In section 5,
the experiments on the development set and the results
are introduced. The official results in NTCIR-7 of oursystem are provided in section 6. Finally, section 7
presents our conclusion and the future work.

2 The system architecture
In the CLIR tasks of the previous sessions of NTCIR,
e.g. NTCIR-5 and NTCIR-6, each query consists of atitle which contains the keywords describing theinformation need briefly, a description which explains
the information need in detail, and a narrative which
supplies more details about the query, such as the
background of the topic and the judgement metric of
document relevance. However, for the test set of IR4QA
in NTCIR-7, each query is a complex question, such as

Algorithm 1. The algorithm of phrase
identification and translation

Figure 1. the overall architecture of ICT-Crossn(EN-CS Task)“What are stem cells”, “List major events in Saddam
Hussein's life”.
For the IR4QA task, there are two important
problems we should pay attention to. The first is that therequirement of translation is more stringent. Generally
the questions are very concise, it leads to wrong
understanding if 1 or 2 words in a question aretranslated improperly. The second is that the judgement
metric of document relevance in QA differs from theone in IR. We think, for QA, the document containing
the answer to the question is more important than the
one just related to the queries derived from the question.
For example, for the question “who is Junichiro
Koizumi”, a document containing the personal historyof Koizumi is more important than the one about ameeting he attended. Thus, the goal of our system is toimplement an effective method to translate the short whquestions or assertive sentences more accurately, and to
modify the current IR models to make them fit for QA
metrics.
We think the traditional CLIR framework is still
suitable for IR4QA because the input and output are
similar with a CLIR one. However, some successfulalgorithms for CLIR should be re-tested and modified
for the new problems in IR4QA. Based on theconsiderations above, we follow the traditional CLIR
methods that split the whole process into 2 steps. One is
responsible for translating the queries into the languageof the corpus. The second step performs ad hoc retrieval
according to the queries. Figure 1 shows the overallarchitecture of ICT-Crossn. As we see, the architecture
is fit for both monolingual and multilingual IR tasks
because the retrieval module doesn’t care how thequeries are made.
However, the specific techniques used in our system
are different from many CLIR systems. We don’t use
dictionary-based translation methods that are widelyused in many CLIR systems. Our system borrows some2

Input: an English question in the test set, the structural
template set T, the phrase table P, the constituent set D
Output: the Chinese key words of the question1) Initialize: D = Φ2) For each rule in T:
Extract the constituents ci from the question
with each template of T;
Add ci into D
End for
3) Remove the repeated items in D
4) For each item ci in D
If | ci | ≤ 3 thenTake ci as a basic phrase and search for thetranslation in the phrase table
If not found, pass on ci to the OOVtranslation
Else
Enumerate the sub-phrases in ci, which
occur in P or are left after others are
enumerated
Choose the combination C of subphrases which make up ci and minimize
the joining times
End If
For each sub-phrase si in C
If si is in P thenchoose the top 3 candidate translations
by the probability p(c|e)
Else
Pass on si to the OOV translation
End if
End forEnd fortechniques of SMT to improve the translation
performance, especially the translation of phrases.
While in retrieval, we are inspired by the idea of system
combination. Results from different IR systems areusually complementary, and achieve more steady but
better performance than a standalone system.
Before translation and retrieval, some preprocessing
operations are executed, including text segmentation,
encoding conversion, etc. We use ICTCLAS 1 toperform word segmentation on the simplified Chinese
documents. Unigram and bigram segmentation are also
performed on the corpus to build the indices. If thequery or the corpus is traditional Chinese, we convert all
texts to simplified Chinese at first, and then perform
word segmentation with ICTCLAS.
In the query translation module, we run the wordalignment toolkit — Giza++ 2 on the Chinese-English
1

ICTCLAS is an open source tool for simplified Chinese
word segmentation, which is developed by ICT and available
in http://www.nlp.org.cn/project/project.php?proj_id=6.
2
Giza++ is a training toolkit of statistical translation models
and word alignment. It is available in
http://www.fjoch.com/GIZA++.html.

parallel corpus and then run our phrase extraction tool
to extract bilingual phrase pairs. The phrase table
generated on a large-scale corpus (e.g. Gigaword
released by LDC) covers most common phrases. Take
the EN-CS subtask of IR4QA as an example. ICTCrossn searches for the available Chinese translations
corresponding to phrases in an English question in thephrase table, sorts the candidates by probabilities andchooses the top ones. If some have no translations in thetable, ICT-Crossn will seek for the help of searchengines. It constructs the queries based on the English
phrases, submits them to a search engine to obtain the
Chinese web pages, and then extracts the possible
translations by statistic information.
In the document retrieval module, we use Lemur3 tobuild our IR system. Furthermore, we build 3 kinds ofIR systems by altering the index unit, i.e. unigram,bigram and word. Each system uses pseudo relevance
feedback based language model and provides a different
ranked document list in most cases. Then we combine
all lists into the final result with a linear function.
The following sections will describe the algorithms
and techniques of our system in detail respectively.

Algorithm 2. Parameter tuning algorithm

3 Module of question translationphrase translation module enumerates the phrases in aquestion and searches for their translations in the
dictionary. However, quite a lot of words and phrases
cannot be identified and translated properly for thedictionary has poor coverage. In addition, dictionaries
don’t provide possibilities to rank the candidates. We
also tried a Stanford parser to recognize the constituents
we need in a question. Although it works well for longsentences, it introduces many errors for short ones. As aresult, we try another way. At first, we extract thephrases and words by the specific structures in thesentences. Then we translate them with a phrase table.
The phrase table we use is trained on 5 million
parallel sentence pairs which is constructed on the
English-Chinese corpus released by LDC for NIST MT
evaluation. The training process is to use Giza++ toalign words on the corpus, and then to use our phrase
extraction tool to extract the parallel phrase pairs and to
estimate the alignment probabilities[5]. The phrase table
occupies up to 17GB in disk. In evaluation, to reduce
the size of the table, we filter it with the test set. So only
the phrases existing in the test set are kept in the table.
In the test set of the dry run and the formal run, it iseasily observed that most questions follow a few fixed
structures. For instance, “Who is Osama Bin Laden?”,“Who is Ronaldo?”, “List the hazards of global
warming.”, “List the relationship between inflation andthe economy.”, etc. Roughly, four types of question are
defined in four kinds of structure. Therefore, wepredefine some structural templates to capture theconstituents we need in the questions. For example, atemplate we define is “what is X” and X is the

3.1 Overview
We don’t intend to perform a full text translation on
the questions with a SMT system, because the question
is usually short and it is difficult for the SMT system totranslate it well without context. Additionally, we think
IR4QA prefers more relevant key words to the besttranslation, and is not sensitive to order of the words.
Therefore, our goal is to translate the key words in thequestion instead of the whole sentence. Our work
focuses on phrase translation because we think words insentences are ambiguous but phrases are more clearly
defined. And translating multiple words as a phrase
introduces much less candidates than translating thesewords separately. For IR4QA, the critical information in
the question is often phrases. If not translated well, we
will retrieve the irrelevant documents. Therefore, how
to recognize a phrase and how to translate themcorrectly are the crucial problems we face.

3.2 Phrase translation
In the beginning, we compile LDC dictionaries,
electronic version of traditional dictionaries and someterminology dictionaries into a huge dictionary. The
3

Lemur is a famous open-source toolkit of language modelingand information retrieval. Its official web site is
http://www.lemurproject.org/.

3

Input : model set Μ ={M 1 , M 2 , ... M n }Output : parameter set Λ = {α1 , α 2 , ..., α n }Let D is the model set with fixed parameter in the
combination.
Let maxMAP to be the best MAP score.
Step 1 : Initialize D = ∅ , max MAP = 0
Step 2:(1) for ∀ < M i , M j > in model set M where i ≠ jtune their parameter <α i , α j > to get the best
MAP score MAPi , j(2) Fix the parameter of the model pair < M f , M s >with the best MAP score. Remove M f , M s frommodel set M and append them into D. Add theparameters of M f , M s into Λ
Step 3 :while M ≠ ∅(1) for each model M k in M
tune its parameter along with the models in D
and their fixed parameter to get the best MAP
score MAPk
(2) Fix the parameter of the model M l with the best
MAP score of the combination model. Remove
M l from M and append it into D.
Add its parameter into Λ.

Figure 2. The results of different index units
and feedback document number
Word

Unigram

8 10 15 20 25 30

feedback doccument number

constituent we want. However, this kind of template hasa rough granularity. E.g. for a question like “what isThree Gorges Dam in China”, we can extract “Three
Gorges Dam in China”, but it’s a compound phrase
consisting of shorter phrases and words and has notranslations in the phrase table. Thus, we divide thecompound phrase into “basic” ones with the help of thephrase table.
Algorithm 1 illustrates the algorithm of phrase
identification and translation. Here we think a phrase
consisting of less 4 words is “basic”. Our idea is to findthe most spanning sub-phrases both in the question andin the phrase table. For example, for the query “what ishuman genetic sequence determination”, “human
genetic sequence determination” is extracted with thetemplate. And then we find 6 sub-phrases in the phrase
table,
i.e.
“human”,“genetic”,“sequence”,“determination”, “human genetic sequence” and“genetic sequence”. However, “human genetic
sequence” and “determination” make up the wholeconstituent and have the minimal joining times. So the
combination is what we want. In order to provide more
relevant key words, the system is designed to output
multiple candidates. We rank the candidates by p(c|e),the probability to translate an English phrase e into a
Chinese one c.
If the phrase/word is a particle after enumeration or it
occurs in many questions, we will remove it from theresult because we think they are useless for retrieval. In

Word

20
0

16
0

12
0

40
2 3 4 5 6 7

Bigram

0.36
0.345
0.33
0.315
0.3
0.285
0.27
90

0.36
0.345
0.33
0.315
0.3
0.285
0.27

60

Bigram

MAP

MAP

UnigramFigure 3. The results of different index units andfeedback term countfeedback term countaddition, we find if the first letter of each word in aword sequence is capitalized, the word sequence islikely to be a proper noun, e.g., “Yasukuni Shrine”,“Edmund Ho Hau Wah”, “Balkan Syndrome”, etc. Thus,
we take the phrase matching the rule as a whole subphrase and search for its translations.
Through experiments, we find that the CLIR systemachieves the optimal performance when 3 translations
with highest probability for each sub-phrase are given.
In order to make each key word equal, we reproduce thetranslation of a phrase three times if it has less than
three candidates. We also conduct a experiment on the
NTCIR-5 test set, which only outputs the besttranslation of the whole phrase, but find it works badly.
We think it is because the SMT system aims to translate
phrase more precise while CLIR need more relevanttranslations to indicate the information need.

3.3 OOV words translation

In our system, an OOV word is defined as the one notappearing in the phrase table. Our OOV translation
algorithm is based on search engines combining withartificial rules and statistics measures.
At first, we construct an English query for Chinese
pages based on the OOV word and submit it to a searchengine, e.g. BAIDU (http://www.baidu.com). To maketrade-off between the relevance and the coverage, wecollect the top 200 snippets returned by the searchengine. A snippet is the abstract of a page extracted by
the search engine, which often includes a title and a
Table 1. The results of different IR models and index units for Chinese monolingual IR on the NTCIR-5 testset
IR modeltf_idffb_tf_idf
okapi
fb_okapi
kl_jm
kl_dirkl_absmixfb_kl_dir

4Unigram
0.24970.27240.26090.2931
0.23690.30310.30450.3443T run
Bigram0.26640.31620.26460.30050.2366
0.26150.25190.2860Word
0.25150.29500.25820.32770.24990.2706
0.25780.3057

Mean Average Precision
D run
UnigramBigramWord
0.21640.24650.20520.25690.30880.26080.20650.23930.17690.1946
0.23450.31300.22180.21510.18510.25880.22700.20130.23020.22170.18230.29830.25130.2933

Unigram
0.27160.28440.28350.16780.29160.31540.31830.3343

N run
Bigram0.3032
0.33620.31720.33900.29700.30190.30130.3378Word
0.27290.28370.27370.27730.26850.27900.26240.2992

short text near the key words. After duplicated snippets
deletion and word segmentation, we extract the possible
translations within a predefined window. The words
occurring around the keywords in less than 20 words
distance are considered as the candidate translations.
We think many factors determine if a Chinese word
is the translation of an English key word. Therefore, we
use a linear feature function to capture the idea:(1)
score(c, e) = ωi f i (c, e)∑iin which f i (c, e) is the feature function and ωi is theweight of the function.
We find the candidates appearing directly behind the
key words with a pair of parenthesis are more likely tobe the right translation. This feature is considered with ahigh priority and the feature function f (c, e) is anindicator one equal to 0 or 1. Another important featureis the similarity between the transliteration of acandidate and the keyword. Each word is transliterated
according to Chinese Pinyin and compared with theEnglish word. We take the longest common sub-string
length as feature value. And of course, the cooccurrence probability of the key words and the Chinese
word is an important feature, too. All weights are tuned
manually in a heuristic and greedy way. Initially, weassign greater weights to some important features. Thenwe fix them and tune the weights for other features.
Following the way in the phrase translation, we choose
the top 3 candidates to increase the coverage.

5

document for a query is a summed interpolation valueof different models as follows:(2)score( d , q ) = ∑ α j score j ( d , q )jwhere scorej(d, q) is the original score assigned by an
IR model and α j is the weight of the model.
In ICT-Crossn, we prefer index models to retrieval
models to construct our combination model. There are
two reasons for our choice:- Various retrieval models often generate very
different document lists which are hard to be combined.
On the contrary, the similar models with various index
units often generate similar lists with minor difference,
which makes the combination easy to carry out.
- Index model combination is simpler than retrieval
model. Various retrieval models have many parameterswhich influence the performance. But not all parameters
are in the same level, so it is hard to adjust them in a
linear function. However, for index model combination,
we can adopt the same IR model but different index
units. Thus, the parameters for each model can be tuned
within a linear formula.
To tune the parameters, we define the object function
in formula 2 as follows:
n

n

i =1

i =1

δ (U Modeli ) = MAP (U Modeli )(3)nhere U Modeli is the set of different models. The goal is
i =1

4 Module of document retrieval

to find the parameter setting maximizing the MAP scoreof the combination model. The algorithm is used as
Algorithm 2.

4.1 Overview

4.2 IR model selection

In NTCIR-7, we investigate the effect of traditional
ad hoc IR methods in the IR4QA task. And then we
propose a linear combination method based on different
index models. A greedy search algorithm is used to tune
the parameter setting to achieve the locally optimal
performance. A lot of experiments on different data set
are conducted to adjust the parameters of the system.
There are two categories of models in information
retrieval. One is retrieval model and the other is index
model. Retrieval model includes Boolean model, vector
space model, probabilistic model, language model etc[6,7, 8, 9]. Index model is specified by the index term, e.g.
unigram, bigram, word, etc. Previous studies show
different models have different performances on
different IR tasks[10]. However, we think results ofvarious models are complementary for each other. If we
combine all results into one in a proper way, the final
result should have a better recall and precision.
Thus, motivated by the idea of system combinationin SMT, we design a combination model for a robustand better IR performance. The final score of a

In order to choose the optimal IR model that fits for
the NTCIR data set, we conduct a series of experimentson the NTCIR-5 test set. Table 1 illustrates our
experiments on the NTCIR-5 C-C tasks. The IR system
we use is Lemur with default settings. We choose 8
candidates: tf_idf (simple tfidf model), fb_tfidf (tfidf
model with feedback), okapi (simple okapi model),fb_okapi (okapi model with feedback), kl_jm (simple
language model with JM smoothing), kl_dir(simple
language model with Dirrichlet smoothing), kl_abs
Table 2. Development set in NTCIR-7
Task
EN-CS

CT-CT

Question Type
Event
Relationship
Biography
Definition
Event
Relationship
Biography
Definition

Question number
14
10
21
33
8
10
24
29

(simple language model with Absolute Discounting
smoothing), mixfb_kl_dir (language model, Dirrichletsmoothing, with feedback). We conduct three groupexperiments. The first group uses the title of each topic(T run). The second group only uses the description (Drun) and the third uses the narrative (N run). Table 1
shows that the simple language modeling, with Dirichletsmoothing and feedback, outperforms other models.
Based on the results, we think the last model(mixfb_kl_dir) is the best for our task.

Figure 4. The results of index models
combination in EN-CS task for dry run

4.4 Index unit combination
Previous studies have shown that index unit
combination can improve IR performance using
unigram and bigram[11]. We further investigate possiblecombination of different index units including unigram(U), bigram (B) and word (W) because we think words
in Chinese are important meaning units.
Two questions arise when combining different index
units. The first is that the scores of a document assigned
by various index models are not theoreticallycomparable and therefore should not be summed
directly. We normalize the score of each document in
the returned list with the highest score, i.e. the score of
the top document. Then all scores fall into the range[0,1]. The second is that when the combination shouldbe performed. There are three strategies: (1) Each index
model completes its initial retrieval and the system
combines their scores before feedback; (2) Each index
model finishes retrieval and feedback respectively andthen the system combines their final score; (3) Eachindex model finishes its initial retrieval, and then thesystem combines their scores before feedback. After
feedback, the system combines all scores again to get
the final results. Through the experiments, we find thesecond works better than the others.

5 Experiments on the dry-run set
5.1 Development set construction
The test set in IR4QA differs from that in previous
collections. Thereby, the metric of performance of IR
systems has changed. So we don’t expect the parameter6

0.5U+0.5W+B

0.
95

0.
8

0.
65

0.
5

0.
35

The mixfb_kl_dir model has parameters of itself andthe default setting may not be the best one. We adjust
the number of feedback document and the count offeedback term in the update stage of the model to get
the locally optimized performance for different indexing
strategies. Figure 2 and Figure 3 illustrate the IR
performance with different feedback document number
and term count for each indexing strategy on the
NTCIR-5 C-C task using only the title of each topic. In
Figure 2, the count of feedback term is set to 100.

W+B

0.22
0.215
0.21
0.205
0.2
0.195
0.19
0.185
0.18
0.
05

4.3 Parameter optimization

U+W

0.
2

MAP

U+B

setting that works well in NTCIR-5 is still useful in
NTCIR-7. Parameters should be adjusted on the test set
similar with the official set. Fortunately, the organizers
released the dry run set before the formal run. We
construct our development set both for EN-CS and CTCT tasks based on the dry run set. We download the dryrun set and the answers through the EPAN4 interface.
For each question, documents that contain an answer aremarked as relevant. Our EN-CS development set
contains 78 topics and CT-CT set contains 71 topics.
Table 2 gives a detailed illustration about ourdevelopment set.

5.2 EN-CS
retrievalcross-language

information

We choose language model with Dirrichlet smoothing
and feedback as our primary retrieval model. Unigram,bigram and word indices are created separately. The
parameter setting is tuned using the methods introduced
in section 4.1. The MAP score is 0.1847 on the unigram
index, with 2 feedback documents and 180 feedback
terms. The MAP score reaches 0.2018 on bigram index,
with 3 feedback documents and 160 feedback terms,
and the MAP score gets 0.1955 on word index, with 9
feedback documents and 80 feedback terms.
Figure 4 shows the combination results of two index
models. The best performance is shown in table 3.
- U+B: the highest MAP 0.2131 when 0.45U+0.55B.
- U+W: the highest MAP 0.2146 when 0.5U+0.5W.
- W+B: the highest MAP 0.2079 when 0.35W+0.65B.
Then we fix the weight of unigram and word in theinterpolation, adjust the weight of bigram for some steps
and get the best performance with MAP score of 0.2168.

5.3 CT-CT
retrievalmonolingual

information

We convert all the documents and topics in CT to CS
because Lemur and ICTCLAS only recognize Chinese4

http://aclia.lti.cs.cmu.edu:8080/epan/index.jsp

6 Results on NTCIR-7 evaluation
We submit 5 runs for EN-CS task and 4 runs for CTCT task. They are listed as following:- MITEL-EN-CS-01-T: our primary system for ENCS task, combined in the scheme of 0.5U+0.5W+0.15B,
which outperforms others in the dry run set and uses
only the question part of each topic.
- MITEL-EN-CS-02-T: the combination scheme is
0.5U+0.5W, using the question part of each question.
- MITEL-EN-CS-03-T: the combination scheme is
0.45U+0.55B, using the question part of each question.
- MITEL-EN-CS-04-D: the combination scheme isthe same as primary system, but only using the narrative
part of each topic.
- MITEL-EN-CS-05-TD: the same as the primarysystem, except using both question and narrative part ofeach topic.
- MITEL-CT-CT-01-T: our primary system for CTCT task. The combination scheme is 0.3U+0.7B, using
the question part of each topic.
- MITEL-CT-CT-02-T: the combination scheme is
0.3U+0.7B+0.05W, using the question part. We use it as
our secondary system because of the poor performanceof word index.
- MITEL-CT-CT-03-D: the same as primary system
except using the narrative part.
- MITEL-CT-CT-04-T: only use bigram index and

Figure 5. The results of index model combinationin CT-CT task for dry run
U+B

U+W

B+W

0.3U+0.7B+W

0.
8

0.
65

0.
5

0.
35

0.
2

0.32
0.31
0.3
0.29
0.28
0.27
0.26
0.25
0.24
0.23
0.22
0.21
0.
05

MAP

characters with GB encoding. The process of CT-CT IR
is the same as the retrieval process of EN-CS CLIR task.
The MAP score is 0.2722 on unigram index, with 3
feedback documents and 10 feedback terms. The scoreof bigram is 0.3077 when 6 feedback documents and 10
feedback terms are used. The MAP score is only 0.2057
on word index, with 6 feedback documents and 80
feedback terms. The reason for the poor performance ofword index is that the encoding conversion is based on
Chinese character but the words for many objects aredifferent in Chinese mainland and Taiwan, especiallyproper nouns. For example, Bush is always translated
into “布什” in simplified Chinese but “布希” is often
used in Taiwan and Hong Kong. So there are many
word segmentation errors in the word index and thus theperformance is degraded. The results of index unit
combination are shown in Figure 5.

the question part.
Table 3 shows the performance of each run of ICTCrossn on the IR4QA subtasks in NTCIR-7. Our runs
achieve a good performance in both subtasks. In the
EN-CS CLIR subtask, MITEL-EN-CS-03-T works welleven when compared to the monolingual runs. In CTCT task, our runs rank top 4.
According to the official report of NTCIR-7[12], ICTCrossn has the highest coverage over relevant
documents. Our system also has a high recall on the dryrun set, but the precision is not satisfying. We think thereason lies in the difference between the metric ofperformance of traditional IR and IR4QA. For example,
given a question like “Who is Hu Jintao”, the user wants
to know some details about Hu. In the dry run set, a
document that introduces the biography of Hu is thebest answer. Even the key word “Hu Jintao” only
appears once. However, The documents about his visit
to Japan are not listed in the answer, but it ranks high in
a traditional IR system because “Hu Jintao” appears
many times. In other words, the traditional IR methods
are not fully suitable in IR for QA tasks.

7 Conclusion and future work
In NTCIR-7, IR4QA is set up as an evaluation task
for the first time. Although it is obvious that thetraditional CLIR methods are not fully suitable for the
new IR4QA test, they are still the most straightforward
scheme to deal with the problem.

Table 3. The performance based on real qrels in the NTCIR-7 formal result
Task
EN-CS

CT-CT

7

Run idMITEL-EN-CS-01-T
MITEL-EN-CS-02-T
MITEL-EN-CS-03-T
MITEL-EN-CS-04-D
MITEL-EN-CS-05-TD
MITEL-CT-CT-01-T
MITEL-CT-CT-02-T
MITEL-CT-CT-03-D
MITEL-CT-CT-04-T

Mean AP0.58490.56930.59590.57890.58980.5791
0.58390.58390.5645

Mean Q
0.60050.58580.61240.59500.60580.59630.60180.60130.5783

Mean nDCG
0.79490.78470.79470.79070.80030.78350.78730.78690.7648

ICT-Crossn is based on the traditional CLIR
framework. First, the system translates the questions
into the target language of the corpus with a phrase table
generated by a SMT system and an OOV wordtranslation module based on search engines. Second, thesystem performs information retrieval on the corpus
with different index units, and combines the document
lists into the final result. To obtain the optimal models
and the parameters, we construct the development set
based on the dry run set. The results on the official set
show our system achieves a good performance.
However, there are still some drawbacks with our
methods. Firstly, our translation module hasn’t yet
solved the translation of OOV words well. E.g., oursystem cannot identify the phrase “natural gas hydrates”
because it neither appears in the phrase table nor has
distinct features, which results in the impropertranslations and the bad performance of retrieval.
Secondly, since the judgment metric of document
relevance of IR4QA is different from the one of CLIR,
the order of the document list should be different fromthe one of a CLIR system. We attempt to add re-ranking
techniques similar with the one of I2R in the document
retrieval module to bring forward the documents
containing the answers[13]. However, the performanceon the development set is not satisfying. Therefore, weplan to improve the method of phrase identification andthe method filtering the candidate translations extracted
from the web pages returned by search engine. In
addition, we try to explore more effective techniques ofre-ranking to increase the precision of our system in the
IR4QA task.

Acknowledgments
The work is supported by Hi-tech research anddevelopment program of China (863 program), Contact
No. 2007AA01Z438 and Contact No. 2006AA010108.
We would thank to Dr. Zhongjun He for his help.

References[1]Teruko Mitamura, Eric Nyberg, Hideki Shima, Tsuneaki
Kato, et al. Overview of the NTCIR-7 ACLIA: Advanced
Cross-Lingual Information Access. To appear in theproceedings of NTCIR-7, 2008.
[2]Ying Zhang, Phil Vines. Using the Web for AutomatedTranslation Extraction in Cross-Language Information
Retrieval. In the Proceedings of the 27th annualinternational ACM SIGIR conference on Research anddevelopment in information retrieval, 2004, pp.162-169.
[3]Gaolin Fang, Hao Yu, and Fumihito Nishino. ChineseEnglish Term Translation Mining Based on Semantic
Prediction. In Annual Meeting of the ACL Proceedings of
the COLING/ACL on Main conference poster sessions,
2006, pp.199 – 206.

8[4]Franz Josef Och, Hermann Ney. The Alignment Template
Approach to Statistical Machine Translation. In
Computational Linguistics, Volume 30, Issue 4, 2008,
pp.417 - 449[5]Chengye Lu, Yue Xu Shlomo Geva. Translation
disambiguation in web-based translation extraction forEnglish-Chinese CLIR. In Symposium on Applied
Computing Proceedings of the 2007 ACM symposium on
Applied computing, 2007, pp.819- 823.
[6]J. Lafferty and C.X. Zhai. Document Language Models,
Query Models, and Risk Minimization for Information
Retrieval. In the Proceedings of ACM SIGIR, 2001,
pp.111-119.
[7]J. Ponte and W.B. Croft. A language modeling approach to
information retrieval. In the Proceedings of ACM SIGIR,
1998, pp.275-281.
[8]C.X. Zhai and J. Lafferty. A Study of Smoothing Methods
for Language models Applied to Ad Hoc Information
Retrieval. In the Proceedings of ACM SIGIR, 2001,
pp.334-342.
[9]C.X. Zhai and J. Lafferty. Model-based Feedback in the
Language Modeling Approach to Information Retrieval. In
the Proceedings of the tenth international conference on
Information and knowledge management, 2001, pp.403410.
[10]J.Y. Nie., J. Gao, J. Zhang and M. Zhou. On the use ofwords and n-grams for Chinese information retrieval. In theFifth International Workshop on Information Retrieval
with Asian Languages, IRAL-2000, 2000, pp.141-148.
[11]L.X. Shi, J.Y. Nie. Using Unigram and Bigram Language
Models for Monolingual and Cross-Language IR. In theProceedings of NTCIR-6 Workshop Meeting, 2007, Tokyo,
Japan, pp.20-25.
[12]Tetsuya Sakai, Noriko Kando, Chuan-Jie Lin, Teruko
Mitamura. Overview of the NTCIR-7 ACLIA IR4QA
Subtask. To appear in the proceedings of NTCIR-7, 2008.
[13]L.P. Yang and D.H. Ji. I2R at NTCIR5. In theProceedings of NTCIR-5 Workshop Meeting, 2005, Tokyo,
Japan.