Proceedings of the Third NTCIR Workshop

Study on Merging Multiple Results from Information Retrieval System
Hiromi itoh OZAKU , Masao UTIYAMA , Hitoshi ISAHARA
Communications Research Laboratory
2-2-2 Hikaridai,
 Seika-cho, Soraku-gun, Kyoto, 619-0289, Japanromi, mutiyama, isahara  @crl.go.jp


Yasuyuki KONO , Masatsugu KIDODENara Institute of Science and Technology
8916-5 Takayama-cho,Ikoma-city, Nara, 630-0101, Japankono, kidode  @is.aist-nara.ac.jp

Abstract
We participated in Web retrieval tasks in order toinvestigate the integration of multiple retrieval results.
We also examined the possibility of using a general retrieval system designed for organized documents, suchas news articles that match users’ queries, to retrieve
unorganized documents, such as Web documents.
Our results show that if information retrieval systems use the OKAPI method, results can be used for
Web retrieval that is based on scored results without
score normalization. On the other hand, if the systems use Rocchio’s feedback method, ranked results
are shown to be the best.
Keywords: OKAPI, Rocchio’s formula, Web retrieval, meta-search

1 Introduction
In general, information retrieval systems use one
index file made by each system in order to calculate
the similarity between the data and the queries. It is
possible to generate one optimal index file by using
documents from the whole Web in the Internet. However, the number of documents on the Internet has increased dramatically. Collecting and managing Web
documents with one machine has become much moredifficult therefore multiple systems have been used in
retrieving Web documents.
The multiple systems are called meta-search systems. The conventional meta-search systems needto combine the returned lists of pages from different
search engines. At that time, the systems either directly show the results with the name of the specificsearch engine or present new results that have been recalculated by using various normalizing methods [7][2].
© 2003 National Institute of Informatics

Generally, score normalization methods are
adopted by the meta-search systems. However, thenormalization process is considered to be a timeconsuming task and is sometimes unrealistic. Ourinterest is to find efficient methods that do not use thescore normalization process, in order to retrieve Web
documents rapidly.
We examined three merging methods without scorenormalization and one normalizing method in Web
tasks. The merging methods depend on the outputs
ranking from the information retrieval systems, using both scores and ranking, or only scores. Results
showed that using the OKAPI search method without automatic feedback, could be efficient for Web retrieval without a score normalization process. In addition, our scored results showed an equivalent levelto the score normalization results. On the other hand,
Rocchio‘s feedback method showed the best-ranked
results.
In this paper, we discuss the merging methods andexamine the results we obtained by submitting Web
tasks at NTCIR3. In our approach, the key points are
not only the precision of each search engine among
the multiple systems, but also the way to merge resultsfrom each system in order to produce the best overall
results.

2 Information Retrieval System
We used an IR package [8] 1 which is opened tothe public as a general retrieval system. This package is considered as a preprocessing tool for natural
language processing tasks, as well as an information
retrieval system. It has two functions: the first is to retrieve documents that match a query, while the second
1 IR package can be downloaded on http://www.crl.go.jp/jt/a132/members/mutiyama/software.html. We used the IR package
version 1.47.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

is to collect statistics on the occurrence frequencies ofcertain words. In this paper, we focus on the retrieval
function.

1      !   ! 
 
 
   !  QR   3! 

2.1 Scoring Function
The IR package uses the Okapi BM25 function [6],which is as a probabilistic-type function. When aquery is given, the score of a certain document is calculated using the following equation:(1)(

Using Rocchio’s feedback, and both have val*
!  in equation (3) beues of in equation (2). Then
comes

where

EPO

O

ISTUWV 1!  U(3)!  Q is the occurrence frequency of a term

corresponding to the initial query.
!  U is the occurrence frequency of a term
contained in the documents ranked amongthe
1 top X of the retrieval documents.
is the number of documents utilized forautomatic
feedback. The default value is
1

-

where
is a query, containing terms " , andis #
the weight of " in the query as expressedin the following equation:

isE a constant set according to previous re*,+.- .
sults,
with a default value ofO) ( *,+.-0/ 2134( 5*6+ -)7839( *,+.-0/ :;3978391  ( 5*6+ -:For the Web retrieval task, we submitted the resultsof the OKAPI retrieval, called the normal method (N )and the Rocchio’s feedback retrieval, called the feedback method (FB).

$&%'

(2)

is the number of documents in theHere,collection;
7
1 is the number of documents containing " ;is the number of documents known to be
relevant( to ;and is the number of documents relevant
to and containing " . 2 . 
Returning <  3>= ?to equation=5@0A  (1), is expressed as =  BDC @0A .
, andare constants set according toprevious results. Thesewere constants=  set andwith default value of
***
FE
E
E
GIH .
is the length of  as a substitute for thenumberof terms contained in  .
JIK GH is the average document length in thecollection.
 is the occurrence frequency of " in  .
!  is the occurrence frequency ofFinally," in .
The retrieval results are ordered by the documentscore based on equation (1).

2.2 Automatic Feedback
Equation (1) implements a retrieval function which
utilizes a feedback process as adapted in equation (2).
In the IR package, another feedback retrieval function
which based on a type called Rocchio’s feedback[3] isimplemented.

2 L and M are fixed by either the user or the IR system’s estimation. In this package, L and M were initially taken as N for convenience.

3 Hardware
We used a Linux system at the open laboratory of
the National Institute of Informatics (NII) to examine
all the processes within our tasks. The Linux machine
had the following characteristics: two 1-GHz Pentium
III CPUs, 1 GB of memory, and 1.2 GB of swap. The
Linux OS was Red Hat 7.1.

4 Web Retrieval Process
We submitted our results to the following tasks:
A1 Topic Retrieval, A2 Similarity Retrieval, and B2Target Retrieval. Two document collections of 100
Gbytes (LARGE) and 10 GB (SMALL) were used,with both raw and cooked data provided by NII. The
cooked data were pre-converted to EUC code, andnon-alphanumeric data such as graphics, movies, andthe like were eliminated. The resulting size of the
LARGE cooked data was approximately 39 GB, andthe size of the SMALL cooked data was about 4.8 GB.
The number of documents in the LARGE collection
was 11,038,720, while the number in the SMALL collection was 1,445,466.
Our process in using the IR package is listed as follows:
Y
Transform the cooked data to formatted data by

Y using the IR package.

Generate index files from the resulting formatted
data by using the ‘irbuilder‘ command of the IR
package.

Proceedings of the Third NTCIR Workshop<NW:DOC><NW:META><NW:DOCID>document ID</NW:DOCID><NW:URL>URL of the page</NW:URL><NW:DATE>gathered date</NW:DATE>...
</NW:META><NW:DATA><NW:DSIZE>data bytes</NW:DSIZE>
page contents</NW:DATA></NW:DOC>
Figure 1. NTCIR cooked data format

warning: NW001696888 of 011965: characters in a line over 25000(32000)
warning: NW001697123 of 011965 is over 1,000,000 (1017805)
warning: NW001607100 of 011195: characters in a line over 25000(517807)
warning: NW001966687 of 013887: characters in a line over 25000(46867)
warning: NW002103675 of 014826: characters in a line over 25000(517807)
warning: NW002707583 of 019598: characters in a line over 25000(30337)
warning: NW002361678 of 016731 is over 1,000,000 (4675648)
warning: NW002475775 of 017593 is over 1,000,000 (2079216)
warning: NW001918225 of 013585 is over 1,000,000 (1687263)
warning: NW002180204 of 015447 is over 1,000,000 (1012667)
warning: NW003174089 of 022989 is over 1,000,000 (1664762)
warning: NW003784469 of 027129: characters in a line over 25000(84316)
warning: NW003784470 of 027129: characters in a line over 25000(84316)
warning: NW003991386 of 028573: characters in a line over 25000(98940)
warning: NW003991400 of 028573: characters in a line over 25000(27275)<DOC NAME=’’document ID’’><TITLE>document ID’’</TITLE><DATE>gathered date<DATE><TEXT>
page contents</TEXT></DOC>Y

Figure 2. IR package format

warning: NW003991694 of 028573: characters in a line over 25000(27275)
warning: NW000224079 of 001359 is over 1,000,000 (1175805)
warning: NW001137337 of 008309: characters in a line over 25000(26080)
warning: NW001175481 of 008581: characters in a line over 25000(69387)
warning: NW006942200 of 045893: characters in a line over 25000(30962)
warning: NW008601480 of 055641 is over 1,000,000 (1869815)
warning: NW010462950 of 068046 is over 1,000,000 (1540714)
warning: NW010768429 of 070084: characters in a line over 25000(26758)
warning: NW010768432 of 070084: characters in a line over 25000(26758)

Y Retrieve results for all queries of the Web tasks.
Transform the results to NTCIR form.

warning: NW011177333 of 072867: characters in a line over 25000(60480)
warning: NW011594459 of 075256: characters in a line over 25000(27858)
warning: NW013419684 of 088115: characters in a line over 25000(47181)
warning: NW014794137 of 097628: characters in a line over 25000(31028)

Each process had some problems. In the following
sections we discuss each process in detail and explain
our proposed solutions to each problem.

warning: NW014401503 of 094677: characters in a line over 25000(44163)
warning: NW014896431 of 098493: characters in a line over 25000(37010)
warning: NW014896431 of 098493: characters in a line over 25000(72044)
warning: NW014896431 of 098493: characters in a line over 25000(51858)

4.1 Transforming the Data
A format of the cooked data is shown in Figure 1.
The cooked data contains Web machine information in
the META fields, and html file data in the DATA fields.
The html file data were pre-converted to EUC code,
non-alphanumeric data such as graphics, movies, were
eliminated.
The page contents in the cooked data included
many space codes and control codes, which we removed, except for Kanji codes. We then transformed
the NTCIR cooked data into the IR package format, asshown in Figure 2. The IR data has the same document ID as the cooked data in DOC NAME fields andtext information in TEXT fields that were transformedby eliminating html tags that did not have related contents, for example font size information, table line information and the like.
In this step, many errors were generated. We did
not analyze all the errors completely, but we considered them as related to the existence of files with incorrectly converted code or file-size limitations that were
processed with the IR package. These errors mightalso be related to hardware limitations.
We next eliminated cooked files and documents according to the following criteria:

Figure 3. List of files eliminated from the
SMALL CollectionY
Y Cooked data files of more than 10 MB;Y

documents with over 25,000 characters in one
line;
and documents of more than 1 MB.

Ultimately, we eliminated 32 documents, as shownin Fig. 3, and three cooked files (000046, 008792,
096956) from the SMALL collection.
From the LARGE collection, 430 documents(NW001696888, etc.) and 170 cooked files (097775,
etc.) were eliminated. This reduced the size of the
SMALL collection to approximately 2.4 GB, and thesize of the LARGE collection to about 21 GB. Thus,
the number of documents in the SMALL collection became 99% of the number of documents in the NTCIR
produced data, while the number of LARGE collection
documents became 94% of the number of documentsin the NTCIR produced data.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

4.2 Generating Indexes
In this step, we encountered one problem related tothe size of the index file.
In general, information retrieval systems use one index file that is made by each system, in order to calculate the similarity between the data and the queries. If
information retrieval systems can create one index fileby using the whole collection of documents in the Internet, it could be said that the index file is the optimal
index file.
However, it is difficult to manage huge documentswith just one index file due to the limitation of the IR
package. On the other hand, search engines for Web
documents currently employ multiple machines, in order to cover a wider range of the Internet or searchmore efficiently. The multiple machines are dispersed
at some places for load sharing, trouble prevention andthe like. In this situation, it is difficult to make one index file and share the one index file with the multiple
machines that are established at different places. We
should merge multiple outputs from each machine andmake one result.
Assuming a distributive situation, we set the input
file size to 500 MB for the SMALL collection and 1
GB for the LARGE collection. We then generated fiveindex files for the SMALL collection and 21 index files
for the LARGE collection.
Index files were not generated when the line lengthof the input file was too long. We set a limitation of
8,000 terms for one line length and eliminated documents with line lengths larger than this limitation.
As a result, 129 documents were eliminated fromthe SMALL collection, while 494,687 documents
were eliminated from the LARGE collection 3 . Ultimately, we had 1,445,267 documents in the SMALL
collection and 9,902,195 documents in the LARGE
collection. The number of documents in the LARGE
collection decreased to 89% of the number in the NTCIR produced data.

4.3 Retrieving Results for All Queries
We used three types of queries: those using only
the description field DESC (called D), those using
only the TITLE field (T), and those using the TITLE
field and the first relevant documents field RDOC[1](TRD). The DESC field provides a basic descriptionof user request, the TITLE field is composed of one
to three words that represent the essence of the user
request. The RDOC field identifies three relevant documents, RDOC[1] means the first relevant documentin the RDOC fields.
3 The documents eliminated from the LARGE collection included all the documents (493,709 documents) in one particular input file. We tried various ways to generate an index file from this
input file but were unable to do so during the formal run term. Thus,
we used 20 index files for the LARGE collection.

In general, the so-called meta-search engines combine the results of multiple search engines. In thiscase, how do meta-search engines merge and sort thescores from each search engine? Conventional metasearch systems combine the returned lists of URLs or
pages from different search engines. At that time, thesystems either directly show the results with the nameof the specific search engine or present new results thathave been recalculated by using various normalizing
methods [7] [2].
For this task, we used the scores and rankings produced by the IR package output. We examined four
methods for using the combined results. All four
methods worked automatically.
The first method, called Score, merges some outputs depending on the scores. It organizes the results
which are based on the IR package’s output scores foreach index file.
The second method, called Each, merges some outputs using the scores and ranking. It consists on two
steps. First, the results are extracted for N ranked documents from all output over 1,000 documents. Second, the final results are organized according to the
output score. For example, for the SMALL collection
we made five index files. As a result, we obtained five
outputs, extracted the top 200 documents from each
output file, and combined them into one file as a final
result according to the score of each document.
The third method, called Top, merges multiple outputs by the ranking. It includes several steps. The firststep is to extract the first-ranked document from each
output file. The next step is to order the extracted documents according to their scores and define the ranked
documents as a result. We repeat this process for thesecond-ranked document from each output file. These
steps are continued until we have over 1,000 documents in the final result. For example, in the SMALL
collection, we had five output files, so we extracted thefive documents that ranked first in each output, orderedthem from first to fifth according to their scores, anddefined them as the top five documents of the final result. Next, we extracted the five documents that ranked
second, ordered them by rank, and defined them as thesixth through tenth documents of the final result. We
then continued this process until we had over 1,000
documents in the final result.
Finally, the third method, which uses score normalization, is called
7 SN. This method consists of rebuildGIH
ing JIK
and from all index files just the same as
making one index using the whole data collection. We
believed the best way was to make one index using thewhole data collection, then retrieve information using
the OKAPI method. However, the system is not able
to make one index using the whole data collection because of the limitations of the machine, file size, text
JIK GHlength7 and the like. Therefore, we recalculated
and from the set of indexes, as if we had made

Proceedings of the Third NTCIR Workshop

one index. The OKAPI score is obtained for the SN
method.

5 Experimental Results
We obtained three official results from NII as EVF,
DCG[5], and MRR[9]. The EVF results were created
by ’trec eval’ [1], which was used to evaluate the official TREC results. We describe the EVF results indetail.

5.1 Survey Retrieval
Tables 1 and 2 show the experimental results for
the SMALL and LARGE collections, respectively. In
the tables, we classified the results into four classes
depending on query length and method. Bold numbers
show the best score for each class.
The label qp-cont refers to the results when theassessor judges the relevance so that high relevance,
relevance, and partial relevance are wholly judged as
the same level of relevance, with consideration of only
the content of the document itself. The label qp-link
refers to the results when high relevance, relevance,and partial relevance are wholly judged as the samelevel, with consideration of both the content of the
document and the out-linked documents on the condition that they are included in the document pools. The
label q-cont refers to the results when high relevanceand relevance are judged as the same level, with consideration of only the content of the document itself,while q-link refers to using the content of the document and the out-linked documents in the document
pools.
We expected that the score normalization method(SN) would have an effect on retrieving relevant documents, and also on the ranking, with a high rank fora strongly related document. We also expected that
the Rocchio’s feedback retrieval method (FB) would
be better than the method without feedback that was
method N.
The official review, however, was contrary to our
expectations. When the query was only DESC (D) in
the SMALL collection, the combination of scores only
and method N produced the best score for qp-cont, qplink, q-cont and q-link. When the query was D in the
LARGE collection, almost the same results were obtained.
On the other hand, when the feedback method isused, the ranked method (TOP) got the best score inspite of the collection size.
In conventional information retrieval systems, better results are always generated by long queries withseveral terms generally. Thus, when the query is TRD,
the combination of the Score method and method N
achieves the best score in the SMALL collection. In
the LARGE collection, we obtained similar results.

When comparing the three methods of Score, Eachand Top, the combination the Score method andmethod N produces the best score in both collections.
On the other hand, the Top method produces the bestscore in both collections on the method FB.
Comparing methods N and FB, we found thatmethod N tended to achieve better scores. This tendency is especially strong for short queries, whichindicates a weak point for method FB. When feedback documents include non-relevant documents, theresults were the worst. If we use the interactive mode
to select the feedback documents, method FB should
produce better scores.
In the automatic mode and the OKAPI method, we
consider that the Score method is the best approachthat is equivalent to SN method. In the feedback
method, the Top method is the best way.

5.2 Target Retrieval
Among the results for R-Precision in the SMALL(Table 3) and LARGE (Table 4) collections, TRD produced the best score for P@5 and P@10 The P@5
and P@10 mean the precision at five and ten documents retrieved that produced by the evaluation program ’trec eval’ [1].
In the SMALL collection, the average R-Precision
score for the Score method combined with method N
showed the best results. In the LARGE collection,
the Score and Each method obtained the best score for
P@5 and for the average in qp-link. For this subtask,
combining the Score method and method N is the mostefficient approach in the automatic mode, especiallyfor long queries. For method FB, the Top method has
the best score.
The reason for the efficiency of the Score method
is considered to be as follows: If each index file includes the same contents, the relevant results obtained
from all index files become similar. This means results can be obtained with similar scores. The relevant
documents get higher scores, and then the highly relevant documents extracted from all index files can bemerged in the order of the highest to the lowest relation
using the Score method.
In general, when systems collect Web documents
and distribute them to index files automatically, thecontents of the index files are distributed uniformly.
In such a situation, the Score method is very effective
and equivalent to the SN method.
However, when the index files are made or classified according to the contents in special subjects, itwill be effective to select an index file that is suitable
for the query. On the other hand, when the search system uses method FB, the Top method is very effective.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

Table 1. Results for Average Precision (SMALL)
Method Used
Average Precision
Query Score Each Top SN qp-cont qp-link q-contDN
0.1844 0.1559 0.1638
DN
0.1844 0.1559 0.1637
DN
0.1656 0.1392 0.1466
DN 0.1834 0.1572 0.1570
D
FB
0.1352 0.1141 0.1213
DFB
0.1386 0.1192 0.1236
DFB
0.1582 0.1366 0.1313
TN
0.1702 0.1414 0.1668
TN
0.1702 0.1413 0.1667
TN
0.1549 0.1270 0.1591
T
N 0.1752 0.1449 0.1723
T
FB
0.1296 0.1155 0.1074
TFB
0.1331 0.1211 0.1095
T
FB
0.1579 0.1357 0.1240
TRD
N
0.2678 0.2161 0.2433
TRD
N
0.2677 0.2159 0.2433
TRD
N
0.2544 0.1785 0.2279
TRD
N 0.2804 0.2235 0.2560
TRD
FB
0.2225 0.1728 0.1945
TRD
FB
0.2239 0.1749 0.1950
TRD
FB
0.2300 0.1627 0.1971

q-link
0.13810.13800.12330.13490.10340.10670.11170.12620.12610.11580.1297
0.09860.10220.10520.19980.19970.16290.20750.15440.15630.1449

No.
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

Table 2. Results for Average Precision (LARGE)
Method Used
Average Precision
Query Score Each Top SN qp-cont qp-link q-cont
DN
0.1278 0.1269 0.0927
DN
0.1262 0.1241 0.0913
DN
0.1075 0.1042 0.0757
D
N 0.1273 0.1270 0.0920
D
FB
0.0729 0.0809 0.0697
DFB
0.0919 0.0984 0.0829
DFB
0.1136 0.1101 0.0899
TN
0.1386 0.1369 0.0916
TN
0.1365 0.1330 0.0901
TN
0.1166 0.1117 0.0719
T
N 0.1381 0.1364 0.0914
TFB
0.0746 0.0859 0.0686
T
FB
0.0923 0.1022 0.0802
TFB
0.1094 0.1089 0.0823
TRDN
0.1515 0.1461 0.1271
TRD
N
0.1507 0.1448 0.1263
TRD
N
0.1232 0.1052 0.1059
TRD
N 0.1581 0.1510 0.1334
TRD
FB
0.0866 0.0863 0.0774
TRD
FB
0.0963 0.0962 0.0850
TRD
FB
0.1033 0.0920 0.0917

q-link
0.0987
0.09610.0777
0.09820.0837
0.0952
0.09220.1007
0.09720.07640.09850.0886
0.09880.08880.12940.1281
0.09110.13330.08210.08970.0802

Proceedings of the Third NTCIR Workshop

No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

No.
22
23
24
25
26
27
28
29
30
31
32
33
36
37
38
39
40
41
42
42
43

Query
D
D
D
D
D
D
D
T
T
T
T
T
T
T
TRD
TRD
TRD
TRD
TRD
TRD
TRD

Table 3. Results for R-Precision (SMALL)
Method Used
qp-cont
Score Each Top SN P@5
P@10 Ave.
P@5
N
0.2174 0.1717 0.1867 0.2723
N
0.2174 0.1717 0.1867 0.2723
N
0.1739 0.1565 0.1795 0.2213
N 0.2130 0.1739 0.1946 0.2681
FB
0.2130 0.1739 0.1664 0.2426
FB
0.2130 0.1739 0.1664 0.2426
FB
0.2130 0.1783 0.1810 0.2511
N
0.1913 0.1761 0.1757 0.2468
N
0.1913 0.1761 0.1757 0.2468
N
0.1783 0.1478 0.1731 0.2255
N 0.2000 0.1804 0.1734 0.2468
FB
0.2000 0.1630 0.1611 0.2340
FB
0.2000 0.1630 0.1611 0.2340
FB
0.2043 0.1826 0.1685 0.2340
N
0.3244 0.2489 0.2655 0.4217
N
0.3244 0.2489 0.2655 0.4217
N
0.3022 0.2333 0.2477 0.3217
N 0.3442 0.2628 0.2820 0.4409
FB
0.2978 0.2267 0.2377 0.3652
FB
0.2978 0.2267 0.2377 0.3652
FB
0.2800 0.2089 0.2449 0.2913

qp-link
P@100.22980.22980.20430.23830.20850.20850.21700.21910.21910.19360.22340.20640.20640.21490.3326
0.3326
0.27170.34550.28910.2891
0.2348Ave.
0.19320.19320.16630.19510.16290.16290.18360.17820.17820.15910.18510.1636
0.16360.1751
0.23930.23930.20050.24490.20500.20500.1982

Query
D
D
D
D
D
D
D
T
T
T
T
T
T
T
TRD
TRD
TRD
TRD
TRD
TRD
TRD

Table 4. Results for R-Precision (LARGE)
Method Used
qp-cont
Score Each Top SN P@5
P@10 Ave.
P@5
N
0.2638 0.2660 0.1886 0.3106
N
0.2638 0.2660 0.1888 0.3106
N
0.2638 0.2617 0.1639 0.3106
N 0.2511 0.2681 0.1916 0.3021
FB
0.2426 0.2234 0.1218 0.2681
FB
0.2426 0.2234 0.1306 0.2681
FB
0.2298 0.2362 0.1774 0.2468
N
0.3064 0.2745 0.1918 0.3617
N
0.3064 0.2745 0.1919 0.3617
N
0.2979 0.2489 0.1660 0.3489
N 0.2894 0.2574 0.1895 0.3404
FB
0.2000 0.2106 0.1225 0.2298
FB
0.2000 0.2106 0.1332 0.2298
FB
0.2468 0.2404 0.1734 0.2638
N
0.4043 0.3043 0.1879 0.4913
N
0.4043 0.3043 0.1877 0.4913
N
0.4348 0.3109 0.1561 0.4609
N 0.3909 0.3250 0.1999 0.4727
FB
0.2478 0.1957 0.1257 0.3000
FB
0.2478 0.1957 0.1309 0.3000
FB
0.2957 0.2413 0.1495 0.3174

qp-link
P@100.30850.30850.30210.31280.25320.25320.25960.31910.31910.29150.30640.24680.24680.25530.38700.38700.33260.40000.2391
0.23910.2609Ave.
0.19930.20010.17180.19510.13410.14730.18510.19980.20000.17300.20040.13940.15430.18120.19400.19420.14770.20340.13100.13730.1432

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

6 Conclusion and Future WorkWe examined four methods of merging multiple results from retrieval systems.
We could find that the combination of the Score
method and method N was the most efficient approach.
Result were similar to the SN method. Furthermore,
we could find that when using method FB, the Top
method is the most efficient approach. This findingmeans that when the Web documents are automatically collected, their contents are uniformly distributed
and the OKAPI method is used by the search engine,
the score information obtained from the results can beused in Web retrieval without score normalization.
In general, we could not get the score informationfrom each search engine, as they use different methods. Our results confirm that the Top method is themost efficient.
Results for average precision are not satisfactory for
an information retrieval system, although the IR package is a satisfactory tool in retrieving news articles.
Thus, we need to develop some revised methods of retrieving unarranged documents. In this study, we did
not use link information or an interactive mode of theIR package. Previous research [4] has shown the effectiveness of using link information. We would liketo investigate the approach of using link information.
Results of average precision for the LARGE collection are worst than for the SMALL collection. It
can be considered to affect elimination of documents
badly. We should cope with processing incorrectly
converted files and text files with long lines. In thisstudy, we eliminated all such files. We would like todevelop ways to deal with text files with long lines.
Further research will include selecting relevant documents with Rocchio’s feedback by using the interactive mode to achieve more effective results. It should
also be possible to improve the precision of information retrieval by applying link information.
In this study, we did not consider the processing
time required for information retrieval. Search engines need to respond quickly to users’ requests. We
should thus make efforts and decrease the processing
time, although we think the processing time was able
to be shortened considerably by omitted score normalization.

References[1] ftp://ftp.cs.cornell.edu/pub/smart/trec eval.v3bgeta.share.
[2] J. A. Aslam and M. Montague. Models for metasearch.
In SIGIR 2001, 2001.
[3] R. Baseza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, 1999.
[4] N. Craswell, D. Hawking, and S. Robertson. Effective
site finding using link anchor information. In SIGIR
2001, 2001.

[5] K. Jarvelin and J. Kekalainen. IR evaluation methods
for retrieving hightly relevant documents. In SIGIR
2001, 2001.
[6] S.E. Robertson and S. Walker. Okapi/keenbow at trec-8.
TREC 8 proceedings.
[7] M. Uehara, N. Sato, T. Yamamoto, Y. Nishida, and
H. Mori. Distributed scoring in cooperative search engine, 1999. Internet Conference 1999 (in Japaneses).
[8] M. Utiyama and H. Isahara. Implementation of an IR
package. IPSJ SIG notes 2001-FI-63 (in Japaneses).
[9] E. M. Voorhees and D. M. Tice. Building a question
answering test collection. In SIGIR 2001, 2001.