Proceedings of the Third NTCIR Workshop

POSNIR: Probabilistic Natural Language Information Retrieval System
Changki Lee Seungwoo Lee Gary Geunbae Lee
Dept. of Computer Science & Engineering, POSTECH
San 31, Hyoja-Dong, Pohang, South Korea, 790-784{leeck, pinesnow, gblee}@postech.ac.kr
Abstract
This paper describes our information retrieval system
participated in CLIR task of NTCIR3 and reports theresults with some observations. Most previous
information retrieval models assume that terms ofqueries and documents are statistically independent
from each another. However, independence
assumption is obviously and openly understood to be
wrong. We presented two method of incorporating
the term dependences in probabilistic retrieval model
to compensate the weakness of the linked
dependence assumption.

Keywords: information retrieval, probabilistic
model, term dependence
1. Introduction
Most previous information retrieval (IR) models
assume that terms of queries and documents are
statistically independent from each another. Although
independence assumption is obviously and openly
understood to be wrong [11], many IR models basedon this assumption have been developed because theassumption leads to a formal representation of themodel more easily, and most IR systems practicallyhave worked well under this assumption.
Many researchers tried to remove theindependent assumption and have incorporated
various term dependence models with diverse
techniques [1][2][7]. However, when a higher ordermodel of term dependence is used, the easily reachedformal representation of the model (in fact, thegreatest merit of term independence) cannot be
maintained and we face extreme difficulties in
inducing the probabilities of the model. Nevertheless,it has been clarified that incorporation of a term
dependence
model
actually
improved
theperformance [3][7].
In this paper, we introduce two methods ofincorporating term dependence in probabilistic
retrieval model. First method is to incorporate term
dependence in probabilistic retrieval model byadapting Bahadur-Lazarsfeld Expansion (BLE) using
term co-occurrence information. Second method is toincorporate term dependence in probabilistic retrieval
model by adapting a structural index system using
dependency parse tree and the Chow Expansion (CE).
Bahadur-Lazarsfeld Expansion and Chow Expansion© 2003 National Institute of Informatics

were originally used in pattern recognition field [12].
This paper is organized as follows. In section 2,
we discuss previous researches on diverse techniques
to incorporate the term dependences in different
retrieval models and compare them with our own
research. In section 3, we describe our POSNIR
system participated in CLIR task of NTCIR3 andreport the results. In section 4, we describe BahadurLazarsfeld Expansion theory and our adaptation of
the theory to 2-Poisson model, particularly Okapi
BM25. In section 5, we describe the Chow
Expansion theory, the dependency parse tree, and ouradaptation of the theory to 2-Poisson model for
dependency structured indexing system. In section 6,
we draw some conclusions and plans for future
works.

2. Previous Researches
Robertson and S. Walker presented an IR model
approximating 2-Poisson model, well known as
Okapi BM series [8], which integrate withindocument term frequency, document length andwithin-query term frequency. While these models
have been widely used in IR, they are based on one
important assumption, i.e., linked dependence
assumption [11]:Pr( A, B | rel ) Pr( A | rel ) Pr( B | rel ) /=Pr( A, B | rel ) Pr( A | rel ) Pr( B | rel )////, where A, B are regarded as properties of documents,
and rel designates the relevance set.
The
linkeddependence
assumptionisconsiderably weaker than the binary independence
assumption, so in most cases, IR systems using theformula based on the linked dependence have shown
relatively good experimental results. Nevertheless, it
is also an unrealistic assumption, and many
researches have tried to address the limitations of thelinked dependence assumption by computing theterm dependences using diverse techniques on thebasis of different retrieval models.
Bollmann-Sdorra and Raghavan showed that, forretrieval functions such as dot products or the cosine
used in the vector space model, weighted retrieval isincompatible with term independence in query space[3]. They also proved that the term independence inthe query space even turned out to be undesirable.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

Croft proposed an approach to integrate Boolean
and statistical systems, where boolean queries areinterpreted as a way of specifying term dependencies
in the relevant set of documents [10].
Losee proposed a probabilistic model integrating
boolean query in CNF (Conjunctive Normal Form),
where most of the dependences exist between the
disjunctions of the terms [5].
Losee also incorporated term dependence
information in estimating Pr(d|rel) using the
Bahadur-Lazarsfeld Expansion (BLE) [7], anddocuments were ranked by Expected Precision (EP)of the documents as follows:Pr( rel | d ) =Pr(d | rel ) Pr(rel ) /Pr( d )

, where d is the vector of a document and rel is arelevant set. Losee performed experiments using
Cystic Fibrosis (CF) for spanning the degree of theterms and showed that the best performance wasobtained when degree 3 and ±3 to ±5 window of theterms were used. However, in his experiment, heestimated the parameters (all probabilities andcorrelations for appropriate relevance class) using the‘retrospective’ technique. That is, before the retrieval
process, all parameters were estimated with the fullknowledge of the characteristics of relevant and nonrelevant documents. In general, however, we cannot
know the relevant and non-relevant documents fully
in real situations. Moreover, because of the relatively
small sized test collection in his experiments, it is not
sufficient to verify that this technique is actuallyeffective in practical situations.
Van Rijsbergen explored one way of removing
the independence assumption [1]. He constructed aprobabilistic model incorporating dependences
between index terms. The extent to which two index
terms depend on one another is derived from the
distribution of co-occurrences in the whole collection
or in the relevant and non-relevant document sets,and used to construct a non-linear weighting function.
In a practical situation, the values of some of theparameters of such a function must be estimated fromsmall samples of documents. So a number of
estimating rules were discussed and one in particular
was recommended.
Turtle described a new formal retrieval model
which uses probabilistic inference networks torepresent documents and information needs [6].
Retrieval is viewed as an evidential reasoning
process in which multiple sources of evidence about
document and query content are combined toestimate the probability that a given document
matches a query. This model generalizes several
current retrieval models and provides a framework
within which disparate information retrieval research
results can be integrated. The chief advantage of themodel is that it allows complex dependencies to be
represented in an easily understood form and allows
networks containing these dependencies to beevaluated without development of a closed form
expression. However the model makes only limited
use of term dependence information (phrase andthesaurus information) and should be extended toincorporate additional dependencies (e.g., term
clustering).
Much of the works done within the TREC on theuse of phrases and passages can be seen as seeking tocapture dependencies by more informal means,though there may be other motivations as well [4].
Thus limiting candidate query expansion terms tothose occurring in the passage neighborhoods ofmatching terms can be seen as a way of concentratingon the co-occurrence information so that it is more
discriminating than the co-occurrence information
computed over extended full texts.

3. POSNIR System
In this section, we describe our POSNIR system
participated in CLIR task of NTCIR3 and report theresults.

3.1

Keyword Extraction

For keyword extraction, we tagged the document
collection and queries using POSTAG/K and
POSTAG/J (the Korean and Japanese Part-Of-Speechtagger based on HMM) in Korean and Japanese. In
the case of Chinese, we used Chinese segmenter
POSTAG/C. The outputs of POSTAG/K and
POSTAG/J are composed of lexis, POS tag, andlemma. From the result of the taggers, we selectedlemma as keywords. Stop words are eliminated using
two kinds of stop list: common stop list and queryspecific stop list which must be removed from thequery.
For constructing noun phrases, we made lexicosyntactic rules based on the POS-tag patterns. Someof the rules are described below.
Term1/{NN|NP} Term2/{NN|NP}
 Term1_Term2
Term1/JJ Term2/{NN|NP} Term3/{NN|NP}
 Term1_Term2_Term3

3.2

Initial Retrieval

Our retrieval system uses 2-Poisson model basedon the probabilistic term distribution. The system
retrieves top-raked documents after giving scores toeach document of a target data collection with eachquery term list made from the keyword extraction
process. For scoring, a rank system uses Okapi
BM25 formula as shown below.

Proceedings of the Third NTCIR Workshop

Run type
K-K-C rigid
K-K-D rigid
K-K-D rigid
J-J-C rigid
J-J-D rigid
J-J-T rigid
C-C-C rigid
C-C-D rigid
C-C-T rigid

Query expansion
PRF
No
PRF
No
No
No
No
No
No

Avg Prec0.215700.174400.195700.24480
0.212800.23760
0.22030
0.17260
0.19280

Prec@100.306700.26000
0.31330
0.323800.295200.321400.316700.28100
0.28100

Prec@1000.178700.126700.14500
0.15480
0.14980
0.156200.143300.118300.12950

R-Prec0.24920
0.213700.243900.255800.210400.243800.252500.201500.22000

Table 1. Results of NTCIR3 CLIR task
adapting TF (Term Frequency) factor as follows: N − n + 0.5 
w(1) = log
 n + 0.5  5/ / / / / / / / / / / / / / / / / / / (1)/
Score(d , q ) = / / (2)/



(
1
)
(
,
)
k
tf
q
t
+
×
(k1 + 1) × tf t


3
q
(1)
w
×
×
∑

dl d 
k 3 + tf q (q, t ) t ∈q k 1 ×  (1 − b) + b × + tf t
avdl , where N is the number of documents in thecollection, n is the number of documents containing
the term, tft is the term frequency of term t in adocument d, dld is the document length, avdl is theaverage document length, tfq(q,t) is the term
frequency of query term t in the query q, and k1 , b, k3
are tunable constant parameters.

3.3

Query Expansion

Query expansion is achieved through PRF(Pseudo Relevance Feedback). In the process of PRF,
top-ranked documents are regarded as relevant andTSV (Term Selection Value) is given to all single
terms except stop words in them. Then, top-ranked
single terms are expanded and added to the original
query term list. In this process, the weights of bothoriginal and expanded query terms are re-weighted
by Eq.(3) reflecting relevance and non-relevance
information [9].
w(1) =

k5 
N 
R 
r + 0.5  k 4 + log
+
 log

N − n  k5 + R 
R − r + 0.5 
k5 + R  k6
N  
S
s + 0.5 log−  −  k + S log S − s + 0.5 
N
n
−
k
S
+
  6

 6

/ / (3)//, where N, n is the same as in the Eq.(1), R is thenumber of documents known to be relevant to a
specific topic, r is the number of relevant documents
containing the term, S is the number of documents
known to be non-relevant, s is the number of nonrelevant documents containing the term, and k5, k6 aretunable constant parameters.
For TSV function, we developed a formula
tf t ,d(1)TSV =  ∑
× w


dl
d
 d∈R k1  (1 − b) + b
 + tf t ,d 

avgdl




/

3.4

Experiments in NTCIR3

We performed experiments on three languages(Korean, Japanese, and Chinese). Because we do nothave sufficient test collections of Japanese andChinese to set parameters of our model, we used PRF
only K-K-C and K-K-D run. So we did not perform
query expansion at other runs except K-K-C and KK-D runs. Table 1 summarizes the CLIR task results.
The results indicate that when a query was expanded
using PRF, the performance was better.

4. BLE Adaptation to 2-Poisson Model
We propose a method of incorporating the term
dependence into probabilistic models, in particular 2Poissonmodels,using
Bahadur-Lazarsfeld
Expansion (BLE) [7][12]. Document probabilities
can also be estimated on the BLE. An exact
probability is calculated when full expansion is used,
but if the expansion is truncated, an estimate of theprobability is computed. The expansion begins withthe estimate of the independent probability, and thenproceeds by multiplying this independent probability
by a correction factor. Each successive
approximation accounts for the correlations of one
more higher order, but naturally requires thecomputation of additional terms. When the full
expansion is used, the exact probability of adocument is computed as follows:Pr( d ) =
Pr1 (d ) 1 + ∑ ρ ij y i y j + ∑ ρ ijk y i y j y k + ... + ρ 12...n y1 y 2 ... y n 
i
<
j
i
<
j
<
k



/ (4)/

The Third NTCIR Workshop, Sep.2001 - Oct. 2002/
Pr1(d) is the probability of a document byassuming the term independence, computed as:n

Pr1 (d ) = ∏ pixi (1 − pi )1− xi /i =1/
, where d is represented as a binary vector
x={x1,x2,…,xn}, pi=Pr(di = 1), and 1-pi = Pr(di = 0).
yi is a normalization variable of xi , computed as:xi − p iyi =pi (1 − pi )/within a sentence of query in our experiments. We
consider a sentence as a window of documents as we
did in the query. We can calculate ρij by counting thenumber of documents where term i and term j appear
in a sentence simultaneously. The counting is basedon the assumption that the co-occurrence terms havea reasonably short distance in the document.
The results of our experiments are listed in the
Table 2 and Table 3.
Cases
BM25
BM25BLE

AvgP
0.23810.2464

P@50.54400.5640P@10
0.49000.5020R-P
0.28260.2903

Table 2. Performance on the HANTEC2.0 collection/

ρij is a correlation coefficient of xi and xj ,computed as:

Cases
BM25
BM25BLE

AvgP
0.20270.2064

P@50.35200.3520P@10
0.28600.2920R-P
0.24420.2495

Table 3. Performance on the WT10g collection

ρ1, 2,...i = E[ y1 y2 m yi ] = ∑ y1 y 2 m yi Pr(d )d//The sum of correction factor of Eq. (4) may bearbitrarily truncated so that one can include alldependences with term pairs, term triples, and so on.
We incorporate the term dependence into thestate-of-the-art 2-Poisson model, in particular Okapi
BM25, using the BLE. We adapt the BLE to BM25
up to the second order as follows [18]:
Score(d , q) =

∑ qtf
i∈q

tf i
i

dl
k1 (1 − b + b ⋅) + tf iavdl


w(1) − β log1 + k 7 + ∑ ρ ij yi y j 
i< j



, where w(1) is the IDF estimated with no relevance
information, and β and k7 are constant parameters.

4.1

BLE Experiments/
We will empirically demonstrate that the 2Poisson model incorporating BLE term dependences
gives a significantly better performance than the 2Poisson model under the conventional linked
dependence assumption. For a more convincing
demonstration, we performed the experiments on two
different languages, Korean and English. For Korean,
we use the HANTEC2.0 collection [17] and 50 topics
for test, and for English, we use WT10g collection
for TREC evaluation and 50 topics of TREC-9 (Topic
451~450). All diverse experiments are carried out by
varying models of POSNIR/K and POSNIR/E, which
is a POStech Natural language Information Retrieval
system for Korean and English respectively.
Term pairs are generated from single terms oforiginal query term list. We extracted the term pairs

Comparing ‘BM25’ with ‘BM25BLE’, theperformance of ‘BM25BLE’ was better than
‘BM25’ on HANTEC2.0 as well as on WT10g
collection. From the above results, the 2-Poissonmodel incorporating term dependence with BLE
techniques significantly improves the performance.

5. Probabilistic Model for Dependency
Structured Indexing System
In this chapter, we introduce a method ofincorporating the term dependence into probabilistic
models, in particular 2-Poisson models, using the
Chow Expansion [14] and a dependency structured
indexing system.

5.1 Chow
Expansion
Dependency Parse Tree

TheoryandWhen the components of the vector
x={x1,x2,…,xn} are binary values, the problem of
estimating a density becomes the problem of
estimating the probability. Since there are 2n possible
vectors x, we must estimate 2n probabilities, which isan enormous task.
If the components of x are statisticallyindependent, the problem is greatly simplified. In thiscase we can write
n

n

i =1

i =1

Pr( x) = ∏ Pr( xi ) = ∏ pixi (1 − pi )1− xi //5/where pi = Pr(xi=1) and 1-pi = Pr(xi=0).
It is natural to ask whether or not there are any
compromise positions betweens being completely
accurate, which requires estimating 2n probabilities,
and being forced to assume statistical independence,Proceedings of the Third NTCIR Workshop

which reduce the problem to one of estimating only nprobabilities. One answer is provided by finding an
expansion for Pr(x) and approximating Pr(x) bypartial sum, e.g. the Rademacher-Walsh Expansion
and the Bahadur-Lazarsfeld Expansion [12].
Another interesting class of approximation to a
joint probability distribution Pr(x) is based on theidentityPr( x) = Pr( x1 ,  , x n )= Pr( x1 ) Pr( x 2 | x1 ) Pr( x3 | x 2 , x1 )  Pr( x n | x n −1 ,  , x1 )/Suppose the variables are not independent, but
we can number the variables so that Pr(xi|xi-1,…,x1) issolely dependent on some preceding variable xj(i).
Then we obtain the product expansionPr( x) = Pr( x1 ) Pr( x2 | x j ( 2) ) Pr( x3 | x j (3) ) Pr( xn | x j ( n ) ) //By substituting 0 or 1 for xi and xj(i), we canverify that[Pr( xi | x j (i ) ) = pix| ij (i ) (1 − pi| j (i ) )1− xi] [p (1 − p ) ]x j(i)xiii

1− xi 1− x j ( i )/, where pi|j(i) = Pr(xi=1|xj(i)=1) and pi = Pr(xi=1| xj(i)=0).
By letting pi = Pr(di=1), taking the logarithm, andcollecting terms, we obtain the Chow Expansion [14].

MST (minimum spanning tree).
We developed a simple dependency parser for
Korean to apply dependency parse trees to the Chow
Expansion. Our dependency parser uses someheuristics which are generally used in dependency
parsing [16] (e.g. Non-crossing condition, Constraint
under surface information and nearest modifyee
principle). Our dependency parser shows about 70%
precision.

5.2 Adapting the Chow Expansion to the
Dependency Structured Indexing System
Chow and Liu suggest the construction of a MST
using mutual information for a dependence tree
which is originally used in the Chow Expansion.
However, we use a dependency parse tree which is
generated by linguistic dependency parser instead ofthe mutual information MST, because a dependency
parse tree intuitively and linguistically represents theterm dependence relations in the syntactic structure.
We consider the use of syntactic structure as one wayof relaxing the independence assumption of the terms.
So, we use a dependency structured indexing system
which consists of the dependency parse tree andChow Expansion to relax the independence
assumption.
We adapt the Chow Expansion to the 2-Poissonmodel using dependency structured indexing system
as follows [19]:Score(d , q)
= ∑ qtfii∈qnnlog Pr( x) = ∑ log(1 − pi ) + ∑ xi log
i =1

i =1

n

1 − p i| j ( i )i =2

1 − pi+ ∑ x j ( i ) log

pi
1 − pinpi| j ( i ) (1 − pi )i =2(1 − pi| j (i ) ) pi

+ ∑ xi x j ( i ) log/

Chow and Liu suggest the construction of a treesuch that the mutual information between a variable
and the variable immediately above it are maximized.
A dependency relationship [15] is an asymmetricbinary relationship between a word called head (orgovernor, parent), and another word called modifier(or dependent, daughter). Dependency grammars
represent sentence structures as a set of dependency
relationships. Normally the dependency relationships
from a tree connect all the words in a sentence. A
word in the sentence may have several modifiers, buteach word may modify at most one word. The root of
the dependency tree does not modify any word. It isalso called the head of the sentence.
Since a dependency parse tree represents the term
dependence relations in the syntactic structure, we
can apply this dependency parse tree which is
generated by linguistic dependency parser in the
Chow Expansion, instead of the mutual informationtfi
k1 (1 − b + b ⋅


qq 
q −q
 xi log N + x j (i ) k7 log j (i ) i , j ( i ) + xi x j (i ) k8 log i j (i ) dlqi , j (i ) q j ( i ) (1 − qi )ni
) + tfi 
avdl/, where d is represented as a binary vector
x={x1,x2,…,xn}. k1, b, k7 and k8 are constantparameters.

5.3

Chow Expansion Experiments

We evaluated the 2-Poisson model incorporating
the Chow Expansion term dependences with the
ETRI-KEMONG test collection which is a Korean
encyclopedia published by the Kemong company[13]. The test set contains 46 natural language
queries and the relevance information of the entry
lists related to each query.
Figure 1 shows the dependency structured
indexing system. Through the dependency structure
analysis of the documents, we extract single terms
and dependency relationship for indexing. In
retrieving, keyword extraction from the query is
performed in the same manner as the indexing
process.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002/
Figure 1. Dependency structured indexing system
The results of our experiments for all the cases
are listed in the Table 4.
Cases
BM25
BM25CE

AvgP
0.60520.6195P@50.43910.4391

P@10
0.28040.2826R-P
0.52410.5534

Table 4. Performance on the ETRI-KEMONG collection
Comparing ‘BM25’ with ‘BM25CE’, theperformance of ‘BM25CE’ was better than ‘BM25’
on ETRI-KEMONG collection. From the above
results, the probabilistic model for dependency
structured indexing system seems to improve theperformance, but more experiments should be calledfor.

6. Conclusions
We described our POSNIR system participated in
CLIR task of NTCIR3 and report the results. And we
presented two other extended methods ofincorporating the term dependences in probabilistic
retrieval model to compensate the weakness of theprevious linked dependence assumption. Due to thetime constraint, however, we couldn’t apply these
new methods to this year’s NTCIR task, but wecarried out some independent experiments to verify
the proposed models. From the results, improvementof the performance was observed on the document
collections in two extended models.
The disadvantage in using the BLE is that the
retrieval cost becomes very high because cooccurrence information between the two terms must
be obtained at the search time when the user query isgiven. The longer the size of the query is, the greater
the number of term pairs are, which incurs a muchhigher retrieval cost. To reduce this cost, useless termpairs can be removed at a certain threshold. However,this is not the general solution, so effective
algorithms or auxiliary DB’s to pre-obtain the cooccurrence information will need to be developed in

the future.
The disadvantage in using the Chow Expansion isthat the retrieval cost of dependence tree becomesvery high because the dependence tree of the user
query is obtained by dependency parser at the searchtime and co-occurrence information between the twoterms are obtained by dependency parser at theindexing time. To reduce this cost, very fast androbust dependency parser will need to be developedin the future.
Another future project will be to apply the BLE
and Chow Expansion to PRF. Many researches onquery expansions using PRF have verified asignificant performance improvement. But it is not
yet known whether BLE and Chow Expansion
techniques indeed work well on the ARF expanded
queries or not, and this question is another interest
point to the BLE and Chow Expansion based term
dependency model.

References[1] C.J. Van Rijsbergen. A theoretical basis for use of cooccurrence data in information retrieval. Journal ofDocumentation, 33(2):106-119, 1977.
[2] Clement T. Yu, Chris buckley, K. Lam, and Gerard
Salton. A generalized term dependence model in
information retrieval. Information Technology:
Research and Development, 2(4):129-154, 1983.
[3] Peter Bollmann-Sdorra and Vijay V. Raghavan. On the
Necessity of Term Dependence in a Query Space for
Weighted Retrieval. Journal of the American Societyof Information Science, 49(13): 1161-1168, 1998.
[4] K. Spark Jones, S. Walker and S.E. Robertson. A
probabilistic model of information retrieval:
Development and status. Technical Report 446,
University of Cambridge Computer Laboratory, 1998.
[5] Robert M. Losee and Abraham Bookstein. Integrating
Boolean queries in conjunctive normal form withprobabilistic retrieval models. Information Processingand Managements, 24(3):315-321, 1988.
[6] H.R. Turtle and W.B. Croft. Inference Networks for
document Retrieval. In Intern. Conf. on Research andDevelopment in Information Retrieval, pages 1-24,
SIGIR, 1990.
[7] Rorbert M. Losee. Term dependence: Truncating the
Bahadur-Lazarsfeld
expansion.
Information
Processing and Managements, 30(2):293-303, 1994.
[8] S.E. Robertson and S. Walker. Some simple effective
approximations to the 2-Poisson model for
probabilistic weighted retrieval. In Proceedings of the
ACM SIGIR’94, 232-241, 1994.
[9] S.E. Robertson and Walker S. On relevance weights
with little relevance information, In Proceedings of the
ACM SIGIR’97. 16-24, 1997.
[10] W. Bruce Croft. Boolean queries and term
dependencies in probabilistic retrieval models. Journalof the American Society for Information Science andTechnology, 37(2):71-77, 1986.
[11] William S. Cooper. Some Inconsistencies andMisnomers in Probabilistic Information Retrieval. In
Proceedings of the ACM SIGIR’91, 57-61, 1991.

Proceedings of the Third NTCIR Workshop[12] Richard O. Duda, Peter E. Hart. Pattern Classificationand Scene Analysis. A Wiley-Interscience publication,
111-113, 1973.
[13] Kemong. The Kemong Company new encyclopedia.
Seoul: Kemongsa Publishing Co. 1992.
[14] Chow, C., and Liu, C. Approximating discrete
probability distributions with dependence trees. IEEE
Transactions on Information Theory, IT-14(3), 462467. 1968.
[15] David Hays. Dependency theory: a formalism andsome observations. Language, 40:511-525, 1964.
[16] Sadao Kurohashi, Makoto Nagao. KN Parser:
Japanese Dependency/Case Structural Analyzer.
Proceedings of the Workshop on Sharable Natural
Language Resources, pp48-55. 1994.
[17] Suk-Hoon Lee, Sung Hyon Myaeng, Ji-Young Kim,
Dong-Hyun Jang, Jeong-Hyun Seo, Hyun Kim.
Packaging Hangul Test Collection as an Evaluation
System of Information Retrieval, In Proceeding of The
5th Korea Science & Technology Infrastructure
Workshop, 31-48, 2000 (in Korean).
[18] Bong-Hyun Cho, Changki Lee, Gary Geunbae Lee.
Exploring Term Dependences in Probabilistic
Information Retrieval Model. Information Processingand Managements, (accepted).
[19] Changki Lee and Gary Geunbae Lee. Probabilistic
Information Retrieval Model for Dependency
Structured Indexing System, In Proceedings of the
ACM SIGIR’02 Workshop on Mathematical/Formal
Methods in Information Retrieval, 2002.