Relevance Feedback Document Retrieval using
Support Vector Machines
Takashi Onoda

Hiroshi Murata

Seiji YamadaComm. & Info. Lab.
Central Research Institute ofElectric Power Industry
2-11-1, Iwado Kita, Komae-shi,
Tokyo 201-8511 JAPANE-mail: onoda@criepi.denken.or.jp

Comm. & Info. Lab.
Central Research Institute ofElectric Power Industry
2-11-1 Iwado Kita, Komae-shi,
Tokyo 201-8511 JAPANE-mail: murata@criepi.denken.or.jp

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-ku,
Tokyo 101-8430 JAPANE-mail: seiji@nii.ac.jp

Abstract— We investigate the following data mining problems
from the document retrieval: From a large data set of documents,
we need to find documents that relate to human interest as
few iterations of human testing or checking as possible. In each
iteration a comparatively small batch of documents is evaluatedfor relating to the human interest. We apply active learningtechniques based on Support Vector Machine for evaluating
successive batches, which is called relevance feedback. Our
proposed approach has been very useful for document retrieval
with relevance feedback experimentally. In this paper, we adopt
several representations of the Vector Space Model and several
selecting rules of displayed documents at each iteration, and thenshow the comparison results of the effectiveness for the document
retrieval in these several situations.

I. I NTRODUCTION
As progression of the internet technology, accessible information by end users is explosively increasing. In thissituation, we can now easily access a huge document databasethrough the WWW. However it is hard for a user to retrieve
relevant documents from which he/she can obtain useful
information, and a lot of studies have been done in information retrieval, especially document retrieval [18]. Active
works for such document retrieval have been reported in
TREC(Text Retrieval Conference) [13] for English documents,
IREX(Information Retrieval and Extraction Exercise) [3] andNTCIR(NII-NACSIS Test Collection for Information Retrieval
System) [6] for Japanese documents.
In most frameworks for information retrieval, a Vector
Space Model(which is called VSM) in which a document
is described with a high-dimensional vector is used [10].
An information retrieval system using a vector space modelcomputes the similarity between a query vector and document
vectors by cosine of the two vectors and indicates a user a listof retrieved documents.
In general, since a user hardly describes a precise queryin the first trial, interactive approach to modify the query
vector by evaluation of the user on documents in a list of
retrieved documents. This method is called relevance feedback[9] and used widely in information retrieval systems. In thismethod, a user directly evaluates whether a document isrelevant or irrelevant in a list of retrieved documents, and asystem modifies the query vector using the user evaluation. A
traditional way to modify a query vector is a simple learning
rule to reduce the difference between the query vector anddocuments evaluated as relevant by a user.
In another approach, relevant and irrelevant document vectors are considered as positive and negative examples, andrelevance feedback is transposed to a binary classification
problem [7]. For the binary classification problem, Support
Vector Machines(which are called SVMs) have shown the
excellent ability. And some studies applied SVM to thetext classification problems [12] and the information retrieval
problems[2].
Recently, we have proposed a relevance feedback framework with SVM as active learning and shown the usefulnessof our proposed method experimentally[8]. Now, we areinterested in which is the most efficient representation forthe document retrieval performance and the learning performance, boolean representation, TF representation or TFIDF
representation, and what is the most useful selecting rule fordisplayed documents at each iteration. In this paper, we adopt
several representations of the Vector Space Model and several
selecting rules of displayed documents at each iteration, andthen show the comparison results of the effectiveness for the
document retrieval in these several situations.
In the remaining parts of this paper, we explain a SVM
algorithm in the second section briefly. An active learningwith SVM for the relevance feedback, and our adopted VSM
representations and selecting displayed documents rules aredescribed in the third section. In the fourth section, in ordercompare the effectiveness of our adopted representations andselecting rules, we show our experiments using a TREC data
set of Los Angeles Times and discuss the experimental results.
Eventually we conclude our work in the fifth section.
II. S UPPORT V ECTOR M ACHINES
Formally, the Support Vector Machine (SVM) [14] like anyother classification method aims to estimate a classification
function f : X → {±1} using labeled training data from X ×{±1}. Moreover this function f should even classify unseen
examples correctly.

Fig. 1. A binary classification toy problem: This problem is to separate
black circles from crosses. The shaded region consists of training examples,the other regions of test data. The training data can be separated with a marginindicated by the slim dashed line and the upper fat dashed line, implicating
the slim solid line as discriminate function. Misclassifying one training
example(a circled white circle) leads to a considerable extension(arrows) of
the margin(fat dashed and solid lines) and this fat solid line can classify two
test examples(circled black circles) correctly.

For SV learning machines that implement linear discriminant functions in feature spaces, the capacity limitation corresponds to finding a large margin separation between the twoclasses. The margin % is the minimal distance of training points(x1 , y1 ), . . . , (xi , yi ), xi ∈ R, yi ∈ {±1} to the separation
surface, i.e. % = mini=1,...,` ρ(zi , f ), where zi = (xi , yi ) andρ(zi , f ) = yi f (xi ), and f is the linear discriminant function
in some feature spacef (x) = (w · Φ(x)) + b =X̀

αi yi (Φ(xi ) · Φ(x)) + b, (1)i=1P`with w expressed as w =i=1 αi yi Φ(xi ). The quantityΦ denotes the mapping from input space X by explicitlytransforming the data into a feature space F using Φ : X → F.
(see Figure 1). SVM can do so implicitly. In order to train
and classify, all that SVMs use are dot products of pairs ofdata points Φ(x), Φ(xi ) ∈ F in feature space (cf. Eq. (1)).
Thus, we need only to supply a so-called kernel function thatcan compute these dot products. A kernel function k allows to
implicitly define the feature space (Mercer’s Theorem, e.g. [1])viak(x, xi ) = (Φ(x) · Φ(xi )).
(2)
By using different kernel functions, the SVM algorithm
can construct a variety of learning machines, some of whichcoincide with classical architectures:
1) Polynomial classifiers of degree d:k(x, xi ) = (κ · (x · xi ) + Θ)d , where κ, Θ, and d areappropriate constants.
2) Neural networks(sigmoidal):k(x, xi ) = tanh(κ · (x · xi ) + Θ), where κ and Θ areappropriate constants.
3) Radial basis function
classifiers:
kx−xi k2
k(x, xi ) = exp − σ, where σ is an appropriate
constant.

Note that there is no need to use or know the formof Φ, because the mapping is never performed explicitly
The introduction of Φ in the explanation above was forpurely didactical and not algorithmical purposes. Therefore,
we can computationally afford to work in implicitly verylarge (e.g. 1010 - dimensional) feature spaces. SVM can avoid
overfitting by controlling the capacity and maximizing themargin. Simultaneously, SVMs learn which of the features
implied by the kernel k are distinctive for the two classes,
i.e. instead of finding well-suited features by ourselves (which
can often be difficult), we can use the SVM to select themfrom an extremely rich feature space.
With respect to good generalization, it is often profitable
to misclassify some outlying training data points in order toachieve a larger margin between the other training points (seeFigure 1 for an example). This soft-margin strategy can also
learn non-separable data. The trade-off between margin size
and number of misclassified training points is then controlled
by the regularization parameter C (softness of the margin).
The following quadratic program (QP) (see e.g. [14], [11]):P
min kwk2 + C `i=1 ξi(3)s.t.
ρ(zi , f ) ≥ 1 − ξi for all 1 ≤ i ≤ `ξi ≥ 0for all 1 ≤ i ≤ `leads to the SV soft-margin solution allowing for some errors.
In this paper, we use VSMs, which are high dimensionalmodels, for the document retrieval. In this high dimension, it
is easy to classify between relevant and irrelevant documents.
Therefore, we generate the SV hard-margin solution by thefollowing quadratic program.
min
kwk2
s.t. ρ(zi , f ) ≥ 1 for all 1 ≤ i ≤ `(4)III. ACTIVE L EARNING WITH SVM IN D OCUMENTR ETRIEVALIn this section, we describe the information retrieval systemusing relevance feedback with SVM from an active learning
point of view, and several VSM representation of documents
and several selecting rules, which determine displayed documents to a user for the relevance feedback.
A. Relevance Feedback Based on SVM
Fig. 2 shows the concept of the relevance feedback document retrieval. In Fig. 2, the iterative procedure is the grayarrows parts. The SVMs have a great ability to discriminate
even if the training data is small. Consequently, we haveproposed to apply SVMs as the classifier in the relevance
feedback method. The retrieval steps of proposed method
perform as follows:
Step 1: Preparation of documents for the first feedback
The conventional information retrieval system based on
vector space model displays the top N ranked documents
along with a request query to the user. In our method, thetop N ranked documents are selected by using cosine distance
between the request query vector and each document vector
for the first feedback iteration.

In itia l S e a rc h
in p u t Q u e ry

I r r elevant d

S e a rc h &R a n k in g

D is p la y to p
N ra n k e dd o c u m e n ts

C h e c k re le v a n t o rirre le v a n t b y th e u s e r

Fig. 2.
Image of the relevance feedback documents retrieval: The grayarrow parts are made iteratively to retrieve useful documents for the user.
This iteration is called feedback iteration in the information retrieval research
area.

ar g

i n ar ea

I r r elevant d

o

c

u

m

d

o

c

u

m

ents

u

m

ents

ar ea

w
ar ea

D

i s

c

r i m

i nant f u

nc

ti o

Relevant
d

o

c

u

m

ents

ar ea

M

ar g

i n ar ea

Fig. 4.
The right side figure shows displayed documents as the resultof document retrieval: Boxes denote non-checked documents which aremapped into the feature space. Circles denotes checked documents which
are mapped into the feature space. The system displays the documents which
are represented by black circles and boxes as the result of document retrieval
to a user.

ents

ar ea

Relevantcw

R e le v a n c e fe e d b a c k

M

o

n

Fig. 3. The figure shows a discriminant function for classifying relevant or
irrelevant documents: Circles denote documents which are checked relevant
or irrelevant by a user. The solid line denotes a discriminant function. The
margin area is between dotted lines.

Step 2: Judgment of documents
The user then classifiers these N documents into relevant
or irrelevant. The relevant documents and the irrelevant documents are labeled. For instance, the relevant documents have”+1” label and the irrelevant documents have ”-1” label after
the user’s judgment.
Step 3: Determination of the optimal hyperplane
The optimal hyperplane for classifying relevant and irrelevant documents is determined by using a SVM which islearned by labeled documents(see Figure 3).
Step 4: Discrimination documents and information retrieval
The documents, which are retrieved in the Step1, aremapped into the feature space. The SVM learned by theprevious step classifies the documents as relevant or irrelevant.
Then the system selects the documents based on the distancefrom the optimal hyper plane and the feature of the marginarea. The detail of the selection rules are described in thenext section. From the selected documents, the top N ranked
documents, which are ranked using the distance from theoptimal hyperplane, are shown to user as the information
retrieval results of the system. If the number of feedback

iterations is more than m, then go to next step. Otherwise,
return to Step 2. The m is a maximal number of feedback
iterations and is given by the user or the system.
Step 5: Display of the final retrieved documents
The retrieved documents are ranked by the distance between
the documents and the hyper-plane which is the discriminantfunction determined by SVM. The retrieved documents are
displayed based on this ranking(see Figure 4).
B. VSM Representations and Selection Rules of Displayed
Documents
We discuss the issue of the term ti in the document vector
dj . In the Information Retrieval research field, this term iscalled the term weighting, while in the machine learning research filed, this term is called the feature. ti states something
about word i in the document dj . If this word is absent in
the document dj , ti is zero. If the word is present in the
document dj , then there are several options. The first option
is that this term just indicates whether this word i is present
or not. This presentation is called boolean term weighting.
The next option is that the term weight is a count of thenumber of times this word i occurs in this document dj .
This presentation is called the term frequency(TF). In theoriginal Rocchio
  algorithm[9], each term TF is multiplied bya term log nNi where N is the total number of documentsin the collection and ni is the number of documents in which
this word i occurs. This last term is called the inverse document frequency(IDF). This representation is called the term
frequency-the inverse document frequency(TFIDF)[18]. The
Rocchio algorithm is the original relevance feedback method.
In this paper, we compare the effectiveness of the document
retrieval and the learning performance among boolean term
weighting, term frequency(TF) and term frequency inverse
document frequency(TFIDF) representations for our relevance
feedback based on SVM.

I r r elevant d

o

c

u

m

ents

ar ea

I r r elevant d

w

o

c

u

m

ents

c

u

m

ents

ar ea

w

RelevantdoRelevant
ar ea

d

M

ar g

i n ar ea

Fig. 5. Mapped non-checked documents into the feature space: Boxes denotenon-checked documents which are mapped into the feature space. Circles
denotes checked documents which are mapped into the feature space. Blackand gray boxes are documents in the margin area. We show the documents
which are represented by black boxes to a user for next iteration. These
documents are in the margin area and near the relevant documents area.

Next, we discuss two selection rules for displayed documents, which are used for the judgment by the user. In
this paper, we compare the effectiveness of the document
retrieval and the learning performance among the following
two selection rules for displayed documents.
Rule 1:
The retrieved documents are mapped into the feature space.
The learned SVM classifies the documents as relevant or
irrelevant. The documents, which are discriminated relevant
and in the margin area of SVM are selected. From the selected
documents, the top N ranked documents, which are ranked
using the distance from the optimal hyperplane, are displayed
to the user as the information retrieval results of the system(seeFigure 5). This rule is expected to achieve the most effective
retrieval performance. This rule is our proposed one for the
relevance feedback document retrieval.
Rule 2:
The retrieved documents are mapped into the feature space.
The learned SVM classifies the documents as relevant or irrelevant. The documents, which are on the optimal hyperplaneor near the optimal hyperplane of SVM, are selected. The
system chooses the N documents in these selected documents
and displays to the user as the information retrieval resultsof the system(see Figure 6). This rule should make the bestlearning performance from an active learning point of view.
IV. E XPERIMENTS
A. Experimental setting
In the reference [8], we already have shown that the utilityof our interactive document retrieval with active learning of
SVM is better than the Rocchio-based interactive document
retrieval[9], which is conventional one. This paper presents
the experiments for comparing the utility for the document
retrieval among several VSM representations, and the effectiveness for the learning performance among the several
selection rules, which choose the displayed documents to judge

o

c

u

m

ents

ar ea

M

ar g

i n ar ea

Fig. 6. Mapped non-checked documents into the feature space: Boxes denotenon-checked documents which are mapped into the feature space. Circles
denotes checked documents which are mapped into the feature space. Blackand gray boxes are documents in the margin area. We show the documents
which are represented by black boxes to a user for next iteration. These
documents are on or near the optimal hyperplane.

whether a document is relevant or irrelevant by the user.
The document data set we used is a set of articles in theLos Angeles Times which is widely used in the document
retrieval conference TREC [13]. The data set has about 130
thousands articles. The average number of words in a article
is 526. This data set includes not only queries but also therelevant documents to each query. Thus we used the queries
for experiments.
We adopted the boolean weighting, TF, and TFIDF as VSM
representations. The detail of the boolean and TF weighting
can be seen in the section III. And the detail of the adopted
TFIDF can be seen in the reference [8]. In our experiments,
we used two selection rules to estimate the effectiveness forthe learning performance. The detail of these selection rules
can be seen in the section III.
The size N of retrieved and displayed documents at each
iteration in the section III was set as twenty. The feedback
iterations m were 1, 2, and 3. In order to investigate theinfluence of feedback iterations on accuracy of retrieval, we
used plural feedback iterations. In our experiments, we used
the linear kernel for SVM learning, and found a discriminantfunction for the SVM classifier in this feature space. The VSMof documents is high dimensional space. Therefore, in orderto classify the labeled documents into relevant or irrelevant,
we do not need to use the kernel trick and the regularization
parameter C(see section II). We used LibSVM [4] as SVM
software in our experiment.
In general, retrieval accuracy significantly depends on thenumber of the feedback iterations. Thus we changed feedback
iterations for 1, 2, 3 and investigated the accuracy for each
iteration. We utilized precision and recall for evaluating thetwo information retrieval methods [5][17] and our approach.

1.0

0.8

0.8

0.6

0.6

0.4

 ftf

Precisi

Precision

0.2
0.0
0.0

0.4
0.2

0.2

0.4 

  

on

1.0

0.6

0.8

1.0

all

Fig. 7. The figure shows the retrieval effectiveness of SVM based feedback(using the selection rule 1) for the boolean, TF, and TFIDF representations: The lines show recall-precision performance curve by using twenty
feedback documents on the set of articles in the Los Angeles Times after
3 feedback iterations. The wide solid line is the boolean representation, thebroken line is TFIDF representation, and the solid line is TF representation.

B. Comparison of recall-precision performance curves among
the boolean, TF and TFIDF weightings
In this section, we investigate the effectiveness for the document retrieval among the boolean, TF and TFIDF weightings,
when the user judges the twenty higher ranked documents ateach feedback iteration. In the first iteration, twenty higher
ranked documents are retrieved using cosine distance between
document vectors and a query vector in VSMs, which are
represented by the boolean, TF and TFIDF weightings. The
query vector is generated by a user’s input of keywords. In theother iterations, the user does not need to input keywords forthe information retrieval, and the user labels ”+1” and ”-1” asrelevant and irrelevant documents respectively.
Figure 7 shows a recall-precision performance curve of
our SVM based method for the boolean, TF and TFIDF
weightings, after two feedback iterations. Our SVM based
method adopts the selection rule 1. The thick solid line is theboolean weighting, the broken line is the TFIDF weighting,and the thin solid line is the TF weighting.
This figure shows that the retrieval effectiveness of theboolean, TF, and TFIDF representations are almost sameroughly. However, when we see this figure closely, the TFIDF
representation is higher than that of the other two representations, i.e., the boolean and the TF representations. Consequently, in this experiment, we conclude that the TFIDF
representation is a useful VSM representation for our proposed
relevant feedback technique to improve the performance of the
document retrieval.

0.0
0.0

hyperplane
0.2

0.4  

0.6

0.8

1.0

all

Fig. 8.
The right side figure shows the retrieval effectiveness of SVM
based feedback for the selection rule 1 and 2: The lines show recall-precisionperformance curves by using twenty feedback documents on the set of articles
in the Los Angeles Times after 3 feedback iterations. The thick solid line is
the selection rule 1, and the thin solid line is the selection rule 2.

C. Comparison of recall-precision performance curves between the selection rule 1 and 2
Here, we investigate the effectiveness for the document
retrieval between the selection rule 1 and 2, which are described in the section III. Figure 8 shows a recall-precisionperformance curves of the selection rule 1 and 2 for theboolean weightings, after two feedback iterations. The thicksolid line is the selection rule 1, and the thin solid line is theselection rule 2.
This figure shows that the precision-recall curve of theselection rule 1 is better than that of the selection rule 2. The
selection rule 1 is our proposed method and the selection rule2 denotes the selection of documents, which are on or near
the hyperplane. In this experiment, we conclude that the our
proposed selection rule is a useful selection for our proposed
relevant feedback technique to improve the performance of the
document retrieval
Table I gives the average number of relevant documents inthe twenty displayed documents for the selection rule 1 and 2
as a function of the number of iterations. In this table, iteration
number 0 denotes the retrieval result based on cosine distance
between the query vector and document vectors, which are
represent by the TFIDF.
We can see from the table I that the average number of
relevant documents in the twenty displayed documents for theselection rule 1 is higher than that of the selection rule 2 ateach iteration. After all, when the selection rule 2 is adopted,
the user have to see a lot of irrelevant documents at each
iteration. The selection rule 1 is effective to immediately puton the upper rank the special documents, which relate to theuser’s interesting. When the selection rule 1 is adopted, theTABLE I
AVERAGE NUMBER OF RELEVANT DOCUMENTS IN THE TWENTYDISPLAYED DOCUMENTS FOR THE SELECTION RULE 1 AND 2 USING THE
BOOLEAN REPRESENTATION

No. of feedback
iterations
0
1
2
3

Ave. No. of relevant documents
selection rule 1 selection rule 2
6.444
6.444
10.3334.444
10.444
8.333
7.778
6.444

TABLE IIAVERAGE TOTAL NUMBER OF RELEVANT DOCUMENTS FOR THE
SELECTION RULE 1 AND 2 USING THE BOOLEAN REPRESENTATION

No. of feedback
iterations
0
1
2
3

Ave. No. of relevant documents
selection rule 1 selection rule 2
6.444
6.444
16.77810.88927.22219.22235.00025.667user do not need to see a lot of irrelevant documents at each
iteration. However, it is hard for the rule 1 to immediately puton the upper rank all documents, which relate to the user’s
interesting. In the document retrieval, a user do not want to
get all documents, which relate to the user’s interest. The
user wants to get some documents, which relate to the user’s
interest as soon as possible. Therefore, we conclude that thefeature of the selection rule 1 is better than that of the selection
rule 2 for the relevance feedback document retrieval.
Table II gives the average total number of relevant documents for the selection rule 1 and 2 at each iteration. In
this table, iteration number 0 denotes the retrieval result basedon cosine distance between the query vector and document
vectors, which are represent by the TFIDF. In table I, theaverage number of relevant documents in the twenty displayed
documents decreases at iteration number 3. We can see fromtable II that almost relevant documents can be found at theiteration number 3. This situation means that there are few
relevant documents in the rest documents and it is difficult tofind new relevant documents in the rest documents. Therefore,
the average number of relevant documents in the twenty displayed documents decreases with the increase of the numberof iterations.
V. C ONCLUSION
In this paper, we adopt several representations of the Vector
Space Model and several selecting rules of displayed documents at each iteration, and then show the comparison resultsof the effectiveness for the document retrieval in these several
situations.
In our experiments, when we adopt our proposed SVM
based relevance feedback document retrieval, the effectivenessof the boolean, TF, and TFIDF representations are almost sameroughly. When our experimental results can be seen closely,
the TFIDF representation makes a little better performance
than the other two representations. And the selection rule 1,

where the documents that are discriminated relevant and in
the margin area of SVM, are displayed to a user, show better
performance of document retrieval. Generally, data mining
applications based SVM to drug discovery[16], bioinformatics
and so on are discussed from active learning point of view.
In the discussion, the learning performance of computers isvery important. In the interactive documents retrieval between
human and computers, however, the retrieval performance ismore important than the learning performance of computers.
Therefore, interactive documents retrieval systems should display the documents which are interesting for users at each
iteration and keep the learning performance of the systems. In
this paper, we showed the selection rule of documents is very
important to display the interesting documents to users andkeep the learning performance in the interactive documents
retrieval.
In future work, we will plan to analyze our experimental
results theoretically. Especially, we will plan to analyze therelation between our proposed method and transductive learning based on SVM[15].
R EFERENCES[1] B. Boser, I. Guyon, and V. Vapnik, “A training algorithm for optimal
margin classifiers,” in 5th Annual ACM Workshop on COLT, (D. Haussler, ed.), (Pittsburgh, PA), pp. 144–152, ACM Press, 1992.
[2] H. Drucker, B. Shahrary, and D. C. Gibbon, “Relevance feedback using
support vector machines,” in Proceedings of the Eighteenth International
Conference on Machine Learning, pp. 122–129, 2001.
[3] IREX, “http://cs.nyu.edu/cs/projects/proteus/irex/,”.
[4] Kernel-Machines, “http://www.kernel-machines.org/,”.
[5] D. Lewis, “Evaluating text categorization,” in Proceedings of Speechand Natural Language Workshop, pp. 312–318, 1991.
[6] NTCIR, “http://www.rd.nacsis.ac.jp/˜ntcadm/,”.
[7] M. Okabe and S. Yamada, “Interactive document retrieval with relational
learning,” in Proceedings of the 16th ACM Symposium on Applied
Computing, pp. 27–31, 2001.
[8] T. Onoda, H. Murata, and S. Yamada, “Relevance feedback with active
learning for document retrieval,” in Proc. of IJCNN2003, pp. 1757–1762,
2003.
[9] G. Salton, ed., Relevance feedback in information retrieval, pp. 313–323.
Englewood Cliffs, N.J.: Prentice Hall, 1971.
[10] G. Salton and J. McGill, Introduction to modern information retrieval.
McGraw-Hill, 1983.
[11] B. Schölkopf, A. Smola, R. Williamson, and P. Bartlett, “New support
vector algorithms,” Neural Computaion, vol. 12, pp. 1083 – 1121, 2000.
[12] S. Tong and D. Koller, “Support vector machine active learning withapplications to text classification,” in Journal of Machine Learning
Research, pp. 45–66, 2001.
[13] TREC Web page, “http://trec.nist.gov/,”.
[14] V. Vapnik, The Nature of Statistical Learning Theory. Springer, 1995.
[15] V. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
[16] M. Warmuth, G. Rätsch, M. Mathieson, J. Liao, and C. Lemmen,“Active learning in the drug discovery process,” in Advances in Neural
information processings systems, 2002.
[17] I. Witten, A. Moffat, and T. Bell, Managing Gigabytes: Compressing andIndexing Documents and Images. New York: Van Nostrand Reinhold,1994.
[18] R. B. Yates and B. R. Neto, Modern Information Retrieval. Addison
Wesley, 1999.