Query Expansion with the Minimum User Feedbackby Transductive Learning
Masayuki OKABE

Kyoji UMEMURA

Seiji YAMADAInformation and Media Center

Information and Computer Sciences

National Institute of Informatics

Toyohashi University of Technology

Toyohashi University of Technology

Tokyo,101-8430, Japan

Aichi, 441-8580, Japan

Aichi, 441-8580, Japanseiji@nii.ac.jp

okabe@imc.tut.ac.jp

umemura@tutics.tut.ac.jp

Abstract
Query expansion techniques generally select new query terms from a set of top
ranked documents. Although a user’s
manual judgment of those documents
would much help to select good expansion
terms, it is difficult to get enough feedback
from users in practical situations. In thispaper we propose a query expansion technique which performs well even if a user
notifies just a relevant document and a
non-relevant document. In order to tackle
this specific condition, we introduce two
refinements to a well-known query expansion technique. One is application of atransductive learning technique in order toincrease relevant documents. The other isa modified parameter estimation method
which laps the predictions by multiple
learning trials and try to differentiate theimportance of candidate terms for expansion in relevant documents. Experimental results show that our technique outperforms some traditional query expansion
methods in several evaluation measures.

1

Introduction

Query expansion is a simple but very useful technique to improve search performance by adding
some terms to an initial query. While many query
expansion techniques have been proposed so far, astandard method of performing is to use relevance

information from a user (Ruthven, 2003). If we
can use more relevant documents in query expansion, the likelihood of selecting query terms achieving high search improvement increases. However it
is impractical to expect enough relevance information. Some researchers said that a user usually notifies few relevance feedback or nothing (Dumais andet al., 2003).
In this paper we investigate the potential performance of query expansion under the condition that
we can utilize little relevance information, especially we only know a relevant document and a nonrelevant document. To overcome the lack of relevance information, we tentatively increase the number of relevant documents by a machine learningtechnique called Transductive Learning. Comparedwith ordinal inductive learning approach, this learning technique works even if there is few training examples. In our case, we can use many documents
in a hit-list, however we know the relevancy of few
documents. When applying query expansion, we use
those increased documents as if they were true relevant ones. When applying the learning, there occurs some difficult problems of parameter settings.
We also try to provide a reasonable resolution forthe problems and show the effectiveness of our proposed method in experiments.
The point of our query expansion method is that
we focus on the availability of relevance informationin practical situations. There are several researches
which deal with this problem. Pseudo relevance
feedback which assumes top n documents as relevant ones is one example. This method is simple andrelatively effective if a search engine returns a hit-

list which contains a certain number of relative documents in the upper part. However, unless this assumption holds, it usually gives a worse ranking than
the initial search. Thus several researchers propose
some specific procedure to make pseudo feedback
be effective (Yu and et al, 2003; Lam-Adesina andJones, 2001). In another way, Onoda (Onoda et al.,2004) tried to apply one-class SVM (Support Vector Machine) to relevance feedback. Their purpose
is to improve search performance by using only nonrelevant documents. Though their motivation is similar to ours in terms of applying a machine learning
method to complement the lack of relevance information, the assumption is somewhat different. Our
assumption is to utilizes manual but the minimum
relevance judgment.
Transductive leaning has already been applied in
the field of image retrieval (He and et al., 2004). In
this research, they proposed a transductive method
called the manifold-ranking algorithm and showed
its effectiveness by comparing with active learning based Support Vector Machine. However, their
setting of relevance judgment is not different frommany other traditional researches. They fix the total
number of images that are marked by a user to 20.
As we have already claimed, this setting is not practical because most users feel that 20 is too much forjudgment. We think none of research has not yet answered the question. For relevance judgment, mostof the researches have adopted either of the following settings. One is the setting of “Enough relevant
documents are available”, and the other is “No relevant document is available”. In contrast to them, we
adopt the setting of “Only one relevant document isavailable”. Our aim is to achieve performance improvement with the minimum effort of judging relevancy of documents.
The reminder of this paper is structured as follows. Section 2 describes two fundamental techniques for our query expansion method. Section 3
explains a technique to complement the smallnessof manual relevance judgment. Section 4 introduces
a whole procedure of our query expansion method
step by step. Section 5 shows empirical evidenceof the effectiveness of our method compared with
two traditional query expansion methods. Section 6
investigates the experimental results more in detail.
Finally, Section 7 summarizes our findings.

2 Basic Methods
2.1 Query Expansion
So far, many query expansion techniques have been
proposed. While some techniques focus on thedomain specific search which prepares expansion
terms in advance using some domain specific training documents (Flake and et al, 2002; Oyama and etal, 2001), most of techniques are based on relevance
feedback which is given automatically or manually.
In this technique, expansion terms are selectedfrom relevant documents by a scoring function. The
Robertson’s wpq method (Ruthven, 2003) is often
used as such a scoring function in many researches(Yu and et al, 2003; Lam-Adesina and Jones, 2001).
We also use it as our basic scoring function. It calculates the score of each term by the following formula.
(

wpqt =)

rt /(R − rt )rtn t − rt
−
∗log
R
N −R
(nt − rt )/(N − nt − R + rt )(1)

where rt is the number of seen relevant documents
containing term t. nt is the number of documents
containing t. R is the number of seen relevant documents for a query. N is the number of documentsin the collection. The second term of this formulais called the Robertson/Spark Jones weight (Robertson, 1990) which is the core of the term weighting
function in the Okapi system (Robertson, 1997).
This formula is originated in the following formula.
pt (1 − qt )wpqt = (pt − qt ) log(2)
qt (1 − pt )where pt is the probability that a term t appears in
relevant documents. qt is the probability that a termt appears in non-relevant documents. We can easily
notice that it is very important how the two probability of pt and qt should be estimated. The firsttformula estimates pt with rRt and qt with NNt −R−R . For
the good estimation of pt and qt , plenty of relevant
document is necessary. Although pseudo feedback
which automatically assumes top n documents asrelevant is one method and is often used, its performance heavily depends on the quality of an initial
search. As we show later, pseudo feedback has limited performance.
We here consider a query expansion techniquewhich uses manual feedback. It is no wondermanual feedback shows excellent and stable performance if enough relevant documents are available,
hence the challenge is how it keeps high performance with less amount of manual relevance judgment. In particular, we restrict the manual judgmentto the minimum amount, namely only a relevant
document and a non-relevant document. In thisassumption, the problem is how to find more relevant documents based on a relevant document and a
non-relevant document. We use transductive learning technique which is suitable for the learning problem where there is small training examples.
2.2 Transductive Learning
Transductive learning is a machine learning technique based on the transduction which directly derives the classification labels of test data without
making any approximating function from training
data (Vapnik, 1998). Because it does not need tomake approximating function, it works well even if
the number of training data is small.
The learning task is defined on a data set X
of n points. X consists of training data set
L = (⃗x1 , ⃗x2 , ..., ⃗xl ) and test data set U =(⃗xl+1 , ⃗xl+2 , ..., ⃗xl+u ); typically l ≪ u. The purposeof the learning is to assign a label to each data pointin U under the condition that the label of each data
point in L are given.
Recently, transductive learning or semisupervised learning is becoming an attractive
subject in the machine learning field. Several
algorithms have been proposed so far (Joachims,
1999; Zhu and et al., 2003; Blum and et al., 2004)
and they show the advantage of this approach invarious learning tasks. In order to apply transductive
learning to our query expansion, we select an algorithm called “Spectral Graph Transducer (SGT)“(Joachims, 2003), which is one of the state of the art
and the best transductive learning algorithms. SGT
formalizes the problem of assigning labels to U with
an optimization problem of the constrained ratiocut.
By solving the relaxed problem, it produces anapproximation to the original solution.
When applying SGT to query expansion, X corresponds to a set of top n ranked documents in a
hit-list. X does not corresponds to a whole document collection because the number of documents

in a collection is too huge1 for any learning system to process. L corresponds to two documentswith manual judgments, a relevant document anda non-relevant document. Furthermore, U corresponds to the documents of X ∩ L whose relevancy is unknown. SGT is used to produce the relevancy of documents in U . SGT actually assigns
values around γ+ − θ for documents possibly being relevant and γ− − θ for√documents possibly√ be1−fp
fp
ing non-relevant. γ+ = +fp , γ− = − 1−fp ,θ = 12 (γ+ + γ− ), and fp is the fraction of relevant
documents in X. We cannot know the true value of
fp in advance, thus we have to estimate its approximation value before applying SGT.
According to Joachims, parameter k (the numberof k-nearest points of a data ⃗x) and d (the numberof eigen values to ...) give large influence to SGT’slearning performance. Of course those two parameters should be set carefully. However, besides them,
fp is much more important for our task because it
controls the learning performance. Since extremely
small L (actually |L| = 2 is our setting) give no
information to estimate the true value of fp , we donot strain to estimate its single approximation value
but propose a new method to utilize the results oflearning with some promising fp . We describe themethod in the next section.

3 Parameter Estimations based on
Multiple SGT Predictions
3.1 Sampling for Fraction of Positive Examples
SGT prepares 2 estimation methods to set fp automatically. One is to estimate from the fractionof positive examples in training examples. Thismethod is not suitable for our task because fp isalways fixed to 0.5 by this method if the numberof training examples changes despite the number of
relevant documents is small in many practical situations. The other is to estimate with a heuristic thatthe difference between a setting of fp and the fraction of positive examples actually assigned by SGT
should be as small as possible. The procedure provided by SGT starts from fp = 0.5 and the next fp is
set to the fraction of documents assigned as relevantin the previous SGT trial. It repeats until fp changes
1Normally it is more than ten thousand.

InputNtr // the number of training examples
OutputS// a set of sampling points

to a real value which can represents the relevancy of
documents in more detail. The main merit of thisapproach in comparison with fixing fp to a single
value, it can differentiate a value of pt if Ntr is small.

piv = ln(Ntr ); // sampling interval
nsp = 0;// the number of sampling points
for(i = piv; i ≤ Ntr − 1; i+ = piv){
add i to ;nsp++;
if(nsp == 10){ exit; }}4 Expansion Procedures
We here explain a whole procedure of our query expansion method step by step.
1. Initial Search: A retrieval starts by inputting aquery for a topic to an IR system.

Figure 1: Pseudo code of sampling procedure for fp
five times or the difference converges less than 0.01.
This method is neither works well because the convergence is not guaranteed at all.
Presetting of fp is primarily very difficult problem
and consequently we take another approach whichlaps the predictions of multiple SGT trials with somesampled fp instead of setting a single fp . This approach leads to represent a relevant document by nota binary value but a real value between 0 and 1. The
sampling procedure for fp is illustrated in Figure 1.
In this procedure, sampling interval changes according to the number of training examples. In our preliminary test, the number of sampling points should
be around 10. However this number is adhoc one,thus we may need another value for another corpus.
3.2 Modified estimations for pt and qt
Once we get a set of sampling points S = {fpi :i = 1 ∼ 10}, we run SGT with each fpi and laps
each resultant of prediction to calculate pt and qt as
follows.
∑ ir∑
pt = i t
(3)
R
i i
∑
nt − rti
qt = ∑ i

i N − Ri

(4)

Here, Ri is the number of documents which SGT
predicts as relevant with ith value of fpi , and rti isthe number of documents in Ri where a term t appears. In each trial, SGT predicts the relevancy of
documents by binary value of 1 (for relevant) and 0(for non-relevant), yet by lapping multiple resultant
of predictions, the binary prediction value changes

2. Relevance Judgment for Documents in a
Hit-List: The IR system returns a hit-list for
the initial query. Then the hit-list is scanned
to check whether each document is relevant or
non-relevant in descending order of the ranking. In our assumption, this reviewing process terminates when a relevant document anda non-relevant one are found.
3. Finding more relevant documents by transductive learning: Because only two judged
documents are too few to estimate pt and qt
correctly, our query expansion tries to increase
the number of relevant documents for the wpq
formula using the SGT transductive learning algorithm. As shown in Figure2, SGT assigns avalue of the possibility to be relevant for the
topic to each document with no relevance judgment (documents under the dashed line in theFig) based on two judged documents (documents above the dashed line in the Figure).
Hit list
1. Document

1

Manually

2. Document0

assigned

3. Document?

4. Document?

:
i. Document

Assigned byTransductive?

Learning:“1” means a positive label

Labels“0” means a negative label“?” means an unknown label

Figure 2: A method to find tentative relevant documents

4. Selecting terms to expand the initial query:
Our query expansion method calculates thescore of each term appearing in relevant documents (including documents judged as relevant
by SGT) using wpq formula, and then selects
a certain number of expansion terms according
to the ranking of the score. Selected terms areadded to the initial query. Thus an expanded
query consists of the initial terms and added
terms.

5.1.2 Retrieval Models
We use two representative retrieval models which
are bases of the Okapi (Robertson, 1997) and
SMART systems. They showed highest performance in the TREC-8 competition.
Okapi : The weight function in Okapi is BM25. It
calculates each document’s score by the following formula.
score(d) =5. The Next Search with an expanded query:
The expanded query is inputted to the IR system and a new hit-list will be returned. One
cycle of query expansion finishes at this step.

w(1) = log(5)(rt + 0.5)/(R − rt + 0.5)
(nt − rt + 0.5)/(N − nt − R + rt + 0.5)
(6)
(
)
dl
K = k1 (1 − b) + b
(7)
avdl

SMART : The SMART’s weighting function is asfollows2 .

This section provides empirical evidence on how
our query expansion method can improve the performance of information retrieval. We compare ourmethod with other traditional methods.

score(d) =∑{1 + ln(1 + ln(tf ))} ∗ log(

T ∈Qpivot =

5.1 Environmental Settings

N +1) ∗ pivot (8)df

1dl
0.8 + 0.2 × avdl

(9)

df is the term’s document frequency. tf , dl andavdl are the same as Okapi. When doing relevance feedback, a query vector is modified by
the following Rocchio’s method (with parameters α = 3, β = 2, γ = 2).

5.1.1 Data set
We use the TREC-8 data set (Voorhees and Harman, 1999) for our experiment. The document corpus contains about 520,000 news articles. Each document is preprocessed by removing stopwords andstemming. We also use fifty topics (No.401-450)
and relevance judgments which are prepared for adhoc task in the TREC-8. Queries for an initial search
are nouns extracted from the title tag in each topic.

(k1 + 1)tf (k3 + 1)qtf(K + tf )(k3 + qtf )where Q is a query containing terms T , tf
is the term’s frequency in a document, qtf isthe term’s frequency in a text from which Q
was derived. rt and nt are described in section 2. K is calculated by (7), where dl andavdl denote the document length and the average document length. In our experiments,
we set k1 = 1.2, k3 = 1000, b = 0.75, andavdl = 135.6. Terms for query expansion are
ranked in decreasing order of rt × w(1) for thefollowing Okapi’s retrieval tests without SGT(Okapi manual and Okapi pseudo) to makeconditions the same as of TREC-8.

The computational cost of transductive learning is not so much. Actually transductive learning takes a few seconds to label 100 unlabeled documents and query expansion with all
the labeled documents also takes a few seconds. Thus our system can expand queries sufficiently quick in practical applications.

Experimentsw(1) ·T ∈Q

In the above procedures, we naturally introduced transductive learning into query expansion as the effective way in order to automatically find some relevant documents. Thus we
do not need to modify a basic query expansion procedure and can fully utilize the potential power of the basic query expansion.

5∑⃗ new = αQ
⃗ old +
Q

∑
γ
β ∑⃗
d−
d⃗ (10)|Drel |
|Dnrel |
Drel

2

Dnrel

In this paper, we use AT&T’s method (Singhal et al., 1999)
applied in TREC-8

Table 1: Results of Initial SearchOkapi ini
SMART ini

P10
0.466
0.460

P30
0.345
0.336

RPREC
0.286
0.271

MAP
0.239
0.229

R05P
0.195
0.187

Table 2: Results of Okapi sgt (5 terms expanded)
20
50
100

P10
0.516
0.494
0.436

P30
0.381
0.380
0.345

RPREC
0.308
0.286
0.283

MAP
0.277
0.265
0.253

R05P
0.233
0.207
0.177

Drel and Dnrel are sets of seen relevant andnon-relevant documents respectively. Terms
for query expansion are ranked in decreasingorder of the above Rocchio’s formula.

Table 3: Results of Okapi sgt (10 terms expanded)

Table 1 shows their initial search results of Okapi(Okapi ini) and SMART (SMART ini). We adopt
five evaluation measures. Their meanings are as follows (Voorhees and Harman, 1999).

Table 4: Results of Okapi sgt (15 terms expanded)

P10 : The precision after the first 10 documents are
retrieved.

Table 5: Results of Okapi sgt (20 terms expanded)

20
50
100

20
50
100

20
50
100

P30 : The precision after the first 30 documents are
retrieved.
R-Prec : The precision after the first R documents
are retrieved, where R is the number of relevant
documents for the current topic.
MAP : Mean average precision (MAP) is the average precision for a single topic is the mean of
the precision obtained after each relevant document is retrieved (using zero as the precision
for relevant documents that are not retrieved).
R05P : Recall at the rank where precision first dips
below 0.5 (after at least 10 documents havebeen retrieved).
The performance of query expansion or relevance
feedback is usually evaluated on a residual collection where seen documents are removed. However
we compare our method with pseudo feedback based
ones, thus we do not use residual collection in thefollowing experiments.
5.1.3 Settings of Manual Feedback
For manual feedback, we set an assumption thata user tries to find relevant and non-relevant documents within only top 10 documents in the resultof an initial search. If a topic has no relevant document or no non-relevant document in the top 10
documents, we do not apply manual feedback, instead we consider the result of the initial search for

P10
0.508
0.520
0.494

P10
0.538
0.528
0.498

P10
0.546
0.520
0.498

P30
0.383
0.387
0.365

P30
0.381
0.387
0.363

P30
0.387
0.385
0.369

RPREC
0.301
0.294
0.283

RPREC
0.298
0.298
0.280

RPREC
0.307
0.299
0.272

MAP
0.271
0.273
0.261

MAP
0.274
0.283
0.259

MAP
0.289
0.282
0.255

R05P
0.216
0.208
0.190

R05P
0.223
0.222
0.197

R05P
0.235
0.228
0.188

such topics. There are 8 topics 3 which we do not
apply manual feedback methods.
5.2 Basic PerformanceFirstly, we evaluate the basic performance of ourquery expansion method by changing the numberof training examples. Since our method is based on
Okapi model, we represent it as Okapi sgt (with parameters k = 0.5 ∗ Ntr , d = 0.8 ∗ Ntr . k is the number of nearest neighbors, d is the number of eigen
values to use and Ntr is the number of training examples).
Table 2-5 shows five evaluation measures of
Okapi sgt when the number of expansion terms
changes. We test 20, 50 and 100 as the number oftraining examples and 5, 10 15 and 20 for the number of expansion terms. As for the number of training examples, performance of 20 and 50 does notdiffer so much in all the number of expansion terms.
However performance of 100 is clearly worse thanof 20 and 50. The number of expansion terms does
not effect so much in every evaluation measures. In
the following experiments, we compare the resultsof Okapi sgt when the number of training examples
is 50 with other query expansion methods.
3450

Topic numbers are 409, 410, 424, 425, 431, 432, 437 andTable 6: Results of Manual Feedback Methods(MAP)
Okapi sgt
Okapi man
SMART man

5
0.265
0.210
0.209

10
0.273
0.189
0.222

15
0.274
0.172
0.220

20
0.282
0.169
0.219

Okapi sgt
Okapi pse
SMART pse

Table 7: Results of Manual Feedback Methods (10terms expanded)
Okapi sgt
Okapi man
SMART man

P10
0.520
0.420
0.434

P30
0.387
0.285
0.309

RPREC
0.294
0.212
0.250

Table 8: Results of Pseudo Feedback Methods
(MAP)

MAP
0.273
0.189
0.222

R05P
0.208
0.132
0.174

5.3 Comparison with other Manual Feedback
Methods
We next compare our query expansion method withthe following manual feedback methods.
Okapi man : This method simply uses only one
relevant document judged by hand. This iscalled incremental relevance feedback (Aalbersberg, 1992; Allan, 1996; Iwayama, 2000).
SMART man : This method is SMART’s manual
relevance feedback (with parameters α = 3,
β = 2, γ = 0). γ is set to 0 because the performance is terrible if γ is set to 2.
Table 6 shows the mean average precision of
three methods when the number of expansion terms
changes. Since the number of feedback documents is extremely small, two methods except for
Okapi sgt get worse than their initial searches.
Okapi man slightly decreases as the number of expansion terms increases. Contrary, SMART man
do not change so much as the number of expansion
terms increases. Table 7 shows another evaluation
measures with 10 terms expanded. It is clear that
Okapi sgt outperforms the other two methods.
5.4 Comparison with Pseudo Feedback
Methods
We finally compare our query expansion methodwith the following pseudo feedback methods.
Okapi pse : This is a pseudo version of Okapi
which assumes top 10 documents in the initial
search as relevant ones as well as TREC-8 settings.

5
0.265
0.253
0.236

10
0.273
0.249
0.243

15
0.274
0.247
0.242

20
0.282
0.246
0.242

Table 9: Results of Pseudo Feedback Methods (10terms expanded)
Okapi sgt
Okapi pse
SMART pse

P10
0.520
0.478
0.466

P30
0.387
0.369
0.359

RPREC
0.294
0.279
0.272

MAP
0.273
0.249
0.243

R05P
0.208
0.206
0.187

SMART pse : This is a pseudo version of SMART.
It also assumes top 10 documents as relevant
ones. In addition, it assumes top 500-1000 documents as non-relevant ones.
In TREC-8, above two methods uses TREC1-5 disks
for query expansion and a phase extraction technique. However we do not adopt these methods inour experiments4 . Since these methods showed thehighest performance in the TREC-8 adhoc task, it
is reasonable to compare our method with them ascompetitors.
Table 8 shows the mean average precision of
three methods when the number of expansion terms
changes. Performance does not differ so much if thenumber of expansion terms changes. Okapi sgt outperforms at any number of expansion. Table 9 shows
the results in other evaluation measures. Okapi sgt
also outperforms except for R05P. In particular, performance in P10 is quite well. It is preferable behavior for the use in practical situations.

6 Discussion
In the experiments, the feedback documents for
Okapi sgt is top ranked ones. However some users
do not select such documents. They may choose
another relevant and non-relevant documents whichrank in top 10. Thus we test an another experiment
where relevant and non-relevant documents are selected randomly from top 10 rank. Table 10 shows
the result. Compared with table 2, the performanceseems to become slightly worse. This shows that a
4

Thus the performance in our experiments is a bit worse than
the result of TREC-8

Table 10: Results of Okapi sgt with random feedback (5 terms expanded)
20
50
100

P10
0.498
0.456
0.452

P30
0.372
0.359
0.335

RPREC
0.288
0.294
0.270

MAP
0.265
0.268
0.246

R05P
0.222
0.200
0.186

M. Iwayama. 2000. Relevance feedback with a small
number of relevance judgements: Incremental relevance feedback vs. document clustering. In Proceedings of SIGIR 2000, pages 10–16.
T. Joachims. 1999. Transductive inference for text classification using support vector machines. In Proceedings of ICML ’99.

user should select higher ranked documents for relevance feedback.

T. Joachims. 2003. Transductive learning via spectral
graph partitioning. In Proceedings of ICML 2003,
pages 143–151.

7

A. M. Lam-Adesina and G. J. F. Jones. 2001. Applying
summarization techniques for term selection in relevance feedback. In Proceedings of SIGIR 2001, pages
1–9.

Conclusion

In this paper we proposed a novel query expansion
method which only use the minimum manual judgment. To complement the lack of relevant documents, this method utilizes the SGT transductive
learning algorithm to predict the relevancy of unjudged documents. Since the performance of SGTmuch depends on an estimation of the fraction of
relevant documents, we propose a method to sample some good fraction values. We also propose amethod to laps the predictions of multiple SGT trials with above sampled fraction values and try todifferentiate the importance of candidate terms for
expansion in relevant documents. The experimental
results showed our method outperforms other query
expansion methods in the evaluations of several criteria.

References
I. J. Aalbersberg. 1992. Incremental relevance feedback.
In Proceedings of SIGIR ’92, pages 11–22.
J. Allan. 1996. Incremental relevance feedback for information filtering. In Proceedings of SIGIR ’96, pages
270–278.
A. Blum and et al. 2004. Semi-supervised learning using
randomized mincuts. In Proceedings of ICML 2004.
S. Dumais and et al. 2003. Sigir 2003 workshop report:
Implicit measures of user interests and preferences. In
SIGIR Forum.
G. W. Flake and et al. 2002. Extracting query modification from nonlinear svms. In Proceedings of WWW
2002.
J. He and et al. 2004. Manifold-ranking based image
retrieval. In Proceedings of Multimedia 2004, pages
9–13. ACM.

T. Onoda, H. Murata, and S. Yamada. 2004. Nonrelevance feedback document retrieva. In Proceedingsof CIS 2004. IEEE.
S. Oyama and et al. 2001. keysword spices: A new
method for building domain-specific web search engines. In Proceedings of IJCAI 2001.
S. E. Robertson. 1990. On term selection for query expansion. Journal of Documentation, 46(4):359–364.
S. E. Robertson. 1997. Overview of the okapi projects.
Journal of the American Society for Information Science, 53(1):3–7.
I. Ruthven. 2003. Re-examining the potential effectiveness of interactive query expansion. In Proceedings of
SIGIR 2003, pages 213–220.
A. Singhal, S. Abney, B. Bacchiani, M. Collins, D. Hindle, and F. Pereira. 1999. At&t at trec-8.
V Vapnik. 1998. Statistical learning theory. Wiley.
E. Voorhees and D. Harman. 1999. Overview of theeighth text retrieval conference.
S. Yu and et al. 2003. Improving pseud-relevance feedback in web information retrieval using web page segmentation. In Proceedings of WWW 2003.
X Zhu and et al. 2003. Semi-supervised learning using
gaussian fields and harmonic functions. In Proceedings of ICML 2003, pages 912–914.