IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,

VOL. 19,

Semisupervised Query Expansion
with Minimal Feedback
Masayuki Okabe and Seiji Yamada
Abstract—Query expansion is an information retrieval technique in which newquery terms are selected to improve search performance. Although useful terms
can be extracted from documents whose relevance is already known, it is difficult
to get enough of such feedback from a user in actual use. We propose a query
expansion method that performs well even if a user makes practically minimum
effort, that is, chooses only a single relevant document. To improve searches inthese conditions, we made two refinements to a well-known query expansion
method. One uses transductive learning to obtain pseudorelevant documents,
thereby increasing the total number of source documents from which expansion
terms can be extracted. The other is a modified parameter estimation method that
aggregates the predictions of multiple learning trials to sort candidate terms for
expansion by importance. Experimental results show that our method outperforms
traditional methods and is comparable to a state-of-the-art method.
Index Terms—Information search and retrieval, query formulation, relevance
feedback, machine learning.

Ç
1

INTRODUCTION

WHEN searching for information, people usually do not obtainmuch relevant information in the initial search and need to modify
queries and search again until they get satisfactory results. Query
expansion is an automatic search technique that extracts usefulterms from selected documents to improve the search. Althoughmany techniques have been proposed, a standard method is to use
relevant information provided by a user [1]. Although a large
amount of relevant information would help in selecting effective
search terms, users generally do not provide a lot of it. They usuallyprovide little or no relevant feedback because identifying relevant
documents is very time consuming [2].
Pseudofeedback, which assumes that top-ranked documents
are relevant, is a standard way of enhancing initial search resultsthat requires no human input regarding relevance. Though thistechnique has been proven to be effective, its performance depends
on the quality of the initial search. If there are no relevant
documents among the search results, pseudofeedback may
improve little on the initial result [3]. There are several improved
pseudofeedback approaches, including use of clustering [4], oneclass support vector machine (SVM) [5], large external corpora [6],and implicit feedback [7].
In contrast to the above approaches, we consider a type ofmanual feedback where users stop the search as soon as the first
relevant document is found. This approach is very importantbecause even a single relevant document is much more effective
than many nonrelevant documents [3]. Although this type of
feedback can be gotten simply by inputting one relevant document
into the traditional retrieval methods, some methods, such as
OKAPI [8] and SMART [9], cannot get enough statistics and variety
to expand query terms because they use frequency of relevant
documents to select expansion terms.

. M. Okabe is with the Information and Media Center, Toyohashi Universityof Technology, Tempaku 1-1, Toyohashi, Aichi, 441-8580, Japan.
E-mail: okabe@imc.tut.ac.jp.
. S. Yamada is with the National Institute of Informatics, 2-1-2
Hitotsubashi, Chiyoda-ku, Toyoko 101-8430, Japan. E-mail: seiji@nii.ac.jp.
Manuscript received 30 Aug. 2006; revised 16 Apr. 2007; accepted 19 July
2007; published online 2 Aug. 2007.
For information on obtaining reprints of this article, please send e-mail to:
tkde@computer.org, and reference IEEECS Log Number TKDE-0407-0806.
Digital Object Identifier no. 10.1109/TKDE.2007.190646.
1041-4347/07/$25.00 ß 2007 IEEE

Published by the IEEE Computer Society

NO. 11,NOVEMBER 2007

1585

We use transductive learning [10] to overcome the problem oflack of relevant information. This learning approach is more
effective than traditional inductive learning, especially when there
are few training examples. Using this technique, we attempt toincrease the number of pseudorelevant documents based on
relevant documents. Once pseudorelevant documents are obtained, they are used to select expansion terms. Simple applicationof transductive learning does not work well because parametertuning is very difficult. Therefore, we modify the basic scoring
function of OKAPI to apply learning to query expansion.
The remainder of this paper is structured as follows: Section 2
introduces our query expansion method step by step and explains
two fundamental techniques: Robertson’s wpq method and theSpectral Graph Transducer (SGT) algorithm. Section 3 explains
how our method overcomes problems that occur when SGT isapplied to it. Section 4 demonstrates the effectiveness of our
method as compared with other query expansion techniques.
Section 5 analyzes the experimental results. Finally, Section 6summarizes our findings.

2

BASIC METHODS

2.1

Query Expansion with Transductive Learning

First of all, we explain our query expansion method step by step. In
the following procedure, (U) represents a step executed by a userand (S) represents a step executed by the system.
1.
2.

3.

Initial search (U). A user starts a retrieval by inputting aquery to an IR system.
Minimal user feedback for relevance judgment (U).
When an IR system returns, a hit list for the initial query,the user checks it and judges whether or not eachdocument is relevant in descending order of the ranking.
At a minimum, the user stops perusing documents after hefinds the first relevant one. Based on this minimum, ourmethod uses one relevant and several irrelevant documents as training examples for transductive learning in thenext step.
[Retrieval/Discovery] of other relevant documents bytransductive learning (S). After the user selects a relevant
document, a transductive learning algorithm is used tofind relevant documents the user did not find. As shown in
Fig. 1, it carries out the retrieval process by assigning alabel (relevant or irrelevant) to each unjudged (unlabeled)document based on a small set of judged (labeled) datafrom the previous step.

Selecting expansion terms (S). After a set of relevant
documents has been retrieved, the system calculates ascore for each term in those documents. This
conventional scoring function will be described later.
The terms with the top m scores are selected for
expansion.
.
Search with an expanded query (S). The initial queryis expanded by the additional terms as describedabove, and the expanded query is inputted to the
IR system to get a new hit list. This is the end of thefirst cycle of query expansion.
In these procedures, we naturally introduced learning intoquery expansion as an effective way of automatically finding
relevant documents. Thus, we do not need to modify the basic
query expansion procedure and can fully utilize the potential
power of the basic query expansion.
.

2.2

Score Function for Query Expansion

Many query expansion techniques have been proposed. Althougha few techniques use precategorized training documents to1586

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,

2.3

Fig. 1. Method of tentatively identifying relevant documents.

prepare expansion terms for domain specific search [11], [12], mostof them are based on relevance feedback that is given automatically or manually at the moment of ad hoc retrieval. How well thistype of method performs depends on the amount of feedback it
receives and on its ability to rank candidate terms for expansion.
The scoring function proposed in Robertson’s wpq method is
generally considered to be the effective one and has been used inmany researches [1], [13], [14], [15]. Our method is based on thisfunction. Since we actually use a modified version of the function,
we introduce it here and discuss it in Section 3.
The function calculates the score of each term using thefollowing equation:
scoreðtÞ ¼

r

t

R



n t  rt 
rt =ðR  rt Þ log;ðnt  rt Þ=ðN  nt  R þ rt Þ
N R

ð1Þ

where rt is the number of relevant documents containing the termt, nt is the number of documents containing t, R is the number of
relevant documents for a query, and N is the number of all
documents in the collection. This equation is based on the
Robertson/Spark Jones weight [16], which is the core of the term
weighting function in the Okapi system [8]. Note that this equation
is derived from a probabilistic framework, which can be shown bytransforming it as follows:scoreðtÞ ¼ ðpt  qt Þ log

pt ð1  qt Þ;
qt ð1  pt Þ

ð2Þ

qt ¼

nt  rt
;
N R

ð3Þ

ð4Þ

where pt and qt represent the probability that a term t appears in
relevant and nonrelevant documents, respectively. Equations (3)and (4) can be understood as estimates of these probabilities. Thismeans that a term’s score becomes higher when it appears more
frequently in relevant documents and less frequently in nonrelevant documents.
However, if a user only identifies a single relevant document, asin this work, pt is 0 or 1 based on (3) because both R and rt are 1.
nt
t 1
or nN1
based on (3). Since scores are mostly
Similarly, qt is N1
determined by the value of nt , there would be little advantage inthe feedback. The value of R and rt must be increased for pt and qt
to be estimated with higher reliability. That is why we use
transductive learning, which is effective even if there are only a
few training examples.

NO. 11,NOVEMBER 2007

Transductive Learning

Transductive learning is a machine learning technique based on
the transduction that creates classification labels for test data
directly without making any approximate function from a training
data [10]. The learning task is defined on a data set X of n points. X
consists of a training data set L ¼ ð~
x1 ; ~
x2 ; . . . ; ~xl Þ and a test data set
U ¼ ð~
xlþ1 ; ~xlþ2 ; . . . ; ~xlþu Þ, typically, l  u. The purpose of thelearning is to assign a label to each data point in U under thecondition that the label of each data point in L is given.
Normal inductive learning consists of two phases, a learning
phase and an inference phase. In the learning phase, a subset of data
points L and labels YL is given as training examples. The learner
will produce a model to predict labels for the rest of the data points
in U. Using the model in the inference phase, we can finally get
those labels for YU . In contrast to inductive learning, transductive
learning uses both L and U. Moreover, the inference phase is notseparated from the learning phase. Learning and inference areconducted at the same time in a transductive setting. The
transductive learner presumes labels of test examples using
similarities between labeled and unlabeled data points to complement the lack of training examples. Transductive learning is well
suited to settings in which jLj is very small.
Transductive learning or semisupervised learning has recently
become a popular subject in the machine learning field. Several
algorithms have been proposed, and their superiority in various
learning tasks has been demonstrated [17], [18], [19]. To apply
transductive learning to our query expansion, we chose SGT [20], astate-of-the-art transductive learning algorithm. SGT formalizes
the problem of assigning labels to U with an optimization problemof the constrained ratio cut in an undirected graph. By solving its
relaxed problem, it produces a solution that approximates theoriginal one.
SGT first makes an undirected graph G, where vertices are
directed from each datum xi to the k data most similar to xi . Then,it divides the graph into two parts. One part includes all the
positive data but no negative data, and the other part contains
negative but no positive data. Finally, it assigns common labels toall data in each part. The learning performance depends strongly
on how graph G is divided.
This problem can be formulated by the following optimization
problem:
min
~
y

s:t:
rt
pt ¼ ;
R

VOL. 19,cutðGþ ; G Þ
jfi : yi ¼ 1gkfi : yi ¼ 1gj
yi ¼ 1;
if xi is positiveyi ¼ 1; if xi is negative~y 2 fþ1; 1gn :ð5Þ

If there is no constraint, this problem is a simple MinCut problem.
Solving it without those constraints raises the possibility of
producing a very biased partitioning in terms of the number ofdata. This partitioning often produces undesirable results. To
avoid this risk, SGT adds the above constraints, which are used tocreate a balanced partitioning. However, adding the constraints
makes the problem NP-hard. SGT transforms it to a relaxed one
and creates an approximate solution.
When applying SGT to our query expansion, a whole data set X
corresponds to a set of top n ranked documents in a hit list. X does
not correspond to a whole document collection because thenumber of documents in a collection is too huge (normally, it ismore than a hundred thousand) for any learning system to process.
L corresponds to a set of manually judged documents, relevant
documents, and nonrelevant documents. U corresponds to a set of
documents whose relevance is unknown. SGT is used to determine
the relevance of documents in U.

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,

VOL. 19,

By solving the problem, SGT assigns a value zi to eachdocumentin U. If the problem is not relaxed,
qﬃﬃﬃﬃﬃﬃ
qﬃﬃﬃﬃﬃﬃ zi takes þ ¼
p
þ 1p
(for
relevant
documents)
or

¼


p
1p (for nonrelevant
documents). Here, p is a fraction of relevant documents in X.
However, the zi s assigned by solving the relaxed problem are realvalues. p is decided by a threshold to make a binary label
assignment. However, the threshold cannot be decided unless p isfixed. Thus, p and threshold are dependent of each other. To
produce the final decision, SGT parameterizes p (we represent it asfpq
inﬃﬃﬃﬃﬃﬃﬃ
thedescription)and estimates þ and  as ^þ ¼
ﬃ following q
ﬃﬃﬃﬃﬃﬃﬃﬃ
1ff
þ fp p and ^ ¼  1fp p , respectively. SGT sets the middle point
between ^þ and ^ as a threshold for the final decision. Here, weneed to pay close attention to the behavior of zi when changing fp .
Although the value of fp does not need to be rigid according to
Joachims’ work [20], we observed that the number of positive datathat SGT assigns differed greatly in our preliminary experiments.
Assuming the minimum of one relevant document, the amount oftraining data is extremely small, so zi does not change as fp
changes. This means that the value of the threshold, namely, thevalue of fp , dominates the learning results in our method. Because
these results directly affect our method’s performance, we need an
appropriate way of setting fp , which will be described in Section 3.

3

PARAMETER ESTIMATIONS BASED ON MULTIPLE
SGT PREDICTIONS

3.1

Sampling Values for the Fraction of Positive
ExamplesSGT uses two estimation methods to set fp automatically [20]. The
first sets the fraction of positive data directly to fp . However,
because the amount of labeled data is so small, we cannot get a
reliable value for fp . The second method takes a heuristic
approach, iteratively approximating fp to an actual fraction of
positive data. In practice, starting from 0.5, SGT resets fp to thefraction of positive data assigned in each learning trial until thedifference between preset fp and an actual fraction converges intoa predefined range. The problem with this approach is that SGT
does not guarantee the convergence. In fact, it did not work well in
our preliminary experiments.
First, it is unrealistic to set fp precisely in advance, especiallywhen there is very little training data as in our condition. If fp is set
to a small value, we are confronted with the problem described in
Section 2, in which many terms have the same pt , and qt is the only
factor that differentiates term candidates for query expansion.
Instead of setting a single fp , we take another approach, whichruns multiple SGT trials with a variable fp based on a sampling
procedure [21]. Algorithm 1 illustrates the procedure that returns a
set of values to which fp can be set. When choosing a samplinginterval for this procedure, we consulted Buckley’s work [22],which tried to approximate a recall-precision value at rank n in a
hit list using linear regression. On the basis of this work, we
assume that the number of relevant documents increases inproportion to the logarithm of the number of n. Thus, we change
fp based on the integral multiplication of lnðnÞ. We sample lessthan 10 different fp in the procedure. Although the number oftimes we should change fp cannot be decided theoretically, it isimportant that the sampling number should not be too small or toolarge in actual use. The objective of this approach is not to find asingle appropriate value of fp but to increase the variety of term
candidates by increasing the number of relevant documents. After

NO. 11,NOVEMBER 20071587

a prediction value assigned by SGT has been aggregated in eachtrial, each document has a real value between 0 and 1 (not a binary
value). This value represents the possibility that a document isrelevant. Based on this value, any document that has a positive
possibility value can be considered a relevant document. However,the score of candidate terms is calculated based on the possibility
value of the document where the candidates appear.
Algorithm 1 Sampling Procedure for fp
1: Input : n// top n documents in a hit list
2: Output: S// a set of values for fp
3: piv ( lnðnÞ;// sampling interval
4: for i ¼ 1 to 10 do
5:v ¼ ðpiv  iÞ=n;6:
if v > 1:0 then
7:
exit;
8:
else
9:
add v to S;
10:
end if
11: end for

3.2

Modified Estimations for pt and qt

Once a set of sampling points S ¼ ffpi : i ¼ 1  10g is determined,
we run SGT using fpi in turn. Then, we aggregate each prediction
value when calculating pt and qt as follows:
P ir
pt ¼ P i t ;
ð6Þ
i Ri
P
ðnt  rit Þ;
qt ¼ P i
i ðN  Ri Þð7Þwhere Ri is the number of documents that SGT predicts as relevantwith the ith value of fpi , and rit is the number of documents in Ri ,where a term t appears. In each trial, SGT predicts the relevance of
documents by a binary value of 1 (for relevant) and 0 (fornonrelevant). However, if multiple predictions are aggregated,the binary prediction value can be changed to a real value that canrepresent rankings of relevance of documents. The main merit ofthis approach in comparison with fixing fp to a single value is that it
can differentiate a value of pt even if the number of training
examples L is small. Since the number of relevant documents isgenerally small, fp increases gradually. Furthermore, by using asmall fp first, we keep the same ranking of documents and avoid therisk of invalid fp .

4

EXPERIMENTS

In this section, we present evidence that demonstrates thesuperiority of our query expansion method. We also compare
our method with traditional methods.

4.1

Data Set and Evaluation Measures

We used the data set from the ad hoc track of the TREC-8
workshop [23] in our experiments. This data set consists of about
520,000 news documents on 50 topics (No.401-450) and relevancejudgments for the topics. Before the experiment, we removed
stopwords from documents and applied stemming. We extracted
query terms for each topic’s initial search from the title field of a
topic.
We used three evaluation measures: P10, P30, and mean
average precision (MAP). P10 and P30 are the precision for the
first 10 and 30 retrieved documents, respectively. MAP is theaverage precision for a single topic. It is the mean of the precisionobtained when each relevant document is retrieved. Zero is used

1588

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,

VOL. 19,

NO. 11,NOVEMBER 2007

Fig. 2. Results for the parameter n.

Fig. 3. Results for the parameter q.

as the precision for relevant documents that are not retrieved. P10
and P30 are user-oriented measures intended to evaluate the upper
part of a ranking. This type of measure is suited to an ad hoc
search, where a user is satisfied if just a few relevant documents
are found. In contrast, MAP is a system-oriented measure that isbased on recall and considers both top-ranked documents andrelatively low-ranked documents. This type of measure is suitable
when a user wants to collect as many relevant documents as
possible. Since our method is not suited to types of retrieval, we
use these three measures.

In addition to comparing each method with manual feedback,
we also attached the results of initial retrieval and pseudofeedback
for reference. A summary of these results is also shown in Table 1.
The columns Man-rsd and Man are manual feedback results, andPse and Ini are pseudofeedback and initial retrieval results,
respectively. The difference between Man-rsd and Man is whether
it is evaluated using the residual collection or the whole collection.
Though manual feedback is usually evaluated using the residualcollection where already seen (judged) relevant and nonrelevant
documents are removed, we added Man as a comparison with Pse
and Ini.
The best parameters for manual feedback (see the column of
Man-rsd in Table 1) are given as follows: SGT: n ¼ 40, q ¼ 6,
OKAPI: q ¼ 0, SMART: q ¼ 0, ð; ; Þ ¼ ð3; 0; 0Þ, and LANG: ¼ 0:8. The relevance judgment procedure is the same as
described in Section 2 in each method. We set q ¼ 0 for OKAPIand SMART because when we set q larger than 1, they got worse
results even than initial retrieval.2 However, SGT is used, OKAPI
performs 12 percent  18 percent better on all measures. SGT
performed 8 percent  12 percent better than SMART on allmeasures. In contrast, SGT performed 6 percent worse than LANG
on P10 and about the same on P30 and MAP.
Although it is not a fair comparison, it can be seen that our
method is better than any of the pseudofeedback methods.
Parameters set for each method in pseudofeedback retrieval are
listed as follows: OKAPI: q ¼ 6, SMART: q ¼ 16, ð; ; Þ ¼ ð7; 7; 4Þ,and LANG:  ¼ 0:8.

4.2

Parameter Tuning

We first tested our method’s performance using various parameters. We used Joachims’ SGT package1 in our experiments.
Parameters for SGT were set to default values except for k (thenumber of nearest neighbors) and d (the number of eigenvalues touse for learning). These were set as k ¼ 0:5  n and d ¼ 0:8  n,respectively, where n is the number of data used for learning (top n
documents in a hit list). Each datum is represented as a document
vector with normal tf  idf weighting. SGT uses the cosine
coefficient as a similarity measure.
Fig. 2 shows the results of our method when changing nfrom 20 to 100. The number of expansion (added) terms q isfixed to 5. Though the graph of P10 is not stable, all measurespeaked at n ¼ 40. Fig. 3 shows the results of our method whenq was changed from 0 to 10. n was set to 40. In this graph, P10
is also unstable until q ¼ 5, but it maintains constant values forq > 6. P30 and MAP gradually increased and reached peaks at
q ¼ 6 and gradually decreased after that. The best parameters
for this data set are n ¼ 40 and q ¼ 6. We used theseparameters in the following comparison.

4.3

Comparison with Other Methods

We compared our method with three others: OKAPI [24], SMART[25], and LANG [26]. Since OKAPI is a basic retrieval model of our
method, we chose it to test the utility of our extension. SMART isan OKAPI competitor, as shown by the TREC-8 results. LANG is alanguage modeling approach that outscored OKAPI on the TREC-8
data set, as shown in [26]. Detailed descriptions of these models
can be found in [24], [25], [26]. We implemented them using thesame scoring formulas described in the above references. Parameters of OKAPI (BM25) are set as k1 ¼ 1:2, k3 ¼ 1; 000, andb ¼ 0:75. We tested various 0  q  30 and selected the best one for
OKAPI and SMART. In addition to q, we tested all combinations ofparameters 10        1 to update the SMART query vector
and to identify the best one. We used the mixture approach for
LANG [26]. We tested various 0:1    0:9 to determine the bestbalance of the probabilistic model in LANG ( is different from theone in SMART). All terms having more than 0.001 are added as
query expansion. Table 1 shows the results with parametersoptimized as described above.
1. http://sgt.joachims.org.

5

DISCUSSION

The amount of manual feedback assumed in our experiment was
clearly not sufficient for OKAPI and SMART whose results were
marginally better than the initial retrieval but worse than thepseudofeedback, which does not use any manual feedback.
Applying our extension to OKAPI with manual feedback produces
better results than pseudofeedback. Although SGT does not
perform as well as LANG on the P10 evaluation measure, it iscomparable on the other two measures.
There is a big difference between LANG and other methods inthe number of expansion terms q. The average q in LANG is about255. In contrast, qs of other methods are less than 20. The
computational cost of working with large numbers of query terms
might be critical in some computer environments. The number of
documents that require manual feedback, which affects the cost ofuser feedback, averaged to 6.02 (minimum is 1, maximum is 121,and variance is 294.7) in our experiments. This means that a user
2. The poor performance is caused by the lack of relevance information.
Query expansion works well if more relevant documents are given. For
example, OKAPI-Man ðq ¼ 20Þ shows P10 ¼ 0:604 and MAP ¼ 0:281, andSMART-Man ðq ¼ 20;  ¼ 3;  ¼ 2; and  ¼ 0Þ shows P10 ¼ 0:666 andMAP ¼ 0:320 when three relevant documents are given in the feedback.

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,

VOL. 19,

NO. 11,NOVEMBER 2007

1589

TABLE 1Results for All Feedback Methods

needs to judge 6.02 documents until he or she finds the first
relevant document when using our method. This seems high.
However, if we eliminate topics with maximum values as
exceptional cases, the average number drops to 3.6 (the maximum
was 27, and the variance was 24.9). LANG’s average is not very
different from the above values. In addition to reducing thenumber of documents that must be judged, we can also reduce thecognitive load required to judge the relevance of documents byrefining the user interface. For example, we can use clustering tocollect up similar documents and have users judge only the
representative documents in a clustered group. Using thisapproach, users can judge more documents with less effort.
Computational time of our query expansion method increases
linearly with the number of unlabeled documents to be used forlearning. For example, it took about 5.0 sec with 40 unlabeled
documents in our environment (Pentium4-M, 1.8-GHz processor, 1Gbye of memory). Since our experimental program is written as
perl script, we can reduce the time with more sophisticatedimplementation.

[2]

6

[13]

CONCLUSION

In this paper, we proposed a query expansion method using
practical manual feedback where relevance of documents is only
given until a user finds one relevant document. Since the amountof relevant information is so small, we use the SGT algorithm, atransductive learning technique, to obtain pseudorelevant documents. We then use the SGT results in the extended wpq method.
That was our original extension where the scores of query term
candidates were calculated by aggregating several SGT results byslightly changing the parameters. In our experiments, we first
showed how the number of documents used in SGT, and thenumber of expansion terms affects our method’s performance. The
results showed that the method performs well when the number isrelatively small. However, our method is not very sensitive to theparameters. We also compared our method with other threetechniques. As for the retrieval with manual feedback, we
confirmed that our method improved the Okapi model and iscomparable to a state-of-the-art method based on the language
model. As a reference, we also compared our method with theresults of a pseudofeedback. Although our method requires humaneffort, it enables users to work more efficiently.

[3]

[4]

[5]
[6]

[7]

[8]
[9]

[10]
[11]
[12]

[14]

[15]

[16]
[17]
[18]

[19]
[20]
[21]

[22]

[23]

ACKNOWLEDGMENTS[24]

The authors would like to sincerely thank the editor and theanonymous reviewers for their valuable comments and suggestions. This research was supported by MEXT, KAKENHI(18700084).

[25]

REFERENCES[1]

I. Ruthven, “Re-Examining the Potential Effectiveness of Interactive Query
Expansion,” Proc. 26th Ann. Int’l ACM SIGIR Conf. Research and Development
in Information Retrieval, pp. 213-220, 2003.

[26]

S. Dumais et al., “Sigir 2003 Workshop Report: Implicit Measures of User
Interests and Preferences,” Proc. SIGIR Forum, 2003.
S.E. Robertson and S. Walker, “On Relevance Weights with Little Relevance
Information,” Proc. 20th Ann. Int’l ACM SIGIR Conf. Research andDevelopment in Information Retrieval, pp. 16-24, 1997.
M. Iwayama, “Relevance Feedback with a Small Number of Relevance
Judgements: Incremental Relevance Feedback vs. Document Clustering,”
Proc. 23rd Ann. Int’l ACM SIGIR Conf. Research and Development inInformation Retrieval, pp. 10-16, 2000.
T. Onoda et al., “Non-Relevance Feedback Document Retrieval,” Proc. 2004
IEEE Conf. Cybernetics and Intelligent Systems, pp. 456-461, 2004.
F. Diaz and D. Metzler, “Improving the Estimation of Relevance Models
Using Large External Corpora,” Proc. 29th Ann. Int’l ACM SIGIR Conf.
Research and Development in Information Retrieval, pp. 154-161, 2006.
X. Shen, B. Tan, and C. Zhai, “Context-Sensitive Information Retrieval
Using Implicit Feedback,” Proc. 28th Ann. Int’l ACM SIGIR Conf. Researchand Development in Information Retrieval, pp. 43-50, 2005.
S.E. Robertson, “Overview of the Okapi Projects,” J. Documentation, vol. 53,
no. 1, pp. 3-7, 1997.
G. Salton, “The Smart Document Retrieval Project,” Proc. 14th Ann. Int’l
ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 356358, 1991.
V. Vapnik, Statistical Learning Theory. Wiley, 1998.
G.W. Flake et al., “Extracting Query Modification from Nonlinear SVMS,”
Proc. 11th Int’l World Wide Web Conf., 2002.
S. Oyama et al., “Keyword Spices: A New Method for Building DomainSpecific Web Search Engines,” Proc. 17th Int’l Joint Conf. Artificial
Intelligence, 2001.
R.W. White and D. Kelly, “A Study on the Effects of Personalization andTask Information on Implicit Feedback Performance,” Proc. 15th ACM Int’l
Conf. Information and Knowledge Management (CIKM ’06), pp. 297-306, 2006.
A. Amir, M. Berg, and H. Permuter, “Mutual Relevance Feedback forMultimodal Query Formulation in Video Retrieval,” Proc. Seventh ACM
SIGMM Int’l Workshop Multimedia Information Retrieval (MIR ’05), pp. 17-24,
2005.
S. Yu et al., “Improving Pseudo-Relevance Feedback in Web Information
Retrieval Using Web Page Segmentation,” Proc. 12th Int’l World Wide Web
Conf., 2003.
S.E. Robertson, “On Term Selection for Query Expansion,” J. Documentation,
vol. 46, no. 4, pp. 359-364, 1990.
T. Joachims, “Transductive Inference for Text Classification Using Support
Vector Machines,” Proc. Int’l Conf. Machine Learning (ICML ’99), 1999.
X. Zhu et al., “Semi-Supervised Learning Using Gaussian Fields andHarmonic Functions,” Proc. 20th Int’l Conf. Machine Learning, pp. 912-914,
2003.
A. Blum et al., “Semi-Supervised Learning Using Randomized Mincuts,”
Proc. Int’l Conf. Machine Learning (ICML ’04), 2004.
T. Joachims, “Transductive Learning via Spectral Graph Partitioning,” Proc.
Int’l Conf. Machine Learning (ICML ’03), pp. 143-151, 2003.
M. Okabe, K. Umemura, and S. Yamada, “Query Expansion with the
Minimum User Feedback by Transductive Learning,” Proc. Human
Language Technology Conf./Conf. Empirical Methods in Natural Language
Processing, pp. 963-970, 2005.
C. Buckley, G. Salton, and J. Allan, “The Effect of Adding Relevance
Information in a Relevance Feedback Environment,” Proc. 14th Ann. Int’l
ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 292300, 1994.
E. Voorhees and D. Harman, “Overview of the Eighth Text Retrieval
Conf.,” Proc. Eighth Text REtrieval Conf. (TREC ’98), 1999.
S.E. Robertson and S. Walker, “Okapi/Keenbow at TREC-8,” Proc. Eighth
Text REtrieval Conf. (TREC ’99), 1999.
A. Singhal, J. Choi, D. Hindle, D.D. Lewis, and F. Pereira, “AT&T at TREC7,” Proc. Seventh Text REtrieval Conf. (TREC ’98), 1998.
C. Zhai and J. Lafferty, “Model-Based Feedback in the Language ModelingApproach to Information Retrieval,” Proc. 10th Int’l Conf. Information andKnowledge Management (CIKM ’01), pp. 403-410, 2001.