Implicit Feedback for Inferring User Preference:
A Bibliography
Diane Kelly
SCILS, Rutgers University
New Brunswick, NJ 08901 USA
diane@scils.rutgers.edu

Jaime Teevan
CSAIL, Massachusetts Institute of Technology
Cambridge, MA 02138 USA
teevan@csail.mit.edu

1

Introduction

Relevance feedback has a history in information retrieval that dates back well over thirty
years (c.f. [SL96]). Relevance feedback is typically used for query expansion during
short-term modeling of a user’s immediate information need and for user profiling duringlong-term modeling of a user’s persistent interests and preferences. Traditional relevance
feedback methods require that users explicitly give feedback by, for example, specifying
keywords, selecting and marking documents, or answering questions about their interests.
Such relevance feedback methods force users to engage in additional activities beyond
their normal searching behavior. Since the cost to the user is high and the benefits are notalways apparent, it can be difficult to collect the necessary data and the effectiveness of
explicit techniques can be limited.
In this paper we consider the use of implicit feedback techniques for query expansion anduser profiling in information retrieval tasks. These techniques unobtrusively obtain
information about users by watching their natural interactions with the system. Some of
the user behaviors that have been most extensively investigated as sources of implicit
feedback include reading time, saving, printing and selecting. The primary advantage tousing implicit techniques is that such techniques remove the cost to the user of providing
feedback. Implicit measures are generally thought to be less accurate than explicit
measures [Nic97], but as large quantities of implicit data can be gathered at no extra costto the user, they are attractive alternatives. Moreover, implicit measures can be combinedwith explicit ratings to obtain a more accurate representation of user interests.
Implicit feedback techniques have been used to retrieve, filter and recommend a variety
of items: hyperlinks, Web documents, academic and professional journal articles, email
messages, Internet news articles, movies, books, television programs, jobs and stocks.
There is a growing body of literature on implicit feedback techniques for information
retrieval tasks, and the purpose of this article is to provide a brief overview of this work.
Our intention is not to be exhaustive, but rather to be selective, in that we present key
papers that cover a range of approaches. We begin by presenting and extending aclassification of behaviors for implicit feedback that was previously presented by Oard
and Kim [OK01], and classifying the selected papers accordingly. A preponderance ofthe existing work clusters into one area of this classification, and we further examine
those papers. We then provide a brief overview of several key papers, and conclude witha discussion of future research directions suggested by our analysis.

2Classification of Implicit Feedback Techniques

Implicit feedback techniques take advantage of user behavior to understand user interestsand preferences. Oard and Kim [OK01] classified observable feedback behaviors
according to two axes, Behavior Category and Minimum Scope. The Behavior Category(Examine, Retain, Reference and Annotate), refers to the underlying purpose of theobserved behavior. Minimum Scope (Segment, Object and Class), refers to the smallest
possible scope of the item being acted upon. This classification scheme is displayed,with example behaviors, in Table 1.

Minimum Scope

Behavior Category

ExamineRetainSegment
View
ListenScroll
Find
Query
Print

Reference

Copy-and-paste
Quote

Annotate

Mark upObjectSelectClass
BrowseBookmark
Save
DeletePurchase
Email
Forward
Reply
Link
Cite
Rate
PublishAuthorSubscribe

OrganizeType
Edit
Table 1: Classification of behaviors that can be used for implicit feedback
from Oard and Kim [OK01]. Our additions have been highlighted.
CreateBased on our examination of the literature, we added a fifth Behavior Category, “Create”,
to Oard and Kim’s [OK01] original four. The “Create” behavior category describes thosebehaviors the user engages in when creating original information. An example of a
“Create” behavior is the writing of a paper. We also added some additional commonly
investigated observable behaviors, and they have been highlighted. Like Oard and Kim[OK01], we make no claim that this table of behaviors is exhaustive. Rather, we suggestthat Table 1 be viewed as a sample of the possible behaviors that users might exhibit. It
should be noted that Table 1 includes both implicit and explicit observable behaviors. In
our discussion of implicit measures, we do not consider explicit observable behavior,such as “rate”.
Categorizing an observable behavior into the appropriate cell in Table 1 can be difficult,because both the intent of the behavior and its scope can be ambiguous. Thus, while the
Behavior Category for saving a newly created document could appear to be “Retain”, thebehavior is probably more appropriately considered “Create”. Similarly, while find andquery behaviors involve the creation of text, they are primarily used to locate informationfor examination, and thus are classified in the “Examine” category. For example, aperson might use find to locate a term or passage to examine in a document. Similarly,
they might perform a query to locate a document for examination. While querying
traditionally applies to documents, the behavior is classified with a Minimum Scope of“Segment” because some systems return best passages rather than documents.
It is also difficult to assign the Minimum Scope of a behavior, as the scope can be
ambiguous. For example, a behavior such as bookmark acts on a Web page, which istraditionally considered an “Object”. However, when a Web page is considered in thecontext of its containing Web site, it can be understood as a “Segment” instead. Note,too, that observable behaviors are classified according to the minimum scope for which
the behavior could be observed. For example, the minimum scope we might observe forthe behavior type is a “Segment” although it is also common for typing to occur during
the creation of an object. Similarly, view is identified in the “Examine Segment”
category. However, most research has investigated viewing as it relates to objects, andthus that research belongs in the “Examine Object” category.

Minimum Scope
Segment
Examine[WRJ02][BP99][BPC00]Behavior Category[MBC+00]
Retain[BLG00][CLP00]Object[CLW+01] [CC02][JFM97] [KB01] [KC02][KOR00] [KSK97][KMM+97] [Lie95][LYM02][MS94]Class[RS01][SZ00][Maes94][RP97]

Reference[Kle99] [PBM+98][THA+97]
Annotate

Create[RP97][GPS99][PGS98][MBC+00][FS91][BH99] [Rho00][HHW+92]
Table 2: Papers classified based on the observed implicit behaviors they discuss.
The papers discussed in greater depth in Section 3 have been highlighted.
We classified the thirty papers we selected to include in this article according to the
Behavior Category and Minimum Scope of the implicit measures addressed by the paper.

The classification is shown in Table 2. Some of the papers, such as [BLG00], [MS94]
and [RS01], overlap a number of categories and are shown in overlapping gray boxes.
Those papers discussed in greater depth in Section 3 are highlighted.
A preponderance of the literature falls into the “Examine Object” category. This is not
surprising, as document selection and viewing time, both measures included in “Examine
Object”, are relatively easy to obtain and are available for every object with which a userinteracts. Other areas of Table 2 contain little or no work, suggesting possible categories
of observable behavior to explore. One likely reason for the dearth of literature acrossthe Minimum Scope categories of “Segment” and “Class” is that for many systems, theunit with which the user interacts is the object. An exception to this is that many
annotation systems consider segments, and this could suggest why much of theannotation literature falls into this category.
We further examined the 18 papers that fell into the “Examine Object” category,classifying them into Table 3 along two additional axes. One axis represents the standard
software lifecycle based on the spiral model of software development (c.f. [Boe88]):
design, implementation, evaluation. Papers in the “Design” category address the issue ofwhat are good implicit measures to use. The “Implementation” category contains papers
about implementing systems that use implicit feedback, and those in the “Evaluation”
category focus on frameworks for evaluation. Of course, there is overlap among all threeof these categories, particularly because the work with implicit measures is still in its
infancy. For example, because there do not yet exist many test beds for systemevaluation, most system implementation research has necessitated the development of anevaluation scheme. We classify the papers according to the stage they primarily address,
but encourage the reader to explore papers from other categories as well.
DesignIndividual [KB01] [KC02][MBC+00]

Implementation[BP99] [BLG00][BPC00] [KSK97][Lie95] [LYM02]

Evaluation

Group[CC02] [CLW+01][JFM97][CLP00][KMM+97][KOR00] [MS94][RS01] [SZ00]
Table 3: Papers from the “Examine Object” cell in Table 2, classified by study type.
The papers discussed in greater depth in Section 3 have been highlighted.
The other axis in Table 3 focuses on whether the research deals with user preferences on
an individual or group level. For example, in the understanding of implicit measures, theamount of time an individual spends reading a document can be compared to that
individual’s explicit relevance judgment to understand if reading time is a good implicit
measure for relevance, or reading times can be averaged across many users, andcompared to a global relevance judgment for that document. Similarly, systems that use
implicit measures can use them to help retrieve, filter and recommend items forindividual users, or they can provide feedback on an aggregate level by, for example,
clustering the documents or highlighting popular articles. Note that many implicit

feedback systems built to support individuals do so based on analysis performed overgroups. For example, a system that infers an individual’s relevance judgments based on
his or her reading time may base the judgment on a threshold derived from averaging thereading time over a group of users. None the less, because such work focuses on
supporting the individual, we classify it as “Individual”.
While the papers from the “Examine Object” category of Table 2 spread evenly acrossseveral of the categories of Table 3, it is evident that little work has focused primarily on
the “Evaluation” category. This is probably because the field is still young, and until now
it has been difficult to determine what sort of evaluation test beds would be appropriate.
3

Examination of Key Papers

In this section, we provide a more in depth analysis of several papers that we believe aregood representatives of the various different areas of Table 2. Our purpose in examining
these papers in more detail is to present the reader with a better idea of how studies of
implicit feedback are conducted, how this feedback is typically used and what the key
issues and problems in this area are. These papers are necessarily biased toward the workthat has examined reading time, as a majority of research has focused on this behavior.
[CLW+01] Claypool, M., Le, P., Waseda, M., and Brown, D. (2001). Implicit interest
indicators. In Proceedings of the 6th International Conference on Intelligent User
Interfaces (IUI ’01), USA, 33-40.

Claypool, Le, Waseda, and Brown provide a categorization of different interest
indicator categories, both explicit and implicit, and address the fundamental questionof which observable behaviors can be used as implicit measures of interest. The
authors create a customized browser and record the online behavior of seventy-fivestudents, who were instructed to use the browser for 20 to 30 minutes of unstructured
browsing. Several behaviors were examined: mouse clicks, scrolling, and time on
page. Mouse clicks and scrolling were measured both as a frequency number (i.e.
number of mouse clicks) and as total time spent. Scrolling was further measured bothat the keyboard and with the mouse. Users were asked to explicitly rate each pagethat they viewed just before the page closed and these ratings were used to evaluate
the implicit measures. Users looked at a total of 2,267 Web pages and made ratings
on 1,823 (80%) of these. The authors found that time spent on a page, the amount ofscrolling on a page (all scrolling measures combined) and the combination of timeand scrolling had a strong positive correlation with the explicit ratings. However, thenumber of mouse clicks and the individual scrolling measures were found to beineffective in predicting the explicit ratings.
[MS94] Morita, M., and Shinoda, Y. (1994). Information filtering based on user behavior
analysis and best match text retrieval. In Proceedings of the 17th Annual International
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’94), Ireland, 272-281.

Morita and Shinoda explored how behaviors exhibited by users while reading articles
from newsgroups could be used as implicit feedback for profile acquisition andfiltering. For six weeks, eight users were required to read all articles that were posted
to the newsgroups of which they were members and to explicitly rate their interest inthe articles. The authors measured the reading time, the saving or following-up of astory and copy for each of the 8,000 articles read by their users. They further
examined the relationship of three variables on reading time: the length of the
document, the readability of the document and the number of news items waiting to
be read in the user’s news queue. Very low correlations (not significant) were foundbetween the length of the article and reading time, the readability of an article andreading time and the size of the user’s news queue and reading time. Although nostatistics are presented, the reading time for articles rated as interesting was longer
than for articles rated as uninteresting. Saving, following-up and copying of anarticle were not found to be related to interests. Based on these results, the authors
examined several reading time thresholds for identifying interesting documents.
When applied to their data set, they found that the most effective threshold was 20
seconds, resulting in 30% of interesting articles being identified at 70% precision.
[RS01] Rafter, R., and Smyth, B. (2001). Passive profiling from server logs in an online
recruitment environment. In Proceedings of the IJCAI Workshop on Intelligent
Techniques for Web Personalization (ITWP 2001), USA, 35-41.

While the two preceding works found reading time to be related to interests, it is often
difficult to effectively deal with reading time distributions because the curves are notnormal. Instead, the curves have long tails, with a majority of points at the low end(toward zero). When collected in natural settings, there are often numerous outliers.
These distributions often make statistical analysis challenging and may require sometransformations. Rafter and Smyth perform a two-step process to prevent spurious
reading times in data collected from the log records of users accessing job postings.
In the first step, the median of median reading time values per individual job accessfor both users and jobs were used to calculate a normal reading time for thecollection. Spurious reading times were then identified using this normal reading
time, and outliers were replaced by this value. Graded reading times per job werethen produced by calculating in each user’s profile the number of standard deviations
each job’s newly adjusted reading time was above or below the user’s mean reading
time. In addition to the reading time, the authors also used raw visits to a job,
incorporating a threshold on revisits, as implicit feedback and used the behaviors ofapplying for a job or emailing the job to oneself to evaluate the implicit measures.
Users who had a profile of at least fifteen jobs were included in the analysis (412 total
users). Using the adjusted revisit data and adjusted reading time data was found toresult in better prediction performance than using their unadjusted counterparts,
suggesting that collection, task and user specific transformations and normalizations
on the raw behavioral data can produce more effective predictions of usefulness.
[WRJ02] White, R. W., Ruthven, I., and Jose, J. M. (2002). Finding relevant documents
using top ranking sentences: An evaluation of two alternative schemes. In Proceedingsof the 25th Annual International ACM SIGIR Conference on Research and Development
in Information Retrieval (SIGIR ’02), Finland, 57-64.

In a controlled laboratory study, White, Ruthven and Jose, examined reading time asa technique for automatically re-ranking sentence-based summaries for retrieved
documents. Users completed simulated tasks using three types of systems, one ofwhich automatically re-ranked the top sentences in the summaries based on the user’sreading time of each summary. They normalized the reading times for individuals byrequiring users to perform a timing task before each search, where they were
presented with a search description and the text of thirty summaries, and asked toread all documents and mark the relevant ones. To derive baseline reading times foreach user, reading times for each summary were normalized by the length of thesummary and divided by the number of characters to arrive at a character based, userspecific reading time for both relevant and non-relevant summaries. Performance
results regarding the implicit system were inconclusive.
[GPS99] Golovchinsky, G., Price, M. N., and Schilit, B. N. (1999). From reading to retrieval:
Freeform ink annotations as queries. In Proceedings of the 22nd Annual International
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’99), USA, 19-25.

Research which uses the text that a user generates, be it an annotation or text from aword processing application, has shown promising results with regard to using thistext as implicit evidence of user interests. Golovchinsky, Price and Schilit
constructed full text queries based on users’ annotated passages of documents andcompared these to queries constructed using standard relevance feedback techniques.
The motivation for this work was that the words and passages that users mark canprovide the system with a more refined, user-specific unit with which to perform
relevance feedback and that these passages can help in establishing a context that isbetter than using just a list of terms. Results from an experiment with ten users
annotating and evaluating documents for six topics found that queries derived fromusers’ annotations produced retrieval performance that was better than standard
relevance feedback techniques.
[BH99] Budzik, J., and Hammond, K. (1999). Watson: Anticipating and contextualizing
information needs. In Proceedings of the 62nd Meeting of the American Society for
Information Science, USA, 727-740.

Budzik and Hammond proposed a system that automatically retrieved documents andrecommended URLs to the user based on what the user was typing. This work was
motivated in part by the observation that users typically pose short queries that arehighly ambiguous and often lack context. Like Golovchinsky, Price and Schilit[GPS99], Budzik and Hammond suggest that evidence of context can be found innumerous other applications with which the user interacts. To initially and informallyprovide some support for their hypothesis, the authors asked ten researchers to submit
an electronic version of a paper that they wrote and then asked these users to evaluate
the documents that their experimental system had retrieved based on these texts. The
results were encouraging, with at least eight of the ten users indicating that at least
one of the retrieved results would have been useful. While Budzik and Hammond
also provide results from several other informal evaluations, a full-scale, formal
evaluation has yet to be performed.
[Kle99] Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5),604-632.

Perhaps the most impressive large-scale use of implicit feedback comes in the form of
Web link analysis. An example of this is Kleinberg's work with hubs and authorities.

Authorities are authoritative information sources on a topic, and hubs are collectionsof authorities. Kleinberg suggested that good hubs could be recognized because they
point to many good authorities, and similarly, good authorities could be recognizedbecause they are pointed to by many hubs. Thus the links that people make in thecourse of Web page authoring are interpreted as endorsement. Link analysis, in theform of PageRank [PBM+98], is used to great success in practice by Google.
From the work reported in this section, it is clear that numerous problems arise whentrying to infer information from observable behaviors, because what can be observed
does not necessarily reflect the user’s underlying intention. For instance, the amount oftime that an object is displayed does not necessarily correspond to the amount of timethat the object is examined, yet display time is traditionally treated as an equivalent toreading time. Further, the amount of time an object is actively examined does notnecessarily correspond to the user’s interest in that object. As is evidenced by the work
described above, it appears that while implicit measures can be useful, they are notnecessarily inherently so. Implicit feedback is often difficult to measure and interpret,and should be understood within the larger context of the user’s goals and the system’sfunctionalities.
4

Future Directions

We have looked at some of the relevant work on implicit feedback, and classified andhighlighted a diverse set of papers that lay a foundation for the field. We believe thatusing implicit feedback is an exciting and promising approach to identifying user
preference, and in this paper we have called attention to the areas where research in thefield has focused, as well as illustrated several areas where there does not exist muchwork. We have had to be brief in our examination of key papers, and regret the exclusionof many interesting papers from this discussion. We did not consider some types of
behaviors that could also be useful, such as those not covered by Table 1 and feedback
from outside the digital domain (e.g., eye movements and gesture). For instance, Maglio,
et al., [MBC+00] suggested using eye movements to infer user interests and there is alarge body of research in the HCI community using eye movements to infer attention.
We encourage the interested reader to explore the references provided in this paper
further and assure that a longer review of implicit feedback is under construction.
To allow for the effective use of implicit feedback, more research needs to be conducted
on understanding what observable behaviors mean and how they change with respect tocontextual factors. Along with the papers discussed in Section 3, there is additional
evidence that individual, task, topic and collection differences have some effect on theuse of reading time as an effective measure of implicit feedback [KB01, KC02]. Whilesome work has limited the particular type of task under investigation, a more systematicinvestigation of the relationship between various contextual factors and potential
behavioral indicators of interests needs to be undertaken.
Not all implicit measures are equally useful and some may only be useful in combination
with others. For instance, the selection of an object is different, and perhaps weaker,
evidence of interests than the printing or saving of an object, and a document with a low
reading time might be printed or saved. It is likely, also, that how implicit measures arecollected influence their effectiveness. More tools that allow for the accurate and reliablecollection of data, such as the browser developed by Claypool, et al. [CLW+01], need to
be developed, tested and shared, and further research should be done into how thecollection process can encourage implicit feedback to closely match the user’s underlying
intent.
An in-depth investigation into the research that looked at object examination as a type of
implicit feedback (Table 3) revealed that implicit feedback is used to recommend,
retrieve and filter objects on both an individual and group level. Our examination further
highlighted the lack of literature on developing test beds and evaluation metrics for
implicit measures. We hypothesize that this is due to the novelty of the field; it isdifficult to develop a good testing framework while all of the assumptions underlying
implicit measures are still being explored. Perhaps now is a good time to look atdeveloping test beds to encourage the further development of implicit measures systems.
5

Acknowledgements

We would like to thank Nick Belkin, David Brown and Doug Oard for their feedback.
This research was supported by NTT, the Packard Foundation, the Oxygen Partnership,
HP and the National Science Foundation.
References[BP99] Billsus, D., and Pazzani, M. J. (1999). A personal news agent that talks, learns
and explains. In Proceedings of the 3rd International Conference on Autonomous
Agents (AGENTS 99), USA, 268-275.
[BPC00] Billsus, D., Pazzani, M. J., and Chen, J. (2000). A learning agent for wirelessnews access. In Proceedings of the 5th International Conference on Intelligent User
Interfaces, (IUI ’00), USA, 33-36.
[Boe88] Boehm, B. (1988). A spiral model of software development and enhancement.
IEEE Computer, 21(5), 61-72.
[BLG00] Bollacker, K. D., Lawrence, S., and Giles, C. L. (2000). Discovering relevantscientific literature on the Web. IEEE Intelligent Systems, 15(2), 42-47.
[BH99] Budzik, J., and Hammond, K. (1999). Watson: Anticipating and contextualizing
information needs. In Proceedings of the 62nd Meeting of the American Society for
Information Science, USA, 727-740.
[CLW+01] Claypool, M., Le, P., Waseda, and M., Brown, D. (2001). Implicit interest
indicators. In Proceedings of the 6th International Conference on Intelligent User
Interfaces (IUI ’01), USA, 33-40.
[CC02] Cooper, M. D., and Chen, H.M. (2001). Predicting the relevance of a librarycatalog search. Journal of the American Society for Information Science, 52(10),813-827.
[CLP00] Cosley, D., Lawrence, S., and Pennock, D. M. (2002). REFEREE: An open
framework for practical testing of recommender systems using ResearchIndex. In
Proceedings of the 28th Very Large Database (VLDB ‘02) Conference, China, 35-46.

[FS91] Fischer, G., and Stevens, C. (1991). Information access in complex, poorly
structured information spaces. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ‘91), USA, 63-70.
[GPS99] Golovchinsky, G., Price, M. N., and Schilit, B. N. (1999). From reading to
retrieval: Freeform ink annotations as queries. In Proceedings of the 24th Annual
ACM Conference on Research and Development in Information Retrieval (SIGIR’99), USA, 19-25.
[HHW+192] Hill, W. C., Hollan, J. D., Wroblewski, D., and McCandless, T. (1992). Edit
wear and read wear. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’92), USA, 3-9.
[JFM97] Joachims, T., Freitag, D., and Mitchell, T. (1997). WebWatcher: A tour guidefor the World Wide Web. In Proceedings of the International Joint Conference on
Artificial Intelligence (ICML ‘97), USA, 770-777.
[KSK97] Kamba, T., Sakagami, H., and Koseki, Y. (1997). ANATAGONOMY: A
personalized newspaper on the World Wide Web. International Journal of HumanComputer Studies, 46, 789-803.
[KB01] Kelly, D., and Belkin, N. J. (2001). Reading time, scrolling and interaction:
Exploring implicit sources of user preferences for relevance feedback during
interactive information retrieval. In Proceedings of the 24th Annual International
Conference on Research and Development in Information Retrieval (SIGIR ‘01),
USA, 408-409.
[KC02] Kelly, D., and Cool, C. (2002). The effects of topic familiarity on information
search behavior. In Proceedings of the Second ACM/IEEE Joint Conference on
Digital Libraries (JCDL ’02), USA, 74-75.
[KOR00] Kim. J., Oard, D.W., and Romanik, K. (2000). Using implicit feedback for usermodeling in internet and intranet searching. University of Maryland CLIS Technical
Report 00-01.
[Kle99] Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment.
Journal of the ACM, 46(5), 604-632.
[KMM+97] Konstan, J., Miller, B. N., Maltz, D., Herlocker, J.L., Gordon, L.R., andRiedl, J. (1997). GroupLens: Applying collaborative filtering to usenet news.
Communications of the ACM, 40(7), 77-87.
[Lie95] Lieberman, H. (1995). Letizia: An agent that assists Web browsing. In
Proceedings of the International Joint Conference on Artificial Intelligence. USA,924-929.
[LYM02] Liu, F., Yu, C., and Meng, W. (2002). Personalized Web search by mappinguser queries to categories. In Proceedings of the Eleventh International Conferenceon Information and Knowledge Management (CIKM ‘02).USA, 558-565.
[Maes94] Maes, P. (1994). Agents that reduce work and information overload.
Communications of the ACM, 37(7), 30-40.

[MBC+00] Maglio, P. P., Barrett, R., Campbell, C. S., and Selker, T. (2000). SUITOR:
An attentive information system. In Proceedings of the 5th International Conferenceon Intelligent User Interfaces (IUI ’00), USA, 169-176.
[MS94] Morita, M., and Shinoda, Y. (1994). Information filtering based on user behavior
analysis and best match text retrieval. In Proceedings of the 17th Annual International
ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’94), Ireland, 272-281.
[Nic97] Nichols, D.M. (1997). Implicit ratings and filtering. In Proceedings of the 5th
DELOS Workshop on Filtering and Collaborative Filtering, Hungary, 31-36.
[OK01] Oard, D. W., and Kim, J. (2001). Modeling information content using observablebehavior. In Proceedings of the 64th Annual Meeting of the American Society for
Information Science and Technology, USA, 38-45.
[PBM+98] Page, L., Brin, S., Motwani, R., and Winograd, T. (1998). The PageRankcitation ranking: Bringing order to the Web. Stanford University Technical Report.
[PGS98] Price, M. N., Golovchinsky, G., and Schilit, B. N. (1998). Linking by inking:Trailblazing in a paper-like hypertext. In Proceedings of the 9th ACM Conference onHypertext and Hypermedia’98 (Hypertext ‘98), USA, 30-39.
[RS01] Rafter, R., and Smyth, B. (2001). Passive Profiling from Server Logs in an Online
Recruitment Environment. In Proceedings of the IJCAI Workshop on Intelligent
Techniques for Web Personalization (ITWP 2001), USA, 35-41.
[Rho00] Rhodes, B. J. (2000). Margin Notes: Building a contextually aware associative
memory. In Proceedings of the 5th International Conference on Intelligent User
Interfaces (IUI ’00), USA, 219-224.
[RP97] Rucker, J., and Polanco, J. (1997). Siteseer: Personalized navigation for the Web.
Communications of the ACM, 40(3), 73-75.
[SZ00] Seo, Y. W., and Zhang, B. T. (2000). A reinforcement learning agent for
personalized information filtering. In Proceedings of the 5th International Conferenceon Intelligent User Interfaces (IUI ’00), USA, 248-251.
[SL96] Spink, A., and Losee, R. M. (1996). Feedback in information retrieval. Annual
Review of Information Science and Technology, 31, 33-78.
[THA+97] Terveen, L., Hill, W., Amento, B., McDonald, D., and Crete, J. (1997).
PHOAKS: A system for sharing recommendations. Communications of the ACM,
40(3), 59-62.
[WRJ02] White, R. W., Ruthven, I., and Jose, J. M. (2002). Finding relevant documents
using top ranking sentences: An evaluation of two alternative schemes. In
Proceedings of the 24th Annual International Conference on Research andDevelopment in Information Retrieval (SIGIR ’02), Finland, 57-64.