Recommender systems for literature selection: A competition of decision making and memory models

Recommender Systems for Literature Selection: A Competition between Decision Making and Memory Models Leendert van Maanen (leendert@ai.rug.nl) Department of Artificial Intelligence, University of Groningen P.O. Box 407 9700 AK Groningen, The Netherlands Julian N. Marewski (marewski@mpib-berlin.mpg.de) Max Planck Institute for Human Development, Center for Adaptive Behavior and Cognition Lentzeallee 94, 14195 Berlin, Germany Abstract We examine the ability of five cognitive models to predict what publications scientists decide to read. The cognitive models are (i) the Publication Assistant, a literature recommender system that is based on a rational analysis of memory and the ACT-R cognitive architecture; (ii-iv) three simple decision heuristics, including two lexicographic ones called take-the-best and naiveLex, as well as unit-weight linear model, and (v) a more complex weighted-additive decision strategy called Franklin’s rule. In an experiment with scientists as participants, we pit these models against (vi) multiple regression. Among the cognitive models, take-the- best best predicts most scientists’ literature preferences best. Altogether, the study shows that individual differences in scientific literature selection may be accounted for by different decision-making strategies. Keywords: Recommender system; ACT-R; rational analysis; simple heuristics; take-the-best; literature search Literature Selection In 2006, the number of scientific publications in the relatively small ISI subject category Information Science & Library Science was 2054. In other words, researchers working in this area had to scan through over 2,000 papers a year to keep up with the current developments. However, this number is, if anything, an underestimation of the total number of potentially relevant papers, as this number only holds if a researcher is interested in a single subject area. In practice, most researchers work on the intersection of multiple domains, increasing the number of potentially relevant papers enormously. Not only professionals in the scientific domain are confronted with masses of potentially relevant information. Also, government or business employees often need to decide which of numerous reports, leaflets, and bulletins to read, and which to ignore—a challenge that is aggravated by the continuously increasing amount of information that is available online. For instance, many press agencies produce over 12.500 bulletins a year. Reporters therefore have to make selections, and although the agencies often tag their bulletins, the sheer mass of information makes that it is easy to miss important ones. In this paper, we will focus on one solution to this problem: recommender systems. Typically, corresponding decision aids automatically come up with a pre-selection of information that is worth further consideration, saving institutions, firms, and people parts of the time and effort otherwise required to separate the relevant from the irrelevant. In particular, here we will evaluate six models that can solve the problem of information selection for the scientific domain. All models select relevant scientific papers from a large collection of scientific abstracts. They include the Publication Assistant (Van Maanen, Van Rijn, Van Grootel, Kemna, Klomp, & Scholtens, in press), a recommender system that was recently developed to assist scientists in identifying relevant articles. We will compare the performance of this system to that of three simple decision heuristics, including a unit-weight linear model (see Dawes, 1979; Dawes & Corrigan, 1974), and two lexicographic rules, called take-the-best (Gigerenzer & Goldstein, 1996), and naiveLex. We will also pit all models against two complex linear weighted-additive models, one being Franklin’s rule (Gigerenzer & Goldstein, 1999) and the other multiple regression. While we do not aim to model the cognitive processes that are actually going on when scientists make literature choices, except for multiple regression all models tested here are grounded in cognitive theories. The Publication Assistant is a memory model that is based on the rational analysis framework (Anderson, 1990; Oaksford & Chater, 1998), as incorporated in the ACT-R cognitive architecture (Anderson & Lebiere, 1998). The heuristics are models of decision making that are grounded in the fast and frugal heuristics framework (Gigerenzer, Todd, & the ABC Group, 1999). The linear weighted additive model, Franklin’s rule, is also a model of decision making (Gigerenzer & Goldstein, 1999). All models are common in the memory and judgment and/or decision making literature. In what follows, we will give an outline of the Publication Assistant. Next, we will introduce the five alternative models. In an experiment, we will then evaluate the models’ performance in predicting scientists’ literature preferences. The Publication Assistant: A Memory Model An example of the problem addressed in this paper is the selection of relevant talks when attending a large, multi- track scientific conference such as the Annual Cognitive Science Conference. The information selection process starts when a researcher registers and receives a copy of the conference program. For instance, a strategy often employed by many conference attendees is to scan talk titles, author