CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

Being Accurate is Not Enough:How Accuracy Metrics have hurt
Recommender Systems
Sean M. McNee

Abstract

GroupLens Research

Recommender systems have shown great potential tohelp users find interesting and relevant items fromwithin a large information space. Most research up to
this point has focused on improving the accuracy ofrecommender systems. We believe that not only has
this narrow focus been misguided, but has even been
detrimental to the field. The recommendations that are
most accurate according to the standard metrics are
sometimes not the recommendations that are mostuseful to users. In this paper, we propose informalarguments that the recommender community should
move beyond the conventional accuracy metrics andtheir associated experimental methodologies. We
propose new user-centric directions for evaluating
recommender systems.

Department of Computer Science and Engineering
University of Minnesota
Minneapolis, MN 55455 USA
mcnee@cs.umn.edu
John Riedl
GroupLens Research
Department of Computer Science and Engineering
University of Minnesota
Minneapolis, MN 55455 USA
riedl@cs.umn.edu
Joseph A. Konstan
GroupLens Research
Department of Computer Science and Engineering
University of Minnesota

Keywords

Minneapolis, MN 55455 USA
konstan@cs.umn.edu

Recommender systems, personalization, collaborative
filtering, metrics

Copyright is held by the author/owner(s).

ACM Classification Keywords

CHI 2006, April 22–27, 2006, Montréal, Québec, Canada.

H3.3. Information Search and Retrieval: Information
Filtering, Search Process, Retrieval Models

ACM 1-59593-298-4/06/0004.

1097

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

Introduction

Similarity

Imagine you are using a travel recommender system.
Suppose all of the recommendations it gives to you arefor places you have already traveled to. Even if thesystem were very good at ranking all of the places you
have visited in order of preference, this still would be apoor recommender system. Would you use such asystem?

More frequently than not, recommendation lists containsimilar items in them. Going to Amazon.com for a bookby Robert Heinlein, for example, will give you arecommendation list full of all of his other books. We
have seen this behavior in algorithms as well. TheItem-Item collaborative filtering algorithm can trap
users in a ‘similarity hole’, only giving exceptionallysimilar recommendations (e.g. once a user rated one
Star Trek movie she would only receive
recommendations for more Star Trek movies) [Rashid2001]. This problem is more noticeable when there isless data on which to base recommendations, such as
for new users to a system. It is these times when apoor recommendation could convince a user to leave
the recommender forever.

Unfortunately, this is exactly how we currently test ourrecommender systems. In the standard methodology,
the travel recommender would be penalized forrecommending new locations instead of places theusers have already visited! Current accuracy metrics,
such as MAE [Herlocker 1999], measure recommender
algorithm performance by comparing the algorithm’s
prediction against a user’s rating of an item. The mostcommonly used methodology with these metrics is theleave-n-out approach [Breese 1998] where apercentage of the dataset is withheld from therecommender and used as test data. In essence, wereward a travel recommender for recommending places
a user has already visited, instead of rewarding it forfinding new places for the user to visit.
By focusing on this way of testing recommenders, are
we really helping users find the items they areinterested in? We claim there are many different
aspects to the recommendation process which current
accuracy metrics do not measure. In this paper, wewill review three such aspects: the similarity ofrecommendation lists, recommendation serendipity,
and the importance of user needs and expectations in arecommender. We will review how current
methodologies fail for each aspect, and provide
suggestions for improvement.

Accuracy metrics cannot see this problem because they
are designed to judge the accuracy of individual item
predictions; they do not judge the contents of entire
recommendation lists. Unfortunately, it is these lists
that the users interact with. All recommendations are
made in the context of the current recommendation list
and the previous lists the user has already seen. The
recommendation list should be judged for its usefulness
as a complete entity, not just as a collection ofindividual items.
One approach to solving this problem was proposed in[Ziegler 2005], with the introduction of the Intra-List
Similarity Metric and the process of Topic Diversification
for recommendation lists. Returned lists can be altered
to either increase or decrease the diversity of items onthat list. Results showed that these altered lists
performed worse on accuracy measures thanunchanged lists, but users preferred the altered lists.

1098

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

Depending on the user’s intentions, the makeup ofitems appearing on the list affected the user’s
satisfaction with the recommender more than thechanges in the accuracy of the item on the list.

Serendipity
Serendipity in a recommender is the experience ofreceiving an unexpected and fortuitous item
recommendation. There is a level of emotional
response associated with serendipity that is difficult tocapture in any metric. But even if we remove thatcomponent, the unexpectedness part of this concept—
the novelty of the received recommendations—is still
difficult to measure. The converse of this concept, theratability of received recommendations, is quite easy tomeasure, however. And it is ratability that a leave-nout approach measures.

recommendation for items with high ratability. Thesealgorithms are good at predicting what else will appear
in that user’s profile. Indeed, many machine learningalgorithms score very well as recommenders based ona leave-n-out methodology [Billsus 1998, Breese1998].

When applied to a classification problem, a leave-n-out
methodology works great: items belonging to aparticular segment are withheld and the classifier isjudged on how it classifies these items given what it
knows of the dataset. Similarly, a leave-n-out
methodology judges a recommender algorithm by how
well it classifies, or recommends, withheld items to theusers they were withheld from.

The implicit assumption is that a user is always
interested in the items with the highest ratability.
While this assumption is true in classification problems,
we claim it isn’t always true in recommenders. Users
often judge recommendations that are for items theywould not have thought of themselves. For instance,
we once were helping an online music store withrecommendations using a User-User Collaborative
Filtering algorithm. The most common
recommendation was for the Beatle’s “White Album”.
From an accuracy perspective these recommendations
were dead-on: most users like that album very much.
From a usefulness perspective, though, therecommendations were a complete failure: every user
either already owned the “White Album”, or had
specifically chosen not to own it. Even though it was
highly ratable, “White Album” recommendations werealmost never acted on by users, because they added
almost no value.

We define the ratability of an item in a recommender tobe the probability that this item will be the next itemthat the user will consume (and then rate) given what
the system knows of the user’s profile. From amachine learning point-of-view, the item with thehighest ratability would be the next item a classifier
would place into the user’s profile. Thus, recommender
algorithms which score well on accuracy metrics using aleave-n-out methodology can generate

Our previous research [McNee 2002, Torres 2004] hasgiven us one more piece to this puzzle: recommender
algorithms generate qualitatively different
recommendations lists from each other. When asked,users preferred lists from different recommender
algorithms based on their current task. Users chosedifferent words to describe the recommendations theyreceived (e.g. User-based collaborative filtering wasconsidered to generate “novel” recommendations).

1099

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, CanadaThis suggests that we need other ways to classify
recommender algorithms. While a ‘serendipity metric’
may be difficult to create without feedback from users,
other metrics to judge a variety of algorithm aspects
would provide a more detailed pictures of thedifferences between recommender algorithms.

User Experiences and Expectations
As we have shown in previous work, user satisfaction
does not always correlate with high recommenderaccuracy [McNee 2002, Ziegler 2005]. There are manyother factors important to users that need to beconsidered.
New users have different needs from experienced users
in a recommender. New users may benefit from analgorithm which generates highly ratable items, as theyneed to establish trust and rapport with a recommender
before taking advantage of the recommendations itoffers. Previous work shows that the choice ofalgorithm used for new users greatly affects the user’s
experience and the accuracy of the recommendations
the system could generate for them [Rashid 2001].
Our previous work also suggested that differences inlanguage and cultural background influenced user
satisfaction [Torres 2004]. A recommender in a user’s
native language was greatly preferred to one in analternate language, even if the items themselvesrecommended were in the alternate language (e.g. a
Portuguese-based research paper recommenderrecommending papers written in English).

Moving Forward
Accuracy metrics have greatly helped the field ofrecommender systems; they have given us a way tocompare algorithms and create robust experimental
designs. We do not claim that we should stop using
them. We just cannot use them alone to judge
recommenders. Now, we need to think closely about
the users of recommender systems. They don’t careabout using an algorithm that scored better on ametric, they want a meaningful recommendation.
There a few ways we can do this.
First, we need to judge the quality of recommendations
as users see them: as recommendation lists. To dothis, we need to create a variety of metrics which act
on recommendation lists, not on items appearing in alist. There are already a few, such as the Intra-List
Similarity metric, but we need more in order tounderstand other aspects of these lists.
Second, we need to understand the differences
between recommender algorithms and measure themin ways beyond their ratability. Users can tell thedifference between recommender algorithms. For
example, when we changed the algorithm running theMovieLens movie recommender, we received manyemails from users wondering why MovieLens had
become so “conservative” with its recommendations.
Both algorithms scored well on MAE measures, butwere clearly different from each other.
Finally, users return to recommenders over a period oftime, growing from new users to experienced users.
Each time they come to the system, they have somereason for coming: they have a purpose. We need tojudge the recommendations we generate for each user
based on whether or not we were able to meet theirneed. Until we acknowledge this relationship with1100

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canadausers, recommenders will continue to generatemismatched recommendations [Zaslow 2002].
In the end, recommenders exist to help users. We, asa community, have created many algorithms andapproaches to studying recommender algorithms. It is
now time to also study recommenders from a usercentric perspective to make them not only accurate andhelpful, but also a pleasure to use.

Acknowledgements
We would like to thank our collaborators, Cai-Nicolas
Ziegler and Roberto Torres, and all of GroupLens
Research. This work was funded by the National
Science Foundation, grants DGE 95-54517, IIS 9613960, IIS 97-34442 IIS 99-78717, and IIS 01-02229.

[4] Herlocker, J.L., Konstan, J.A., Borchers, A., andRiedl, J. An algorithmic framework for performing
collaborative filtering. In Proc. of ACM SIGIR 1999,
ACM Press (1999), 230-237.
[5] McNee, S.M., Albert, I., Cosley, D., Gopalkrishnan,P., Lam, S.K., Rashid, A.M., Konstan, J.A. and Riedl, J.
On the Recommending of Citations for Research Papers.
In Proc. of ACM CSCW 2002, ACM Press (2002), 116125.
[6] Rashid, A.M., Albert I., Cosley D., Lam, S.K.,
McNee, S.M., Konstan, J.A. and Riedl, J. Getting toknow you: learning new user preferences inrecommender systems. In Proc. of ACM IUI 2002, ACMPress (2002), 127-134.
[7] Torres, R., McNee, S.M., Abel, M., Konstan, J.A.,
and Riedl, J. Enhancing digital libraries with
TechLens+. In Proc. of ACM/IEEE JCDL 2004, ACM
Press (2004) 228-236.

References[1] Billsus, D., and Pazzani, M.J. Learning
Collaborative Information Filters. In Proc. of ICML
1998, Morgan Kaufmann (1998), 46-54.

[8] Zaslow, J. “If TiVo Thinks You Are Gay, Here's How
To Set It Straight --- Amazon.com Knows You, Too,
Based on What You Buy; Why All the Cartoons?” The
Wall Street Journal, sect. A, p. 1, November 26, 2002.

[2] Breese, J.S., Heckerman, D., and Kadie, C.
Empirical Analysis of Predictive Algorithms for
Collaborative Filtering. In Proc. of UAI 1998, MorganKaufmann (1998), 43-52.

[9] Ziegler, C.N., McNee, S.M., Konstan, J.A., andLausen, G., Improving Recommendation Lists throughTopic Diversification. In Proc. of WWW 2005, ACM
Press (2005), 22-32.

[3] Herlocker, J.L., Konstan, J.A., Terveen, L.G., andRiedl, J.T. Evaluating Collaborative Filtering
Recommender Systems. ACM TOIS 22, 1 (2004), 5-53.

1101