XXXXThe MovieLens Datasets: History and Context
F. MAXWELL HARPER and JOSEPH A. KONSTAN, University of Minnesota

The MovieLens datasets are widely used in education, research, and industry. They are downloaded hundreds of thousands of times each year, reflecting their use in popular press programming books, traditionaland online courses, and software. These datasets are a product of member activity in the MovieLens movie
recommendation system, an active research platform that has hosted many experiments since its launch in1997. This paper documents the history of MovieLens and the MovieLens datasets. We include a discussionof lessons learned from running a long-standing, live research platform from the perspective of a researchorganization. We document best practices and limitations of using the MovieLens datasets in new research.
CCS Concepts: rInformation systems → Recommender systems; rHuman-centered computing →Collaborative filtering; Collaborative and social computing systems and tools; rGeneral and reference → Surveys and overviews;
General Terms: Human factors, Experimentation, Design
Additional Key Words and Phrases: Datasets, MovieLens, Ratings, Recommendations
ACM Reference Format:
F. Maxwell Harper and Joseph A. Konstan, 2015. The MovieLens Datasets: History and Context. ACM
Trans. Interact. Intell. Syst. V, N, Article XXXX ( 2015), 20 pages.
DOI: 0000001.0000001

1. INTRODUCTION

The MovieLens datasets, first released in 1998, describe people’s expressed preferences
for movies. These preferences take the form of <user, item, rating, timestamp> tuples, each the result of a person expressing a preference (a 0-5 star rating) for a movie
at a particular time. These preferences were entered by way of the MovieLens web
site1 — a recommender system that asks its users to give movie ratings in order toreceive personalized movie recommendations.
The MovieLens datasets are heavily downloaded (140,000+ downloads2 in 2014)
and referenced in the research literature (7,500+ references to “movielens” in Google
Scholar). This popularity is, to a certain degree, a reflection of the incredible rate of
growth of personalization and recommendation research, where datasets such as thesehave substantial value in exploring and validating ideas. The popularity might also
be attributed to the flexibility of ratings data, which is naturally suited not just to
1 http://movielens.org
2 http://grouplens.org/datasets/movielens

This material is based on work supported by the National Science Foundation under grants DGE-9554517,
IIS-9613960, IIS-9734442, IIS-9978717, EIA-9986042, IIS-0102229, IIS-0324851, IIS-0534420, IIS-0808692,
IIS-0964695, IIS-0968483, IIS-1017697, IIS-1210863. This project was also supported by the University of
Minnesotas Undergraduate Research Opportunities Program and by grants and/or gifts from Net Perceptions, Inc., CFK Productions, and Google.
Author’s addresses: F. Maxwell Harper and Joseph A. Konstan, Department of Computer Science and Engineering, University of Minnesota; emails: {harper, konstan}@cs.umn.edu.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights for components of this work owned
by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
c 2015 ACM. 2160-6455/2015/-ARTXXXX $15.00⃝
DOI: 0000001.0000001
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:2

Harper and Konstan

recommendation technology, but also to a more general family of data science techniques concerned with summarization, pattern identification, and visualization. Also,
because movie preferences are highly subject to personal tastes, the movie domain iswell-suited to testing personalization technology. Finally, the popularity may reflect
the perceived accessibility of movies as a content domain: movies are a common interest, making algorithmic output easy to discuss.
This paper explores the history of the MovieLens system in order to document thefactors that have shaped the resulting datasets. Along the way, we also share lessons
learned from operating a long-running research platform, and we document best practices for conducting research using the datasets. We include two main sections. In
section 2, we share history and lessons from the MovieLens system; in section 3, weshare descriptions and guidelines concerning the datasets.
2. THE MOVIELENS SYSTEM

The MovieLens datasets are the result of users interacting with the MovieLens online
recommender system over the course of years. As with most long-lived and dynamic
online systems, MovieLens has undergone many changes — both in design and infunctionality. These changes necessarily impact the generation of ratings: users only
rate movies that appear on their screens; users’ ratings are also influenced by thepredicted ratings themselves [Cosley et al. 2003].
In this section, we present a history of MovieLens with an eye towards illustrating
those changes most likely to have had an impact on the datasets’ content.
2.1. Origins and Landmarks

University of Minnesota researchers began developing MovieLens in the summer of
1997, the result of Digital Equipment Corporation’s (DEC) decision to close their own
movie recommender, EachMovie. At this time, DEC contacted the recommender systems community looking for an organization to develop a replacement site to carry on
the same mission, and perhaps serve the same users (but without using DEC’s proprietary code); GroupLens volunteered. Legal issues blocked directly transferring user
accounts, but DEC did transfer an anonymized dataset to GroupLens, who used thisdataset to train the first version of the MovieLens recommender.
MovieLens launched in the fall of 1997. As much as possible, this first version wasdeveloped to look like the EachMovie interface it was replacing (see Figure 5 for ascreenshot). EachMovie used a proprietary recommendation algorithm, so MovieLens
instead incorporated user-user collaborative filtering (CF) as implemented in the GroupLens usenet news recommender system [Konstan et al. 1997].
Use of MovieLens increased significantly during late 1999 when it received attention from the mass media. These events — an article in the New Yorker by Malcom
Gladwell [Gladwell 1999], an episode of ABC’s Nightline, and a favorable mention by
noted film critic Roger Ebert — each prominently featured MovieLens as an exampleof the future of online computing and personalization.
Since that time, the growth of the MovieLens system has been remarkably stable,
especially considering the nearly total absence of marketing efforts. MovieLens has
for a long time averaged 20-30 new user sign ups every day, and these are largely
the result of word-of-mouth or unsolicited press. See Figure 1 for a visualization of
the growth of MovieLens since its launch, Figure 2 for a visualization of the numberof monthly active users, Figure 3 for a visualization of the number of users in eachcalendar year logging in more than n times, and Figure 4 for a visualization of the
distribution of ratings data over time.
Steady growth does not mean that MovieLens has not changed since the late 90s.
In fact, since its initial launch, MovieLens has seen five major releases (v0-v4), each
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context
250,000
200,000
150,000
100,000
50,000035,00030,00025,00020,00015,00010,0005,000
025,000,000
20,000,000
15,000,000
10,000,000
5,000,0000
600,000
500,000
400,000
300,000
200,000
100,000
0

XXXX:3users
A

moviesB

ratings

A

tags
C
1999

2001

2003

2005

2007

2009

2011

2013

2015

Fig. 1. A 17 year view of growth in movielens.org, annotated with events A, B, C. User registration andrating activity show stable growth over this period, with an acceleration due to media coverage (A). The
rate of movies added to MovieLens grew (B) when the process was opened to the community. There is anacceleration in tag applications with the release of the “tag expression” feature (C).

monthly active users

6,000
5,000
4,000
3,000
2,000
1,000
0

1999

2001

2003

2005

2007

2009

2011

2013

2015

Fig. 2. Number of users that logged into MovieLens each month. Data before the dashed line (July, 1998)
is missing.

release representing a full reimplementation of the server- and client-side code. Tosummarize the most significant changes, we include Table I, which includes dates andshort summaries of some of the most significant changes to MovieLens. Also, to provide
a visual frame of reference, we provide historical screenshots in Figures 5-8 and ascreenshot of the current interface in Figure 9.
In the remainder of this section, we discuss key changes to MovieLens features thatcan most inform the shape and research use of the MovieLens datasets.
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:4

Harper and Konstan

users with at least n logins

0.7

n=2
n=5
n=10

0.6

n=25
n=100

0.5
0.4
0.3
0.2
0.1
0.0

2000

2002

2004

2006

2008

2010

2012

2014

Fig. 3. Fraction of users that logged in to MovieLens at least n times during each full calendar year. E.g.,
in 2014, 10% of the users who visited MovieLens logged in 25 or more times.

5.0

monthly median rating (dotted) and average rating (solid) with standard deviation

4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5

1999

2001

2003

2005

2007

2009

2011

2013

2015

Fig. 4. Rating distribution, aggregated by month. The solid line represents each month’s average rating.
The gray area covers the average plus or minus the standard deviation of that month’s ratings. The dottedline shows each month’s median rating. The two vertical dashed lines represent the launch of v3 (withhalf-star ratings) and v4 (with a simpler star ratings widget).

2.2. Changes to Key Features
2.2.1. Exploring. The mechanisms for searching, filtering, and ordering movies affectwhich movies a user can rate, and therefore have a fundamental influence on the shapeof the MovieLens datasets. The core user experience in MovieLens revolves around a
feedback loop between rating movies and viewing movie recommendations: the user
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:5

Table I. MovieLens landmarks. MovieLens has changed constantly since its release in
1997; these are some of the most meaningful events in its history.
Date
8/1997
9/199911/1999
2/20002/20036/2005
12/2005
9/20081/2012
11/2014

Landmark
v0 interface; EachMovie seed; GroupLens user-user recommenderv1 interface
media exposure; Net Perceptions user-user recommenderv2 interface; movie groups; additional external movie metadata (e.g.,box office, DVD releases); reviews
v3 interface; multilens item-item recommender; ajax ratings widget
discussion forums
tagging
member-based movie adding
lenskit item-item recommenderv4 interface; user-selectable recommender; external movie data

Fig. 5. Screenshot of MovieLens v0, circa 1998.

views items in a list of recommendations, rates those items, which in turn alters theitems shown on subsequent page views.
This rating/recommendation cycle is enabled by collaborative filtering (CF) algorithms [Resnick et al. 1994]. MovieLens, until the release of v4, ordered lists of movies
strictly based on user-personalized predicted rating values — the output of its underlying collaborative filtering algorithms. In other words, the movies that appear first in
search results for a user are those movies that the algorithm predicts that user will
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:6

Harper and Konstan

Fig. 6. Screenshot of MovieLens v1, circa 1999.

rate the highest. v4 blends a popularity factor in with predicted rating to order therecommendation lists [Ekstrand et al. 2015; Harper et al. 2015].
The algorithm itself has changed over time, which has had mostly unstudied effects
on user satisfaction and rates of contribution:— 1997: user-user CF via the GroupLens Usenet recommender [Konstan et al. 1997]— 1999: user-user CF via Net Perceptions3— 2003: item-item CF via MultiLens [Miller 2003]— 2012: item-item CF via LensKit [Ekstrand et al. 2011]— 2014: users able to select [Ekstrand et al. 2015] from: a non-personalized algorithm,
a recommender that supports new users [Chang et al. 2015], item-item CF and
FunkSVD via LensKit
The main page of MovieLens is a focus of user attention, and a prime spot to show
top-recommended movies to users. Early versions (v0, v1) simply displayed a list of
top recommendations on this page. v2 and v3 were more complex, showing several
pre-filtered lists of top picks (recent movies in theaters, recent movies on DVD/VHS),
along with a list of links out to different site features. v4 simply shows a list of lists,headed by overall “top picks” at the top of the page. Versions v2–v4 all encourage the
3 Net Perceptions was a recommender systems company co-founded in 1996 by GroupLens faculty and stu-

dents.

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:7

Fig. 7. Screenshot of MovieLens v2, circa 2000.

user to begin exploring a particular slice of the movie database, though the available
search customizations grew over time.
The MovieLens interface has always supported searching and filtering operations
that allow users to restrict their view to particular groups of movies. All versions havesupported title search; v4 includes an autocomplete function. Starting with v0, the interface supported filtering by movie genre and release date. v3 expanded the filtering
options by allowing users to also filter movies by actor, director, and other attributes. In
mid-2004, MovieLens added an “advanced search” page that brought an extremely expressive (and complicated) control panel for finding movies. v4 launched with a smaller
set of search filters, but an expanded array of pre-defined searches with previews on
the front page.
Many recommendation interfaces offer a “more like this” function; MovieLens beganoffering a version of this in late 2009 with the introduction of the “movie tuner” [Viget al. 2012]. This feature allowed users to navigate the information space differently, by
jumping from movie to movie directly, outside the influence of the primary explorationinterface. This feature, which persists in v4 with a different interface design, is unique
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:8

Harper and Konstan

Fig. 8. Screenshot of MovieLens v3, circa 2003.

in MovieLens for scoring similarities using a content-driven algorithm (using tags)
rather than ratings-based techniques.
MovieLens users have typically had access to only a limited set of movie metadata
in the site. Until mid-2005, movies only appeared in lists with very basic informationsuch as release dates and actors. In mid-2005, MovieLens added “movie detail” pages
into the site design to support linking with the discussion forum (see Section 2.2.6).
The addition of movie tagging in late 2005 (see Section 2.2.4) provided both objective
and subjective data for describing movies. In Spring 2009, MovieLens integrated with
Netflix (Netflix stopped supporting this API in late 2014), incorporating poster art anda plot synopsis into the movie details pages. v4 integrated MovieLens with The Movie
Database (TMDb), bringing plot summaries, movie artwork, and trailers directly into
the site.
2.2.2. Rating. Since MovieLens launched, ratings have been expressed as a “star”
value, which is a standard user interface pattern for users to input preferences. The
biggest change to the ratings interface came with the launch of v3 (February, 2003)
when the interface shifted from “whole star” to “half star” ratings, the most-requested
feature in a user survey, doubling the range of preference values from five (1–5) to ten(0.5–5.0). The release of v3 also upgraded the ratings widget to submit values asynchronously, without requiring the user to click a submit button.
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:9Fig. 9. Screenshot of MovieLens v4, 2015.

Versions v0–v3 employed two separate visual elements for the ratings widget. Theview element was an image showing a certain number of stars, where the color represented prediction or actual rating. The input element was an html <select> element
allowing the user to choose the star value. With the release of v4, the user interface combined these two elements into a single five star representation that accepts
touch/click events.
Throughout the life of v1–v3, a small legend describing different star values (seeFigures 6-8) sat in the top of each screen. The labels describing the star values changedin v3. It is possible that these labels influenced rating behavior substantially, given
the body of literature pointing to anchoring effects (e.g., [Lynch et al. 1991]), but there
has been no empirical study of this effect in MovieLens. At a coarse level of analysis,
Figure 4 shows no visible global changes in ratings distributions resulting from theseinterface changes.
2.2.3. Bootstrapping. The ratings-based CF algorithms that MovieLens is built on do
not work well for users until they have provided some initial ratings to the system
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:10

Harper and Konstan[Rashid et al. 2002]. MovieLens, until the release of v4, required users to cross someratings barrier [Drenner et al. 2006] before entering the main system.
Early versions, starting in v0, required users to rate 5 movies before entering thesystem. MovieLens showed users movies 10 at a time, where nine were randomly
chosen from the database, and the tenth chosen from a hand-chosen list of highlyrecognizable titles [Rashid et al. 2002].
In v3, the process was changed to require 15 ratings from users. Users were again
shown movies 10 at time, but now movies were selected for their popularity, excluding
the most popular 50-150 movies. Other methods were briefly evaluated for experimental purposes [Rashid et al. 2008] but did not have a lasting effect.
Version v4 dropped the 15 rating requirement, instead developing a special newuser recommender, based on a faster group-based personalization process [Chang et al.
2015]. We do not yet know the long-term benefits and costs of dropping this barrier.
2.2.4. Tagging. MovieLens introduced tagging in December, 2005; tags appear in thelater MovieLens datasets (10M and 20M). This feature allows users to apply tags —words or short phrases — to movies. MovieLens displays tags next to movies. Tags areclickable to show a list of movies where that tag has been applied.
The visibility and ordering of tags differed by user in early versions to provide A/B
testing data [Sen et al. 2006], but by 2006 the interface was consolidated, sorting tagsby the likelihood that the tag is “high value” according to a published metric [Sen et al.
2009]. In January, 2007, MovieLens launched a tag rating feature that put clickable
thumbs up and down icons next to tags [Sen et al. 2007].
In Spring, 2009, the tagging interface was given a new feature called “tag expressions” [Vig et al. 2010] that dramatically influenced tagging behavior. This interface
allowed users to re-tag movies much more easily, and caused an increase in the rate oftagging activity, and an increase in tag diversity.
2.2.5. Movie Adding and Editing. The presence or absence of movies shapes rating andtagging behavior, as does the accuracy and completeness of the associated metadata.
Early versions of MovieLens relied on a small group of administrators and a content
expert to curate the movie database. During this phase, MovieLens represented a relatively narrow universe of movies: those with a widespread United States theatricalrelease.
Starting in mid-2004, member control of the database increased. Soon after the release of v3, MovieLens added a link that allowed members to suggest titles, and in
mid-2004, MovieLens began to allow members to edit movie details directly [Cosley
et al. 2005]. After early experiments (circa 2004) allowing members to add movies, aninterface for movie adding became a permanent feature of the system in late 2008.
In v4, control of the metadata shifted from member input to an external source —http://themoviedb.org.
2.2.6. Interacting with Other Users. The MovieLens interface has always emphasized interacting with movies over interacting with other users. However, through the historyof MovieLens, features have come and gone that draw users’ attention towards moresocial uses of the system.
MovieLens offered two iterations of a feature designed for groups of people to receive
joint recommendations. v2 included a feature called “groups” [O’Connor et al. 2001]
which were persistent named collections of users that could receive jointly-optimized
recommendations. This structure was redesigned in v3, when it (now called “buddies”)
was reoriented to behave more like a social network. This redesign recognized thatgroup creation was too hard and groups were too inflexible — users could have pairwise
relationships and decide which buddies to include in a request for recommendations.
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:11The buddies feature was used by approximately 9,000 users to connect with a medianof one other MovieLens user (avg=1.8).
MovieLens offered three iterations of a feature designed to support conversations
around movies. The first of these, launched with v2, was a simple threaded discussion
forum. This version was not deeply connected with the main site, and was barely used.
It was discontinued when v3 was released.
The second iteration, released in mid-2005, was a threaded discussion forum designed to integrate more deeply with the movie database. The custom forum software
automatically detected movie titles in the text, and placed prediction/rating widgets
for each referenced movie next to it in a sidebar. These widgets linked to the movie details pages about each movie, and the movie detail pages linked to the five most recent
posts referencing the movie (or similar movies) [Drenner et al. 2006].
The third iteration, released in 2009, replaced the forum interface with one that supported question-asking. This redesign diminished user interest in the feature, because
the question asking and answering format did not fit as naturally with the community’s desired uses.
The v3 discussion forums hosted a moderate amount of activity, most of the content
from a small group of users. Over the span of 4.5 years, about 900 users posted about
5.5 times each day, creating about 10,000 posts and 11,000 links between the movies
interface and the discussion forum. The question-oriented design had less use; over the
span of 5 years, about 900 users posted about 1.5 times each day, creating about 2,000
posts and 1,500 links between the movies interface and the question-asking area.
Several experiments — affecting thousands of users over the life of the discussion forums — created short-lived features that engaged users socially. The first ofthese, which ran from early 2005 until early 2006, added personalized messages to thehomepage inviting users to participate in the discussion forums [Harper et al. 2007a].
Following this work, from late 2006 until late 2007, was a large-scale intervention
that added additional social features, including individual profile pages, group profile
pages, and home-page recommendations. This design was the result of a large-scale
field study of the differential influences in an online community of building interpersonal friendships or group identities [Ren et al. 2012; Harper et al. 2007b].
2.3. Lessons Learned

During the more than 17 years our research lab has operated MovieLens, we havelearned several lessons about the costs and benefits of operating a “real” system insupport of academic research.
The primary benefit of running a live research system is clear: we are able to run
online field experiments (e.g., [Ekstrand et al. 2015; Ren et al. 2012; Harper et al.
2007a; Sen et al. 2006; Drenner et al. 2006; Cosley et al. 2005; Rashid et al. 2002]),
gaining an ecological validity and a larger number of subjects that can be very difficult
to achieve in a lab setting. Because we control the operation of the system, we havecomplete flexibility in designing experiments, from modifications to the user interface
to the backing algorithms. This flexibility makes it easy to ask for user judgments andopinions, often a key element in determining if an experimental change is successful.
Even better, we have discovered that MovieLens users are open to experiments on
a broad range of features outside of the core feature set of the system, enabling ourgroup’s work on tagging, online communities, and user motivation. Because of this
openness, we are also able to email MovieLens users to participate in one-off user
studies [Shani and Gunawardana 2011] to test features that are not ready to scale or
integrate into the main user interface.
The primary cost in running a live system is the time required by researchers on
non-research tasks such as software development, hardware maintenance, user comACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:12

Harper and Konstan

munication, database maintenance, and marketing. There are many hidden tasks that
consume the time of researchers, such as restarting a server after a power failure, responding to an angry user’s email when their favorite feature is changed, or maintaining secure backups of user data. However, some of these tasks carry indirect benefits
for researchers: MovieLens developers gain large-scale, team-based software development experience. Moreover, interacting with and managing a community of users often
reveals new research ideas.
There are several lessons that we have learned over the course of running MovieLensthat may generalize to other research organizations hoping to build active researchplatforms.
— Lessons from the technology startup community apply. It is difficult to build a systemthat is popular with users because the world is full of high-quality, free services thatcompete for their time. Our research lab has launched several other systems, ranging
from Cyclopath, which was successful enough to run several small experiments (e.g.
[Priedhorsky et al. 2010]) to GopherAnswers, which failed to attract enough users
to advance our research goals. Thus, we have learned that many of the same ideas
from the world of startup technology companies (e.g., [Ries 2011]) also apply in thiscontext: launch quickly, and fail fast — most of the time spent getting a system up
and running is not productive from a research perspective.
— Running a live site implies continual work, not one-time effort. User expectations advance quickly in response to technological norms, and this creates pressure for the
research site to keep up or lose users. For example, innovations by search engine companies have shifted the norms of the behavior of search boxes; in response MovieLens
has seen significant changes to its implementation of search since its launch. While
keeping up with these technological changes may be important to retain a healthyuser base, it is also important in attracting graduate students or other researchers,
who may be uninterested in working with code that is several years out-of-date.
— Encourage good experimental code through code reuse and social coding conventions.
The code for any given experiment can harm a system over the long run, for example
by introducing bugs or by worsening code quality. A natural response to this problem
is to build a research framework into the code base to contain and manage experiments. However, we have found that experiments take such varied forms that it isdifficult to capture all of the requirements in any framework. We have found better
success through two efforts. First, we provide tools to make common research tasks— e.g., logging and condition assignment — easier and more consistent. Second, we
have established social coding conventions including code review to ensure that newexperiments follow established norms and minimize impact to the core code.
— Invest in tools that allow the community of users to help. In many cases, it is possible to build features that allow users to perform actions without intervention froman administrator. For example, while researchers in our lab have little interest in
maintaining a movie database, our users are passionate about the subject.
3. THE MOVIELENS DATASETS

There have been four MovieLens datasets released, known as 100k, 1m, 10m, and 20m,
reflecting the approximate number of ratings in each dataset. These datasets, first released in 1998, and with a major release every 5-6 years since, have grown in size alongwith the MovieLens system. Along with the release of the 20m dataset, GroupLens began hosting additional, non-archival datasets (an unabridged version for completenessalong with a small version for speed) that are refreshed periodically to include up-todate movies: latest and latest-small.
Summary statistics for the MovieLens datasets are shown in Table II.
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:13

Table II. Quantitative summary of the MovieLens ratings datasets. The sole computed column, Density, represents the percentage of cells in the full user-item matrix that contain rating values.
Name
ML 100K
ML 1M

Date Range
9/1997 - 4/1998
4/2000 - 2/2003

ML 10M

1/1995 - 1/2009

ML 20M

1/1995 - 3/2015

Rating Scale
1-5, stars
1-5, stars
0.5-5,
half-starsa
0.5-5,
half-starsa

Users
943
6,040

Movies
1,682
3,706

Ratings
100,000
1,000,209

Tag Apps
00Density
6.30%
4.47%

69,878

10,681

10,000,054

95,5801.34%

138,49327,27820,000,263

465,564

0.54%

a MovieLens changed from a 1-5 star scale to a 0.5-5.0 half-star scale on Feb 18, 2003.

500,000
400,000
300,000
200,000
100,0000
500,000
400,000
300,000
200,000
100,0000
500,000
400,000
300,000
200,000
100,0000
500,000
400,000
300,000
200,000
100,000
0

100k

1m

10m

20m

1998

2000

2002

2004

2006

2008

2010

2012

2014

Fig. 10. Number of ratings across time in the four MovieLens datasets, aggregated by month.

All of the currently-released MovieLens datasets share several characteristics. They
each represent rating tuples of the form <user, item, rating, timestamp>. Ratingsand other data are attributed to anonymized user ids (these ids do not map acrossdatasets). Movies are listed along with their titles in MovieLens, along with zero or
more genres (movie ids do map across datasets). Only users with at least 20 ratings
are included.
These four datasets differ in their sampling methods. Though all four required aminimum of 20 ratings to include a user, 100k also required complete demographic
data. 1m sampled users who joined the system in the year 2000, but the sample wascollected in early 2003, leading to a sparse set of ratings from 2001 and 2002. 10m and20m selected random users across the entire history of the system. These collection
methods created datasets that reflect the bursts of ratings activity that happened inMovieLens. See Figure 10 for a visualization of the density of ratings within the four
datasets across time.
Several notable changes have affected the structure of the datasets between releases. The 100k and 1m datasets include users’ demographic data (age, gender, occupation, zip code), while the 10m and 20m datasets do not include any demographic
information (it stopped being collected in the site). Only the 10m and 20m datasets inACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:14

Harper and Konstan

clude tag applications (the earlier datasets were released before the tagging featureexisted). 20m includes a table mapping MovieLens movie IDs to movie IDs in two external sites, to allow dataset users to build more complete content-based representationsof the items.
As noted above, these datasets are the result of actual people interacting with MovieLens, a steadily evolving system. Some of the design changes, e.g., changing to halfstar ratings, have a directly-measurable effect on the contents of the datasets. Otherchanges, e.g., adding a minimum of 15 ratings to the sign-up process, may have a more
subtle influence. Changes such as these may have led different users to reach the 20
ratings minimum, or to rate different types of movies. These subtle differences are anecessary artifact of generating datasets from a long-running and changing system.
3.1. Impact

As has become increasingly well-recognized by both researchers and funders of research, shared datasets can have a substantial positive impact on research, education,and practice. By publishing datasets covering more than 17 years of usage of a single
evolving system, the MovieLens datasets have been able to make a distinctive contribution in the field of recommender systems as well as related fields.
3.1.1. Contributions to Research— In Spring, 2015, a search for “movielens” yields 2,750 results in Google Books and7,580 results in Google Scholar.
— We are particularly proud that the MovieLens datasets were used to develop and test
many of the core algorithmic advances in recommender systems, including item-item
collaborative filtering [Sarwar et al. 2001; Karypis 2001; Deshpande and Karypis2004], dimensionality reduction collaborative filtering [Sarwar et al. 2000], trustbased recommenders [O’Donovan and Smyth 2005; Massa and Avesani 2007], recommenders on rapidly-changing sets of items [Das et al. 2007], and cold start algorithms[Schein et al. 2002].
— The datasets continue to be actively used today (1,112 search results in Google
Scholar are dated 2014), and we hope the recent release of our 20m rating dataset
will help many other researchers.
3.1.2. Contributions to Education and Practice— In 2014, the datasets were downloaded more than 140,000 times from the GroupLens
web site (http://grouplens.org/datasets/movielens); while some of this usage reflects
research use, we believe the vast majority of it is educational. We hear from faculty
who use the datasets in their courses (in some cases seeking our permission to makecustom-formatted versions for their students), from students using the datasets in assignments or self-study, and from textbook authors and others who use the datasetsin their materials. From the beginning we have made MovieLens datasets free forany non-commercial use (including internal research and education by commercial
entities).
— The datasets have found a niche as test data for sample code in popular-press
programming and data science books. For example, the datasets have been used
to demonstrate building an item-item collaborative filtering algorithm in Python[Segaran 2007], as well as to demonstrate loading, exploring, and visualizing data
in a high-performance distributed computing platform [Pentreath 2015].
— We have disseminated the MovieLens datasets ourselves through a MOOC (massive online open course) on recommender systems [Konstan et al. 2014] and throughautomated build tool integration with the LensKit recommender system toolkit(http://lenskit.org) [Ekstrand et al. 2011].
ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:15— We also are aware of fairly extensive use by commercial entities. We have been contacted by companies using the MovieLens datasets to test their own recommender
algorithms and systems, to benchmark external systems, and for training and demonstration purposes.
3.2. Limitations

The extensive changes to the MovieLens user experience inevitably have led to less“clean” ratings data than if we had kept the interface constant. As discussed above,changes to the searching tools, recommendation algorithms, and available features allchange the content that is shown to users, which in turn affects the resulting ratings
data. Further, our extensive experimentation on the MovieLens platform has further
changed user rating behavior in a variety of ways. However, MovieLens is not uniquein this problem — recommendation sites like Amazon and Netflix also have introduced
substantial interface changes and perturbations through A/B testing.
The MovieLens datasets only include data from users with at least 20 ratings, andtherefore are inherently biased towards “successful” users. That is, the users who areless interested in rating movies, were unable to find enough ratable content, or did
not enjoy their initial experience in the system are not included in the datasets. It is
possible that these users are fundamentally different from the users in the datasets.
The datasets associate timestamps with each rating, but these timestamps do not
represent the date of consumption. Ratings in MovieLens can happen any time, possibly many years after watching a movie. Often, users will enter a large number ofratings in a single session, backfilling their ratings history for personal satisfaction orin the hopes of getting more personalized recommendations. As discussed above, thesebackfilled ratings are often prompted by the recommendations we display, further reducing the value of the timestamps as even an estimate of when the user thought
about the movie.
3.3. Recommendations for Usage

In this section we offer some experience-based guidelines for making more effective
use of the MovieLens datasets, to further our aim of making them generally valuable
to researchers, students, and practitioners.
For those performing algorithmic research, much of the power of these datasets liesin the ability to compare results with prior research papers or with other algorithms.
We strongly suggest using the same dataset used in a published study when comparing
against those published results. Alternatively, in many cases it is possible to use tools
such as LensKit to replicate prior results and apply them — and your new algorithm— to the new dataset.
For those not comparing with historical published results, we strongly recommend
using the 20m rating dataset because of the greater number of tags and the links toexternal metadata sources. Publishing results based on this dataset will make it easierfor others to compare your results with the results of approaches that make greateruse of metadata and content-based techniques.
Those who are interested in combining ratings or tagging data with rich movie
content data should consider using the 20m dataset in combination with external resources. The 20m dataset includes a map of MovieLens IDs to IMDb4 (“Internet Movie
Database”) IDs. Though IMDb does not provide APIs or encourage access to their data,
their IDs are recognized across many different movie sites, including several with API
access to interesting sources of metadata. Since MovieLens movie IDs are stable across
4 http://imdb.com

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:16

Harper and Konstan

datasets, the 20m linking file can be used in combination with any of the earlier released datasets.
Educators should consider using the MovieLens latest-small dataset. Its size on
disk makes it quick to download, parse, and process. It contains recent movies, which
students may enjoy (as compared with the traditional 100k dataset, that contains nomovies past 1998). The dataset is also formatted as a .csv file with a header row, whichis consistent with modern dataset conventions.
3.4. Alternative Datasets

There are several other datasets with explicit ratings that have been frequently usedwith (or instead of) the MovieLens datasets. These alternatives differ from MovieLens
in terms of their size and shape, their domain, and their context of user interactions.
Researchers might consider alternative datasets when they are a better match withthe online system they wish to simulate [Shani and Gunawardana 2011], or if theywish to evaluate an approach across multiple datasets. There are too many alternatives to mention; in this section we discuss several of the most frequently cited or
currently prominent alternatives. See Table III for an overview, ordered by number ofratings.
Table III. Quantitative summary of the prominent alternative datasets that include explicit ratings data. The
sole computed column, Density, represents the percentage of cells in the full user-item matrix that containrating values.
Name
Book-Crossing
EachMoviea
Jester (dataset1)
Amazon
Netflix Prizea
Yahoo Music (C15)

Date Range
2001 - 2004
1995 - 1997
1999 - 2003
1996 - 2014
1998 - 2005
1999 - 2009

Domainbooks
moviesjokes
manymoviesmusiccRating Scale0 - 10, 11 discrete values
0 - 14, 26 discrete valuesb-10 - 10, continuous
1 - 5, 5 discrete values
1 - 5, 5 discrete values
0 - 100, 101 discrete valuesd

Ratings
1.1m
2.7m
4.1m
82.8m
100.5m262.8mDensity
0.003%
2.872%
57.463%< 0.001%
1.178%
0.042%a No longer available.
b The result of using multiple ratings scales, rescaled for the final dataset.
c Includes ratings for songs, albums, artists, and genres.
d 98% of rating values are multiples of 10.

Book-Crossing. The Book-Crossing dataset5 [Ziegler et al. 2005] is a snapshot of an
online book-rating community. The dataset contains 100,000 users, 340,000 books, and
1.1 million ratings, collected in 2004. It has low density (3.2x10−5 ). The ratings take
11 possible values, on a 0-10 scale. The snapshot was taken in 2004, but the ratings
are not associated with timestamps. However, the dataset reveals users’ locations andages.
EachMovie. The EachMovie dataset was heavily used in research in the early 2000s
before it was made unavailable by DEC due to legal concerns about the potential reidentifiability of the users in the dataset. This dataset contains 2.7 million ratings from59,000 users across 1,500 movies, collected from 1995-1997. The ratings take 26 possible values in the range of 0-14, reflecting the use and subsequent rescaling of several
approaches to collecting ratings. As mentioned earlier, this dataset was the original
seed data for the MovieLens recommender, but this ratings data is not included in anyof the MovieLens datasets.
Jester. The Jester datasets6 [Goldberg et al. 2001] contain continuous, explicit ratings of jokes on a -10 to 10 scale — users rate jokes using a slider widget. There are
5 http://www2.informatik.uni-freiburg.de/∼cziegler/BX/
6 http://eigentaste.berkeley.edu/dataset/

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:17

several versions currently available; the largest (“Dataset 1”) contains 4.1 million ratings from 73,421 users, collected from 1999 to 2003. Because there are only 100 jokes,
this dataset is very dense relative to other options. This dataset does not contain ratingtimestamps.
Amazon. Amazon review datasets7 [McAuley et al. 2015b; McAuley et al. 2015a] arenotable for several reasons. The ratings are associated with textual reviews, span 18
years, and encompass a wide variety of products, from instant videos to baby clothes.
The full (“aggressively deduplicated”) dataset is also very large and sparse: 82 million
ratings from 21 million users across nearly 10 million items (density 3.9x10−7 ). The
ratings take 5 possible values, 1-5, and are associated with timestamps.
Netflix. The Netflix Prize dataset was made available in 2006 as part of the Netflix
Prize to improve the accuracy of predicitons8 . It was taken down in 2009 for legal
reasons. The training dataset contains 480,000 users, 17,000 items, and 100 million
ratings; it has density comparable to the MovieLens 10M dataset. The ratings take 5
possible values, 1-5, and are associated with timestamps.
Yahoo Music. Yahoo! Labs provides a number of music datasets9 extracted from theirMusic product. Their “C15” dataset, released for the KDD Cup [Dror et al. 2012] provides user ratings on songs, albums, artists, and music genres on a 101 point (0-100)
rating scale (though 98% of the ratings are multiples of 10). The dataset contains 1
million users, 620,000 music items (the item types listed above are mixed together),and 262 million ratings. The dataset contains partial timestamps that disclose time
but obfuscate the absolute date.
4. CONCLUSION

In this paper, we provide a historical view of the MovieLens system and datasets, whichhave had a substantial impact in education, research, and industry. The datasets area product of the MovieLens system, which has changed substantially in the 17 years
since its first release. We discuss the key features of the system in terms of their impact on users, their impact on the research literature through field studies, and theirdownstream impact on the datasets.
We recently released a new version of MovieLens (v4) and a new benchmark dataset(20m). We hope these releases continue to encourage the development of high-quality
educational materials, software systems, startup companies, and academic research.
ACKNOWLEDGMENTS
Many have dedicated themselves to building and improving MovieLens and the MovieLens datasets. In particular we would like to thank John Riedl for his instrumental contributions and leadership. Other key contributors include Istvan Albert, Al Borchers, Dan Cosley, Brent J. Dahlen, Rich Davies, Michael Ekstrand,
Dan Frankowski, Nathaniel Good, Jon Herlocker, Daniel Kluver, Shyong (Tony) Lam, Michael Ludwig, Sean
McNee, Chad Salvatore, Shilad Sen, and Loren Terveen. We also gratefully acknowledge the MovieLens
members, who make this project possible.

REFERENCES
Shuo Chang, F. Maxwell Harper, and Loren Terveen. 2015. Using Groups of Items for Preference Elicitation in Recommender Systems. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). ACM, New York, NY, USA, 1258–1269.
DOI:http://dx.doi.org/10.1145/2675133.2675210
7 http://jmcauley.ucsd.edu/data/amazon/
8 http://www.netflixprize.com/community/viewtopic.php?id=68
9 http://webscope.sandbox.yahoo.com/catalog.php?datatype=c

and

http://webscope.sandbox.yahoo.com/

catalog.php?datatype=r

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:18

Harper and Konstan

Dan Cosley, Dan Frankowski, Sara Kiesler, Loren Terveen, and John Riedl. 2005. How Oversight Improves Member-maintained Communities. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems (CHI ’05). ACM, New York, NY, USA, 11–20.
DOI:http://dx.doi.org/10.1145/1054972.1054975
Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is Seeing Believing?: How Recommender System Interfaces Affect Users’ Opinions. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’03). ACM, New York, NY, USA, 585–592.
DOI:http://dx.doi.org/10.1145/642611.642713
Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google News
Personalization: Scalable Online Collaborative Filtering. In Proceedings of the 16th International Conference on World Wide Web (WWW ’07). ACM, New York, NY, USA, 271–280.
DOI:http://dx.doi.org/10.1145/1242572.1242610
Mukund Deshpande and George Karypis. 2004. Item-based top-N Recommendation Algorithms. ACM Transactions on Information Systems 22, 1 (Jan. 2004), 143–177. DOI:http://dx.doi.org/10.1145/963770.963776
Sara Drenner, Max Harper, Dan Frankowski, John Riedl, and Loren Terveen. 2006. Insert Movie Reference
Here: A System to Bridge Conversation and Item-oriented Web Sites. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’06). ACM, New York, NY, USA, 951–954.
DOI:http://dx.doi.org/10.1145/1124772.1124914
Gideon Dror, Yahoo Labs, Noam Koenigstein, Yehuda Koren, and Markus Weimer. 2012. The Yahoo! music
dataset and KDDCup11. In Journal of Machine Learning Research Workshop and Conference Proceedings: Proceedings of KDD Cup 2011. 3–18.
Michael D. Ekstrand, Daniel Kluver, F. Maxwell Harper, and Joseph A. Konstan. 2015. Letting
Users Choose Recommender Algorithms: An Experimental Study. In Proceedings of the 9th
ACM Conference on Recommender Systems (RecSys ’15). ACM, New York, NY, USA, 11–18.
DOI:http://dx.doi.org/10.1145/2792838.2800195
Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John T. Riedl. 2011. Rethinking the
Recommender Research Ecosystem: Reproducibility, Openness, and LensKit. In Proceedings of theFifth ACM Conference on Recommender Systems (RecSys ’11). ACM, New York, NY, USA, 133–140.
DOI:http://dx.doi.org/10.1145/2043932.2043958
Malcolm Gladwell. 1999. The Science of the Sleeper. The New Yorker (Oct. 1999). http://gladwell.com/
the-science-of-the-sleeper/
Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. 2001. Eigentaste: A Constant Time Collaborative Filtering Algorithm. Information Retrieval 4, 2 (July 2001), 133–151.
DOI:http://dx.doi.org/10.1023/A:1011419012209
F. Maxwell Harper, Dan Frankowski, Sara Drenner, Yuqing Ren, Sara Kiesler, Loren Terveen, Robert Kraut,
and John Riedl. 2007a. Talk Amongst Yourselves: Inviting Users to Participate in Online Conversations.
In Proceedings of the 12th International Conference on Intelligent User Interfaces (IUI ’07). ACM, New
York, NY, USA, 62–71. DOI:http://dx.doi.org/10.1145/1216295.1216313
F. Maxwell Harper, Shilad Sen, and Dan Frankowski. 2007b. Supporting Social Recommendations with
Activity-balanced Clustering. In Proceedings of the 2007 ACM Conference on Recommender Systems(RecSys ’07). ACM, New York, NY, USA, 165–168. DOI:http://dx.doi.org/10.1145/1297231.1297262
F. Maxwell Harper, Funing Xu, Harmanpreet Kaur, Kyle Condiff, Shuo Chang, and Loren Terveen. 2015. Putting Users in Control of Their Recommendations. In Proceedings of the 9th
ACM Conference on Recommender Systems (RecSys ’15). ACM, New York, NY, USA, 3–10.
DOI:http://dx.doi.org/10.1145/2792838.2800179
George Karypis. 2001. Evaluation of Item-Based Top-N Recommendation Algorithms. In Proceedings of theTenth International Conference on Information and Knowledge Management (CIKM ’01). ACM, New
York, NY, USA, 247–254. DOI:http://dx.doi.org/10.1145/502585.502627
Joseph A. Konstan, Bradley N. Miller, David Maltz, Jonathan L. Herlocker, Lee R. Gordon, and John Riedl.
1997. GroupLens: Applying Collaborative Filtering to Usenet News. Commun. ACM 40, 3 (March 1997),
77–87. DOI:http://dx.doi.org/10.1145/245108.245126
Joseph A. Konstan, J.D. Walker, D. Christopher Brooks, Keith Brown, and Michael D. Ekstrand. 2014.
Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid
MOOC. In Proceedings of the First ACM Conference on Learning @ Scale Conference (L@S ’14). ACM,
New York, NY, USA, 61–70. DOI:http://dx.doi.org/10.1145/2556325.2566244
John G. Lynch, Jr., Dipankar Chakravarti, and Anusree Mitra. 1991. Contrast Effects in Consumer Judgments: Changes in Mental Representations or in the Anchoring of Rating Scales? Journal of Consumer
Research 18, 3 (Dec. 1991), 284–297. http://www.jstor.org/stable/2489340

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

The MovieLens Datasets: History and Context

XXXX:19

Paolo Massa and Paolo Avesani. 2007. Trust-aware Recommender Systems. In Proceedings of the2007 ACM Conference on Recommender Systems (RecSys ’07). ACM, New York, NY, USA, 17–24.
DOI:http://dx.doi.org/10.1145/1297231.1297235
Julian McAuley, Rahul Pandey, and Jure Leskovec. 2015a. Inferring Networks of Substitutable
and Complementary Products. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’15). ACM, New York, NY, USA, 785–794.
DOI:http://dx.doi.org/10.1145/2783258.2783381
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015b. Image-Based Recommendations on Styles and Substitutes. In Proceedings of the 38th International ACM SIGIR Conference
on Research and Development in Information Retrieval (SIGIR ’15). ACM, New York, NY, USA, 43–52.
DOI:http://dx.doi.org/10.1145/2766462.2767755
Bradley Norman Miller. 2003. Toward a personal recommender system. Ph.D. University of Minnesota, United States – Minnesota. http://search.proquest.com/dissertations/docview/305324342/
abstract/A46BCC87FC4D4DD4PQ/1?accountid=14586
Mark O’Connor, Dan Cosley, Joseph A. Konstan, and John Riedl. 2001. PolyLens: A Recommender System
for Groups of Users. In Proceedings of the Seventh Conference on European Conference on Computer
Supported Cooperative Work (ECSCW’01). Kluwer Academic Publishers, Norwell, MA, USA, 199–218.
http://dl.acm.org/citation.cfm?id=1241867.1241878
John O’Donovan and Barry Smyth. 2005. Trust in Recommender Systems. In Proceedings of the 10th International Conference on Intelligent User Interfaces (IUI ’05). ACM, New York, NY, USA, 167–174.
DOI:http://dx.doi.org/10.1145/1040830.1040870
Nick Pentreath. 2015. Machine Learning with Spark. Packt Publishing Ltd.
Reid Priedhorsky, Mikhil Masli, and Loren Terveen. 2010. Eliciting and Focusing Geographic Volunteer
Work. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW’10). ACM, New York, NY, USA, 61–70. DOI:http://dx.doi.org/10.1145/1718918.1718931
Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, andJohn Riedl. 2002. Getting to Know You: Learning New User Preferences in Recommender Systems. In
Proceedings of the 7th International Conference on Intelligent User Interfaces (IUI ’02). ACM, New York,NY, USA, 127–134. DOI:http://dx.doi.org/10.1145/502716.502737
Al Mamunur Rashid, George Karypis, and John Riedl. 2008. Learning Preferences of New Users in Recommender Systems: An Information Theoretic Approach. ACM SIGKDD Explorations Newsletter 10, 2(Dec. 2008), 90–100. DOI:http://dx.doi.org/10.1145/1540276.1540302
Yuqing Ren, F. Harper, Sara Drenner, Loren Terveen, Sara Kiesler, John Riedl, and Robert Kraut. 2012.
Building Member Attachment in Online Communities: Applying Theories of Group Identity and Interpersonal Bonds. Management Information Systems Quarterly 36, 3 (Sept. 2012), 841–864. http://aisel.aisnet.org/misq/vol36/iss3/11
Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens:
An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW ’94). ACM, New York, NY, USA, 175–186.
DOI:http://dx.doi.org/10.1145/192844.192905
Eric Ries. 2011. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2000. Application of dimensionality reduction in recommender system-a case study. Technical Report. DTIC Document. http://oai.dtic.mil/oai/
oai?verb=getRecord&metadataPrefix=html&identifier=ADA439541
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based Collaborative Filtering
Recommendation Algorithms. In Proceedings of the 10th International Conference on World Wide Web(WWW ’01). ACM, New York, NY, USA, 285–295. DOI:http://dx.doi.org/10.1145/371920.372071
Andrew I. Schein, Alexandrin Popescul, Lyle H. Ungar, and David M. Pennock. 2002. Methods and Metrics
for Cold-start Recommendations. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’02). ACM, New York, NY, USA,
253–260. DOI:http://dx.doi.org/10.1145/564376.564421
Toby Segaran. 2007. Programming Collective Intelligence: Building Smart Web 2.0 Applications. ”O’Reilly
Media, Inc.”.
Shilad Sen, F. Maxwell Harper, Adam LaPitz, and John Riedl. 2007. The Quest for Quality Tags. In Proceedings of the 2007 International ACM Conference on Supporting Group Work (GROUP ’07). ACM, New
York, NY, USA, 361–370. DOI:http://dx.doi.org/10.1145/1316624.1316678
Shilad Sen, Shyong K. Lam, Al Mamunur Rashid, Dan Cosley, Dan Frankowski, Jeremy Osterhouse,
F. Maxwell Harper, and John Riedl. 2006. Tagging, Communities, Vocabulary, Evolution. In Proceed-

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.

XXXX:20

Harper and Konstan

ings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (CSCW ’06).
ACM, New York, NY, USA, 181–190. DOI:http://dx.doi.org/10.1145/1180875.1180904
Shilad Sen, Jesse Vig, and John Riedl. 2009. Learning to Recognize Valuable Tags. In Proceedings of the
14th International Conference on Intelligent User Interfaces (IUI ’09). ACM, New York, NY, USA, 87–96.
DOI:http://dx.doi.org/10.1145/1502650.1502666
Guy Shani and Asela Gunawardana. 2011. Evaluating Recommendation Systems. In Recommender Systems
Handbook, Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Springer US, 257–
297. http://link.springer.com/chapter/10.1007/978-0-387-85820-3 8
Jesse Vig, Shilad Sen, and John Riedl. 2012. The Tag Genome: Encoding Community Knowledge to Support
Novel Interaction. ACM Transactions on Interactive Intelligent Systems 2, 3 (Sept. 2012), 13:1–13:44.
DOI:http://dx.doi.org/10.1145/2362394.2362395
Jesse Vig, Matthew Soukup, Shilad Sen, and John Riedl. 2010. Tag Expression: Tagging with Feeling. In
Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST’10). ACM, New York, NY, USA, 323–332. DOI:http://dx.doi.org/10.1145/1866029.1866079
Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving Recommendation Lists Through Topic Diversification. In Proceedings of the 14th International Conference on World
Wide Web (WWW ’05). ACM, New York, NY, USA, 22–32. DOI:http://dx.doi.org/10.1145/1060745.1060754

ACM Transactions on Interactive Intelligent Systems, Vol. V, No. N, Article XXXX, Pub. date: 2015.