Automated Recommendation Systems Collaborative Filtering Through Reinforcement Learning

Within this work we explore the topic of large scale, automated recommendation systems. We focus on collaborative filtering approaches, wherein a system suggests new products to users based on their viewing history as well as other known demographics. There are several approaches to this in current literature, the simplest of which treat it as a matrix completion problem. We explore the setting from a reinforcement learning perspective by applying traditional algorithms for reinforcement learning to the problem. I. PROBLEM FORMULATION Numerous online services such as Netflix, Amazon, Yelp, Pandora, online advertisings, etc. provide automated recommendations to help users to navigate through a large collection of items. Every time a user queries the system for a new item, a suggestion is made on the basis of the user’s past history and (when available) their demographic profile. Two typical ways of producing these recommendations are collaborative filtering and content-based filtering. There are two simultaneous goals to be satisfied: helping the user to explore the available items and probing the user’s preferences. One of the models that captures this setting well is the multiarm bandit, an important model for decision making under uncertainty. In this model a set of arms with unknown reward profiles is given and, at each time slot, the decision maker must choose an arm to maximize his expected reward. Clearly, the decision at each time slot should depend on previous observations. Thus, there is a trade-off between exploration, trying arms with more uncertain reward in order to gather more information, and exploitation, pulling arms with relatively high reward expectations. For our purposes the arms have a very specific structure and this setting has previously been referred to as the “linearbandits” model (see [3]). Here, it is assumed that the underlying matrix of preferences (which contains the rating user i gives to item j at entry (i, j)) has a low-rank structure. Hence, ratings made by user i to item j can be approximated by a scaler product of two feature vectors ai, bj ∈ R, characterizing user and item respectively. In other words our observations, rij can be viewed as rij = a T i bj + zij where zij represents the unexplained factors. In the general setting, both the user and item feature vectors are treated as unknown, and our recommendation algorithms must estimate them over time. However, some works (like [1]) make a simplifying assumption that the item feature vectors are known. We explore both settings, but find more meaningful results in the case where the item feature vectors are known. In this case, the item feature vectors can be either constructed explicitly, or derived from users’ feedback using matrix factorization methods. With the item latent vectors in hand, we can treat each user independently and throughout the explore-exploit trade-off, we can try to estimate and exploit the users latent vectors. These feature vectors can depend on users’ demographic information and their past behavior in rating items. The goal of our system is to develop a recommendation policy, which suggests items to users. This policy will, at each time slot, output a recommendation based on the previous observations. This policy must properly adjust for the exploreexploit trade-off, and classically there are two types of policies, which differ in the way they perform exploration: optimistic policies, e.g. upper confidence bound (UCB), and probabilistic policies, e.g. posterior sampling. UCB algorithms have been applied to this problem in the past, but posterior sampling is less common. Posterior sampling (also known as Thompson sampling) was introduced in 1933 and offers significant advantages over UCB methods (as shown in [2]), however until recently it has not been very popular or feasible. Our primary objective is to explore the feasibility of collaborative filtering through posterior sampling. We analyze its performance on real world data, specifically the freely available MovieLens datasets, and compare it to existing methods such as UCB and the work done in [1] II. SYSTEM MODELS AND ALGORITHMS In this section we will introduce some notation used throughout the rest of this work, as well as the algorithms that we seek to implement.