Neural Model-Based Reinforcement Learning for Recommendation

1 INTRODUCTION . Recommendation systems have become a crucial part of almost all online service platforms . A typical interaction between the system and its users is — users are recommended a page of items and they provide feedback , and then the system recommends a new page of items . A common way of building recommendation systems is to estimate a model which minimizes the discrepancy between the model prediction and the immediate user response according to some loss function . In other words , these models do not explicitly take into account the long-term user interest . However , user ’ s interest can evolve over time based on what she observes , and the recommender ’ s action may significantly influence such evolution . In some sense , the recommender is guiding users ’ interest by displaying particular items and hiding the rest . Thus , a recommendation strategy which takes users ’ long-term interest into account is more favorable . Reinforcement learning ( RL ) is a learning paradigm where a policy will be obtained to guide the actions in an environment so as to maximize the expected long-term reward . Although RL framework has been successfully applied to many game settings , such as Atari ( Mnih et al. , 2015 ) and GO ( Silver et al. , 2016 ) , it met a few challenges in the recommendation system setting because the environment will correspond to the logged online user . First , a user ’ s interest ( reward function ) driving her behavior is typically unknown , yet it is critically important for the use of RL algorithms . In existing RL algorithms for recommendation systems , the reward functions are manually designed ( e.g . ±1 for click/no-click ) which may not reflect a user ’ s preference over different items ( Zhao et al. , 2018a ; Zheng et al. , 2018 ) . Second , model-free RL typically requires lots of interactions with the environment in order to learn a good policy . This is impractical in the recommendation system setting . An online user will quickly abandon the service if the recommendation looks random and do not meet her interests . Thus , to avoid the large sample complexity of the model-free approach , a model-based RL approach is more preferable . In a related but a different setting where one wants to train a robot policy , recent works showed that model-based RL is much more sample efficient ( Nagabandi et al. , 2017 ; Deisenroth et al. , 2015 ; Clavera et al. , 2018 ) . The advantage of model-based approaches is that potentially large amount of off-policy data can be pooled and used to learn a good environment dynamics model , whereas model-free approaches can only use expensive on-policy data for learning . However , previous model-based approaches are typically designed based on physics or Gaussian processes , and not tailored for complex sequences of user behaviors . To address the above challenges , we propose a novel model-based RL framework for recommendation systems , where a user behavior model and the associated reward function are learned in unified minimax framework , and then RL policies are learned using this model . Our main technical innovations are : 1 . We develop a generative adversarial learning ( GAN ) formulation to model user behavior dynamics and recover her reward function . These two components are estimated simultaneously via a joint mini-max optimization algorithm . The benefits of our formulation are : ( i ) a more predictive user model can be obtained , and the reward function are learned in a consistent way with the user model ; ( ii ) the learned reward allows later reinforcement learning to be carried out in a more principled way , rather than relying on manually designed reward ; ( ii ) the learned user model allows us to perform model-based RL and online adaptation for new users to achieve better results . 2 . Using this model as the simulation environment , we also develop a cascading DQN algorithm to obtain a combinatorial recommendation policy . The cascading design of action-value function allows us to find the best subset of items to display from a large pool of candidates with time complexity only linear in the number of candidates . In our experiments with real data , we showed that this generative adversarial model is a better fit to user behavior in terms of held-out likelihood and click prediction . Based on the learned user model and reward , we show that the estimated recommendation policy leads to better cumulative long-term reward for the user . Furthermore , in the case of model mismatch , our model-based policy can also quickly adapt to the new dynamics with a much fewer number of user interactions compared to model-free approaches . 2 RELATED WORK . Commonly used recommendation algorithms typically use a simple user model . For instance , Wide & Deep networks ( Cheng et al. , 2016 ) and other methods such as xgboost ( Chen & Guestrin , 2016 ) and DFM ( Guo et al. , 2017 ) based on logistic regression essentially assume a user chooses each item independently ; Collaborative competitive filtering ( Yang et al. , 2011 ) takes into account the context where a user makes her choice but assumes that user ’ s behaviors in each page view are independent . Session-based RNN ( Hidasi et al. , 2016 ) and session-based KNN ( Jannach & Ludewig , 2017 ) improve upon previous approaches by modeling users ’ history , but this model does not recover a users ’ reward function and can not be used subsequently for reinforcement learning . Bandit based approaches , such as LinUCB ( Li et al. , 2010 ) , can deal with adversarial user behaviors , but the reward is updated in a Bayesian framework and can not be directly used by a reinforcement learning framework . Zhao et al . ( 2018b ; a ) ; Zheng et al . ( 2018 ) used model-free RL for recommender systems , which may require many user interactions and the reward function is manually designed . Model-based reinforcement learning has been commonly used in robotics applications and resulted in reduced sample complexity to obtain a good policy ( Deisenroth et al. , 2015 ; Nagabandi et al. , 2017 ; Clavera et al. , 2018 ) . However , these approaches can not be used in the recommendation setting , as a user behavior model typically consists of sequences of discrete choices under a complex session context . 3 SETTING AND RL FORMULATION . We will focus on a simple yet typical setting where the recommendation system and its user interact as follows : a user is displayed to a page of k items and she provides feedback by clicking on one or none of these items , and then the system recommends a new page of k items . Our model can be extended to settings with more complex page views and user interactions , but these settings are left for future studies . Since reinforcement learning can take into account long-term reward , it holds the promise to improve users ’ long-term engagement with an online platform . In the RL framework , a recommendation system wants to find a policy π ( s , I ) to choose a set I of k items based on user state s , such that the long-term expected reward to the user is maximized , i.e . π∗ = arg max π ( st , It ) E [ ∞∑ t=0 γtr ( st , at ) ] , where s0 ∼ p0 , At ∼ π ( st , It ) , st+1 ∼ P ( ·|st , At ) , at ∈ At , ( 1 ) where several key aspects of this RL framework are as follows : ( 1 ) Environment : will correspond to a logged online user who can click on one of the k items displayed by the recommendation system in each page view ( or interaction ) ; ( 2 ) State st ∈ S : will correspond to an ordered sequence of a user ’ s historical clicks ; ( 3 ) Action At ∈ ( It k ) of the recommender : will correspond to a subset of k items chosen by the recommender from It to display to the user . ( It k ) means the set of all subsets of k items of It . It ⊂ I is the subset of available items to recommend at time t among all items I . ( 4 ) State Transition P ( ·|st , At ) : S × ( I k ) 7→ P ( S ) : will correspond to a user behavior model which returns the transition probability for st+1 given previous state st and the set of items At displayed by the system . It is equivalent to the distribution φ ( st , At ) over a user ’ s actions , which is defined in our user model in section 4.1 . ( 5 ) Reward Function r ( st , At , at ) : S × ( I k ) × I 7→ R : will correspond to a user ’ s utility or satisfaction after making her choice at ∈ At in state st . Here we assume that the reward to the recommendation system is the same as the user ’ s utility . Thus , a recommendation algorithm which optimizes its long-term reward is designed to satisfy the user in a long run . One can also include the company ’ s benefit to the reward , but in this paper we will focus on users ’ satisfaction . ( 6 ) Policy At ∼ π ( st , It ) : S × 2I 7→ P ( ( I k ) ) : will correspond to a recommendation strategy which takes a user ’ s state st and returns the probability of displaying a subset At of It . Remark . We note that in the above mapping , Environment , State and State Transition are associated with the user , the Action and Policy are associated with the recommendation system , and the Reward Function is associated with both the recommendation system and the user . Here we use the notation r ( st , At , at ) to emphasize the dependency of the reward on the recommendation action , as the user can only choose from the display set . However , the value of the reward is actually determined by the user ’ s state and the clicked item once the item occurs in the display set At . In fact , r ( st , At , at ) = r ( st , at ) · 1 ( at ∈ At ) . Thus , in section 4.1 where we discuss the user model , we simply denote r ( st , at ) = r ( st , At , at ) and assume at ∈ At is true . The overall RL framework for recommendation is illustrated in Figure 1 . Since both the reward function and the state transition model are not provided , we need to learn them from data . Once these quantities are learned , the optimal policy π∗ in Eq . ( 1 ) can be estimated by repeated querying to the model using algorithms such as Q-learning ( Watkins , 1989 ) . In the next two sections , we will explain our formulation for estimating the user behavior model as well as the reward function and design an efficient algorithm for learning the RL policy for the recommendation .

Summary: The authors propose a deep reinforcement learning based recommendation algorithm. Instead of manually designing reward function for RL, a generative adversarial network was proposed to learn the reward function based on user's dynamic behavior. The authors also try to provide an efficient combinatorial recommendation algorithm by designing a cascade DQN. The authors hold their experiments on the Movielens and Ant Financial news dataset. The authors adopt logistic regression (LR) and collaborative competitive filtering(CCF) as comparison baseline to evaluate recommendation performance. The authors also compared their proposed RL policy CQDN with LinUCB.