Towards Learning User-Adaptive State Models in a Conversational Recommender System

Typical conversational recommender systems support interactive strategies that are hard-coded in advance and followed rigidly during a recommendation session. In fact, Reinforcement Learning techniques can be used in order to autonomously learn an optimal (user-adaptive) strategy, basically by exploiting some information encoded as features of a state representation. In this regard, it is important to determine the set of relevant state features for a given recommendation task. In this paper, we address the issue of feature relevancy, and determine the relevancy of adding four different features to a baseline representation. We show that adding a feature might not always be beneficial, and that the relevancy could be influenced by the user behavior. The results motivate the application of our approach online, in order to acquire the right mixture of online user behavior for addressing the relevancy problem. 1 Background and Motivation Recommender systems [Resnick and Varian, 1997] are online applications that assist users in their informationseeking tasks by offering personalized product recommendations during an interaction session. In order to support more interactive sessions, conversational recommender systems [Ricci et al., 2003; Thompson et al., 2004; Shimazu, 2001] have been recently proposed. These systems adopt the conversation style of real-life dialogues in order to guide users through complex product spaces. Mainly, at each dialogue stage, the system selects one from amongst a set of available system actions, e.g., recommend some product, ask the user for some information etc. The particular action selection is specified by the system’s recommendation strategy. For instance, suppose that a conversational recommender is queried for hotels that would suit the user preferences. Then, it could employ the following two strategies (among many others): 1) initially query the user in detail for her preferences, in order to extract a small product subset, or 2) recommend a set of products, and exploit the user feedback to refine future recommendations. Conversational systems typically employ a rigid strategy, i.e., one which is hard-coded inside the system and followed rigidly during a session [Ricci et al., 2003; Thompson et al., 2004; Shimazu, 2001]. A limitation of this approach is that there could be numerous rigid strategies for a given recommendation task. Furthermore, as system designers rely on intuition and previous experience in order to select a strategy, it is is possible that they shall have to evaluate several strategies before they discover a (almost) “good” one. This is an infeasible process in terms of budget, time, task force etc. These limitations would be tamed if conversational systems could be capable of determining by themselves the best (optimal) strategy for assisting the users. In a previous paper [Ricci and Mahmood, 2007], we have tackled these requirements by proposing a new type of recommender system which is able to autonomously improve an initial strategy in order to learn and adopt an optimal one. We validate our approach by applying it within the NutKing travel recommender system [Ricci andMahmood, 2007], and by improving NutKing’s (current) rigid strategy in order to learn an optimal one. We learn the strategy through the Reinforcement Learning (RL) paradigm [Sutton and Barto, 1998] in the context of the Markov Decision Process (MDP) framework (we avoid a lengthy explanation due to space constraints). Briefly, our system, at each stage of the session, observes the current state (or situation) of the user and the session activity. In one or more states (labeled as System Decision Points (SDPs)), multiple actions could be available for execution. The system executes these different actions as the interaction proceeds, and in return, receives a numerical reward informing it of the action’s acceptability for the user. As the interaction continues, the system learns to maximize the reward it receives in each possible state, i.e., it learns to avoid the unacceptable actions and to take the acceptable ones, until it learns the optimal action in each state, i.e., the optimal policy (strategy). Also, in the non-SDP states, only a single action is available which we consider as the optimal action. The optimal behavior is hence characterized by information in the observed state, which is represented as a combination of feature values. Considering that online user behavior is generally detailed and complex, the selection of the relevant state features for a given recommendation task becomes quite crucial. In this paper, we address several issues related to the feature relevancy problem. We introduce two relevancy criteria, and use them to determine the relevancy for four different feature sets (or state representations). We show that adding a feature might not always be relevant for a given task. We also investigate relevancy across different (simulated) user behaviors models, and show that relevancy is influenced by the user behavior. These results motivate the application of our approach in an online context, where we will be able to acquire behavior for a real user population, and to learn an optimal policy for this population.