Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog

1 INTRODUCTION . In order to scale deep reinforcement learning ( RL ) to safety-critical , real-world domains , two abilities are needed . First , since collecting real-world interaction data can be expensive and timeconsuming , algorithms must be able to learn from off-policy data no matter how it was generated , or how little correlation between the data distribution and the current policy . Second , it is often necessary to carefully test a policy before deploying it to the real world ; for example , to ensure its behavior is safe and appropriate for humans . Thus , the algorithm must be able to learn offline first , from a static batch of data , without the ability to explore . This off-policy , batch reinforcement learning ( BRL ) setting represents a challenging RL problem . Most deep RL algorithms fail to learn from data that is not heavily correlated with the current policy ( Fujimoto et al. , 2018b ) . Even models based on off-policy algorithms like Q-learning fail to learn in the offline , batch setting , when the model is not able to explore . If the batch data is not sufficient to cover the state-action space , BRL models can suffer from extrapolation error , learning unrealistic value estimates of state-action pairs not contained in the batch ( Fujimoto et al. , 2018b ) . It can be impossible to correct for extrapolation error when there is a mismatch in the distribution of stateactions pairs in the batch data , and the distribution induced by the learned policy . For example , if the policy learns to select actions which are not contained in the batch , it can not learn a reasonable value function for those actions . Figure 1 illustrates this concept , where the batch only covers a subset of possible policies . Extrapolation error is particularly problematic in high-dimensional state and action spaces ( such as those inherent in language generation ) . We propose to resolve these issues by leveraging a pre-trained generative model of the state-action space , p ( a|s ) , trained on known sequences of interaction data . While training with RL , we penalize divergence from this prior model with different forms of KL-control . This technique ensures that the RL model learns a policy that stays close the state-action distribution of the batch , combating extrapolation error . We also propose using dropout to obtain uncertainty estimates of the target Qvalues , and use this lower bound to alleviate overestimation bias . We benchmark against a discrete adaptation of Batch Constrained Q-learning ( BCQ ) ( Fujimoto et al. , 2018b ) , a recently proposed state-of-the-art BRL algorithm for continuous domains , and show that our Way Off-Policy algorithm achieves superior performance in both a traditional RL domain , as well as in a challenging , underexplored , real-world reinforcement learning problem : using implicitly expressed human reactions in chat to improve open-domain dialog systems . When a machine learning system interacts with humans , ideally we would like to learn about the humans ’ preferences in order to improve its performance . Yet having humans manually indicate their preferences through explicit means like pressing a button ( e.g . Christiano et al . ( 2017 ) ) or submitting a feedback report , does not scale . Instead , we would like to be able to use humans ’ implicit reactions , such as the sentiment they express , or the length of the conversation , in order to improve the policy . However , applying off-policy batch RL to language generation is challenging because the number of potential combinations of words and sentences leads to a combinatorial explosion in the size of the state space . The action space – the set of frequent vocabulary words in the English language – is 20,000-dimensional . This compounds extrapolation error , making BRL even more difficult . However , when learning from human interactions in the wild , it is crucial to be able to learn offline and test the policy before deploying it , lest it learn inappropriate behaviors ( e.g . Horton ( 2016 ) ) . To support this work , we developed an interactive online platform that allows humans to chat with deep neural network dialog models running on a GPU ; the BRL models trained for this study are available live at https : //neural.chat/rl/ . Through this platform we collected human responses to a set of over 40 different dialog models over the course of several months . Using our Way Off-Policy algorithm , we are able to effectively learn from this batch of data , in spite of the fact that it was generated with a vastly different set of model architectures , which were trained on different datasets . Further , we use the batch to learn from many different reward functions designed post-hoc to extract implicit human preferences , something that is only possible with effective off-policy BRL . In summary , the contributions of this paper are : • A novel algorithm , Way Off-Policy learning , which is the first to propose using KL-control from a pre-trained prior model as a way to reduce extrapolation error in batch RL . • Experiments showing the effectiveness of WOP above strong baselines based on prior work ( e.g . Fujimoto et al . ( 2018b ) ) , on both traditional RL tasks and on the challenging problem of open-domain dialog generation . • A set of novel conversation rewards based on how human preferences are implicitly expressed in text . We are the first work to learn from implicit signals in conversation offline using batch RL . 2 RELATED WORK . The approach we propose is based on KL-control , a branch of stochastic optimal control ( SOC ) ( Stengel , 1986 ) where the Kullback-Leibler ( KL ) divergence from some distribution is used to regularize an RL policy ( e.g . ( Abdolmaleki et al. , 2018 ; Kappen et al. , 2012 ; Rawlik et al. , 2012 ; Todorov , 2007 ) ) . Well-known examples include Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) , and use conservative , KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy ( e.g . ( Haarnoja et al. , 2018 ; Kakade , 2002 ; Peters et al. , 2010 ; Rawlik et al. , 2012 ) ) . KL-control can also be applied to entropy maximization ( e.g . ( Ziebart et al. , 2008 ; Nachum et al. , 2017 ; Haarnoja et al. , 2017 ) ) ; for example , G-learning penalizes KLdivergence from a simple uniform distribution in order to cope with overestimation of Q-values ( Fox et al. , 2016 ) .KL-control has also been used to improve transfer learning between maximum likelihood estimation ( MLE ) training on data , and training with RL ( Jaques et al. , 2017 ) . To the best of our knowledge , our work is the first to propose penalizing KL-divergence from a learned prior model of the state-action space as a way to improve offline batch RL . Other strategies to improve off-policy learning have been proposed , but differ from this work in key respects . Many focus on scenarios where the policy is able to explore and collect more data ( e.g . Degris et al . ( 2012 ) ; Riedmiller ( 2005 ) ) ; for example , when learning online from an outdated replay buffer ( e.g . Munos et al . ( 2016 ) ) . In contrast , we learn entirely offline , from a fixed batch of data , without the ability to explore . Methods proposed for this setting have often not been used in conjunction with modern function approximation techniques ( e.g . Thomas et al . ( 2015 ) ) . Many other works focus on off-policy policy evaluation ( rather than policy learning ) , for example using importance sampling or model estimation ( e.g . Farajtabar et al . ( 2018 ) ; Jiang & Li ( 2016 ) ; Precup ( 2000 ) ; Thomas & Brunskill ( 2016 ) ) . In the deep BRL setting , Liu et al . ( 2019 ) have proposed a correction to policy gradients , Gelada & Bellemare ( 2019 ) have proposed covariance-shift methods , and Bhatt et al . ( 2019 ) have proposed normalized feature representations.Kumar et al . ( 2019 ) use maximum mean discrepancy to cope with extrapolation error in BRL , while Agarwal et al . ( 2019 ) use a Random Ensemble Mixture ( REM ) Q-network . Most similar to our work is Batch Constrained Q-learning ( BCQ ) ( Fujimoto et al. , 2018b ) , which tackles off-policy deep BRL in continuous action domains by training a generative model of the batch , p ( a|s ) , sampling from this model , and selecting the best action based on aQ-estimate . Unlike our approach , this does not integrate information about the distribution p ( a|s ) directly into the policy , or allow the model to learn when to strategically deviate from the prior in order to obtain more reward . We propose using dropout to approximate model uncertainty of the target Q-network . The idea of using dropout to estimate uncertainty in neural networks was proposed by Gal & Ghahramani ( 2016 ) . Different forms of uncertainty estimates have been used in RL ( e.g . Kahn et al . ( 2017 ) ; Osband et al . ( 2016 ) ) ; for example , Bayesian uncertainty estimates have been proposed as an alternative to double DQN ( Azizzadenesheli et al. , 2018 ) . 2.1 RL FOR LANGUAGE GENERATION . Improving dialog systems with RL has largely been restricted to task-oriented dialog systems , which have a limited number of task-specific actions ( e.g . Fatemi et al . ( 2016 ) ; Gašić et al . ( 2011 ) ; Liu & Lane ( 2017 ) ; Liu et al . ( 2018 ) ; Su et al . ( 2017 ) ) . These approaches may incorporate human input , usually through explicit , manual feedback ( e.g . Shah et al . ( 2018 ) ) , but sometimes with more implicit signals , such as the user interrupting the system or starting over ( Shi & Yu , 2018 ) . Efforts to expand RL to the open-domain dialog setting , such as those of Li et al . ( 2016b ; 2017 ; 2018 ) , are less numerous , and do not involve learning from human feedback . Even in the open-domain setting , authors may choose to use a highly restricted action space ; for example , using RL to choose which scripted or MLE dialog model to invoke to answer a user ’ s query ( Serban et al. , 2017a ) . Since the posting of the preprint of this paper , Ziegler et al . ( 2019 ) have used explicit human feedback to improve the summarization and text continuation performance of a large-scale language model . Although they do not study dialog or the batch RL setting ( instead learning online from a trained model of human feedback ) , they do make use of our proposal to penalize KL-divergence from a pre-trained language model , and find that this is important to achieving good performance . Although implicit signals such as sentiment ( Hancock et al. , 2019 ) and conversation length ( Zhou et al. , 2018 ) have been used in MLE systems , the idea of using such signals as a reward for RL is relatively unexplored . Shin and colleagues uses on-policy learning in conjunction with a usersentiment approximator to improve a seq2seq model ( Shin et al. , 2019 ) , but are unable to learn directly from user feedback . To the best of our knowledge , we are the first to use batch RL to train open-domain dialog models on implicit cues gained from real human interactions .

Summary: The paper introduces an off-policy batch deep reinforcement learning method for learning human preferences in dialog generation. The key technical contribution is by controlling the KL divergence of the learned policy with a prior policy that was learned on other dialogs. The method is able to constrain the policy not deviate too much from the prior to avoid extrapolation error. Another technical contribution is the use of dropout uncertainty estimates to estimate the Q values, which is more scalable compared to double Q-learning. The authors also incorporate intermediate implicit rewards in dialogs to encourage more positive conversations.