Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog
Authors: Natasha Jaques
Asma Ghandeharioun
Judy Hanwen Shen
Craig Ferguson
Agata Lapedriza
Noah Jones
Shixiang Gu
Rosalind Picard
Venue:  (2020)
Keywords: batch reinforcement learning
deep learning
dialog
off-policy
human preferences

Abstract: Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms which use KL-control to penalize divergence from a pre-trained prior model of probable actions. This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space. WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these. We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.


Under review as a conference paper at ICLR 2020
WAY OFF-POLICY BATCH DEEP REINFORCEMENT LEARNING OF HUMAN PREFERENCES IN DIALOG
Anonymous authors Paper under double-blind review
ABSTRACT
Most deep reinforcement learning (RL) systems are not able to learn effectively from off ofllicy data, especially they cannot explore online in the environment. This is critical shortcoming for applying RL to real-world problems where col- lecting data expensive, and models must be tested offline before being deployed to interact with the environment e.g. systems that learn from human interaction. Thus, we develop novel class of off-policy batch RL algorithms which use KL- control to penalize divergence from pre-trained prior model of probable actions. This KL-constraint reduces extrapolation error, enabling effective offline learn- ing, without exploration, from fixed batch of data. We also use dropout- coopout-based uncertainty estimates to lower bound the target Q-values as a more efficient alter- native to Double Q-Learning. This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAl Gym, and on the problem of open-domain dialog generation; challenging reinforcement learning problem with a 20,000 dimensional action space. WOP allows for the extraction of multiple different re- ward functions post-hoe from collected human interaction data, and can learn ef- fectively from all of these. We test real-world generalization by deploying dialog models live to converse with humans an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.
INTRODUCTION
In order to scale deep reinforcement learning (RI) to safety-critical, real-world domains, two abil- ities are needed. First, since collecting real-world interaction data can be expensive and time- consuming, algorithms must be able to learn from off-policy data no matter how it was generated, or how little correlation between the data distribution and the current policy. Second, it is often necessary to carefully test a policy before deploying it to the real world; for example, to ensure its behavior is safe and appropriate for humans. Thus, the algorithm must be able to learn offline first, from a static batch of data, without the ability to explore.
This off-policy batch reinforcement learning (BRL) setting represents challenging RL problem. Most deep RL. algorithms fail to learn from data that is not heavily correlated with the current policy (Fujimoto et al. 2018b). Even models based on off-policy algorithms like Q-learning fail to learn in the offline, batch setting, when the model is not able to explore. If the batch data is not sufficient to cover the state-action space, BRL models can suffer from extrapolation error, learning unrealistic value estimates of state-action pairs not contained in the batch (Fujimoto et al., 2018b). It can be impossible correct for extrapolation error when there mismatch in the distribution of state- actions pairs in the batch data, and the distribution induced by the learned policy. For example, if the policy learns to select actions which are not contained the batch, it cannot learn a reasonable value function for those actions. Figure [ illustrates this concept, where the batch only covers subset of possible policies. Extrapolation error is particularly problematic in dige--imensional state and action spaces (such as those inherent in language generation).
We propose to resolve these issues by leveraging pre-trained generative model of the state-action space, p(a)s s). trained known sequences of interaction data. While training with RL, we penalize divergence from this prior model with different forms of KL-control. This technique ensures that the RL model learns a policy that stays close the state-action distribution of the batch, combating

---

Under review as a conference paper at ICLR 2020
Pe (s)
KL-constraint
?
9 cn
Behavior
Optimal
In this example batch RL problem, the robot's goal is travel the minimum distance around the black walls to get to the red flag. A trained behavior policy generated the batch data; the probability of each of the states appearing in the batch, PD (s), is in yellow (white locations are not contained in the batch). If the offline RL policy estimates the value of going up or left from the start position is high, it will have no way to refine this estimate using the batch data, or learn good policy in this region of state space. The KL-constraint ensures that the RL policy will stay within the support of the batch data. However, the behavior policy is suboptimal, so using behavior cloning to directly imitate the batch data will result in suboptimal return. Instead, the KL-constrained model can learn to find the optimal policy, which is within the support of the batch.
Figure 1:
extrapolation error. We also propose using dropout to obtain uncertainty estimates of the target Q- values, and use this lower bound to alleviate overestimation bias. We henchmark against a discrete adaptation of Batch Constrained Q-learning (BCQ) (Fujimoto et al.] 2018b), recently proposed taae-oof-the-ar BRL algorithm for continuous domains, and show that our Way Off-Policy algorithm achieves superior performance in both traditional RL domain, well as in challenging, under- explored, real-worke reinforcement learning problem: using implicitly expressed human reactions in chat to improve open-domain dialog systems.
When a machine learning system interacts with humans, ideally we would like to learn about the humans' preferences in order improve its performance. Yet having humans manually indicate their preferences through explicit means like pressing a button (e.g. [Christiano et al.) (2017) or submitting a feedback report, does scale. Instead, we would like to be able to use humans' implicit reactions, such as the sentiment they express, or the length of the conversation, in order to improve the policy. However, applying off-policy batch RL to language generation challenging because the number of potential combinations of words and sentences leads to combinatorial explosion in the size of the state space. The action space the set of frequent vocabulary words the English language is 20.000-dimeesionall This compounds extrapolation error, making BRL even more difficult. However, when learning from human interactions the wild, crucial to be able to learn offline and test the policy before deploying it, lest it learn inappropriate behaviors (c.g. Horton (016)).
To support this work, we developed an interactive online platform that allows humans to chat with deep neural network dialog models running on GPU; the BRL models trained for this study are available live at https: /ttp:::://nuurll. /nnuuallll.. chat/rhl sponses set of over 40 different dialog models Through this platform we collected human re- the course of several months. Using our Way Off-Policy algorithm, we are able to effectively learn from this batch of data, in spite of the fact that it was generated with a vastly different set of model architectures, which were trained different datasets. Further, we use the batch to learn from many different reward functions designed post-hoc extract implicit human preferences, something that is only possible with effective off-policy BRL.
In summary, the contributions of this paper are:
A novel algorithm, Way Off-Policy learning, which is the first to propose using KL-control from a pre-trained prior model way reduce extrapolation error in batch RL.
Experiments showing the effectiveness of WOP above strong baselines based prior work (e.g. Fujimoto et al. (2018b)), on both traditional RL tasks and on the challenging problem of open-domain dialog generation.
A set of novel conversation rewards based how human preferences implicitly pressed in text. We are the first work to learn from implicit signals in conversation offline using batch RI.
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
The approach
we propose is based KL-control, branch of stochastic optimal control
(SOC) [Stengel 1986) where the Kullback-Leibler (KL) divergence from some distribution is used
to regularize RL policy (e.g. (Abdolmaleki al. 2018 Kappen et al. 2012 Rawlik et al.)
2012
Todorov| 2007 ). Well-known examples include Trust Region Policy Optimization (TRPO)
(Schulman al. 2015). and use conservative, KL-regularized policy updates to restrict the RL al- Peters et al.) Rawlik et al., 2012). ). KL-control can also be applied to entropy maximization (e-g. (Ziebart 2008 Nachum et al. 2017] Haarnoja et al. (017]); for example, G-learning penalizes KL- et al.
gorithm to stay close to its own prior policy (e.g. (Haarnoja et al., 2018
2010
divergence from a simple uniform distribution in order to cope with overestimation of Q-values [Fox et al. 2016). KL-control has also been used to improve transfer learning between maximum likelihood estimation (MLE) training on data, and training with RL. (Jaques et al., (017). To the best of our knowledge, our work is the first to propose penalizing KL-divergence from a learned prior model of the state-action space as a way to improve offline batch RL.
Other strategies to improve off-policy learning have been proposed, but differ from this work in key respects. Many focus on scenarios where the policy is able to explore and collect more data (e.g. Deerris et al.] (2012); Riedmiller (2005)): for example, when learning online from an outdated replay buffer (e.g. Munos et al. (2016).. In contrast, we learn entirely offline, from a fixed batch of data, without the ability to explore. Methods proposed for this setting have often not been used
in conjunction with modern function approximation techniques (e.g. Thomas et al. (2015). Many other works focus on off-policy policy evaluation (rather than policy learning). for example using importance sampling or model estimation (e.g. Farajtabar et al. (2018); Jiang & Li (2016); Precup (0000; TThoms & Brunskill (2016)) In the deep BRL setting, Liu et al. (2019) have proposed a (Tooma correction to policy gradients, Gelada & Bellemare (2019) have proposed covariance-shift methods, and Bhatt et al. (2019) have proposed normalized feature representations Kumar et al. (2019) use maximum mean discrepancy to cope with extrapolation error in BRL, while Agarwal et al. (2019) use a Random Ensemble Mixture (REM) Q Q-network. Most similar to our work is Batch Constrained Q-learning (BCQ) (Fujimoto al. 2018b), which tackles off off-policy deep BRL in continuous action domains by training a generative model of the batch, p(a(s), sampling from this model, and selecting the best action based on a Qeettmmate. Unlike our approach, this does not integrate information about the distribution p(a|s) directly into the policy, or allow the model to learn when to strategically deviate from the prior in order obtain more reward.
We propose using dropout to approximate model uncertainty of the target Q- Q-network. The idea of using dropout estimate uncertainty in neural networks was proposed by Gal & Ghahramani (2016). Different forms of uncertainty estimates have been used in RL (e.g. Kahn et al. (2017); Os- band al. (2016)); for example, Bayesian uncertainty estimates have been proposed as an alternative to double DQN (Azizzadenesheli et al. 2018).
2.1 RL FOR LANGUAGE GENERATION
Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of aask-specific actions (e.g. Fatemi et al. (2016); Gašić et al. (2011); Liu & Laane (011)); Liu al. (2018); ; SS et al. (2017). These approaches may incorporate human input, usually through explicit, manual feedback (e.g. Shah et al. (2018)) but sometimes with more implicit signals, such as the user interrupting the system or starting over (Shi & Yu! 2018). Efforts to expand RL. to the open-domain dialog setting, such as those of [ et al. (2016b) 2017]2018) are less numerous, and do not involve learning from human feedback. Even the open-domain setting, authors may choose to scripted or MLE dialog model to invoke to answer a user's 's query (Serban et al., 2017a).
highly restricted action space; for example, using RL to choose which
Since the posting of the preprint of this paper, Ziegler et al. (019)) have used explicit human feed- back to improve the summarization and text continuation performance of a large-scale language model. Although they do not study dialog or the batch RL setting (instead learning online from trained model of human feedback), they do make use of our proposal to penalize KL-divergence from pre-trained language model, and find that this is important to achieving good performance.
3

---

Under review as a conference paper at ICLR 2020
Although implicit signals such sentiment (Hancock et al., 2019) and conversation length (Zhou et al. 2018 have been used in MLE systems, the idea using such signals as a reward for RL relatively unexplored. Shin and colleagues uses on-policy learning in conjunction with user- sentiment approximator to improve a seq2seq model (Shin et al., 2019), but are unable to learn directly from user feedback. To the best of our knowledge, open-domain dialog models on implicit cues gained from real human interactions.
the first to use batch RL train
3 METHODS
We employ typical RL notation in which 8, represents the environment state at time t, the agent takes action a according its policy T(de(s)), and receives reward r(s,, az). The agent's goal is to maximize reward over an episode trajectory т, with a discount factor of applied to future rewards. Q-learning learns an action-value estimate of the total expected discounted future reward, Qx (ar, 8c) = En 12 7t "r(se, ar)) through iterative updates based on the Bellman equation:
Qa, (0t, ax) r(t,, ax) + P (plle) [max Qa (S141, (141] ax+1
In deep Q-learning MMiih et al. 2013), Q-network approximates Qo, (St, at) and drives the policy т. A second target Q-network approximates the expected reward from the next state, Qo (441,0+411 (Van Hasselt et al. 2016).
3.1 BATCH RL AND EXTRAPOLATION ERROR
In batch RL, we are given a fixed batch of data B, and assume that no further interaction with the environment possible. To train Qv., we sample (st, Oe, I',, 8141) ~ B, and update the weights of the Q-network to approximate Eq. [T] Because Q-learning is an off-policy algorithm, in principle it should be able to learn from data collected by any behavior policy. However, extrapolation error can occur if the BRL policy learns to favour a state-action pair (s, a) that is unlikely, or not contained, in the batch data. In this case, the estimate Q(s', x (s')) can be arbitrarily bad (Fujimoto et al.) 2018b). Such errors can then accumulate through the Bellman backup operator (Kumar et al.][2019) Experiments from Fujimoto et al. (2018b) show that extrapolation error can be highly detrimental to learning off-policy in BRL.
These problems are compounded by the fact that algorithms based on the Bellman operator are inherently optimistic in the face of uncertainty. When value estimates for some region of the state- action space are noisy (because too few experience samples have been used to refine them), the maximum operation in Eq. [ will lead to an overestimation of expected future reward. In normal RL setting, this overestimation bias drives the model to explore areas of the state-action space for which the value estimates have the highest variance, thus enabling it refine them; in essence, creating built-in drive to explore. However, in a batch setting where exploration is not possible, the model is instead driven to value parts of the state-action space for which it has little to no data to leam good policy (see Figure]) D.
3.2 DROPOUT FOR UNCERTAINTY ESTIMATION OF TARGET Q-VALUES
The overestimation of Q-values in the BRL setting necessitates other methods for estimating the future reward via the Target Q-network. Clipped Double Q-learning (Fujimoto et al. 2018a) main- tains two independent pairs of Q-networks, and takes the minimum of their estimates of future reward. This approach is computationally expensive and memory intensive. Instead, we leverage the fact that a network trained with dropout can be used to approximate a Bayesian uncertainty estimate of the output value (Gal & Ghahramani (016). Given the target Q-network Qay we com- pute Q (((++1,.1p $141) using a Monte Carlo (MC) estimate of the lower-bound of (0 by running M stochastic forward passes of the network, each with new dropout mask di W:
Q(a++1, 11+ min [Quz (ar+1, 8141; d4)]
i=1...
Using the minimum operator penalizes high variance estimates and leads the algorithm to be pes- simistic in the face of uncertainty, rather than optimistic. Such a bias will push the model to favour actions that lead states well covered by the batch data.
4

---

Under review as a conference paper at ICLR 2020
3.3 DISCRETE BATCH CONSTRAINED Q
Batch Constrained Q-learning (BCQ) (Fujimoto et al. 2018b) proposes to address the BRL problem by constraining the actions of the Q- Q-network to be close the data contained within the batch. This is accomplished by learning a generative model of the batch, G p(a[s), and sampling from this model during learning and inference. Because BCQ is designed for continuous action domains, applics a learned perturbation model 5 {s, a; ②) which is allowed to alter the action within the range [-0, $). BCQ learns -esstmmates that incorporate the perturbation model, Qo(s, a [(s, a; +)). To act, n possible actions are sampled from the generative model, {a: Gw)(s) ]=)) perturbed, and the action with the maximum Q-value is selected, giving the BCQ policy:
BB) = arg max Qa a; + E(8, az; ())
(3)
a, a,+,(e,0.;))
We propose an adaptation of BCQ to discrete action spaces (DBCQ) which does not use a continuous perturbation model. Since BCQ relies on Double Clipped Q-learning (Fujimoto al., (018a). here we use dropout-basse based uncertainty estimates as in Eq. 2 Thus the DBCQ policy is:
TDBCQ (s)
arg max Q8, (s,
(4)
aum=p(als)
3.4 KL-CONTROL FROM PRE-TRAINED PRIOR
Rather than simply sample from the prior, we would like the Q-learning algorithm to directly incor- porate the prior into the policy. Thus, we use KL-control to penalize divergence between the learned prior p(a(s), and the Q-network policy Te. while still maximizing reward. Given a trajectory of ac- tions, 7 = {ai,, 02 a2,-....-111 1.. let ((T) = I[=_ (o 8t) be the policy of our Q-learning algorithm at the trajectory level. Similarly, let p(7) - II p(a,]s) (ss) be the prior distribution over the trajectory, and r(T) be the return. We seek to maximize the following KL-regularized objective:
[(q)= (B
(5)
Since DKL [p] = En (Iog ((x) - log p(z)), we can see that this is equivalent to maximizing pariglle) the following expected value function of the policy To at the action level:
T
(" = Ex r(sv, ar) )/c + logp(arlss (se) - log < [se|st)]
(6)
The two terms introduced in Eq. 6 have clear motivations. The p(a|s) term rewards the model for choosing actions that have high probability under the prior, biasing the model to state-action pairs that are likely to be in the batch. The log T (a is) term analogous to entropy regularization. Maintaining diversity in the action space through entropy regularization is important for generative models like dialog systems, which are known to collapse to an uninteresting, small number re- peated samples (Li et al. 2016a). Re-stating Eq. 6 entropy regullazeee Q-function, we obtain:
T
Q(5:, 0.) = E I> r(se, av) c log p(a (w)) + H(-1ss
One can derive soft version of the entropy- regulaieee Q-function that uses Boltzmann distri- bution to estimate future reward (Haarnoja et al. (2077). We refer to it as a V U-function following previous work (Jaques et al. 2017). which derived this function as a generalization of the V-leaaning proposed by (Rawlik et al. 2012). The optimal V-function and policy are:
V (st. a)) rrre,,(s//ee) ar)/c p(gr((asss + y log Σ (xp((""ss'aa')
(8)
(arls)) exp( (st, az))
(9)
Because it avoids taking a hard max over noisy estimates, V-learning leads to less overestimation of future reward (Abdolmaleki et al. 2018: Haarnoja et al. 2017). This improves learning through more stable temporal difporaldifference (TD) updates. Thus, may be especially useful in the BRL setting for reducing optimism in the face of uncertainty. The Way Off-Policy (WOP) algorithm combines Monte Carlo (MC) target estimation, V-learning, and KL--oottool from pre-trained prior.
5

---

Under review as a conference paper at ICLR 2020
500
Trinioa sieo
Wh 50ep
102 113 5000
Traning 1010
(a) Full buffer
(b) Concurrent
(c) Expert demonstrator
(d) Noisy demonstrator
Figure 2: Comparison of batch RI. algorithms for different offline learning conditions. WOP con- sistently exceeds the performance of Batch Q-earniig. Behavioral Cloning (BC), DBCQ. and the Behavior policy used to generate the batch data. Error bars show 95% CI of the mean over 50 trials.
4 TRADITIONAL RL EXPERIMENTS
To demonstrate the effectiveness of these techniques, conduct series of experiments in tra- ditional RL tasks using the OpenAl gym (Brockman et al., 2016). Here we show results for the CartPole-vO environment; learning Behavior policy, and store all (s, a, r, s') experience samples into replay buffer. We use this buffer to train prior model of p(als) using a Variational Auto-encoder (VAE) (details in Ap- pendix). This model is used as part of both the DBCQ and WOP algorithms. We can use the prior for imitation learning, by sampling actions directly from p(als) to obtain Behavioral Cloning (BC). We benchmark all of these techniques against vanilla Q-learning on the batch data (Batch Q).
results
available in the Appendix. We first train online Q-
We experiment with four different conditions which vary the quality of the Behavior policy and the replay buffer data: a) Full buffer: all experience samples experienced during online training are used for offline learning; b) Concurrent: the offline learning algorithms see a sliding window of experience samples in the same order that the online learner experienced them; c) Expert demon- strator: the buffer only contains experience generated by a fully trained online learner; and d) Noisy demonstrator: the online learner has high probability of acting randomly (c 0.3) and is thus bad model of the optimal policy.
Figure 2 shows the results. Across conditions, imitation learning (BC), DBCQ, and the original behavior policy. As expected, Imitation learning (BC) underperforms other techniques when the batch contains noisy inexpert experience samples. However, when the batch contains only expert trajectories, Batch Q fails to learn, because the batch does not cover the full state-action space well, increasing extrapolation error (as illustrated in Figure D. DBCQ matches outperforms BC and Batch Q in all scenarios. However, because DBCQ acts by sampling from p(ass) as learned by the BC model, its performance suffers when the batch data is noisy or imperfect. In contrast, WOP is able to learn to trade-off staying close to the prior and obtaining higher reward, and consistently outperforms all other algorithms in this environment.
that WOP is able outperform Batch Q,
5 BATCH RL FOR LEARNING DIALOG FROM HUMAN FEEDBACK
Here, tackle the problem of training an open-domain dialog model from human feedback. We consider human interaction to represent the 'environment'. The response of a human to the bot's utterance is used compute reward signal to train the model. The state is the conversation history, composed of series of conversation turns or utterances, UI...f where each utterance is composed of vocabulary tokens. The model attempts construct a response utterance uf+ [a1, .2...., a,) by iteratively choosing an action as the next token. Applying RL to dialog generation is challenging due to the large state-action space. The number of tokens the vocabulary our pre-trained model is 20,000, making the action space very high-dimensional;; this further compounds the problem of extrapolation error.
We trained over 40 dialog models with different architectures (e.g. [erban et al. (2077))) on dif- ferent datasets, generating models that varied significantly terms of the distribution language they learned. We deployed these models to users via web server that hosts neural network dialog models on GPU for fast, real-time inference: https: pettps:/nneuraal chat The code for the models and the server is available in open-source <redacted>. Using the server, collected batch

---

Under review as a conference paper at ICLR 2020
of human interaction data containing 14232 pairs of user input and agent response. Because learn- ing language online from humans on the internet can result in inappropriate behavior (see Horton 2016)), ) learning offline using BRL is imperative.
The batch data was used to train the RL models as described in Section Here, we use a pre-trained language model to estimate p(als). We also initialize the weights of the Q-network and target Q-network are from the pre-trained model, to combat extrapolation error. The trained RL models were then re-deployed to the web. We recruited 90 Mechanical Turk workers provide total of 718 7-point Likert scale ratings of the bots' quality, fluency, diversity, contingency (relatedness), and empathy, after interacting with each bot for least 3 turns. Participants also had the option to provide explicit feedback through upvoting or downvoting a particular utterance within the interface. We sum these manual votes to create an overall votes score. We note that using this platform to test our models "in the wild" with humans represents more meaningful test of generalization than testing an RI. model in the same limited (game) environment in which it was trained, since humans are not restricted in the text they can type as input to the model.
5.1 LEARNING FROM IMPLICIT HUMAN PREFERENCES
We seek to improve a dialog model's ability to engage in natural conversation with human by learning from the signals implicit in the way that the human responds. Rather than having the human manually label good performance which we show in this work does not scale the agent should recognize informative cues within the user's responses, like sentiment, and the amount of time they spend chatting. Essentially, want to create agent that is intrinsically motivated to produce positive reactions in its human conversation partner. To this end, we reward the model for: 1) eliciting positive sentiment, 2) eliciting longer conversations and more words typed (a sign of engagement). 3) eliciting laughter (in the form of typed "ha's), 4) high semantic similarity between the human input and bot response, and 5) asking questions, since this is an important active listening skill (Bodic et al. 2012). The total reward given to the agent is a combination of these, with details (and coefficients) in the Appendix. Note that the first types of rewards depend on eliciting positive responses from a human user; we call these the implicit human reward. The 5th reward is easily exploitable by the agent itself. These rewards were designed and extracted post-hoe from the batch of human data, and thus learning from them is only possible with effective batch RL, since they had no effect on the policies used to generate the batch.
DIALOG RESULTS
Table 1: Interactive human evaluation of batch RL. techniques. KL-control models strongly outper- form other techniques. Ratings are Likert scale, votes and human reward are z-scores.
Human
Model type
Quality
Fluent
Diverse
Related
Empathy
Total
Votes
reward
DBCQ
1.64 1.29
1.87 1.34
3.13 1.58
1.84 1.34
2.09 1.38
10.58 1
-2228
-050
Batch
1.87 1.30
2.36
4.42
1.41
1911 1.32
2.58
1.47
11.91
1158
-163
-005
Batch Q MC
1.85 4.39
2.46
2.44
2,46
#.52
1.98 +.39
2.34
4.49
11.07
1182
-068
.005
KL--contol Q
2.38 +.39
3.24
+.47
3.42
+.54
2.38 +.45
2.56
+.43
13.98
+181
.016
.004
KT.-contool V (WOP)
2.33 +.41
+.53
2.82
+.50
2.31 +.44
3.47
+.50
14.67
+182
.128
.061
Table Purely reward-maximizing methods like Batch Q (left) diverge away from realistic language (saying phrases like "where did you say to me?") in order to trivially exploit the reward function by asking a question every turn, and using the maximum number of tokens in every sentence. In contrast, KL-control methods (right) output plausible language by staying close to the prior, but shift to using polite, cheerful language to maximize implicit human reward.
[User]:: hey, what's up?
[Batch Q]: where did you ?? that sounds so exciting, how do work for you...
[User]: hey, what's up?
you to
me?... you
[KL-contro]]: i'm sure, but glad see you,
[User]: yes, want to talk to you! me?... you want to talk you about you...
i'm so happy that you' re not alone
[Batch Q] where did you
[User]: oh thanks, that's very nice of you'
you good for you... you are so good...
[KL-contro]]: i'm sorry,
i'm glad [Uer]]: doe't be sorry, you did a good job.
you think.
[User]: thankss, appreciate that.
[Batch Q]: where did you get??, that sounds so exciting. how do work for you...
[KI-control): that's really nice to me.
you you...
7

---

Under review as a conference paper at ICLR 2020
To compare models, we not only look human users' ratings and votes, but also consider the automatic signals detectable from the text itself. This implicit human reward metric aggregates the measures listed in items 1-4 in Section 5.I and measures the ability to elicit positive responses from a human. Table [ shows the results of the human evaluation, comparing WOP to ablations of itself, Batch Q. and DBCQ. MC Target Q estimation leads modest improvements in votes and human reward, but does not improve ratings. Using V-learning improves all three. However, the most notable difference in performance comes from KL-control. The KL-control models show substantial gains over the baseline models across both ratings and human reward. We perform a one- way analysis of variance (ANOVA) comparing the KL-control models to the Batch Q baselines and DBCQ on the total human rating score, and find that the KL-control models are significantly better, F(x) = 4.781, p < .05. This validates the hypothesis that KL-control with strong, pre-trained prior can be used to improve batch RI.
As shown in Figure [] without KL-regularizztton, the bascline RL models diverge quickly and continuously from the prior, losing information about realistic se- quences. This helps explain the poor performance of DBCQ in Table [T] The underlying Q-network in DBCQ does not directly integrate the prior. As Q-learning causes the model to diverge from the prior, the Q-estimates of language generated according to the prior become unreal- istic, and Eq. 4 selects unrealistic actions. This results in highly 'diverse" (random) generated utterances. Although DBCQ performed well in simple domains in Section 4 it does not scale effectively to dialog.
O
0.0
A000250
8000075
Training stop
Figure 3: KL-divergence of the policy from the prior is lower with KL-control throughout training. Bands show σ.
The pre-trained prior may be especially important in a generative domain like dialog, where the true re- ward function is unknown, and so purely maximiz- ing a heuristic reward may lead to lower quality con- versations. Table[sshows shows examples of conversations with a Batch Q and KL-controol model. Because the Batch Q model has no incentive to stay close to real- istic language, it learns to exploit the reward by ask- ing a question and outputting the maximum number of tokens (30) every utterance. These sentences con- tain implausible phrases that do not represent realis- tic language (c.g. "wheee did you say to me?"). In contrast, the KL-control model uses fluent language, but shifts its distribution towards cheerful and polite speech, presumably because this is what led to posi- tive human responses in the batch data.
Upvote Implict reward
125
Conversation
Semantic similanty
User leughter (ha)
Words elitted
Sentime transttoon
0.5
7 Broand
Sentiment
Polite
Supportive
Cheerful
0.5
-025
-050
oRco
Batch
Datth MC.
aL contzul O
RL control
Model
In fact, we noticed that all models trained with the implicit human rewards described in Section 5.1 learned to use more cheerful and supportive lan- guage. Therefore, we create post-hoc metrics to measure this effect (see the Appendix for details). Figure how these metrics, well the im- plicit rewards, differ across models. Without KL- control, baseline methods like Batch Q exploit sim- ple rewards like asking questions at the expense of realistic language, explaining their poor quality rat- ings. In contrast, KL-control models learn to rely more on realistic but polite, supportive, and cheerful dialog to elicit higher total human reward.
Figure 4: Z-scored reward. Red metrics were used training rewards, green are post- hoc. Traditional RL methods like Batch Q exploit simple action-based rewards, like asking questions. In contrast, KL-control methods shift their distribution towards po- lite, supportive, and cheerful conversation, allowing them to elicit higher human reward (blue).
Table presents the results of WOP models trained with only a single reward function, ordered from lowest to highest quality. Notably, extracting multiple different reward functions post-hoc from a batch of data and training on these independently is only possible with effective BRL. Investigating which rewards presented are most critical to achieving high-quality conversations with humans, note that maximizing positive and minimizing negative sentiment in the user turns out to lead
8

---

Under review as a conference paper at ICLR 2020
Table 3: Interactive human evaluation different reward functions (models trained with WOP).
Reward
Human
Quality
Fluent
Diverse
Related
Empathy
Total
Votes
function
reward
Cony. len.
2.20 +.40
3.61
+.53
3.02 =.52
2.25 = 46
2.48 = 4.45
13.57
#1.84
-003
-003
Semantic sim.
1.93 +.34
3.50
+.45
2.37 =.45
2.11 4.45
2.52 4.48
12.43
#175
-020
.012
User laughter
1.96 1.38
3.56
1.48
2.33 1.51
1,93
1.42
3.20
1.55
12.98
11.60
-14
-003
Words elicited
2.11 4.32
3.96
4.44
3.04
#.45
2.04
1.35
2.55
1.46
13.70
11.44
.059
.024
Manual votes
2.14 +.38
3.47
+.45
2.91
+.47
2.07
+.39
2.42
+.46
13.00
+1.6
-030
.010
Sent. trans.
2.02 +.31
3.71
1:49
2.98
1.0
2.04
4:42
2.84
1.48
13.60
1:663
.031
.014
Question
2.29 1.37
4.31
1.50
3.31
1.52
2.20
1.40
2.60
1:41
14.71
1:163
.057
.012
Sentiment
2.47
+.32
4.05
+.45
3.23
+.46
2.42
+.39
3.23
+.55
15.40
+1.49
.085
.045
to the highest quality bot. This underscores the importance of affective signals as cues for good conversation. Bots trained the manual upvotes and downvotes provided by users the utterance level fail to achieve similarly high performance. Even though users were instructed to make use of the vote feature, the task burdensome, and users did vote frequently enough to provide good training signal. This validates the hypothesis that implicit signals of human enjoyment (such as sentiment) are a more scalable way to learn from human preferences.
7 CONCLUSION
This paper presents the Way Off-Policy (WOP) algorithm, which improves performance when learn- ing off-policy without the possibility to explore - i.e. batch RI. (BRL). We are the first to propose using KL-control from strong prior model pre-trained on data way to avoid extrapolation and instability BRL. Our results on traditional RL tasks demonstrate that our WOP algorithm provides performance improvements generation show that KL-control is critical to achieving good performance in this real-world, high- dimensional setting. In generative domain such dialog, the true reward function is not known, and trivially exploiting the rewards can actually lead to worse performance. Thus, KL-control may be particularly necessary to ensure samples remain realistic and close the data distribution. We propose several reward functions that could allow open-domain dialog generation model to learn from rich cues implicit in human interaction, where learning from expressed sentiment was most promising. We find that maximizing implicit rewards leads to better performance than relying on explicit feedback. We hope that the techniques presented here can improve learning with RI. from offline data, making it easier to apply RL to safety-critical settings such as human interaction.
ofte--of-the-arl BRL techniques, and the results in dialog
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint 1rXiv:1806.063 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity off-policy deep reinforcement learning. arXiv preprint arXiv: arXiv:1907.0455 /907.04547, 2019.
Kamyar Avizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep g-networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Aditya Bhatt, Max Argus, Artemij Amiranashvili, and Thomas Brox. Crossnorm: Normalization for off-policy td reinforcement learning. arXiv preprint 1rXiv:190..05605, 2019.
Graham D Bodie, Kellie St. Cyr, Michelle Pence, Michael Rold, and James Honeycutt. Listening competence in initial interactions Distinguishing between what listening is and what listeners do. International Journal of Listening, 26(1):1-28) 2012.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

---

Under review as a conference paper at ICLR 2020
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 2299-4307, 2017.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Cristian Danescu- Nieullscu-Miiii and Lillian Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pp. 76-87. Association for Computational Linguistics, 2011.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 179-186. Omnipress, 2012.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pp. 1446-1455, 2018.
Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. Policy networks with two-stage training for dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 101-110, 2016.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sar- casm. In 2017 Conference on Empirical Methods in Natural Language ProcessingConference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Shirty-Second Conference on Uncertainty in Artificial Intelligence, pp. 202-211, AUAI Press, 2016.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1582-1591, 20.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 2018b.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning, In international conference on machine learming, pp. 1050-1059, 2016.
Milica Gašić, Filip Juröfček, Blaise Thomson, Kai Yu, and Steve Young. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In 20/ IEEE Workshop on Automatic Speech Recognition & Understanding, pp. 312-317. IEEE, 2011.
Carles Gelada and Marc G Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. arXiv preprint arXiv:190I.0%4 2019.
Asma Ghandeharioun, Judy Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. Approximating interactive human evaluation with self-play for open-domain dialog systems. arXiv preprint arXiv: 09300,308, 2019.
Tuomas Haarojja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1352-1361. 115..3R...M org, 2017.
Tuomas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1856-1865, 2018.
10

---

Under review as a conference paper at ICLR 2020
Braden Hancock, Antoine Bordes, dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05411 2019.
Mazare, and Jason Weston. Learning from
Jennifer Hay. Functions of humor in the conversations of men and women. Journal of pragmatics, 32(6):709-742, 2000.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv
preprint arXiv:1503.0255 02.531,.. 2015.
Helena Horton.
Microsoft deletes
'teen
girl' ai after became a hitler- URL
loving
robot within 24 hours.
In Telegraph UK, 2016.
https:// htt::://ww..tee. http://wwwweegga.....o.. co wtt..//teeeeeecccclg//////////// lovinnr-lovig sex- robot wit/
microsofts-teen-ggie ottp::ennnaaiaaaaaattttooaaaaaaa
Molly E Ireland, Richard B Slatcher, Paul Eastwick, Lauren E Scissors, Finkel, and James W Pennebaker. Language style matching predicts relationship initiation and stability, Psychological science, 22(1):39 44, 2011.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato. Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- els with kl-control. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1645-1654. JMLR. org, 2017.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 552-661, 2016.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty- aware reinforcement learning for collision avoidance. arXiv preprint 17Xiv:1702.01 2017.
Sham M Kakade. natural policy gradient. In Advances neural information processing systems (NIPS), volume 14, pp. 1531-1538, 2002.
Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model infer- ence problem. Machine learning, 172211199-182, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14126694 2014.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 19906000949 2019.
Jiwei Li, Alexander H Miller, Sumit Chopra, Mare' Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in- -n-tt---otoo-...... arXiv preprint arXiv:1611.099 1.09220,,,. 2016a.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, 2016b.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, 2017.
Ziming Li. Julia Kiseleva, and Maarten de Rijke. Dialogue generation: From imitation learning to inverse reinforcement learning. arXiv preprint frXiv:1812.03509, 2018.
Bing Liu and Ian Lane. Iterative policy learning in end-to-end trainable task-oriented neural dialog models. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp- 482-489. IEBE, 2017.
Bing Liu, Gokhan Tür, Dilek Hakkani- Tür, Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback in end-to-ens trainable task-oriented dialogue systems. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2060-2069, 2018.
11

---

Under review as a conference paper at ICLR 2020
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off- policy gradient with state distribution correction. arXiv preprint arXiv: 1904.08473, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:/312.355 2013.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1064-1062, 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances Neural Information Processing Systems, pp. 2775-2785, 2017.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-402240 2016.
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI, pp- 1607-1612. Atlanta, 2010.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Robert R Provine. Laughter. American scientist, 84(1):38-48, 1996.
Konrad Rawlik. Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. In Robotics: science and systems, 2012.
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural reinforce- ment learning method. In European Conference on Machine Learning, pp. 317-328. Springer, 2005.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897, 2015.
Julian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang. Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. A deep reinforcement learning chatbot. arXiv preprint arXiv: /000:000000022 02.44, 2017a
lulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for gen- crating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence, 2017b.
Pararth Shah, Dilek Hakkani-Tus Tur, Bing Liu, and Gokhan Tur. Bootstrapping neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pp. 41-51, 2018.
Weiyan Shi and Zhou Yu. Sentiment adaptive end-to-end dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1509-1519, 2018
Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. Happybot: Generating empathetic di- alogue responses by improving user experience look-ahead. arXiv preprint arXiv: wwXiv:/906.... 1906.084877 2019.
Robert F Stengel. Stochastic optimal control. John Wiley and Sons New York, New York, 1986.
Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 147-157, 2017.
12

---

Under review as a conference paper at ICLR 2020
Philip Thomas and Emma Brunskill. Data-efficient off off-policy policy evaluation for reinforcement learning. International Conference on Machine Learning, pp. 1139-114,, 2016.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement, In International Conference on Machine Learning, pp. 2380-2388, 2015.
Emanuel Todorov. Linearly-solvable markov decision problems. In Advances in neural information processing systems (NIPS), pp. 1369-1376, 2007.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Li Zhou, Jianfeng Gao, Di Li, and Heung- Yeung Shum. The design and implementation of xiaoice, an empathetic social chatbot. arXiv preprint arXiv:1812.0080 2018.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume pp. 1433-1438. Chicago, IL. USA, 2008
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1rx:v: X90.0099.00 2019.
A APPENDIX
A.1 DETAILS ABOUT IMPLICIT METRICS
The total reward used to train the bots is combination of the rewards described below, in the following proportions:
+ 0 0.15313553lauuhhter + 0.14206642+ transition
0. 115682657-question + 0. 0.188755B-emaaticccccheee
0.147601 48wwords elicited + 0. 0.1199222cconverstttin......
0..14206422
A.1.1 SENTIMENT-BASED
To compute sentiment on short texts like conversation utterances, we leverage state-of-the-at sentiment-detection model, which was trained on a massive amount of Twitter data to predict the emojis in tweets (Felbo et al. 2017). Transfer learning from this model to other tasks showed that was able to significantly outperform a series of sentiment, irony, and sarcasm benchmarks. This DeepMoji model outputs probability distribution over 64 most-frequently used emojis shown in Figure 5 After observing the performance of the model in detecting users' emotions in the domain of online chat, we define set of weights over the emojis and calculate the weighted sum an emotion embedding vector to derive a sentiment reward which higher for positive sentiment and lower for negative sentiment. These weights a centiment-transitte occurred later in the conversation than the peak negative sentiment, reasoning that sentiment should improve over the course of the conversation.
shown in Figure (b). We also compute reward using the same score based on whether the peak positive sentiment
A.1.2 ENGAGEMENT-BASED
Based on prior work (Zhou al. 2018), we use the number of turns in the conversation an indicator of the quality of the bot's 's performance. To distribute this reward over every utterance in the conversation, utterance n < as y N-"N N. We also reward each utterance with the number of words in the user's response, which we refer as the words elicited.
take the total conversation length N, and compute the discounted reward for
13

---

Under review as a conference paper at ICLR 2020
1.00
0.75
0.50
-0.50
-0.75
-1.00
(b)
Figure 5: (a) 64-most frequent emojis as predicted by (Felbo al. 2017 used for calculating emo-
tion embeddings. (b) Assigned weights used in producing the sentiment reward from the predicted emoji values.
A.1.3 LAUGHTER
Laughter has been shown to be very important to human affiliation (Provine) 1996> and solidarity (Hay] 2000). Therefore, detect the number of occurrences of the string 'ha" the user's response, and use this as reward. Interestingly, we find that bots trained to maximize user laughter learn to be extremely supportive and cheerful compared other bots (for definitions of supportive and cheerful, see Section ??).
A.1.4 SEMANTIC SIMILARITY
Language style matching has been shown be strong predictor of relationship initiation and stability (Ireland et al. 2011). While it would be ideal if our chatbots could intelligently adapt their conversation style to a new user, in reality most baseline dialog models struggle maintain topic coherence, even over few utterances (for an analysis of this effect, see (Ghandeharioun et al., 2.19) Therefore we reward semantic similarity between the user's input and the bot's response, to encourage the bot to stay topic and produce reasonable answers. This score is computing by leveraging a state-of-the-art sentence embedding model (Conneau et al. 2017), and penalizing distance in embedding space.
A.1.5 QUESTIONS
Asking questions is an important listening skill, and is linked to conversation management, atten- tiveness, and responsiveness (Bodie et al. 2012). Therefore, we give the bot reward of 0.5 if the utterance contains question word (how; what, where, why, when, who), and an additional 0.5 if contains question mark.
1. POST-HOC METRICS
After training the bots on these rewards, we noticed shift in the distribution of their language towards more polite, cheerful, and supportive speech. Therefore, we designed post-hoe metrics to measure these qualities, which are based on counting whether subset of phrases present in an utterance.
Politeness phrases: if may; may I; please; thanks; no worries; if you don't mind; have a great day; I'm sorry.
Supportive phrases: you're we right; you are right; you re not alone; you are not alone; congrats; that's 's good idea; that is good idea; you'l 'Il be fine; you will be fine; you 'Il be okay; you will be okay; it will get better; sorry you'l re going through; sorry you are going through; if it makes you feel better; if it makes you feel any better; keep your head up; keep it up: I'm in a similar situation; am in a similar situation; you'll 'Il get it; you will get it; happy for you; I'm in the same boat; I am in the same boat; you feel like you need to vent.
Cheerful phrases: nice to hear; happy: excited; really nice; glad; the best; great; good time; looking forward; beautiful.
14

---

Under review as a conference paper at ICLR 2020
A.2
TRAINING DETAILS AND HYPERPARAMETERS
A.2.1 TRADITIONAL RL EXPERIMENTS
All Q-networks shared the same underlying architecture: three fully-connected layers of size [256, 128, 64], with ReLU activation between. The model of p(als) was learned with Variational Auto- encoder which attempted to reconstruct the next state given the current state, p(s'ss). using a mean- squared error loss. Both the encoder and decoder were made up of two linear layers with 750 neurons each. The latent dimension of the VAE was size 256. The next action, was predicted from the latent emhedding 2, meaning the model learned three functions: = All models were trained with the Adam optimizer Kingma & Ba (0144).
f((s), 8' fa(z), and a = fa(2)
For each experiment, we ran 50 trials of each model with different random seed each time. The Behavior policy was trained for total of 20,000 steps in the environment, so in the Full buffer condition offline agents saw 20,000 experience samples. The Behavior policy typically converged before 10,000 steps, so in the Expert demonstrator condition the offline agents received the last 10,000 experience samples from the trained agent. In the Concurrent condition, offline agents saw moving window of 1000 samples, since the online learner only used the most recent 1000 samples in the buffer for learning. The learning rate was ,001, 7 = .99, and 6 decayed linearly from 1.0 computed as Dx [g(r)]l((T)
to ,01 over 2000 steps. The KL--onstrriitt
log p(als) 33 log T (a]ss) where = 0.5 and 3 = 0.1. DBCQ sampled n = 2 actions before selecting the best action based on the maximum Q-value; note that in this environment there are only 2 actions.
A.2.2 DIALOG EXPERIMENTS
The underlying architecture of the language models employed for this work is a Variational Hier- archical Recurrent Encoder Decoder (VHRED) (Serban et al. 2017b), with additional knowledge distillation to improve the model's ability to track the sentiment and semantics of the conversa- tion, proposed by Ghandeharioun et al. (2019). The language models were originally trained on two datasets: movie dialogs (Danescu-Niculese) Nicless--iiiil & Lee 2011 and a dataset scraped from reddit com/ r/ casual http/aaasaaoaaaeaaaaeaeeaa wtw._loovveereeeeeeeeeee (Ghandcharioun et al. (2019). The RL models were initialized with the weights of the best model trained on the Reddit dataset.
RL models were trained for between 800 and 1000 batches of data, where the batch size fixed at 32. Early stopping was used to determine the number of training iterations of the best checkpoint. All other hyperparameters were shared between RL models, and were as follows: discount 7 0.5, weight placed on RI. reward vs. Kl.-divergence term e = 2, number of Monte Carlo samples of the Target Q-network M = 5, target network update rate a .005, learning rate r = .0001. We used smooth L1 loss function to approximate the Q-values, and clipped gradients at a value of 1.0.
The underlying parameters of the VHRED model were as follows: Context RNN hidden size 1000, decoder hidden size- = 1250, encoder hidden size = 1250, embedding size 600, gradient clip- 1.0, dropout 0.2. The maximum conversation length was fixed 5 utterances (context from more than 5 utterances ago was discarded), and the maximum sentence length was 30 tokens.
We also added layers to the Context RNN and regularized it to be able to predict the semantic content of the input utterance using a form of knowledge distillation (Hinton et al. 2015) from a state-of- the-art sentence- centence-embedding model (Conneau et al. (017). There were additional feedforward semantic prediction prediction layers of size 128, which used ReLu activation.
A.3 ADDITIONAL RESULTS
A.3.1 TRADITIONAL RL EXPERIMENTS
We ran additional experiments in another standard OpenAl gym environment, Acrobot-vI; Figure 6 shows the results. The experimental setup and hyperparameters remained the same as in the first experiment, except that we used smaller VAE (the encoder and decoder had only one layer of size 256 each, and the latent dimension was 64), the weight on the prior was a = 0.5 and the entropy term was 3 = 0, and we used the Q-learning loss rather than -elearning.
15

---

Under review as a conference paper at ICLR 2020
Nean eotrock
(a) Full buffer
(b) Concurrent
(c) Expert demonstrator
(d) Noisy demonstrator
Figure 6: Comparison of batch RL algorithms for different offline learning conditions in Acrobot-
v/. WOP exceeds the performance of Batch Q-learning, Behavioral Cloning (BC), DBCQ, and the Behavior policy used to generate the batch data. Error bars show 95% C/ of the mean over 10 trials.
Conversation length
0.018
-0.003
-0.046
-0.048
0.023
-0.053
0.008
-0.543
-0.102
-0.035
-0.037
-0.143
-0.277
Semantic similarity
0.037
User laughter (ha)
0.029
-0.03
-0.00
0.000
0.023
00.016
00.05
00.08
-0.050
-0.149
-0.210
0.800
0.828
Words elicited
0.072
0.024
0.133
0.043
-0.093
0.003
-0.038
-0.126
0.01.
0.059
0.5.7
-0.285
-0.016
Sentiment transition
0.036
0..04
0.299
0.123
Question
0.012
0.230
0.210
Sentiment
0.123
0.045
0.092
0.021
0.030
0.068
0.099
0.346
0.294
0.085
0.098
0.270
0.073
Manual ratings-
0.064
0.010
-0.065
0
0.062
0.026
-0.071
-0.029
-0.064
-0.030
-0.112
-0.207
-0.275
Total reward-
Impiccit human reward
Conversation length-
Sentiment similarity
User laughter (ha)-
Words elicited-
Sentiment transition
Question
Sentiment-
Manual ratings-
Politeness-
Supportive
Cheerful
Figure 7: Normalized reward scores obtained by models trained with respect different rewards. We see that the bot trained to ask questions is easily able to exploit this reward, and similarly the bot trained elicit positive sentiment does so successfully. For the of the bots, the relationship is less clear. For example, the bot trained to elicit laughter becomes the most supportive and cheerful, while the bot trained to elicit more words very polite.
A.3.2 DIALOG EXPERIMENTS
Figure shows the normalized reward scores obtained bots trained with respect to different rewards. We see that testing the bots in the wild, with new of users, bots trained optimize certain types of human response (e.g. words elicited) do not perform best on this task. We hypothesize this because the relatively small size of batch date were able collect (s 14, 000 utterances) does not give the bots enough information about how to elicit long responses from users.
In addition the comparisons between RL models, also benchmarked the RL methods against the MLE prior. While we found that the KL-control methods exceed the prior in obtaining high implicit reward signals from humans (e.g. eliciting positive sentiment, laughter, more words, etc), we found that this did not lead to them being rated significantly higher in quality by human judges. Even though the RL bots successfully learned to conduct conversation to optimize these implicit signals, we believe the fact that this did not lead to higher quality ratings highlights the idea that the reward functions we are optimizing do not fully cover what means to have a high quality conversation with human user. We note that these rewards only first step, and hope other researchers will be able to use the techniques proposed in this paper to learn from better measures of human enjoyment.
16

---

Under review as a conference paper at ICLR 2020
B INTERACTIVE BOT PLATFORM DETAILS
To collect data from humans interacting with our bots, we buil::ttts:///euuaaa chat a plat- form for hosting deep neural network dialog models online GPU for fast, real-time inference. Figure shows an example of the interface, in which users are able to rate the bots after talking to them for least three turn..
Chat Hi story
Please Rate Your Chat Experience!
goáng?
IR. XL conteol Pr1l: @ honectly, you
Overall. hor was the QUALITY o1 the chat?
goang have keep talk about what than.
Very
Wey High
break:
Ds. AL conteol Pail: & well, you know that
1ike
HoN DIVERSE (non-Tepetstiv) NEDE The chat bot's responses?
pralls,
conteol Psil: today going
good time.
[User]: yeah? what are you goling to do7 conteol Psil: con't sodng, rappy
PLIENT
bot?
Figure 8: Interactive evaluation ratings page available at whttps://neuraaa hhaat
Figure 9 is example conversation within the platform that interactive evaluation participants see. Annotators can optionally click the up and down arrows beside each chatbot response to give feed- back the specific utterance. Once 3 or more turns of the conversation has taken place, participants may click "Close Chat and Rate" to get to the rating screen.
[User] just work.
you Close Chat:
glad
[Usce] you're baring.
Rodel Padl:
would this? what's you?l 7171711 11811811
[uee]]: sarmy!
you.
(RL Model avg Psil: i'' not. suze, and think i' going to do this, and a'm suze you' 11 gol
[User]: will
being werrd)
same, con't understanc that, [User]] ]: we are not the same. i hunan and you ie a bet.
Pst?:
that's
SEAD
Figure 9: Interactive evaluation chat interface.
B. WEBSITE SERVER SETUP AND CONHIURATTIN
The server hosted Google Cloud Platform virtual instance with 64GB of RAM and NVIDIA Tesla P100 graphics card. The backend was Django program being served by NGINX and uWSGI. For simplicity, we opted to have the Django process import the chatbots into the same Python process as Django, rather than have the two connect to each other via other means such as sockets. This configuration decreased development time and increased reliability, but it would need to be revisited if the server needed to scale several orders of magnitude past what was required for this study. The current configuration was still able to support hundreds of simultaneous users and host more than 30 bots concurrently.
The chatbots were kept in separate project from the Django project and maintained separately from the server code. Each chatbot extended abstract class that defined key methods for the Django program to use, and was registered to a globally accessible dictionary via a decorator. The Django project was provided the path to the Chatbots project in its PYTHONPATH, so it could import the dictionary in which all the chatbot objects had been registered and use that to dynamically determine which chatbots were available and to access them in its views.
is important to note that the chatbots used PyCUDA, and PyCUDA does not work in a multipro- cessing environment. Because of this, uWSGI needed be configured only have one python
17

---

Under review as a conference paper at ICLR 2020
process and to disable any attempt at multiprocessing. Furthermore, the chatbots required substan- tial startup times, so all chatbots are kept in memory at all times in the Django process. In order to needed very high amount of RAM on server
keep all the chatbots in memory concurrently, and opted for a 64GB virtual instance, and GPU with 16GB RAM. This combination of CUDA to the chatbots time resulted in incredibly fast server response times, with effectively no increase in response time when using the bots in requests compared requests that did not.
the GPU with high amount of RAM keep all bots in memory the same
For further information and instructions on server configuration, please read the server documenta- tion available j ;URL redacted for anonymity.. We hope that this platform will allow others to host their own bots and evaluate them in an interactive setting.
18


Reviews:
[{"review_id": "rkx10Mr1cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the Way Off-Policy (WOP) algorithm which penalize divergence fro a strong pre-trained prior model as a way to avoid extrapolation error and instability in batch reinforcement learning (BRL). KL-divergence penalties have previously been explored extensively in the on-policy setting (mainly with policy gradient methods) and the authors claim they are the first to apply this idea to off-policy RL. I am not familiar enough with related work to evaluate this statement, but given it’s true, this is an interesting application of these ideas. I also like the application to dialog because pre-trained language models have been shown to offer very good priors for high-quality natural language generation.\n\nUnfortunately, before any other comments I would like to point out that this paper is not properly anonymized. Both the submitted code and the online demo that is linked to from the paper contain the author names and their affiliation (the page linked to in Section 1 contains the author names but the other one in Section 5 does not—it mentions they have been redacted). Apparently there is also an arXiv version of the paper which I was not aware of and which would be fine if this version was not cited from the README file in the submitted code. Please be more careful with respect to anonymizing your submission!\n\nOverall the paper proposes an interesting algorithm that combines existing ideas in a new way and arrives at an algorithm that achieves good performance, based on the experimental evaluation provided in the paper. The experiments performed on dialog are good and I really like that human evaluation was used. However, the traditional RL experiments are lacking. The authors only present results for the CartPole environment in the main paper, which is a very easy problem and so not that interesting. In the appendix they also show results for the Acrobot environment, which are a bit weaker for WOP. Both are quite simple problems and it’s quite easy to learn a good prior model. Why not present results for harder problems? It should be easy to run on more Gym environments and see if WOP still does well in harder problems. Other than that, I like the main ideas in the paper, but I think they could be presented better. Given this and the anonymization issue I lean towards rejecting this paper for now.\n\nOther comments:\n\n(1) I don’t necessarily agree with the statement “Because learning language online from humans on the internet can result in inappropriate behavior (see Horton (2016)), learning offline using BRL is imperative.” I would rather just say that learning offline is one way of dealing with this issue but it’s not the only one or necessarily the best one. In general, the paper defends the batch RL setting quite extensively, but I feel some of the statements may be overly strong. There is a lot of prior work that shows when and why the batch RL setting is useful and so I would add more references to such work.\n\n(2) In Section A.1 you present the implicit metrics that were used and the weight of each metric. These weights seem *extremely* arbitrary to me so could you please provide some insight (and references) as to how they were chosen? I like the results presented in Table 3, in terms of the insight they provide. Are these what you used to derive the weights presented in Section A.1?\n\n(3) The yellow shading in Figure 1 is not really visible. You could try using a different color for the shading (e.g., use gray and make the walls blue or something like that).\n\n(4) The axis labels in the plots of Figure 2 are too tiny. These plots should be made bigger to be readable in printed form. Also, I would recommend you order the labels in the legend in maybe chronological order or something like, which would also put WOP at the top to clarify it’s the proposed method.\n\nDisclaimer: I am not too familiar with this area of research and the related work and so my comments and evaluation should be taken into account in that context.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SkgumSL19S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an off-policy batch reinforcement learning algorithm, Way Off-Policy(WOP). The authors address the extrapolation error which is a general problem of batch reinforcement learning. WOP algorithm uses KL-control to penalize divergence between prior and policy. Detailed comments are as follows:\n\n- KL penalize is the first in batch RL, but many approaches have already been proposed in RL. Moreover, I'm not sure if it is novel because I think that it is not much different from KL penalize proposed in RL.\n- And DBCQ seems to simply remove the perturbation of BCQ. Is there any other difference? (If so, a detailed explanation needs to be added in the paper.)\n- Are there any comparisons with more recent batch RL algorithms for discrete actions? (ex. SPIBB(Safe Policy Improvement with Baseline Bootstrapping, Laroche R., et al 2019))\n- Although the main experiment is about dialog, the section of ‘traditional RL experiments’ seems not enough for cartpole only.\n\nThis paper is well organized and clearly written, but I cannot agree with the main contribution (KL-control penalize) of this paper (because the KL penalize idea is very similar to proposed approaches in RL). And, it seems to be compared with more recent batch RL algorithms in the experiments.\n\n[Minor errors]\n- c first appears in Equation (6), which seems to have no definition and explanation in the paper.\n- It is shown that the normalization constant is missing in Equation (9).\n- In section 3.4, line 4, pi_\\theta(a_t, s_t) -> pi_\\theta(a_t | s_t)\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgQVvQ0tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an off-policy batch deep reinforcement learning method for learning human preferences in dialog generation. The key technical contribution is by controlling the KL divergence of the learned policy with a prior policy that was learned on other dialogs. The method is able to constrain the policy not deviate too much from the prior to avoid extrapolation error. Another technical contribution is the use of dropout uncertainty estimates to estimate the Q values, which is more scalable compared to double Q-learning. The authors also incorporate intermediate implicit rewards in dialogs to encourage more positive conversations. \nIn terms of methodology, the use of dropout as uncertain estimates seem to be firstly used for DRL to my knowledge. The experiments seem to be properly conducted, with particular focus on inferring what rewards encourage positive conversations.\n\nHowever, I am not fully convinced by the motivation of using KL control. If there is too much weight on the rewards of learning of human preferences compared to reasonable sentences, one could imagine other compared methods such as batch Q-learning could fail since they will learn to encourage positive conversations and diverge away from realistic language. Using KL to control its generated dialog to the prior is a way to prevent it from diverging. But I would think that with a proper tuning of the different type of rewards could also prevent the policy from diverging to generating non-realistic dialogs especially when there is a large amount of dialogs available. If possible I would hope to see tuning on the weights of different types of rewards used in the problem setup. Also I am curious if you use the same initialization (using the same pretrained language model) for other baselines? I did not seem to find it the paper. \nAlso the proposed method using KL control shares similarity to the one in Distral [1] where a distilled policy is used as prior policy and discounted KL divergence is used as regularizer, which limits it novelty in this perspective.\n\nOverall, the paper is fairly well-written. The paper develops a method for learning human preferences in dialogs in off-policy reinforcement learning and use KL control to avoid extrapolation error issues. The technical novelty of the method is limited as there is a similar method proposed in [1]. The experiments are illustrative but more comparisons with the other methods will be appreciated to make it more convincing. \n\n[1] Teh, Yee, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. \"Distral: Robust multitask reinforcement learning.\" In Advances in Neural Information Processing Systems, pp. 4496-4506. 2017.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a possibly novel approach to regularizing policy learning to make it suitable for large-scale divergence in the underlying domain.  Unfortunately all the reviewers are unanimous that the paper is not acceptable in present form.  Insufficient clarity regarding the contribution relative to several references, some of which were missing from the submitted version, is perhaps the most significant issue in the view of the AC.
Decision: Reject