SlowGuess/NAACL_2022_OCR huggingface dataset:
Justin Fu
Mengjiao Yang
Sergey Levine
vsiddharth@berkeley.edu
{justinjfu,sherry,svlevine}@eecs.berkeley.edu
Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties. However, dialogue can also be regarded as a goal directed process, where speakers attempt to accomplish a specific task. Reinforcement learning (RL) algorithms are designed specifically for solving such goal-directed problems, but the most direct way to apply RL – through trial-and-error learning in human conversations, – is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents that better accomplish task goals.
Constructing fluent and intelligent dialogue agents could pave the way for intuitive interfaces and automation of human-interactive tasks. However, this requires dialogue agents that both generate fluent, natural responses and effectively pursue the goals of the given dialogue task. A predominant approach to training dialogue agents is through supervised learning, where an agent is tasked with imitating language provided by humans. While this can provide for fluent responses, it becomes difficult to ensure that such agents systematically pursue the goals of the dialogue interaction. If we instead view dialogue as a control problem, frameworks such as reinforcement learning (RL) could allow agents to automatically optimize dialogue with respect to a task goal through a trial-and-error process and improve over human behavior.
However, implementing an RL system in practice, where an agent learns online from interacting with real humans, can be prohibitively expen
sive and time-consuming. This is in stark contrast to supervised learning approaches, where we can cheaply construct datasets for training imitation agents by simply logging conversations. Therefore, existing RL approaches for dialogue often rely on interacting with a learned model of a human (Li et al., 2016b; He et al., 2018), from which experience can be generated inexpensively. However, naive training in this manner can result in the dialogue agent exploiting the model, which can degenerate into non-intelligible language. To mitigate this, algorithms must typically enforce strong priors to keep generated language similar to those seen in the dataset (Li et al., 2016b; Jaques et al., 2019), or adopt dialogue management and template-based approaches which directly re-use language seen in the dataset (He et al., 2018).
Issues such as model exploitation and distribution shift when training on static datasets are a primary concern of offline RL (Levine et al., 2020), and can provide a formalized approach to tackling these problems. While offline RL is motivated by scaling RL to large datasets, annotated datasets for dialogue are still small compared to the large amount of raw text datasets available today. Therefore, we propose an offline, model-free approach to dialogue generation that leverages language models. Because the size of unlabeled language datasets dwarfs that of curated datasets for dialogue, using a pre-trained language model as a central component of our method allows it to learn aspects of language fluency from unlabeled datasets, while learning higher-level strategies for goal-directed dialogue via RL on a smaller annotated datasets. This combined approach enables us to utilize the large amounts of existing language data that standard RL methods cannot.
The main contribution of this work is CHAI (CHatbot AI), an algorithm for learning task-oriented dialogue that utilizes a language model in conjunction with offline RL. We show that this
Human Language Technologies, pages 4471 - 4491
leads the policy to generate goal-oriented dialogue that is both realistic and functional, and does not require training against a simulated model of human language. We evaluate our method on a negotiation task, which requires the model to both reason about strategic aspects of conversation along with generating fluent language. We show that CHAI consistently bargains for better prices and with higher rates of successful negotiation than prior RL approaches to goal-oriented dialogue.
Recent developments in deep learning have led to end-to-end approaches to dialogue using supervised learning, such as sequence-to-sequence models (Dusek and Jurcicek, 2016; Eric and Manning, 2017), hierarchical models (Serban et al., 2017), attention (Mei et al., 2017; Chen et al., 2019), and Transformer-based models (Wu et al., 2021; Hosseini-Asl et al., 2020; Peng et al., 2020; Adiwardana et al., 2020). However, supervised learning only allows an agent to imitate behaviors, requires optimal data, and does not allow agents to exceed human performance. Supervised learning for dialogue generation also has well-known issues such as outputting commonplace responses (e.g., I do not know) regardless of the inputs (Li et al., 2016a). Therefore, additional training of the dialogue agent is required for performing goal-oriented tasks.
Task-oriented dialogue has been formulated as a sequential decision making problem in a Markov Decision Process (MDP) since the 1990s (Smith and Hipp, 1994; Singh et al., 1999; Williams and Young, 2007; Young et al., 2013; Paek, 2006; Henderson et al., 2008; Gao et al., 2018; Pieraccini et al., 2009; Young et al., 2013; Su et al., 2015; Chen et al., 2020). Dialogue is converted into abstract states and actions from which an agent is trained using reinforcement learning (RL) (Eckert et al., 1997; Levin et al., 2000; Chung, 2004; Georgila et al., 2006; Schatzmann et al., 2007; Heeman, 2009; Georgila and Traum, 2011; Su et al., 2016; Fatemi et al., 2016; Asri et al., 2016; Zhao et al., 2019; Zhang et al., 2020; Wang et al., 2020). These methods differ in how the abstract states/actions are designed and whether the simulated environment for training the policy is hand created, learned as a fixed model, or is an agent itself. For instance, Eckert et al. (1997); Levin et al. (2000) learn a fixed transition model from human conversations and Georgila and Traum (2011)
learn negotiation agents where each agent is the user simulator for the other agent. These methods also differ in how the decision making policy is trained, e.g., online (Gašić et al., 2011) or off-policy/offline (Yu et al., 2016; Pietquin et al., 2011) using actor-critic (Su et al., 2017), policy gradient (He et al., 2018), or fitted Q-iteration (Pietquin et al., 2011). Regardless of the RL method used, since policies are trained on abstract states and actions, these methods lack the ability to generate natural language (i.e., response is created via templates depending on an abstract action).
To overcome these limitations, recent work has trained policies directly on text, using a recurrent neural network to output language tokens, and using self-play for policy training while interacting with another learned agent (Li et al., 2016b; Lewis et al., 2017; Liu et al., 2018). To further improve the generated language quality, hierarchical methods decouple the strategic high-level dialogue decisions from generation (Yarats and Lewis, 2018; He et al., 2018; Saleh et al., 2020). These model-based approaches require accurate estimation of the environment/human (e.g., the trained self-play agent needs to mimic complex human behavior), which is beyond current capability of model-based reinforcement learning algorithms. Similar to our proposal, Jaques et al. (2019) use offline RL based on KL-control for text generation in open-domain dialogue. Our work differs in that our model is able to utilize large amounts of unsupervised data through the use of pre-trained language models, and that our work focuses on task-oriented (as opposed to open-domain) dialogue tasks. Goal-oriented tasks have clearly defined objectives that can be quantified, allowing us to provide an objective comparison between our method and prior approaches.
In this section, we describe our evaluation task and cover the necessary background and notation.
3.1 Negotiation via Task-Oriented Dialogue
We evaluate our approach on the CraigslistBargain task (He et al., 2018). CraigslistBargain consists of 6682 advertisements scraped from Craigslist, along with dialogues for each advertisement collected via Amazon Mechanical Turk where two users play the role of buyer/seller. An example advertisement from this dataset is shown in Fig. 1, along with a sample conversation between a human and CHAI.
Title: iPhone 5S 16 GB black silver - $135
Description: Great condition. No scratches. I upgraded to iPhone 7. Please reply with phone number if you are interested. Thank you.
Buyer: Hi, I would like to buy the iPhone 5S
CHAI: Hi, great! The phone is in great condition. I'm asking $135.00, it's in mint condition.
Buyer: Does it come with a charger?
CHAI: Yes it comes with a charger, as I upgraded. It's in mint condition.
Buyer: How long have you been using this phone?
CHAI: Just a month. It's in great condition and has a charger included.
Buyer: Did you buy this phone secondhand or from the Apple store?
CHAI: I bought it brand new from Apple store. I only used it a month, I'm happy I did not have to use this one.
Buyer: I can buy it for $100
CHAI: I'll take that if you can pick it up in a few hours.
Buyer: Where will you be at 5pm today?
CHAI: I'm at the Apple store so about an hour.
Buyer: Okay, I can meet you at the Apple store.
CHAI: Awesome!
Buyer: accept
Figure 1: An example advertisement from the CraigslistBargain task, with a conversation between CHAI (blue) and a human buyer (green).
During each round of interaction, the buyer and seller can execute four possible response types. A message allows one player to send an utterance to the other. An offer allows one player to propose a price at which to conduct the transaction. Once an offer is made, the other player can either accept or reject the offer, which ends the episode. A reward is then computed based on the transaction price. Our bot receives a reward equal to the normalized price the item is sold for at the end of an episode (normalized by the list price), scaled by a constant factor of 10. Additionally, we penalize the bot by a constant of -20 for episodes resulting in a reject to incentivize the agent to make deals.
We selected this task because it provides a clear objective, allowing us to illustrate our approach with quantifiable metrics. Of course, practical applications of CHAI to goal-directed dialogue may tackle other problems, including non-adversarial problems such as helping a user to answer a question or fulfill a request. However, our choice of tasks was constrained by the limited availability of public datasets for dialogue tasks that are goal-directed and have objective task goals.
3.2 Reinforcement Learning Setup
We formulate the task-oriented dialogue problem as an RL problem, where the agent serves the role
su	utterance	Hello!
scontext	context	Selling a new mountain bike.
sprice	price	0.5 (of list price)
stype	action_type	message
Figure 2: An example of the state space used in the CraigslistBargain task described in Section 3.1.
of seller in the CraigslistBargain problem, and the environment serves the role as the buyer. We consider a Markov decision process defined as a tuple  $(S, A, T, r, \gamma)$ . The state and action spaces,  $S$  and  $A$ , consist of three main components: the action type type (one of the four described in Sec. 3.1), an utterance  $u$  (used only for message responses), and a normalized price price, expressed as a fraction of the list price. Additionally, the context component  $s_{\text{context}}$  contains the advertisement listing description. The price component is used in two ways. First, because we do not wish to represent prices in the dialogue as discrete tokens, we instead replace prices in the utterance with a placeholder token understood to be substituted with the price component. Second, the price component is used in offer response type to communicate the desired transaction price. In all other cases, it is ignored. An example of the state space is shown in Fig. 2.
We write individual states as  $s = \{s_u, s_{\text{type}}, s_{\text{price}}, s_{\text{context}}\}$  and actions as  $a = \{a_u, a_{\text{type}}, a_{\text{price}}\}$ , where  $s_u, a_u$  denote the utterances, or sequences of words generated by the environment and agent.  $\{s_{\text{type}}, a_{\text{type}}\}$  denote the action types,  $\{s_{\text{price}}, a_{\text{price}}\}$  denote the prices, and  $s_{\text{context}}$  denotes the listing context. The transition distribution  $\mathcal{T}(s'|s, a)$  governs the distribution over responses generated by the buyer agent (environment), and the reward  $\mathcal{R}$  defines the task objective. The goal of RL is to find a policy  $\pi(a|s)$  that maximizes the expected returns:
$$
\mathbb {E} _ {\pi , \mathcal {T}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} r (s _ {t}, a _ {t}) \right],
$$
where  $r: S \times \mathcal{A} \mapsto \mathbb{R}$  is the reward function, and  $\gamma \in (0,1]$  is the discount factor. In online RL, the agent is interacts with the environment to maximize this objective. In contrast, offline RL uses
a dataset  $\mathcal{D}$  of pre-collected interaction. This is a natural approach in many settings, such as dialogue, where online RL could require interacting with real humans for impractically long periods of time, whereas offline RL can utilize pre-recorded dialogues between humans.
4 Offline Reinforcement Learning with Language Models
Offline RL potentially allows reinforcement learning methods to leverage large datasets for policy learning. However, it still requires datasets to be annotated with rewards, and for conversations to come from the task at hand. Because of this, annotated dialogue datasets, such as the CraigslistBargain dataset presented in Section 3.1, are many orders of magnitude smaller than unlabeled datasets collected for unsupervised and language modeling tasks. In order to utilize these large unlabeled datasets, we propose an algorithm that combines offline RL with fine-tuned language models.
Our approach begins with training a language model, such as GPT-2 (Radford et al., 2018), and fine-tuning it on our task-specific dialogue corpus (Sec. 3.1). We use  $\mathrm{LM}(u|s_{1:t})$  to denote a distribution over utterances  $u$  produced by the language model given the dialogue history, denoted  $s_{1:t}$ . We then train a critic or Q-function as described in Sec. 4.1, which is responsible for scoring good and bad responses and is used to select responses from a pool of candidates generated from the language model. Our approach can be viewed as using a Q-function to steer a language model (which has no concept of a task) towards producing language that accomplishes some task-specific goal.
4.1 Q-Learning with Language Models
In this section we describe how to train a Q-function that can score candidate responses based on their potential to maximize returns. We implement and evaluate three different training procedures, each utilizing different offline RL methods. In the overall Q-learning framework, we sample a batch of transitions (consisting of states, actions, rewards, and successor states) from our dataset and perform updates based on minimizing a modified Bellman loss:
$$
J (\theta) = \left(Q _ {\theta} (s, a) - Q _ {\text {t a r g e t}} (s, a)\right) ^ {2}, \tag {1}
$$
where target value  $Q_{\mathrm{target}}(s,a)$  is typically computed via the Bellman operator defined as
$$
Q _ {\text {t a r g e t}} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \left[ \max  _ {a} \bar {Q} \left(s ^ {\prime}, a\right) \right]. \tag {2}
$$
However, using this update directly can lead to problems if we only have access to offline datasets. A widely studied issue in offline RL is the challenge of handling out-of-distribution actions: when the maximization over the action in the target value is not constrained in any way, it is easy to obtain actions for which the Q-value predictions are erroneously high (Levine et al., 2020). In dialogue, this issue is greatly exacerbated, since the Q-function is only trained on responses in the dataset, and therefore is unlikely to make accurate predictions for arbitrary strings. The following modifications to Eqn.1 and Eqn.2 address this issue.
Proposal sampling (CHAI-prop) In the proposal sampling approach, the target value  $Q_{\mathrm{target}}(s,a)$  is computed via a modified Bellman operator that utilizes a proposal distribution based on the language model,  $\mu (a_t|s_{1:t})$ , to generate  $N$  response proposals, and then uses the target Q-function,  $\bar{Q}$ , to score those responses and selects the highest one:
$$
\begin{array}{l} Q _ {\text {t a r g e t}} ^ {\text {p r o p}} (s, a) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {\{a _ {i} \} ^ {N} \sim \mu} \left[ \max _ {i} \bar {Q} (s ^ {\prime}, a _ {i}) \right]. \\ \end{array}
$$
This sampling scheme serves a dual purpose: it both constrains the responses to be naturalistic, and it also prevents out-of-distribution inputs to the Q-function in the target value calculation. This approach resembles a number of prior offline RL methods that also employ proposal distributions (Kalashnikov et al., 2018; Kumar et al., 2019; Fujimoto et al., 2019; Wu et al., 2019). Similarly to several prior works, we use samples from a proposal distribution for the target value, without an explicit actor (Kalashnikov et al., 2018; Ghasemipour et al., 2020). Unlike these approaches, our method leverages a pretrained and finetuned language model LM, which additionally makes use of extensive unsupervised prior datasets during the pretraining stage and enables our method to handle the complex and combinatorial action space of dialogue generation. In addition, following prior work, we use a separate target network  $\bar{Q}$  whose weights are updated to track those of Q using a soft update rule as done in prior methods (Lillicrap et al., 2016; Haarnoja et al., 2018).
Figure 3: An overview of our dialogue generation and selection process. First (a), a finetuned language model is used to generate candidate responses conditioned on the conversation history, along with sampled prices. Then (b), each action is scored using a critic, and (c) a final response is returned by sampling the Q-value scores.
The proposal distribution  $\mu (a_{t}|s_{1:t})$  represents a distribution over actions  $a = \{a_u,a_{\mathrm{type}},a_{\mathrm{price}}\}$ . We use the language model in order to sample utterances  $a_{u}\sim \mathrm{LM}(\cdot |s_{1:t})$ . To make training more computationally efficient, we pre-generate a batch of 5 utterances per transition in the dataset using the language model, and resample these during training as an approximation of directly sampling from the language model. For the prices  $a_{\mathrm{price}}$ , we uniformly sample a value between  $70\%$  to  $100\%$  of the previously offered price, which roughly matches the distribution of the seller's offers in the dataset. Finally, we infer the message type based on the utterance sampled using a simple heuristic, as the CraigslistBargain task requires us to specify a type for each response. During language model fine-tuning, we replaced each offer, accept, or reject action with the utterances "offer", "accept", and "reject", respectively. We then check if the language model generated any of these tokens and return the corresponding action type, and label the action as a message otherwise. This simplifies our method and allows us to use the language model to generate the action types as well as the utterances.
Conservative Q-learning (CHAI-CQL) Conservative Q-learning (CQL) (Kumar et al., 2020) proposes a complimentary approach to reducing the harmful effect of out-of-distribution Q-values by explicitly penalizing the Q-value of actions not seen in the dataset. We adapt CQL as an additional regularizer on the Q-value in addition to the proposal sampling scheme. Specifically, we use the  $\mathrm{CQL}(\mathcal{H})$  variant, which add an additional regularizer  $f^{CQL}$  to the Q-learning objective:
where the regularizer is defined as:
$$
f ^ {C Q L} (\theta) =
$$
$$
\mathbb {E} _ {s \sim \mathcal {D}} \left[ \log \sum_ {a} \exp (Q _ {\theta} (s, a)) - \mathbb {E} _ {a \sim \mathcal {D}} [ Q _ {\theta} (s, a) ] \right]
$$
Our adaptation of CQL differs from Kumar et al. (2020) in that they propose an actor-critic method which trains an explicit actor. Rather, the remainder of our method is identical to the proposal sampling variant, specifically in regards to the computation of  $Q_{\mathrm{target}}$  and language model sampling.
Behavior-regularized Q-learning (CHAI-BRAC) Behavior-regularized actor-critic (BRAC) (Wu et al., 2019) proposes an alternative method for regularizing the Q-function such that out-of-distribution Q-values are penalized. Adapting this method to the setting of dialogue with language models, we use this approach to regularize the price proposal mechanism. Rather than uniformly sampling prices as described for proposal sampling, we train an additional price proposal network  $\pi_{\phi}(a_{\mathrm{price}}|s_{1:t})$  that outputs a Gaussian distribution over prices given the conversation states. Using the notation  $a^\prime \sim \pi_\phi ,\mu$  to denote sampling prices from the proposal network, utterances from the language model, and action types uniformly, the proposal network is trained according to the objective
$$
\begin{array}{l} \max  _ {\pi_ {\phi}} \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ \mathbb {E} _ {a ^ {\prime} \sim \pi_ {\phi}, \mu} [ Q (s, a ^ {\prime}) ] \right. \\ \left. - D _ {K L} \big (\pi_ {\phi} (\cdot | s), \pi_ {B} (\cdot | s) \big) \right]. \\ \end{array}
$$
The prior proposal network,  $\pi_B$ , is estimated as a univariate conditional Gaussian of the current offer given the previous offer, where the mean and standard deviations are linear functions of the previous offer. The target value is then computed as:
$$
J ^ {C Q L} (\theta) = \left(Q _ {\theta} (s, a) - Q _ {\text {t a r g e t}} (s, a)\right) ^ {2} + \alpha f ^ {C Q L} (\theta),
$$
$$
Q _ {\text {t a r g e t}} ^ {\text {B R A C}} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}, a ^ {\prime} \sim \pi_ {\phi}, \mu} \left[ \bar {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right].
$$
This adaption of the behavior regularized objective controls out-of-distribution queries on the Q-value by regularizing the price towards those seen within the dataset. This prevents the target Q-value from being queried in low-data regimes which can cause inaccuracies during training.
4.2 Dialogue Generation
Once the Q-function has been trained, dialogue generation from our model is a three-phase process. The first phase is sampling: given a response from the buyer, we query the language model to sample 5 candidate utterances, and sample 5 candidate prices. The next step is scoring: we then take the cross-product of these sets, and score each potential action with the Q-function. Finally, in the selection phase, several methods are considered in order to select an action. A straightforward method is to return the action that had the highest Q-value. However, we found that this approach resulted in behavior with low diversity. Instead, we opted to follow the approach in soft Q-learning (Haarnoja et al., 2018) and sample actions from a softmax distribution over the Q-values,  $p(a|s) \propto \exp \{Q(s,a)\}$ , which increases diversity in the responses as suboptimal actions are occasionally sampled. This decoding process is depicted in Fig. 3.
4.3 Architecture Details