Year: 2021

Abstract:
Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus. It is very challenging in terms of RL since the natural language action space is infinitely large, while feasible (syntactically and semantically correct) actions are very sparse. Thus, standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a powerful pre-trained language model. In this paper, we introduce GPT-Critic, an offline RL method for task-oriented dialogue. GPT-Critic is built upon GPT-2, fine-tuning the language model through behavior cloning of the critic-guided self-generated sentences. GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model. In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in the task-oriented dialogue benchmarks including MultiWOZ and ConvLab.

heading: 1 INTRODUCTION
text: Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing (NLP) tasks, yet challenging due to large language action spaces and limited availability of human-annotated data. Recently, large-scale pre-trained language models (LM) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Raffel et al., 2019). The current best performing end-to-end conversational agents for a task-oriented dialogue system utilize a pre-training on largescale corpus and fine-tuning on downstream tasks (Ham et al., 2020; Yang et al., 2021; Lin et al., 2020; Peng et al., 2021). This combination of pre-training and fine-tuning significantly improves overall performance in the task-oriented dialogues. However, supervised fine-tuning (i.e. imitation learning of the dialogue corpus) alone may not be sufficient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels. Thus, in order to optimize the task performance of the conversational agent, goaloriented training (i.e. reinforcement learning) is an essential and promising direction to pursue.
Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as offline reinforcement learning (RL) problem (Levine et al., 2020; Fujimoto et al., 2019; Jaques et al., 2020), which offers the prospect to optimize the policy solely from the fixed dataset without online environment interaction. Most of the existing offline RL methods are built on the off-policy ActorCritic framework, which performs iterative optimization of the policy (i.e. actor) and the actionvalue function (i.e. critic) (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020). Yet, a naive application of these offline RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language (Lewis et al., 2017; Zhao et al., 2019; Jang et al., 2020).
To mitigate the aforementioned problem, discrete latent representation models for language actions have been proposed to disentangle the semantics of the utterance and the natural language generation (Zhao et al., 2019; Yarats & Lewis, 2018). In that framework, goal-based training was performed in the space of the discrete latent variables instead of directly optimizing utterances. While these approaches can prevent degenerate responses in principle, they cannot be straightforwardly applied to widely used large-scale LMs since most of them are not trained for discrete latent variables. One
could make modifications to the LM to work with discrete latent variables but it would need to be pre-trained from scratch with a tremendous amount of data and time, which is undesirable.
In this paper, we present an offline RL algorithm for task-oriented dialogue, which can be adopted for any generative pre-trained language model. Our algorithm, GPT-Critic, does not rely on policy gradient unlike actor-critic methods and is essentially free from the issue of diverging from human language. It starts with fine-tuning the GPT-2 model and learning the action-value function (critic) using the dialogue corpus. Then, GPT-Critic generates a strategically promising action that is selected based on the value estimated by the critic. GPT-Critic updates the policy through behavior cloning of the critic-guided self-generated responses. This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus, where the action choice is restricted to the support in the dataset (Wang et al., 2020). Since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2, it thus inherits GPT-2’s ability to generate human-like responses. In the experiments, we demonstrate that GPT-Critic outperforms the state-of-the-art end-to-end dialogue agent in the task-oriented dialogue benchmarks including MultiWOZ (Budzianowski et al., 2018) and ConvLab (Zhu et al., 2020).

heading: 2 BACKGROUND

heading: 2.1 OFFLINE REINFORCEMENT LEARNING FOR TASK-ORIENTED DIALOGUES
text: We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process (POMDP) defined by tuple 〈S,A,O, T, Z,R, γ〉 where S is the set of environment states s = 〈g, h〉 (underlying state that consists of the user goal g and dialogue history h), A is the set of actions a (a sequence of tokens which represents dialogue act and system response), O is the set of observations o (user utterance), T (s′|s, a) = Pr(st+1 = s′|st = s, at = a) is the transition function, Z(o|s′, a) = Pr(ot+1 = o|st+1 = s′, at = a) is the observation probability, R(g, h, a) is the reward function indicating the utility of executing action a in history h and the user goal g, and γ ∈ (0, 1) is a discount factor. The history at time step t, ht = {o0, a0, . . . ot−1, at−1, ot}, is a sequence of all previous observations and actions. Since the underlying state s (e.g. user goal) is not directly observable, the agent makes decisions based on the entire observation-action history. The policy π(at|ht) is mapping from history ht to a probability distribution over A. The goal is to find an optimal policy π∗ that maximizes the expected cumulative rewards, i.e. π∗ = arg maxπ Eπ [ ∑∞ t=0 γ
tR(g, ht, at)]. The action-value function of policy π is defined as Qπ(h, a) := Eπ [ ∑∞ t=0 γ
tR(g, ht, at)|h0 = h, a0 = a], where Qπ is a unique solution of the Bellman equation: Qπ(h, a) = Eg[R(g, h, a)] + γEπ [Qπ(h′, a′)].
Using offline RL for dialogue policy optimization, the agent optimizes the policy from the precollected dataset D = {{(gj , hjt , a j t , r j t , h j t+1) T t=0}Nj=1} without online environment interaction during the intermediate stages of training. Prior offline RL algorithms (Fujimoto et al., 2019; Janner et al., 2019; Kumar et al., 2020) rely on off-policy actor-critic method, where the critic network is trained by minimizing the temporal differnce error with respect to the target policy π:
arg min φ
E(ht,at,rt,ht+1)∼D [( rt + γEat+1∼π(ht+1) [ Qφ̄(ht+1, at+1) ] −Qφ(ht, at) )2] (1)
where φ̄ is the parameters of the target network. As discussed in the prior work (Fujimoto et al., 2019; Kumar et al., 2020), optimizing this loss can be challenging in the offline RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution (OOD) actions to evaluate the value of the next state.

heading: 2.2 END-TO-END TASK-ORIENTED DIALOGUE SYSTEM
text: We focus on the MultiWOZ dataset (Budzianowski et al., 2018), which is a representative benchmark for task-oriented dialogue. The MultiWOZ dataset is a fully-annotated corpus of human-human task-oriented conversations, which is collected via the Wizard-of-Oz setting (Kelley, 1984). The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of the following four modules: 1) A natural language understanding (NLU) module (Kim et al., 2017; Zhu et al., 2020) identifies the user’s intent and extracts the information of slots and their values, 2) A Dialogue state tracking (DST) module (Williams et al., 2013) infers the belief state, 3) A dialogue policy (POL) module decides the system action, 4) A natural language generation (NLG) module (Wen et al., 2015) generates the system response corresponding to the system action. Recently, end-to-end task-oriented dialogue methods leveraging the pre-trained language model have been proposed (Yang et al., 2021; Ham et al., 2020; Lin et al., 2020; Peng et al., 2021; Hosseini-Asl et al., 2020), and significantly improves overall performance in the task-oriented dialogues. In this paper, our algorithm is built upon UBAR (Yang et al., 2021), which is based on GPT-2 (Radford et al., 2019) and currently the state-of-the-art end-to-end dialogue agent for the MultiWOZ domain.

heading: 3 OFFLINE REINFORCEMENT LEARNING FOR END-TO-END TASK-ORIENTED DIALOGUE SYSTEMS
text: The corpus collected from human-human conversations inevitably contains unsuccessful dialogues in terms of task completion. For example, approximately 20% dialogues of the MultiWOZ dataset fail to meet the user goal. Therefore, a naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot of unsuccessful dialogues: an agent that imitates failure would be inevitably suboptimal. Yet, dropping the unsuccessful dialogues from the corpus is also undesirable, since they may contain some task-specific information that is useful to properly respond to user requests. We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while improving the task performance.
In this section, we present GPT-Critic, an offline RL algorithm for task-oriented dialogue. Our GPTCritic is analogous to Actor-Critic method: GPT (Actor) decides which action to take while the Critic informs how good the action was and provides a signal for policy improvement. Still, GPTCritic is distinct from the Actor-Critic methods in that it does not rely on the policy gradients, which are generally known to cause the issue of diverging from human language (Lewis et al., 2017; Zhao et al., 2019). Instead, we sample a set of action candidates using GPT-2 and pick the best one using the critic, which constitutes a revised dialogue corpus. Then, we perform supervised fine-tuning of the GPT-2 on the revised dialogue corpus. This learning procedure of our GPT-Critic does not hurt the agent’s capability to generate human-like sentences, given that the generated action candidates were all natural-looking sentences due to the power of large pre-trained LM. Our algorithm is built upon the GPT-2 but it can be adopted for any generative pre-trained language model.

heading: 3.1 POLICY EVALUATION
text: Our GPT-Critic starts by training the action-value function (i.e. critic), which can evaluate the candidates for the response. The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value. The parameterization of the critic network Qφ is designed to share the parameters of the Transformer (Vaswani et al., 2017) layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step. The critic network of GPT-Critic is trained by minimizing the temporal difference error with respect to the dataset D:
arg min φ
E(ht,at,rt,ht+1,at+1)∼D [( rt + γQφ̄(ht+1, at+1)−Qφ(ht, at) )2] (2)
where φ̄ is the parameters of the target network. Note that Eq. (2) is an on-policy evaluation on the dataset D, which can be optimized very stably since every at+1 is always an in-distribution sample of D. This is in contrast to Eq. (1), which requires evaluation of out-of-distribution actions sampled from the target policy π. The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset.
This kind of on-policy evaluation has been explored in the offline RL context for stable policy optimization (Brandfonbrener et al., 2021; Goo & Niekum, 2021), but they are limited to only one-step policy improvement: once the policy π is improved by the initial on-policy Q-function (i.e. π(s) = arg maxaQ(s, a)), the new policy deviates from the dataset policy, thus it requires off-policy evaluation for further policy iteration. In contrast, our GPT-Critic performs policy improvement by generating an improved dataset based on the learned critic, where we can perform on-policy evaluation on the new dataset again. As a consequence, GPT-Critic can enjoy the stable multi-step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset, which will be discussed in the following section.

heading: 3.2 POLICY IMPROVEMENT VIA DATASET REVISION
text: In the task-oriented dialogues, the reward is given by the external program provided as a part of the dataset, which checks whether the user goal is satisfied by examining the dialogue history. To generate the improved dataset, we adopt the common approach for automatic evaluations of dialogue systems, where the agent generates dialogue state, dialogue act, and system response on every system turn with fixed user utterances. More formally, the GPT-Critic generates a new dataset containing revised responses by:
Di+1 = {(g, ht, a∗t , r∗t , h∗t+1) | a∗t = arg max a∈{ak}N
{ak}N∼πiθ(ht)
Qφ(ht, a) where ht ∈ Di} (3)
where {ak}N is a set of N response candidates generated from the policy π (i.e. fine-tuned GPT-2), and Di is the dataset at i-th iteration. In the task-oriented dialogues, a reward function R(g, h, a) is provided that can compute a reward given a user goal, dialogue history, and system action. The revised reward r∗t = R(g, ht, a ∗ t ) is computed by given user goal, dialogue history, and revised system action a∗t . The dialogue history is a sequence of all previous observations and actions, thus the revised history h∗t+1 = {o0, a0, . . . , ot, a∗t , ot+1} is defined by replacing the original action at of ht+1 with the revised action a∗t . The examples of revised responses can be found in Appendix B.
Algorithm 1 GPT-Critic Input: Training dataset D0 = {{(gj , hjt , a j t , r j t , h j t+1) T t=0}Nj=1}, policy network (GPT) πθ , critic network Qφ Fine-tune the initial policy represented by GPT-2 model (e.g. UBAR) for each iteration i do
Update critic by minimizing the temporal difference error until convergence:
argmin φ E(g,ht,at,rt,ht+1,at+1)∼Di
[( rt + γQφ̄(ht+1, at+1)−Qφ(ht, at) )2] Update dataset by critic-guided self-generation:
Di+1 = {(g, ht, a∗t , r∗t , h∗t+1) | a∗t = argmax a∈{ai}N
{ai}N∼πiθ(ht)
Qφ(ht, a) where ht ∈ Di}
Update policy by behavior cloning of critic-guided self-generated dataset: (Early stop according to the loss on the validation set)
argmin θ E(ht,at)∼Di+1 [− log πθ(at|ht)]
end for
In order to address the prohibitively large language action spaces, we explicitly consider the set of response candidates that are generated from the fine-tuned GPT-2. The GPT-Critic selects the most promising response by calculating the Q-values over the response candidates. GPT-Critic then performs behavior cloning of critic-guided self-generated dialogues:
arg min θ E(ht,at)∼Di+1 [− log πθ(at|ht)] (4)
where θ is the parameters of GPT-2. The policy improvement of GPT-Critic is performed by behavior cloning of generated dialogues from the GPT-2, thus GPT-Critic inherits GPT-2’s ability to generate human-like responses.
We can theoretically show that the updated policy by the above policy improvement step has a higher value than the old policy. Furthermore, we can also theoretically show that updated policy by the higher number of candidate actions has a higher value than the policy updated by the lower number of candidate actions. We formalize this result in Theorem 1.
Theorem 1. (Policy Improvement) Given a policy π and the number of sampling actions N ≥ 1, If we update the new policy πnewN by
∀s, πnewN (·|s) = arg max a∈{ak}N
{ak}N∼π(s)
Qπ(s, a)
then Qπ new N (s, a) ≥ Qπ(s, a) ∀s, a always holds. Furthermore, for any N,M such that N ≥ M ≥ 1, Qπ new N (s, a) ≥ QπnewM (s, a) ∀s, a always holds. (Proof in Appendix A.)
We describe our algorithm, GPT-Critic, in Algorithm 1, that alternates between policy evaluation and policy improvement via revising the dataset until the policy performance converges.

heading: 4 RELATED WORK
text: End-to-End Task-Oriented Dialogue Systems. The traditional approach to building a task-oriented dialogue system adopts a modular pipeline, which consists of natural language understanding, dialogue state tracking, dialogue policy, and natural language generation. Recently, pre-trained LMbased end-to-end task-oriented dialogue agents that all sub-tasks recast as a single sequence prediction problem have been proposed (Ham et al., 2020; Hosseini-Asl et al., 2020), and significantly improved overall performance in the task-oriented dialogues. There are a number of variants of GPT-2-based end-to-end task-oriented dialogue agents. Yang et al. (2021) leverage the entire dialogue session of every dialogue turn. Peng et al. (2021) adopt transfer learning and machine teaching
for training a GPT-2-based dialogue agent. Lin et al. (2020) present efficient dialogue state tracking with a minimal generation length, then leverage pre-trained language models for task-oriented dialogues.
Reinforcement Learning for Task-Oriented Dialogue Systems. Applying the standard RL methods straightforwardly to optimize a task-oriented dialogue agent causes the issue of diverging from human language. To address this problem, interleaving reinforcement learning with supervised learning has been proposed but it is still not free from the issue of diverging from human language (Lewis et al., 2017). Recently, the latent representation models for language actions have been introduced to address the aforementioned problem (Zhao et al., 2019; Yarats & Lewis, 2018). They disentangle the semantics of the utterance and the natural language generation, and then perform goal-based training in the space of the latent variables instead of directly optimizing utterances. However, they cannot be directly applied to large-scale pre-trained language models that are not designed in a way that works inherently with discrete latent variables. Jaques et al. (2020) use KLcontrol to restrict the policy to stay close to its prior policy, but it still suffers from divergence from human language even with carefully chosen hyper-parameters. Furthermore, Jang et al. (2020) adopt Bayes-adaptive Monte-Carlo planning to negotiation dialogue then use it as a policy improvement operator. This approach can prevent the issue of diverging from human language through the policy improvement based on behavior cloning of self-generated dialogues. However, they assume a user model that is difficult enough to be considered another problem.
Offline Reinforcement Learning. There have been extensive studies on offline RL (Fujimoto et al., 2019; Levine et al., 2020; Kumar et al., 2020; Wang et al., 2020). Most of prior works are built on the off-policy actor-critic framework, and they focus on the overestimation issue by taking the OOD actions (Kumar et al., 2019; Lee et al., 2020; Fujimoto et al., 2019; Jaques et al., 2020; Kumar et al., 2020). However, a naive application of these offline RL methods suffer from the issue of diverging from human language in the task-oriented dialogues (Lewis et al., 2017; Zhao et al., 2019; Jang et al., 2020). On the other hand, there are a number of recent works on weighted behavior cloning, where a policy is trained by a variant of supervised learning loss (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020). The weighted behavior cloning approaches filter out bad actions, then perform behavior cloning on high-quality data. However, in the task-oriented dialogues, simply dropping the unsuccessful dialogues from the corpus is undesirable, since they may contain some task-specific information that is useful to properly respond to user requests. Our GPT-Critic aims to revise unsuccessful dialogues into successful ones, which is in contrast to the weighted behavior cloning on the fixed training dataset, where the action choice is restricted to the support in the dataset (Wang et al., 2020; Peng et al., 2019; Siegel et al., 2020). More recently, Chen et al. (2021) introduce Decision Transformer, a Transformer-based architecture that casts the problem of RL as conditional sequence modeling. These offline RL methods based on behavior cloning are directly applied to the task-oriented dialogues without aforementioned issue, but their results are similar to that of behavior cloning in the task-oriented dialogues.

heading: 5 EXPERIMENTS
text: In this section, we show the experimental results of GPT-critic on both automatic evaluation and human evaluation. First, we evaluate the performances of GPT-Critic on the MultiWOZ domain (Budzianowski et al., 2018) as dataset-based automatic evaluation, compared with baseline methods including offline RL algorithms. Second, for more realistic evaluation, we conduct a simulatorbased evaluation on the ConvLab framework (Zhu et al., 2020). Third, we also conduct the human evaluation to evaluate the quality of generated responses. Finally, we give a qualitative analysis of our method using generated dialogue examples on the training dataset of MultiWOZ, which shows how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues. The qualitative analysis with generated dialogue examples can be found in Appendix B.

heading: 5.1 EXPERIMENTAL SETUP
text: We implement GPT-Critic based on the HuggingFace Transformers library (Wolf et al., 2019) and codebase of UBAR (Yang et al., 2021), which is a GPT-2-based current state-of-the-art end-to-end task-oriented dialogue agent for the MultiWOZ domain. For the generative pre-trained language model, we use DistilGPT2 (Sanh et al., 2019), a distilled version of GPT-2. Figure 2 shows the ar-
chitecture of our policy and critic network based on GPT-2. We design the parameterization of the critic network to share the parameters of the Transformer layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step. For the hyperparameters of fine-tuning the GPT-2 model, we follow the setting in the public code of UBAR (Yang et al., 2021). We use N = 5 for the number of candidate actions {ak}N , and the set of candidate actions are constructed by vanilla softmax sampling from the policy, rather than beam search, to collect diverse actions. For each behavior cloning iteration, all models are fine-tuned with a training dataset from the pre-trained GPT-2 and early stop according to the loss on the validation set.

heading: 5.2 EVALUATION ON THE MULTIWOZ DATASET
text: We evaluate our algorithm on the MultiWOZ dataset, which is one of the representative task-oriented dialogue benchmarks. The MultiWOZ is a large-scale multi-domain Wizard-of-Oz dataset, where a tourist (i.e. user) converses with a clerk (i.e. system) at the information center in a touristic city. It consists of 8438/1000/1000 dialogues for training/validation/testing. For end-to-end evaluation on the MultiWOZ dataset, we use the following automatic evaluation metrics: 1) Inform: evaluates whether the system provides an appropriate entity, 2) Success: evaluates whether the system answers all the requested information, 3) BLEU: measures the fluency of the generated response (Papineni et al., 2002). We also report the Combined Score as an overall quality measure (Combined = (Inform + Success)× 0.5 + BLEU). We compare the performance of GPT-Critic with the following algorithms: 1) SFN+RL (Mehri et al., 2019), a seq2seq network that incorporates several pre-trained dialogue modules into a neural dialogue model, 2) DAMD (Zhang et al., 2020), the domain-aware multi-decoder network with multiaction data augmentation method, 3) SimpleTOD (Hosseini-Asl et al., 2020), a GPT-2-based endto-end dialogue agent that all sub-tasks recast as a single sequence prediction problem, 4) SOLOIST (Peng et al., 2021), a GPT-2-based end-to-end dialogue agent with transfer learning and machine teaching, 5) MinTL (Lin et al., 2020), an efficient dialogue state tracking method with a minimal generation length by predicting the difference between old and new states, 6) UBAR (Yang et al., 2021), a GPT-2-based end-to-end dialogue agent that leverages the entire dialogue session of every dialogue turn. We implement our algorithm into the codebase of UBAR (Yang et al., 2021), and the result of UBAR is reproduced by adapting its code to the same evaluation settings as other papers1. Moreover, we also compare the data augmentation method, DATA AUGMENTATION, which is naively fine-tuning the GPT-2 model with additionally generated data by vanilla softmax sampling from the trained policy.
In addition, we also compare with recent offline RL algorithms that are free from the issue of diverging from human language: 1) CRR (Wang et al., 2020), a value-filtered regression method that performs weighted behavior cloning of offline dataset, 2) Decision Transformer (Chen et al., 2021), a Transformer-based architecture that casts the problem of RL as conditional sequence modeling. For a fair comparison, we use the same pre-trained GPT-2 model as a policy network to train the
1The score reported in the UBAR paper is the result of using the true dialogue state for DB search. In order to compare under the same conditions with other algorithms, we record the result of UBAR to use the predicted dialogue state for DB search.
CRR and the Decision Transformer. Moreover, to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language, we also provide examples of responses generated by policy-gradient-based standard RL algorithm in Appendix C.
Table 1 and Table 2 show the results of policy iteration. Table 1 shows the performance of training dataset and critic-guided self-generated dialogues being used for each policy improvement step. Table 2 reports the intermediate performance of behavior cloning of training dataset and criticguided self-generated dialogues in each of the policy iteration. As shown in Table 1 and Table 2, the performance of critic-guided self-generated dialogues is improved gradually; the performance of GPT-Critic is also consistently improved through the behavior cloning of improved dataset.
Table 3 summarizes the overall performance of GPT-Critic and baseline algorithms in end-to-end response generation setting, where the generated dialogue state and generated dialogue act are used for the DB search and response generation. The results show that GPT-Critic achieved the best performance in terms of inform rate, success rate, and combined score. Moreover, the performance of GPT-Critic on the BLEU score matches those of other pre-trained LM-based methods, since GPT-Critic inherits GPT-2’s ability to generate human-like responses through the behavior cloning of responses generated by GPT-2. The results show that GPT-Critic improves the task performance of the agent without the issue of diverging from human language. In addition, as can be shown in Table 3, the naive data augmentation is not effective since it will not change the GPT’s sampling distribution in principle.
For the results of offline RL baselines, CRR and Decision Transformer show the results that do not diverge from human-language, since their policy is also trained by behavior cloning. However, both algorithms show limited performance because they perform behavior cloning on a fixed dataset. CRR has achieved remarkable success in continuous control tasks by performing weighted behavior cloning of training dataset filtered by critic, but it does not effectively perform in the task-oriented dialogues because of data scarcity. Furthermore, to evaluate the Decision Transformer, we adopt a delayed return where the agent receives the cumulative reward at the end of dialogue, since the agent cannot observe user goal. Therefore, without observing the user goal at test time, Decision Transformer reduces to the behavior cloning of successful dialogues.

heading: 5.3 EVALUATION ON CONVLAB EVALUATOR
text: In order to evaluate the performance of dialogue agents in an end-to-end fashion, we conduct simulator-based evaluation on ConvLab (Zhu et al., 2020). ConvLab is an open-source toolkit that enables to build task-oriented dialogue systems and perform an end-to-end evaluation. The simulator-based evaluation is more reliable than dataset-based automatic evaluation because it evaluates the performance while interacting with the user simulator. To interact with dialogue systems, ConvLab provides an agenda-based user simulator (Schatzmann et al., 2007) that consists of a BERT (Devlin et al., 2019) for NLU, a rule-based policy, and a template-based NLG. We compare the performance of GPT-Critic with baseline algorithms interacting with the same user simulator and user goals. We report the results with the following metrics: 1) Complete: evaluates whether the system completes the goal, 2) Success: evaluates whether all the user requests have been informed and the booked entities satisfy the constraints, 3) Book: evaluates how many booked entities satisfy the
user constraints, 4) Inform (Precision / Recall / F1): evaluates how many user requests have been informed, 5) Turn (success / all): evaluates the average turn number for successful/all dialogues.
We describe the performance of GPT-Critic and baselines in Table 4. Each algorithm is tested for 1000 runs with randomly sampled user goal. The results show that GPT-Critic achieves the best performance in all metrics related to task accomplishment. However, they also show that GPT-Critic takes longer dialogue turn for the task accomplishment because GPT-Critic is trained by maximizing the success rate without considering the dialogue turn.

heading: 5.4 HUMAN EVALUATION
text: We also conduct human evaluation on Amazon Mechanical Turk (AMT) to assess the quality of generated responses of GPT-Critic and baseline algorithms, using the evaluation protocol as in (Yang et al., 2021; Lin et al., 2020; Zhang et al., 2020). Specifically, human workers on AMT were asked to read the context and generated response by interactive simulation via ConvLab, then score the following two evaluation metrics on a Likert scale (1-5): 1) Appropriateness: evaluates whether the generated responses are appropriate for the given context, 2) Fluency: evaluates whether the generated responses are comprehensible and human-like. We compare the performance of GPT-Critic with same baselines on ConvLab evaluation. Figure 3 summarizes the overall results of human evaluation, where 60 workers evaluate the quality of 30 randomly selected dialogues for each algorithm. The results show that GPT-Critic significantly outperforms baseline algorithms in appropriateness which is related to task accomplishment. Moreover, the result of fluency shows that GPT-Critic does not hurt the agent’s capability to generate human-like sentences.

heading: 6 CONCLUSION
text: We presented GPT-Critic, an offline RL algorithm for task-oriented dialogue system, which can be adopted for any generative pre-trained language model. GPT-Critic aims to learn an end-to-end task-oriented dialogue agent without the issue of diverging from human language. GPT-Critic starts with fine-tuning the GPT-2 model and learning the critic using the dialogue corpus. Then, GPTCritic updates the policy through the behavior cloning of the critic-guided self-generated responses, thus it is essentially free from the issue of diverging from human language. In the experiments, we demonstrated that GPT-Critic outperforms the state-of-the-art algorithms in the task-oriented dialogue benchmarks including MultiWOZ and ConvLab.

heading: A POLICY IMPROVEMENT THEOREM
text: Theorem 1. (Policy Improvement) Given a policy π and the number of sampling actions N ≥ 1, If we update the new policy πnewN by
∀s, πnewN (·|s) = arg max a∈{ak}N
{ak}N∼π(s)
Qπ(s, a)
then Qπ new N (s, a) ≥ Qπ(s, a) ∀s, a always holds. Furthermore, for any N,M such that N ≥ M ≥ 1, Qπ new N (s, a) ≥ QπnewM (s, a) ∀s, a always holds.
Proof. For any s, a,N ≥M ,
Qπ new M (s, a) = EP [ R(st, at) + γEat+1∼πnewM (st+1)[Q πnewM (st+1, at+1)]|st = s, at = a ]
= EP [ R(st, at) + γE{ai}M∼π(st+1)[ max
a′∈{ai}M Qπ(st+1, a
′)]|st = s, at = a ]
≤ EP [ R(st, at) + γE{ai}N∼π(st+1)[ max
a′∈{ai}N Qπ(st+1, a
′)]|st = s, at = a ]
= EP [ R(st, at) + γEanewt+1∼πnewN (st+1)[Q π(st+1, a new t+1)|st = s, at = a ] = EP [ R(st, at) + γ ( Eanewt+1∼πnewN (st+1)[R(st+1, a new t+1)] + γEat+2∼π(st+2)[Q π(st+2, at+2)] ) |st = s, at = a
] = EP
[ t+1∑ τ=t Eaτ+1∼πnewN (sτ+1)[γ τ−tR(sτ , aτ )] + γ 2Eat+2∼π(st+2)[Q π(st+2, at+2)]|st = s, at = a ] ≤ EP
[ t+1∑ τ=t Eaτ+1∼πnew(sτ+1)[γ τ−tR(sτ , aτ )] + γ 2Eanewt+2∼πnew(st+2)[Q π(st+2, a new t+2)|st = s, at = a ] = EP
[ t+2∑ τ=t Eaτ+1∼πnew(sτ+1)[γ τ−tR(sτ , aτ )] + γ 3Eat+3∼π(st+3)[Q π(st+3, at+3)|st = s, at = a ] ...
≤ EP [ ∞∑ τ=t Eaτ+1∼πnew(sτ+1)[γ τ−tR(sτ , aτ )]|st = s, at = a ] = Qπ new N (s, a)
For the case of N = 1, note that πnew1 is simply reduced to the π, which concludes the proof:
Qπ new N (s, a) ≥ Qπ new M (s, a) ≥ Qπ new 1 (s, a) = Qπ(s, a) for all s, a,N ≥M ≥ 1.

heading: B QUALITATIVE ANALYSIS OF SELF-GENERATED DIALOGUES
text: In this section, we provide a qualitative analysis on the critic-guided self-generated responses in GPT-Critic. We show the critic-guided self-generated dialogue examples in Table 5 that illustrates how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues, compared with unsuccessful dialogues on the training dataset of MultiWOZ. Each example demonstrates the critic-guided self-generated dialogue act and delexicalized system response, compared with the original dialogue. The generated responses contain all the requests of user with abundant information, whereas the original responses of unsuccessful dialogues do not contain all the requested information. GPT-Critic improves the performance through the behavior cloning of these
revised responses. Moreover, Table 5 shows that the generated dialogues do not diverge from human language. Since GPT-Critic updates the policy through behavior cloning of the self-generated human-like responses, GPT-Critic is essentially free from the issue of diverging from human language.

heading: C QUALITATIVE EXAMPLES OF STANDARD REINFORCEMENT LEARNING ALGORITHM
text: In this section, we provide examples of responses generated by standard RL algorithm (REINFORCE) to show that the policy-gradient-based standard RL algorithms suffer from diverging from human language. As can be shown in Table 6, policy-gradient-based RL algorithm generates responses which is diverging from human language.

Review summaries:
This paper presents a reinforcement learning-based approach to building a task-oriented dialogue agent. Given a dialogue dataset annotated with rewards, the state-action value function is first trained by minimizing temporal differences. Then, a new training dataset is created by using the best actions selected among the candidates generated by the current policy (i.e., the language model). The policy is then updated by behavior cloning using the created dataset, and the whole process is repeated. The authors have conducted experiments using MultiWOZ and ConvLab and shown that this iterative process improves the performance of the agent.

This paper proposes an offline RL method applied to an end-to-end task-oriented dialogue model, where the proposed GPT-Critic is built on GPT-2 and fine-tuned on the self-generated sentences for policy updating.  The paper claims that it is free from the issue of diverging from human language (a common issue in standard RL), because it learns from the sentences directly sampled from the pre-trained language model. The conducted experiments show that the proposed model achieves better performance compared to other task-oriented end-to-end dialogue models in both offline and online settings (MultiWOZ and ConvLab respectively).

The paper works on offline reinforcement learning for natural language action space setting, particularly for task-oriented dialogue management. The paper nicely incorporate the policy network (to sample agent response) and the q network (to evaluate the agent response) into a single GPT-2 network and propose a policy interation algorithm to optimize both the q and policy network. During policy evaluation, the q network is updated with sampled system actions and responses. During policy improvement, the sampled system actions and responses with maximum q values are used as labels to update the policy network. The model achieves SoTA performance on MultiWOZ dataset.