GPT-Critic: Offline Reinforcement Learning for End-to-End Task-Oriented Dialogue Systems
Authors: Youngsoo Jang
Jongmin Lee
Kee-Eung Kim
Venue: ICLR 2022 Poster (2022)
Keywords: task-oriented dialogue
pre-trained language model
offline reinforcement learning

Abstract: Training a task-oriented dialogue agent can be naturally formulated as offline reinforcement learning (RL) problem, where the agent aims to learn a conversational strategy to achieve user goals, only from a dialogue corpus. It is very challenging in terms of RL since the natural language action space is astronomical, while feasible (syntactically and semantically correct) actions are very sparse. Thus, standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a powerful pre-trained language model. In this paper, we introduce GPT-Critic, an offline RL method for task-oriented dialogue. GPT-Critic is built upon GPT-2, fine-tuning the language model through behavior cloning of the critic-guided self-generated sentences. GPT-Critic is essentially free from the issue of diverging from human language since it learns from the sentences sampled from the pre-trained language model. In the experiments, we demonstrate that our algorithm outperforms the state-of-the-art in the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.

Published as a conference paper at ICLR 2022
GPT-CRITIC: OFFLINE REINFORCEMENT LEARNING FOR END-TO-END TASK-ORIENTED DIALOGUE SYS-
TEMS
Youngsoo Jang', Jongmin Lee', Kee-Eung Kim
'School of Computing, KAIST, Daejeon, Republic of Korea Graduate School of AI, KAIST, Dacjcon, Republic of Korea (ystang, jmlee) kaist. ac.kr, kexim@kaist.ac. ac.
ABSTRACT
Training task-oriented dialogue agent can be naturally formulated as offline rein- forcement learning (RI. problem, where the agent aims learn conversational strategy to achieve user goals, only from a dialogue corpus. It is very challenging in terms of RL since the natural language action space is astronomical, while feasi- ble (syntactically and semantically correct) actions are very sparse. Thus, standard RL methods easily fail and generate responses diverging from human language. even when introduce BPT-CCiitic an offline RL method for task-oriented dialogue. GPT-Critic is built upon GPT-2, fine-tuning the language model through behavior cloning of the critic-guided self-generated sentences. GPT-Critic essentially free from the issue of diverging from human language since learns from the sentences sam- pled from the pre-trained language model. In the experiments, we demonstrate that our algorithm outperforms the -fate-of-the-art the task-oriented dialogue benchmarks including MultiWOZ 2.0 and ConvLab.
a powerful pre-trained language model. In this paper, we
INTRODUCTION
Building an end-to-end task-oriented dialogue agent is one of the promising applications of natural language processing (NLP) tasks, yet challenging due to large language action spaces and limited availability of hunan-andotated data. Recently, large-scale pre-trained language models (LM) have achieved remarkable successes in various NLP tasks with prohibitively large vocabulary (Devlin) ett al. 2019: Radford et al. 2019 Brown et al. 2020 Raffel et al. 2019). The current best performing end-to-end conversational agents for a task- oriented dialogue system utilize a pre-training on large- scale corpus and fine- tuning downstream tasks (Ham et al. 2020 Yang al.] [021] Lin al.) 2020 Peng et al. (022)) This combination of pre-training and fine-tuning significantly improves overall performance in the task-oriented dialogues. However, supervised fine-tuning (i.e. imitation learning of the dialogue corpus) alone may not be sufficient to learn an optimal dialogue strategy since the corpus often contains suboptimal dialogues collected from human participants of diverse expertise levels. Thus. in order to optimize the task performance of the conversational agent, goal- oriented training (i.e. reinforcement learning) is an essential and promising direction to pursue.
Training a task-oriented conversational agent from a dialogue corpus can be naturally formulated as offline reinforcement learning (RL) problem (Levine al.) 2020 Fujimoto al. 20que al.) 2020), which offers the prospect to optimize the policy solely from the fixed dataset without online environment interaction. Most of the existing offline RI. methods are built on the off- off-policy Actor- Critic framework, which performs iterative optimization of the policy (i.c. actor) and the action- value function (i.e. critic) (Fujimoto et al.) [011 Janner et al.) 2019 Kumar et al.) 2020). Yet, a naive application of these offline RL methods generally results in poor dialogue strategies which generate responses in no way similar to human language (Lewis et al. 2017 Zhao et al. 2019 Jang et al. 2020).
Weighted behavior cloning (BC) Wang et al. 2020) is one of the representative offline RL algo- rithms, which is free from the issue of diverging from human language. Weighted BC amounts

---

Published as a conference paper at ICLR 2022
Critic-guided Self-generation
42
Policy
Critic
D1+1
Policy To(a(h)
Hac me(p)))
max Qa(he.a) wefes)" [sx] wnj(hee
Updated dataset [Th aj airyj 1
Critic Qo(h,a)
Dit1
ang arg min D-i log no(a:[ls))
ong min Ext, Aaii ( Rathe-r. awi)) Rolbrooi))
Update Policy and Critic
Figure I: Overall architecture of GPT-Critic.
to filtering out bad actions and imitating good actions. In the context of task-oriented dialogues, that would be equivalent simply dropping the unsuccessful dialogues from the corpus. However, dropping a whole dialogue from training would be wasteful, since they may still contain some task- specific information that is useful properly respond
requests in the intermediate steps.
In this paper, we present an offline RL algorithm for task-oriented dialogue, which can be adopted for any generative pre-trained language model. Our algorithm, GPT-Critic, aims revise unsuccess- ful dialogues into successful ones, rather than removing them as done in weighted BC. It starts with fine-tuning the GPT-2 model and learning the action-valus value function (critic) using the dialogue cor- pus. Then, GPT-Critic generates strategically promising action that is selected based on the value estimated by the critic. GPT-Critic updates the policy through behavior cloning of the critic-guided self- relf-generated responses. This is in contrast to the previous methods that perform weighted behavior cloning on the dialogue corpus, where the action choice is restricted to the support in the dataset (Wang et al. 2020). Compared to traditional actor-critic methods, since GPT-Critic does not rely on policy gradient and updates the policy within the support of generated actions from the GPT-2, thus inherits GPT-2's ability to generate human-likk responses. In the experiments, we demonstrate that GPT-Critic outperforms the state-of of------tthrrrr end-to-ent dialogue agent in the task-oriented dia- logue benchmarks including MultiWOZ. 2.0 (Budzianowski et al. 2018) and ConvLab (Zhu al., 2020
2 BACKGROUND
2.1 OFFLINE REINFORCEMENT LEARNING FOR TASK- -ASK-ORIENTED DIALOGUES
We consider the task-oriented dialogue system that can be modeled as a partially observable Markov decision process (POMDP) WWilliams & Young 2007) defined by tuple (S, A, O,T, Z, R, 7) where the set of environment states s = (g. h) (underlying state that consists of the user goal g and dialogue history h), A is the set of actions (a sequence of tokens which represents dialogue act and system response), O the set observations o (user utterance), T a) = Pr(#++1 a) is the transition function, Z(ols', a) Pr(o++1 o|s s ag a) is the
s'|x, = observation probability, R(g, J,, a) is the reward function indicating the utility of executing action in history h and the user goal g. and h4 = (Op, ag.... 4,-1, is a sequence of all previous observations and actions. Since the underlying state (e.g. goal) is not directly observable, the agent makes decisions based on the entire observation-action history. The policy x(aihh is mapping from history h to probability distribution over A. The goal is find optimal policy x that maximizes the expected cumulative rewards, i.e. n² = arg max, Ex
(0. 1) is discount factor. The history at time step t,
[>> 20 R(g, hu, at)]. The action-value function of policy π is
defined as QT (h, a) := E, I2= R'R(g. h, a;) = h, ag = a). where Q is a unique solution of the Bellman equation: Q* (h, a) R( h a) + 7EE, [Q" (N', (N'aa))
2

---

Published as a conference paper at ICLR 2022
Critic
( at)
Value
Action (at)
Value
85
Act
Resp
Linear
B5
Act
Resp
Liner
B5
GPT-2
f
Linear
Linear
Linear
Linear
Linear
Linear
Linear
Transformer
Dialogue history (ht)
Figure 2: Architecture of policy and critic network based on GPT-2.
Using offline RL for dialogue policy optimization, the agent optimizes the policy from the pre- collected dataset DD = {((r), Ni, ";, 11 ing the intermediate stages of training. Prior offline RL algorithms (Fujimoto al. 2019 Janner et al. 2019; Kumar et al.) (020) rely on off-po actor-critic method, where the critic network is trained by minimizing the temporal differnce error with respect to the target policy m:
1.W without online environment interaction dur-
arg min Ecb (ne yB,, (m1())))) [Qa(he+1,
(o(h),a))
(1)
where the parameters of the target network. As discussed in the prior work (Fujimoto al,, 2019 Kumar et al. 2020), optimizing this loss can be challenging in the offline RL setting due to the overestimation issue in the bootstrapping process by taking out-of-distribution (OOD) actions to evaluate the value of the next state.
2.2 END-TO-END
DIALOGUE SYSTEM
We focus on the MultiWOZ 2.0 dataset (Budzianowski et al. 2018), which is a representative bench- mark for task-oriented dialogue. The MultiWOZ dataset fully- annottedd corpus of comman-hummn task-oriented conversations, which is collected via the Wizard-of-Or setting (Kelley) 1984). The tra- ditional approach to building task-oriented dialogue system adopts modular pipeline, which consists of the following four modules: 1) A natural language understanding (NLU) module (Kim) et al. 2017 Zhu et al. (2020) identifies the user's intent and extracts the information of slots and their values, 2) A Dialogue state tracking (DST) module (Williams et al. (213)) infers the belief state, 3) A dialogue policy (POL) module decides the system action, 4) natural language generation (NLG) module (Wen et al.) 2015) generates the system response corresponding to the system action. Re- cently, end-to-end mask-oriented dialogue methods leveraging the pre- trtitrrieed language model have been proposed (Yang et al. 2021 Ham et al., 2020) Lin et al. 2020, Peng et al. [202 (Hoseini-Asl al., 2020), and significantly improves overall performance in the task-oriented dialogues. this paper, our algorithm is built upon UBAR (Yang et al.) (001), which is based GPT-2 (Radford
ctt al. 2019) and currently the state-of- -feth-rrrt end-to-end dialogue agent for the MultiWOZ domain.
OFFLINE REINFORCEMENT LEARNING FOR END-TO-END TASK- DIALOGUE SYSTEMS
The corpus collected from human-humma conversations inevitably contains unsuccessful dialogues terms of task completion. For example, approximately 20% dialogues of the MultiWOZ. dataset fail to meet the user goal. Therefore, naive behavior cloning of the whole dataset would limit the performance of the conversational agent since the dataset includes a lot unsuccessful dialogues: an agent that imitates failure would be inevitably suboptimal. Yet, dropping the unsuccessful dia- logues from the corpus as done in weighted BC also undesirable, since they may contain some spsk-spe information that is useful to properly respond to user requests. We thus aim to revise unsuccessful dialogues into successful ones in order to prevent repeating the past failure while im- proving the task performance.

---

Published as a conference paper at ICLR 2022
In this section, we present GPT-Critic, offline RL algorithm for task-oriented dialogue. Our GPT- Critic is analogous to Actor-Criie Critic method: GPT (Actor) decides which action to take while the Critic informs how good the action was and provides signal for policy improvement. Still, GPT- Critic is distinet from the Actor-Critic methods in that it does not rely on the policy gradients, which generally known to cause the issue of diverging from human language (Lewis et al. 2017] Zhao iet al.) 2019). Instead, we sample set of action candidates using GPT-2 and pick the best one using the critic, which constitutes a revised dialogue corpus. Then, we perform supervised fine-tuning of the GPT-2 on the revised dialogue corpus. This learning procedure of our GPT--ritic does not hurt the agent's 's capability to generate human-like sentences, given that the generated action candidates were all natural-lookin
sentences due to the power of large pre-trained LM. Our algorithm is built upon the GPT-2 but can be adopted for any generative pre-trained language model.
3.1 POLICY EVALUATION
Our GPT-Critic starts by training the action-value function (i.e. critic), which can evaluate the candi- dates for the response. The architecture of the critic network basically follows GPT-2 with employing different last layers to compute the Q-value. The parameterization of the critic network Qs is de- signed to share the parameters of the Transformer (Vaswani et al., 2017 layers of GPT-2, where the parameters of the Transformer layers are only updated during the policy improvement step. The critic network is trained by minimizing the temporal difference error with respect to the dataset D:
(1 + (2 ax+1) 2 a:))
arg min ar. (441)))))
(2)
where Ø is the parameters of the target network. Note that Eq. (2) is an on-policy evaluation on the dataset D, which can be optimized very stably since every a++1 always an in-distribution sample actions sampled
of D. This is in contrast to Eq. (I. which requires evaluation of from the target policy æ. The OOD action-value estimation can be very unreliable if the target policy deviates much from the dataset.
This kind of on-policy evaluation has been explored in the offline RL context for stable policy optimization (Brandionbrener et al.) [021 Goo & Nickum, 202I). but they are limited to only one-step policy improvement: once the policy я is improved by the initial on-policy Q-function (i.e. (s))= arg max, Q(s, a)), the new policy deviates from the dataset policy, thus it requires off-policy evaluation for further policy iteration. In contrast, our GPT-Critic performs policy im- provement by generating an improved dataset based on the learned critic, where we can perform on-policy evaluation on the new dataset again. As a consequence, GPT-Critic can enjoy the stable multi-step step policy iteration through alternation between on-policy evaluation and policy improvement via revising dataset, which will be discussed in the following section.
3.2 POLICY IMPROVEMENT VIA DATASET REVISION
In the task-oriented dialogues, the reward is given by the external program provided a part of the dataset, which checks whether the user goal is satisfied by examining the dialogue history. To generate the improved dataset, we adopt the common automatic evaluation of dialogue systems, where the agent generates dialogue act and system response on every system turn with fixed user utterances. More formally, the GPT-Critic generates a new dataset containing revised responses by:
Di+1 [(g his at r7 hi+1) l a arg max Q0(he. a) where bę E DD}
(3)
aE(ar) [ax) (he
where (ax) J.N is set of N response candidates generated from the policy (i.e. fine-tuned GPT-2), and D, is the dataset at i-th iteration. In the task-triented dialogues, a reward function R(g, h, a) is provided that can compute a reward given user goal, dialogue history, and system action. The revised reward rf = Rig. hy, at) is computed by given user goal, dialogue history, and revised system action at. The dialogue history is sequence of all previous observations and actions, thus the revised history h;_ = for, ap. ....... Ot, at ot+1} is defined by replacing the original action o of h with the revised action a;. The examples of revised responses can be found in Appendix B
In order to address the prohibitively large language action spaces, we explicitly consider the set of response candidates that generated from the fine-tuned GPT-2. The GPT--ritti selects the
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 GPT-Critic
Input: Training dataset Do = {((g), ri, (o) policy network (GPT) no, critic network Qa
Fine-tune the initial policy represented by GPT-2 model (e.g. UBAR)
for each iteration i do Update critic by minimizing the temporal difference error until convergence:
7 - (Qull,,
arg min ae. 1
Update datasct by critic guided self- geffraeneration:
Di+1 = {(g, he, ai ri, h(+1) oz
arg max (o( where he E D
(a;a) {oz] (he)
Update policy by behavior cloning of critic-guided self-generated dataset: (Early stop according to the loss on the validation sel)
arg min (hr. [- log To (ax [[+]]
end for
most promising response by calculating the Q-values over the response candidates. GPT-Critic then performs behavior cloning of critic-guidest guided self-generated dialogues:
arg min (((h,.r)>-D.41 [- log (a( [h)))
where f is the parameters of GPT-2. The policy improvement of GPT-Critte performed by behavior cloning of generated dialogues from the GPT-2, thus GPT-Critic inherits GPT-2's ability to generate human-like responses.
We can theoretically show that the updated policy by the above policy improvement step has a higher value than the old policy. Furthermore, we can also theoretically show that updated policy by the higher number of candidate actions has higher value than the policy updated by the lower number of candidate actions. We formalize this result Theorem I
Theorem 1. (Policy Improvement) Given policy π and the number of sampling actions N 2 we update the new policy π new by
1s, x (-18) arg max Q"
aEfax) |N
{ar)"~mt ~sT()
then Q (8, 2 (" a) Vs, a always holds. Furthermore, for any N, M such that 2 M Q (s,0) 2 Q "t" (s, a) Vs. a always holds. (Proof in Appendix
We describe our algorithm, GPT-Critic, Algorithm 1 that alternates between policy evaluation and policy improvement via revising the dataset until the policy performance converges.
4 RELATED WORK
End-to-Ean End Task-Oriented Dialogue Systems. The traditional approach to building a task-oriented dialogue system adopts modular pipeline, which consists of natural language understanding, di- alogue state tracking, dialogue policy, and natural language generation. Recently, pre-trained LM- based end-to-end task-oriented dialogue agents that all sub-tasks recast a single sequence pre- diction problem have been proposed (Ham et al.) 2020); Hosseini-Ass et al.) (020), and significantly improved overall performance in the lask-oriented dialogues. There are number of variants of GPT-2-based end-to-end task-oriented dialogue agents. Yang et al. (2021) leverage the entire dia- logue session of every dialogue turn. Peng et al. (2021) adopt transfer learning and machine teaching for training a GPT-2-based dialogue agent. Lin et al. 2020 present efficient dialogue state track- ing with a minimal generation length, then leverage pre-trained language models for oriented dialogues.
5

---

Published as a conference paper at ICLR 2022
Reinforcement Learning for Task-Oriented Dialogue Systems. Applying the standard RL meth- ods straightforwardly to optimize a task-oriented dialogue agent causes the issue of diverging from human language. To address this problem, interleaving reinforeement learning with supervised learning has been proposed but it is still not free from the issue of diverging from human language (Lewis al. (.17). Recently, the latent representation models for language actions have been in- troduced to address the aforementioned problem (Zhao et al., 2019; Yarats & Lewis] 2018). They disentangle the semantics of the utterance and the natural language generation, and then perform goal-based training in the space of the latent variables instead of directly optimizing utterances. However, they cannot be directly applied to large-scale pre-trained language models that are not designed in a way that works inherently with discrete latent variables. [Jaques et al. (2020) use KL- control to restrict the policy to stay close to its prior policy, but still suffers from divergence from human language even with carefully chosen syper-parameten Furthermore, Dang et al. (2020) adopt Bayes-adaptive Monte-Carlo planning to negotiation dialogue then use as a policy improvement operator. This approach can prevent the issue of diverging from human language through the policy improvement based on behavior cloning of self- eeff-genrrate dialogues. However, they assume a user model that is difficult enough to be considered another problem.
Offline Reinforcement Learning. There have been extensive studies on offline RL (Fujimoto et al., 2019 Levine et al., 2020 Kumar et al., 2020 Wang et al.) 2020). Most of prior works are built on the off-policy actor-critic framework, and they focus on the overestimation issue by taking the OOD actions (Kumar et al. 2019 Lee et al. 2020; Fujimoto et al. 2019 Jaques et al.. 2020 Kumar iet al., 2020). However, a naive application of these offline RL methods suffer from the issue of diverging from human language in the task-oriented dialogues (Lewis al. 2017 Zhao al. 2019 Jang et al. 2020). On the other hand, there are a number of recent works on weighted behavior cloning, where a policy is trained by variant of supervised learning loss (Wang et al., 2020: Peng al. 2019; Siegel et al., 2020). The weighted behavior cloning approaches filter out bad actions, then perform behavior cloning on high-quality data. However, in the task-triented dialogues, simply dropping the unsuccessful dialogues from the corpus is undesirable, since they may contain some task-specific information that is useful to properly respond to user requests. Our GPT-Critic aims to revise unsuccessful dialogues into successful ones, which is in contrast the weighted behavior cloning on the fixed training dataset, where the action choice is restricted to the support in the dataset (Wang et al. 2020 Peng al.) 2019 Siegel et al. (020). More recently, Chen al. (22T)) introduce Decision Transformer, a Transforee former- architecture that casts the problem of RL as conditional sequence modeling. These offline RL methods based on behavior cloning are directly applied the
task- oriented dialogues without aforementioned issue, but their results are similar to that of behavior cloning in the task-oriented dialogues.
EXPERIMENTS
In this section, we show the experimental results of GPT-critic on both automatic evaluation and human evaluation. First, evaluate the performances of GPT-Critic on the MultiWOZ. 2.0 (Budzianowski et al. 2018) as datasct- automatic evaluation, compared with baseline methods including offline RL algorithms. Second, for more realistic evaluation, we conduct simulator-based evaluation on the Lab framework (Zhu et al., 2020). Third, we also conduct the human eval- uation to evaluate the quality of generated responses. Finally, we give a qualitative analysis of our method using generated dialogue examples on the training dataset of MultiWOZ 2.0, which shows how GPT-Critic improves the performance through the behavior cloning of self-generated dialogee.. The qualitative analysis with generated dialogue examples can be found in Appendix B
5.1 EXPERIMENTAL. SETUP
We implement GPT-Critic based on the HuggingFace Transformers library (Wolf et al.) 2019) and codebase of UBAR (Yang et al., 2021), which is a GFT-2-based current state-of-ther -ht---rttt--tttt- end-to-end task-oriented dialogue agent for the MultiWOZ 2.0 dataset. For the generative pre-trained language model, we use DistilGPT2 (Sanh et al. 2019). a distilled version of GPT-2. Figure 2 shows the ar- chitecture of our policy and critic network based on GPT-2. We design the parameterization of the critic network to share the parameters of the Transformer layers of GPT-2, where the parameters of the Transformer layers only updated during the policy improvement step. For the hyperparame-
6

---

Published as a conference paper at ICLR 2022
Inform
Success
TRAINING DATA
92.10
80.45
UPDATED DATA (1)
93.76
83.40
UPDATED DATA (2)
95.35
86.04
UPDATED DATA (3)
95.85
87.05
Table 1: The performance of training dataset and self- eeff-generated dialogues being used for each iter- ation of policy improvement step.
Inform
Success
BLEU
Combined Score
POLICY ITERATION (1)
87.47
74.43
17.61
98.56
POLICY ATERATION (2)
88.60
76.43
17.72
100.23
POLICY ITERATION (3)
89.83
77.33
17.36
100,95
POLICY ITERATION (4)
90.07
76.63
17.78
101.13
Table 2: Experimental results for the policy improvement with GPT-Critic MultWWOZ dataset.
ters of fine-tuning the GPT-2 model, we follow the setting in the public code of UBAR (Yang et al, 2021 We use N = 5 for the number of candidate actions {ax 1 and the set of candidate actions are constructed by vanilla softmax sampling from the policy, rather than beam search, to collect diverse actions. For each behavior cloning iteration, all models fine-tuned with training dataset from the traitre GPT-2 and early stop according to the loss on the validation set.
5.2 EVALUATION ON THE MULTIWOZ DATASET
We evaluate our algorithm on the MultiWOZ 2.0 dataset, which is one of the representative task- oriented dialogue benchmarks. The MultiWOZ 2.0 is large-scale multi-domain Wizard-of- Oz dataset, where a tourist (i.e. user) converses with a clerk (i.e. system) at the information center in a touristic city. It consists of 4438/1000/1000 dialogues for training/valiidationnestnng For end-to-end evaluation on the MultiWOZ. 2.0 dataset, we use the following automatic evaluation metrics: 1) In- form: evaluates whether the system provides an appropriate entity, 2) Success: evaluates whether the system answers all the requested information, 3) BLEU: measures the fluency of the generated response (Papineni et al. 2002). We also report the Combined Score as an overall quality measure (Combined = foneeedd=(nnoooo (Inform- + Success) x 0.5 + BLEU).
We compare the performance of GPT-Critie with the following algorithms: SFN+RL (Mehri al. 2019), seq2seq network that incorporates several pre-trained dialogue modules into a neural dia- logue model, 2) DAMI) (Zhang et al., 2020), the domain-aware multi-decode network with multi- action data augmentation method, 3) SimpleTOD (Hosseini-Asl et al.) 2020). a GPT-2-based end- to-end dialogue agent that all sub-tasks recast as single sequence prediction problem, 4) SOLOIST (Peng et al., 2021), GPT-2-based end-to-end dialogue agent with transfer learning and machine teaching, 5) MinTL (Lin et al., 2020), an efficient dialogue state tracking method with a minimal generation length by predicting the difference between old and new states, 6) UBAR (Yang al. 2021]. a GPT-2- dialogue turn. We implement our algorithm into the codebase of UBAR (Yang et al., 2021), and the result of UBAR is reproduced by adapting its code to the same evaluation settings as other pa- per:"] Moreover, we also compare the data augmentation method, DATA AUGMENTATION, which is naively fine-tuning the GPT-2 model with additionally generated data by vanilla softmax sampling from the trained policy.
end-to-end dialogue agent that leverages the entire dialogue session of every
In addition, we also compare with recent offline RL algorithms that are free from the issue of di- verging from human language: 1) CRR (Wang et al., 2020). faluerfilteedd regression method that
performs weighted behavior cloning of offline dataset, 2) Decision Transformer (Chen et al.) (02T), a Transformer-basee architecture that casts the problem of RI as conditional sequence modeling- For fair comparison, we use the same pre-trained GPT-2 model as policy network to train the CRR and the Decision Transformer. Morcover, to show that the policy-gradient-bare algorithms suffer from diverging from human language, we also provide examples of responses generated by seiiccggradiettii based standard RL algorithm in Appendix D
standard RL
'The score reported in the UBAR paper is the result of using the true dialogue state for DB search. In order to compare under the same conditions with other algorithms, we record the result of UBAR to use the predicted dialogue state for DB search.
7

---

Published as a conference paper at ICLR 2022
Algorithms
Inform
Success
BLEU
Combined Score
S MMerri 2019)
73.80
53.60
16.90
83.10
DAMD (hhagg ct al.
2020)
76.40
60.40
16.60
85.00
SIMPLETOD (Hossein- Ass et al. 2020)
84,40
70.10
15.01
92.26
SOLOIST* (Peng et al. 2021)
85.50
72.90
16.54
95.74
MINTL * (Lin et al.
2020)
84,88
74.91
17.89
97.78
UBAR (Yang et al.
2021
87.47
74.43
17.61
98.56
DATA AUGMENTATION
87.27
74.23
17.33
98.08
CRR (Wang et al.
2020)
87.00
74.23
16,52
97.14
DECISION TRANSFORMER (Chen al. 2021)
87.53
74.33
17.64
98.57
GPT-CRITIC
90.07
76.63
17.83
101.13
Table 3: End-to-end response generation results on MultiWOZ. 2.0 dataset. The results with are from original papers. All other results are averaged over three independent runs.
Table D and Table 2 show the results of policy iteration. Table I shows the performance of training dataset and critic-guided self-generated dialogues being used for each policy improvement step. Table reports the intermediate performance of behavior cloning of training dataset and critic- guided gelf-generated dialogues in each of the policy iteration. As shown in Table [ and Table 2] the performance of critic-guided self- self-generated dialogues is improved gradually; the performance of GPT-Critic is also consistently improved through the behavior cloning of improved dataset.
Table summarizes the overall performance of GPT-Critic and baseline algorithms in end-to-end response generation setting, where the generated dialogue state and generated dialogue act are used for the DB search and response generation. The results show that GPT-Critic achieved the best performance in terms of inform rate, success rate, and combined score. Moreover, the performance of GPT-Critic the BLEU score matches those of other pre-trained LM-based methods, since GPT-Critic inherits GPT-2's s ability to generate human- like responses through the behavior cloning of responses generated by GPT-2. The results show that GPT-Critic improves the task performance of the agent without the issue of diverging from human language. In addition, as can be shown in Table the naive data augmentation is not effective since it will not change the GPT's sampling distribution in principle.
For the results of offline RL baselines, CRR and Decision Transformer show the results that do not diverge from human-language, since their policy is also trained by behavior cloning. However, both algorithms show limited performance because they perform behavior cloning on a fixed dataset. CRR has achieved remarkable success in continuous control tasks by performing weighted behavior cloning of training dataset filtered by critic, but it does not effectively perform the task-oriented dialogues because of data scarcity. Furthermore, to evaluate the Decision Transformer, we adopt a delayed return where the agent receives the cumulative reward at the end of dialogue, since the agent cannot observe user goal. Therefore, without observing the user goal at test time, Decision Transformer reduces to the behavior cloning of successful dialogues.
5.3 EVALUATION ON CONVLAB EVALUATOR
In order to evaluate the performance of dialogue agents in an end-to-end fashion, we conduct simulator-based evaluation on ConvLab (Zhu et al.] 2020). ConvLab is an open-source toolkit that enables to build task-oriented dialogue systems and perform an end-to-end evaluation. The simulator-based evaluation is more reliable than dataset-based automatic evaluation because eval- uates the performance while interacting with the user simulator. To interact with dialogue systems, ConvLab provides an agenda-basee user simulator (Schatzmann et al.) (007) that consists of a BERT (Devlin et al.) 2019) for NL.U, rule-based policy, and a template-basse based NLGG We compare the per- formance of GPT-Critic with baseline algorithms interacting with the same user simulator and user goals. We report the results with the following metrics: 1) Complete: evaluates whether the sys- tem completes the goal, 2) Success: evaluates whether all the user requests have been informed and the booked entities satisfy the constraints, 3) Book: evaluates how many booked entities satisfy the user constraints, 4) Inform (Precision Recall / F1): evaluates how many user requests have been informed, 5) Turn (success / all): evaluates the average turn number for successful/all dialogues.
We describe the performance of GPT-Critic and baselines in Table 7 1000 runs with randomly sampled user goal. The results show that GPT-Critic achieves the best
Each algorithm is tested for
8

---

Published as a conference paper at ICLR 2022
Algorithms
Complete
Success
Book
Inform
(P/R
Turn (suce all))
DAMD
49.40
44.40
50.78
0.55 /..55
13.72/ / 26.82
MINTL
71.40
68.10
65,38
0.65 0,80 0.69
15.67 / 20.72
UBAR
79.80
74.30
80.82
0.74 0.86 0..76
14.22 / 18.08
CRR
78,20
72.60
82.21
0.74 0.84 0.76
13.61 / 17.96
DECISION TRANSFORMER
81.30
75.30
83.54
0.7370.87 70.77
14.81718.00
GPT-CRITIC
84.30
77.70
85.42
0.74 0.90 0.79
16.27 19,42
Table 4: Experimental results of simulator-based evaluation on ConvLab. All results are averaged over 1000 dialogues.
DAMD
CRR
MINTL
DECISIGN-TRANSFOREE TRANSFORMER
GPT-CRITIC
4.
4.0
3.5
Apprepriateness
Fluency
Figure 3: Human evaluation results for appropriateness and fluency. The results represent the mean and standard error for each algorithm.
performance in all metrics related to task accomplishment. However, they also show that GPT-Critic takes longer dialogue turn for the task accomplishment because GPT-Critte is trained by maximizing the success rate without considering the dialogue turn.
5.4
HUMAN EVALUATION
We also conduct human evaluation on Amazon Mechanical Turk (AMT) assess the quality of gen- erated responses of HPP--ritic and baseline algorithms, using the evaluation protocol as in (Yang) jet al. 2021: Lin et al. 2020 Zhang et al., 2020). Specifically, human workers on AMT were asked to read the context and generated response by interactive simulation via ConvLab, Lab, then score the following two evaluation metrics on a Likert scale (1-5): 1) Appropriateness: evaluates whether the generated responses appropriate for the given context, 2) Fluency: evaluates whether the gen- crated responses are comprehensible and human-like. We compare the performance of GPT-Critic with same baselines ConvLab evaluation. Figure B3] summarizes the overall results of human eval- uation, where 60 workers evaluate the quality of 30 randomly selected dialogues for each algorithm. The results show that GPT-Critic significantly outperforms baseline algorithms in appropriateness which is related to task accomplishment, Moreover, the result of fluency shows that GPT-Critic does not hurt the agent's capability to generate human-like sentences.
CONCLUSION
We presented GPT-Critic, an offline RL algorithm for task-oriented dialogue system, which can be adopted for any generative pre-trained language model. GPT-Critic aims to learn an end-to-end task-oriented dialogue agent without the issue of diverging from human language. GPT-Critic starts with fine-tuning the GPT-2 model and learning the critic using the dialogue corpus. Then, GPT- Critic updates the policy through the behavior cloning of the critic-guided -effraeneeteed responses, thus it is essentially free from the issue of diverging from human language. In the experiments, we demonstrated that GPT-Critic outperforms the otate-of-the-aar algorithms in the task-oriented dialogue benchmarks including MultiWOZ. 2.0 and ConvLab.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation (NRF) of Korea (NRF- 2019R A22111224187634, NRF-201MG31A11999555 and Institute of Information & communications Technology Planning & Evaluation (IIT)) grant funded by the Korea (MVeMmeen(MSIIT) (No.2019- 0-00075, No.2020-0-00940, No. 2002000----006)
REFERENCES
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In Thirly-Fifth Conference on Neural Information Processing Systems, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2rXiv:2005.141100 2020.
Pawel Budzianowski, Ssung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz-a a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. arXiv preprint 1rXiv:1810.000
2018.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint urXiv:2106.01344 2021.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 14233 moclaatholooy.ozg//11
https:
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2002-20222020 2062. PMLR, 2019.
Wonjoon Goo and Scott Niekum. You only evaluate once: a simple baseline algorithm for offline RL. In 5th Anmual Conference on Robot Learning, 2021.
Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2. Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pp. 583-592 2020.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for fask-triented dialogue. arXiv preprint 2rXiv:2005.00796 2020.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. Bayes-adaptive monte-carlo planning and learn- ing for goal-oriented dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint arXiv: (wt.//966.....2 2019.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandcharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline rein- forcement learning. arXiv preprint arxiv:2010.05848, 2020.
John F Kelley. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS), 2(1):26 41, 1984.
Young-Bud -um Kim, Sungjin Lee, and Karl Stratos. OneNet: Joint Domain, Intent, Slot Prediction for Spoken Language Understanding. In 2017 IEEE Automatic Speech Recognition and Understand- ing Workshop, 2017.
10

---

Published as a conference paper at ICLR 2022
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. arXiv preprint arXi::1906.00949. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.4477 2020.
Byungjun Lee, Jongmin Lee, Peter Vranex, Dongho Kim, and Kee-Eung Kim. Batch reinforcement learning with hyperparameter gradients. In International Conference on Machine Learning, pp- $725-5735 PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint WrXiv:2005.01645, 2020.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruy Batra. Deal or no deal? end- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, 2017.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. Mintl: Minimalist transfer learning for task-oriented dialogue systems. arXiv preprint arXiv:2009. 12005. 2020.
Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. Structured fusion networks for dialog. arXiv preprint arXiv:1907.10000 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Baolin Peng. Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. Soloist: Buildingtask bots at scale with transfer learning and machine teaching. Transactions of the Asso- ciation for Computational Linguistics, 9:807-824, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 19Xiv:1910.00177, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. Agenda- based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pp. 152, 2007.
Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmalcki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ziyu Wang. Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari. Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006.1511 2020.

---

Published as a conference paper at ICLR 2022
Tsung-Hsibe Hsien Wen, Milica Gašié, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. Semantically Conditioned LSTM-basee Natural Language Generation for Spoken Dialogue Sys- tems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pro- cessing, 2015.
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. The Dialog State Tracking Challenge. In Proceedings of the SIGDIAL 2013 Conference, 2013.
Jason D. Williams and Steve Young. Partially observable markov decision processes for spoken dialog systems, Computer Speech and Language, 21(2):393-422, 2007. ISSN 0885-2308.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the the--ttherrttt- natural language processing. arXiv preprint arXiv:19100037771, 2019.
Yunyi Yang, Yunhao Li, and Xiaojun Quan. Ubar: Towards fully end-to-end task-oriented dialog systems with gpt-2. In AAAI, 2021.
Denis Yarats and Mike Lewis. Hierarchical text generation and planning for strategic dialogue. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Yichi Zhang, Zhijian Ou, and Zhou Yu. Task-triented dialog systems that consider multiple appro- priate responses under the same context. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 9604-9611 1, 2020.
Tiancheng Zhao, Kaige Xie, and Maxine Eskénazi. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. Convlab-2: An open- toolkit for building, evaluat- ing, and diagnosing dialogue systems. arXiv preprint arXiv:2002.04793, 2020.
12

---

Published as a conference paper at ICLR 2022
A POLICY IMPROVEMENT THEOREM
Theorem I (Policy Improvement) Given a policy a and the number of sampling actions N 2 1, If update the new policy N by
new vs, N
(-|s) = arg max ("(s.a) a) aEfas) {ax) ~x(s) (ax))~<(())
then Q new (s, (s, a) Vs, always holds. Furthermore. any N, M such that N 2 M Q (8,0) 2 QTM (s, a) Vs, a always holds.
hew
Proof. For any 8, 2 M,
Q n (8,a)
Ep R(st, ae) + 7Bu 2f
" (1 1Q 35th (s*+1; 8, =
Ep [(st, ux) + 7 (m,)) 2 -n (n
max Q" Q"*sst1,a']] = 8, d'E(a,) AM
I Ep [(st, at) + 7BB
max d'E(a))
Q Q"(ss1,,a')l [|ss 8, a
Er R(se + yT - 2 (*++1, pow offT) = 8, =
Er R(se, ar) + + ( Bupgy and ( (R(8)=1; (1l
10" 1s 8,
++1 Ep
Bor ( (er+s))) - "H(""""H((_ ar + + Eo Q2" ((4))l= 8,0r 5,0r=
S Ep
Σ Bay+i (1.41)17 T- (R(,,, a7) + +g 2 Ennny now (8 o(4)) 8,
Ep
Σ B
"R(s, a.)) + 2 E (8143, (143)|s; = 8, =
S E.
E.
(14 T "R(o, )lss 8, =
Q" N (8)
For the case of N = 1, note that w-low is simply reduced to the ग, which concludes the proof:
Q a) 2 3 (s.a) 2 T (s, a) == (s, a) for all s,a, N 2 M 2 L.
13

---

Published as a conference paper at ICLR 2022
B QUALITATIVE ANALYSIS OF SELF-GENERATED DIALOGUES
WOZ20322
Original Dialogue Self-generated Dialogue
would like a moderately prices restaurant in the cast part of town. would like the phone number and postcode, please.
User Utterance
Dialogue State
[restaurant] pricerange moderate area east [restaurant] [select] food [inform] choice area have [value choice] matching your request in the [value areal part of town. do you prefer [value food] [value food]? [restaurant] [inform] food price area phone name postcode [value name] is a [value price]] [value_ food] restaurant in the [value area] part of town. their phone number is salue_phone]] and their postcode is [value.pone -aluuepostccoee..
Dialogue Act
System Response
Dialogue Act *
System Response
SNG01237
Original Dialogue / SSelf-generated Dialogue
User Utterance
am looking for the nearest police station
Dialogue State
Dialogue Act
[police] [inform] phone
the nearest police station is in parkside cambridge, the telephone number is [value_phone].
System Response
Dialogue Act *
[police] [iiffrm] name phone address postcode
the nearest police station is vvaue_namel, located in [value [ailue__ddess]]] pratue_postoode).
System Response
their phone number is [value phone]. is there anything else can help you with?
PMUL22640
Original Dialogue / SSef--eneratee' Dialogue
User Utterance
heard cityroomz is good hotel. you give information this hotel?
Dialogue State
[hotel] name cityroomz [hotel] [inform] area price
Dialogue Act
System Response
it is located in the [value_ares and is priced in the [value [value.price] range.
Dialogue Act
[hotel] [inform] area name internet price parking type stars [offerbook]
[value_name] is [value-type] in the [value area]. it is [value-price]] has ivalue_stars)
System Response
stars, and offers free wifi and parking. would you like me to book it for you?
MUT22024
Original Dialogue / Self self-generated Dialogue
User Utterance
am looking for a hotel called the and guest house. can you help me out?
Dialogue State
[hotel] name a and b guest house
Dialogue Act
System Response
yes, what would you like know about it?
Dialogue Act
[hotel] [inform] area address name price
System Response
[value_name] is located in the [valuc_arca] at [valuc_address), it is [valuccprice].
Table 5: Examples of the original dialogue and the critic-guided self-generated dialogue on the training dataset of MultiWOZ. The dialogue act and system response with * denote examples of self- generated actions (dialogue act and system response) by GPT-Critic. All system responses represent delexicalized responses.
In this section, we provide a qualitative analysis on the critic-guided self-generated responses in GPT-Critic. We show the critic-guided self-generater dialogue examples in Table that illustrates how GPT-Critic improves the performance through the behavior cloning of self-generated dialogues, compared with unsuccessful dialogues the training dataset of MultiWOZ. Each example demon- strates the critic-guided self-generated dialogue act and delexicalized system response, compared with the original dialogue. The generated responses contain all the requests of user with abundant information, whereas the original responses of unsuccessful dialogues do not contain all the re- quested information. GPT-Critic improves the performance through the behavior cloning of these
14

---

Published as a conference paper at ICLR 2022
revised responses. Moreover, Table 5 shows that the generated dialogues do not diverge from hu- man language. Since GPT-Critic updates the policy through behavior cloning of the self-generated human-like responses, GPT-Critic is essentially free from the issue of diverging from human lan- guage.
C QUALITATIVE EXAMPLES OF STANDARD REINFORCEMENT LEARNING ALGORITHM
In this section, provide examples of responses generated by standard RL algorithm (REIN- FORCE) to show that the graery-ggddeeettbgee standard RL algorithms suffer from diverging from human language. As can be shown in Table 66] salicy-gradeent-asee RL algorithm generates re- sponses which is diverging from human language.
WOZ.20010
Original Dialogue/ / Generated Dialogue
User Utterance
need restaurant.
Dialogue State
Dialogue Act
[restaurant] [request] area in what part of town?
System Response
Dialogue Act *
[restaurant] [request] name do you know what you are looking for? can direct you to something you will need. if you think of it, please let me know, goodbye. hope everything works. thank you for calling, goodbyc. hope everything works.
System Response
PMUL.0799
Original Dialogue/ / Generated Dialogue
awesome. also need a place to eat the moderate range. am thinking maybe the city centre.
User Utterance
[restaurant] pricerange moderate area centre [train] day wednesday arrive 17:00 destination cambridge departure london liverpool street people I [restaurant] [select] [inform] area food choice price there are (value_choice] [value pprce] restaurant in the [value area), 4 serve [value food], and serves [value food] food. which would you prefer?
Dialogue State
Dialogue Act
System Response
Dialogue Act
alright, have wonderful day! you have wonderful day well. have great day!!!! you have nice day as well!!!!!!!!!!!!!
System Response
PMUL,3700
Original Dialogue Generated Dialogue
User Utterance
need hotel and need free wifi and parking.
Dialogue State
[hotel] parking yes internet yes [hotel] [inform] choice [request] stars type have found [value_choice] place that fit your criteria. to further narrow your scarch, please provide specific hotel type and the amount of of the hotel you would like. hemp bar ask ask ask ask ask ask ask ask ask ask ask ask ask ask ask cause get call us carizza our service chatosa! we haill us love!WeXT do!Win!
Dialogue Act
System Response
Dialogue Act *
DrawLineWeTi have online we rock! sign us hemp! sign us hemp! us hemp! sign hemp! sign hemp! sign hemp! sign hemp! sign hemp! sign hemp
System Response
Table 6: Examples of the original dialogue and the generated dialogue by RI. algorithm (REINFORCE) on the training dataset of MultiWOZ. The dialogue act and system response with denote examples of generated actions by REINFORCE, All system responses rep- resent delexicalized responses.
based
15

---

Published as a conference paper at ICLR 2022
D
EVALUATION FOR THE QUALITY OF GENERATED DIALOGUE STATES AND DIALOGUE ACTS
Algorithms
Joint accuracy
Slot accuracy
Dialogue act fl
UBAR
47.50 + 1.39
95.77 + 0.13
50.94 + 0.17
GPT-CRITIC
47.37 1.14
95.80 + 0.11
51.81 0.09
Table Experimental results for the quality of generated dialogue states and dialogue acts Mul- tOW 2.0 dataset.
We additionally conducted the experiments with UBAR and GPT-Critic explicitly evaluate the generated dialogue states and dialogue acts (rather than evaluating the final system response). The table below shows the performance for predicted dialogue state(Joint accuracy / Slot accuracy) and predicted dialogue act (Dialogue Act fl), where the mean performance and the standard eтror are reported. As the table presents, GPT-Critic outperforms UBAR on Dialogue Act fll, which measures the performance of dialogue policy prediction. However, in the case of dialogue state tracking, there no significant performance gap between GPT-Crite and UBAR since our GPT-Critic revises only the dialogue act and system response (which are considered as action in GPT-Critic) but not the dialogue state in the dataset.
16


Reviews:
[{"review_id": "X6wet93oBR", "reviewer": "Reviewer_JTos", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a nice way to handle large scale natural language action space for RL. An action in this model is a system action plus response (a sequence of tokens) rather than pre-specified variables in discrete latent models for dialogue (LaRL, etc.), so the proposed model can be easily incorporated into a transformer-based language model (e.g. GPT-2).\n\nMy biggest concern is the q-value estimation. Estimating the Q value for an offline dataset is hard due to the problem of overestimation, which is already pointed out by the authors in the paper. However, the authors didn't describe in detail how they mitigate this problem. The only sentence in the paper about this is \"However, we avoid this OOD problem by xxx ... revised by generated system response and evaluated reward using offline automatic evaluation\". More details are needed, and ablation study about this is needed. Otherwise, I don't know if the proposed method can generalize to other natural language action space applications where the ground-truth reward function is unknown for offline automatic evaluation.\n\nMy second concern is the exploration perspective of self-generation. The paper mentioned that {a_k}^N is a set of N response candidates generated from the current policy \\pi. What number is N set to? What is the impact of N to the final performance? Moreover, is{a_k}^N sampled by beam search or vanilla sampling? I'd assume that we want to sample {a_k}^N that is diverse enough, rather than similar system responses with only one or two different words. More discussion about the generation is needed.\n\n", "summary_of_the_paper": "The paper works on offline reinforcement learning for natural language action space setting, particularly for task-oriented dialogue management. The paper nicely incorporate the policy network (to sample agent response) and the q network (to evaluate the agent response) into a single GPT-2 network and propose a policy interation algorithm to optimize both the q and policy network.\nDuring policy evaluation, the q network is updated with sampled system actions and responses. During policy improvement, the sampled system actions and responses with maximum q values are used as labels to update the policy network. The model achieves SoTA performance on MultiWOZ dataset.", "summary_of_the_review": "1. Good method to incorporate policy network and q network into a GPT-2 model to handle large scale natural language action space.\n2. Have concern on offline q-value estimation and sampling of response candidates. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NKM-5v8he7h", "reviewer": "Reviewer_ujyM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on improving the dialogue policy together with the responses by utilizing a pre-trained language model and offline RL.\nThe claimed contributions include:\n1) The proposed method is free from the common issue of diverging from human language, because it learns from the sentences sampled from the pre-trained LM.\n2) The proposed method outperforms other SOTA models in offline and interactive online settings, MultiWOZ and ConvLab respectively.\n\nThe proposed method is reasonable and moderately novel. The experimental results are promising for both settings.\nHowever, there are unclear parts to be addressed or clarified.\n\n- Because the policy learning procedure utilizes the additionally generated dialogue acts and corresponding responses, it is easy to think that naively fine-tuning the GPT-2 model on the additional generated data may also improve the dialogue model performance in terms of its policy and responses (similar to a data augmentation method). Did the authors try this as another compared baseline? This method should be included in the experiments in order to justify the proposed RL approach is necessary.\n\n- The paper mentioned that the standard RL methods easily fail and generate responses diverging from human language, even when fine-tuning a pre-trained LM. Hence, it will be better to additionally include the results of other standard RL algorithms for better justifying this claim. (Current experiments only include the results of models that are free from this issue.)\n\n- In the experiments in MultiWOZ, this paper only evaluates the response generation results. However, evaluating dialogue policy is also important to justify the learned policy is suitable. It is unclear why the authors only show the response generation results.\n\n- The experiments contain two setups, one is offline response evaluation via MultiWOZ, and another is interactive simulation via ConvLab. Both settings show the better performance of the proposed method. The paper can be better if adding the real-user interactions, because the performance may be different between the simulation environment and the real-user interactions reported by prior results (DSTC in ConvLab). Conducting real-human interactions can better justify the effectiveness of the proposed RL method in practical scenarios.\n\nIn sum, the proposed method is relatively novel and the idea is reasonable. The performance seems promising in both settings.\nHowever, the paper does not include detailed descriptions about the proposed method, making readers not easy to understand.\nAlso, some additional experiments need to be added in order to better justify its claims.", "summary_of_the_paper": "This paper proposes an offline RL method applied to an end-to-end task-oriented dialogue model, where the proposed GPT-Critic is built on GPT-2 and fine-tuned on the self-generated sentences for policy updating. \nThe paper claims that it is free from the issue of diverging from human language (a common issue in standard RL), because it learns from the sentences directly sampled from the pre-trained language model.\nThe conducted experiments show that the proposed model achieves better performance compared to other task-oriented end-to-end dialogue models in both offline and online settings (MultiWOZ and ConvLab respectively).", "summary_of_the_review": "The idea is reasonable and moderately novel. The conducted experiments demonstrate the better performance of the proposed model in two different settings.\nThe paper misses some details when describing the proposed method, making readers difficult to fully understand its idea.\nAlthough the experiments already include a lot of baselines, there are still some results to be included for fair comparison (simple data augmentation method).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ey06tojuBuq", "reviewer": "Reviewer_fa2f", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper introduces a critic value function on the top of the pretrained GPT-2 task-oriented dialogue agent, to guide the response generation. \n\nWeakness:\n1. The novelty may be limited by only adding a critic value function on the top of the existing work.\n\n2. Is it possible to add human evaluation for at least of the datasets? (fixed in the rebuttal)\n", "summary_of_the_paper": "To tackle the response generation diverging issue from human language, this paper introduces a critic on the top of the pretrained GPT-2 task-oriented dialogue agent, and demonstrates promising empirical results on two datasets (MultiWoz and ConvLab).  ", "summary_of_the_review": "The motivation (tackling the response generation diverging issue from human language) of this work is clear, and the solution is also good, by adding a critic on the top of existing pre-trained GPT-2 dialogue agent, and shows promising empirical results on the automatic evaluation of two datasets. However, the main concern for this work, is is quite incremental with limited novelty, and also lacks human evaluation to verify its effectiveness, may recommend for a workshop paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0D3R6eAVOWo", "reviewer": "Reviewer_u1n3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is very simple and should be easy to implement. The experimental results seem promising. However, I do have a couple of concerns.\n\nFirstly, I think that some important details are missing in the paper. For example, what is the action space of the agent? Do you treat the conjunction of a dialogue act and a system response as a single action? If so, how exactly are the candidate actions generated? Is some kind of beam search employed? I think some actual examples of actions (and rewards) would help the reader understand the proposed method more clearly.\n\nThe paper does not really describe how the rewards are computed, either. In particular, I am wondering how the reward for the newly selected action is computed. Is it given to the agent by an external program?  Then, it seems to me that the whole training procedure is more like (a somewhat restricted version of) actor-critic-based reinforcement learning than offline reinforcement learning (in which the agent cannot interact with the environment). If that is the case, what is the novelty of the proposed method?\n\nThe authors claim in Table 3 that their proposed approach gives much better results than UBAR, but the original paper of UBAR (Yang et al., 2021) reports much better results (e.g., Inform score of 95.4). Why is there such a big difference?\n\nAlgorithm 1 states that the policy is updated by behavior cloning until \"convergence\". I am wondering if it causes any overfitting problem. Is overfitting the reason why the whole training process is stopped at the fourth iteration (Table 2)?\n\nMinor comments:\n\np. 1: outperforms the state-of-the-art -> outperforms the state of the art;\np. 1: not trained to for -> not trained for?\np. 2: fine-turning the GPT-2 -> fine-turning GPT-2? fine-turning the GPT-2 model?\np. 2: generates strategically -> generates a strategically;\np. 2: Pr(O_{t+1} ... ) -> should not be italic?\np. 3: by training action-value -> by training the action-value;\np. 3: of critic network -> of the critic network;\np. 4: for $i$-th -> for the $i$-th;\np. 4: in task-oriented -> in the task-oriented?\np. 4: generated system response -> generated system responses?\np. 4: from (Zhao et al., 2019) -> from Zhao et al. (2019);\np. 4: prohibitory -> prohibitively?\np. 4: over response candidates -> over the response candidates?\np. 4: updated policy by above -> the updated policy by the above?\np. 5: on MultiWOZ domain -> on the MultiWOZ domain;\np. 5: ConvLab framework -> the ConvLab framework;\np. 5: HuggingFace Transforms library -> the HuggingFace …;\np. 7: prices -> priced?\np. 8: user goal -> the user goal?\np. 8: with following -> with the following;\np. 8: the all -> all the;\np. 8: whereas original -> whereas the original?\np. 9: straightforward -> straightforwardly?\np. 8: large scale -> large-scale?\n", "summary_of_the_paper": "This paper presents a reinforcement learning-based approach to building a task-oriented dialogue agent. Given a dialogue dataset annotated with rewards, the state-action value function is first trained by minimizing temporal differences. Then, a new training dataset is created by using the best actions selected among the candidates generated by the current policy (i.e., the language model). The policy is then updated by behavior cloning using the created dataset, and the whole process is repeated. The authors have conducted experiments using MultiWOZ and ConvLab and shown that this iterative process improves the performance of the agent.", "summary_of_the_review": "The proposed algorithm is simple and seems effective. However, some important details are missing and the novelty is not very clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are all weakly positive. The author response clarified important aspects of the paper. The new human evaluation was critical. However, the human evaluation result presentation is flawed: presenting Likert scores as means does not reflect them well. The authors should use something similar to a Gantt chart to fully reflect the distribution across Likert categories. Another detail in the human evaluation that are troubling: it does not reflect interaction with the system, but judgements through observation. Therefore, the human evaluation does not reflect the ability of the learned dialogue system to interact with users. Overall, the paper makes a nice, original contribution, but despite author improvement there are evaluation flaws (even if they are common in papers using these benchmarks).
Decision: Accept (Poster)