Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems
Authors: Yihao Feng
Shentao Yang
Shujian Zhang
Jianguo Zhang
Caiming Xiong
Mingyuan Zhou
Huan Wang
Venue: ICLR 2023 poster (2023)
Keywords: task-oriented dialogue
reinforcement learning
reward learning

Abstract: When learning task-oriented dialogue (ToD) agents, reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user-specific goals. Prior works mainly focus on adopting advanced RL techniques to train the ToD agents, while the design of the reward function is not well studied. This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end (E2E) ToD agents. Specifically, we introduce two generalized objectives for reward-function learning, inspired by the classical learning-to-rank literature. Further, we utilize the learned reward function to guide the training of the E2E ToD agent. With the proposed techniques, we achieve competitive results on the E2E response-generation task on the Multiwoz 2.0 dataset. Source code and checkpoints are publicly released at https://github.com/Shentao-YANG/Fantastic_Reward_ICLR2023.

Published as a conference paper at ICLR 2023
FANTASTIC REWARDS AND HOw TO TAME THEM: A CASE STUDY ON REWARD LEARNING FOR TASK- ORIENTED DIALOGUE SYSTEMS
Yihao Feng"
Shentao Yang" S Shujian Zhang", Jianguo Zhangt Caiming Xiong', Mingyuan Zhou?, Huan Wang' Salesforce Research The University of Texas at Austin
ABSTRACT
When learning task-oriented dialogue (ToD) agents, reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user- specific goals. Prior works mainly focus on adopting advanced RL techniques train the ToD agents, while the design of the reward function is not well studied. This paper aims at answering the question of how to efficiently learn and lever- age reward function for training end-to-end (E2E) ToD agents. Specifically, we introduce two generalized objectives for reward-function learning, inspired by the classical learning-to- to-rkkkk literature. Further, utilize the learned reward function to guide the training of the E2E ToD agent. With the proposed tech- niques, achieve competitive results the E2E the Multiwoz 2.0 dataset. Source code and checkpoints are publicly released at http://ww..oooeseeaaaaaaa////e/-
task on
INTRODUCTION
The bloom of pre-training language models (e.g., Devlin et al., 2018; Lewis et al., 2019; Radford et al., 2019; Zhang et al., 2022c) have significantly pushed the boundaries of natural language pro- cessing (NLP) on real-world tasks. Among all the promising potentials, one important application is the task-oriented dialogue (ToD) systems, which interact with the users in multiple turns via natural languages accomplish tasks such weather inquiry, ticket booking, or schedule planning (Chen et al., 2017; Kwan et al., 2022).
Traditionally, the problem of ToD is decomposed into several sub-task (Smith & Hipp. 1994; Young et al., 2013): natural language understanding (NLU) for understanding turn-level user intents or slot values (Tur & De Mori, 2011; Casanueva et al., 2020), dialogue state tracking (DST) for track- ing user belief state across multiple dialogue turns (Zhang et al., 2019: Zhu et al., 2020). dialogue management (DM) for choosing system actions to take (Peng et al., 2017: Zhao et al., 2019), and natural language generation (NLG) for mapping system actions to natural language responses (Wen et al., 2015; Zhang et al., 2020). This pipeline approach, however, requires intensive structural de- signs and comprehensive data annotation for model training (Kwan et al., 2022). Recently, there has been a growing interest in building end-to and-to-end (E2E) ToD agents, which directly generate responses based the natural language conversation mixing user utterances and past responses. Apart from this structural simplicity, many of the E2E ToD models can utilize the pre-trained language models and are simply trained by supervisedly fine-tuning the pre- praitrdined models on the ToD datasets (e.g., Hosseini- Asl et al., 2020: Ham et al., 2020; Lin et al., 2020; Peng et al., 2021).
Due to the intrinsic similarity between dialogues and sequential decision-making, reinforcement learning (RL) methods are naturally employed to train dialogue systems and have achieved some success (e.g., Williams Young, 2007; Georgila & Traum, 2011: Zhao al., 2019). Since inter- acting with users during the training process is mostly impractical, offline RL (Lange et al., 2012; Levine al., 2020), i.e., RI. on static datasets, has recently been adopted to train E2E ToD mod- els (e.g., Jaques et al., 2019; 2020; Ramachandran et al., 2021; Snell et al., 2022a;b; Jang et al., 2022). Although this direction already presents promising empirical results, an open question exists on how to properly design the reward function for the underlying (offline) RL. Existing works (e.g.,
"Equal Contribution. Corresponds to {yihao.ac www.g..aa..oooo. com, Daaettaoyang@mmen uuw.xs.eeet.

---

Published as a conference paper at ICLR 2023
Wu et al., 2019c; Jang et al., 2022; Snell et al., 2022b) manually design sparse reward function that only indicates whether the agent achieves the goal or not. Unfortunately, due to the delayed feed- back, learning from such sparse reward signal is itself challenging for RL agents (Andrychowicz et al., 2017; Liu et al., 2019; Durugkar et al., 2021). When applied to train the more complicated ToD agents, the sparse reward signal could lead to poor empirical performance (Takanobu et al., 2019; Wang et al., 2020a). To address this issue, we aim at answering the following question in this paper:
How to efficiently learn reward function and leverage it for training E2E dialogne agents?
We answer the first half of this question by introducing two reward-learning objectives, Rewa rdNet and Rewa roMLE, based on the classical learning-to-rank literature (Cao et al., 2007; Xia et al., 2008). Our desiderata a reward function that can "explain" some non-trivial preference- based ordering among multiple alternative dialogue trajectories, thus potentially allowing the re- sulting RL-trained ToD agents to have better-than-demt performance. We accomplish this goal by learning parameterized reward function on dialogue turns, from which the accumulated reward of a dialogue trajectory can reflect the preference among multiple alternatives. We answer the second half of the question by utilizing the learned reward function to guide the training of the E2E ToD sys- tem, with special considerations on the training stability. With these answers to the above question, achieve competitive results the E2E apponse-generation task the widely-used dialogue benchmark MultWWOZ 2.0 (Budzianowski et al., 2018). Several ablation studies and analyses are conducted to provide further insights into the proposed techniques.
BACKGROUND
Task-oriented dialogue as reinforcement learning. We formulate the problem of the ToD system as a partially observable Markov decision process (POMDP) (Kaelbling al., 1998), specified by M = (S, A. 0, P, R, 7), where state 8 E S consists of the previous dialogue history h and the user intended goal g specified prior to the start of the dialogue: o E O is the observation that can be the user utterance; action E A can be the system response or dialogue act; P(s s, a) is the underlying transition probability; R.h,, a, .) the intermediate reward function for taking action under dialogue history h and goal g; and E [0, 1] is the discount factor.
The dialogue history h4 at timestep t consists of all the previous observations and actions, i.e. h - {oo, ag...... 0 at-1, ot). Since the ToD agent cannot directly observe the user goal g, makes a decision based on the entire dialogue history ht so far. Specifically, the policy π is defined as mapping from h2 to probability distribution over A, i.e., π _ ((; hha). The training objective to find a policy that maximizes the expected (discounted) cumulative reward
((T) e E E 2 (atee,,a..))
9c.
where Ag is the distribution of goals and T is the number of turns in the dialogue trajectory.
Reward design and learning in ToD systems. Unlike the classical RL problems where the interme- diate reward function is well designed and provided, in ToD systems we can only get the evaluation
results at the end of the dialogue (Budzianowski et al., 2018). Consequently, most of the existing works adopt the manually designed intermediate reward function that only gives binary reward to indicate whether the dialogue agent achieves the goal or not (e.g., Weisz al., 2018; Wu et al., 2019c; Jang et al., 2022):
{{- Reonst 0, if goal g is achieved at timestep L if goal g is not achieved at timestep t,
R(h (1.9) = (RRcosst:
-Rconst:
where Reonst is a positive constant that can be 1. However, such sparse reward signal can be one of the reasons that the ToD agents from RL often have poor performance (Takanobu et al., 2019; Wang et al., 2020a). A similar issue is also observed in goal-oriented RL (Andrychowicz et al., 2017).
To address the above issue, few recent works focus on learning intermediate reward func- tion from demonstrations or mechanical dialogue assessments (e.g., Wang et al., 2020a; Ramachan- dran al., 2021), inspired by the woward-learning-fee wew---ro---peeeeeee
in RL (e.g. Christiano et al., 2017; Brown et al., 2019; 2020). More precisely, suppose we are given two dialogue trajec-
tories 7; and Tj, taking the form t; - (g))), (o6') .00 ('"),,...
...... (o oq )) We want learn ...
(3) (8)
2

---

Published as a conference paper at ICLR 2023
parametrized reward function Ro( at. with parameter 0,' such that Σi= Roloi " gg
(
Ei=o Ro(o," ( _a." .(0) when is preferred over Tj (denoted as > T,) and vice versa. Then one can follow the Bradley-Terry Terry model of pairwise preferences (Bradley & Terry, 1952) to train the reward function by minimizing the loss
exp EEi Rolo aaj (l1)))
((0) = - log
(I)
(ke() exp E7_o Ro(o (R) aa (g(
((O) can be interpreted as a pairwise ranking loss, which formalized binary classification in the problem of learning to rank (Herbrich et al., 1999; Freund et al., 2003; Burges et al., 2005).
MAIN METHOD
In this section, we first introduce two objectives for reward-function learning based on the classical approaches in the corming-to-rante (LTR) literature (Liu, 2009). Then we use MinTL (Lin et al., 2020) as an example to demonstrate how we can use the learned reward function as a plugin module to improve existing methods of training the E2E ToD models.
3.1 Two GENERALIZED OBJECTIVES FOR REWARD LEARNING
We introduce two objectives, RewardNet and RdME,, both of which can utilize multiple dialogue trajectories on each update for optimizing the reward function. Our motivation that, compared with the pairwise approach described in Eq. (1), these two objectives consider more in- formation at each training step, and thus can be more effective for reward learning and may lead to better solution under the stochastic training setting.
Setup. Assume that there N 2 2 dialogue trajectories, denoted by DN & (TI, .2..... .N). and each trajectory Ta has an automatic evaluation score S(r,). For simplicity, we assume that these N dialogue trajectories are of equal length T and already sorted by the automatic evaluation scores, ie.,, T1 > 72 > > TN. or equivalently, 5(1)) > S(12) > > S(TN). J. We denote the accumulated reward of dialogue trajectory т; from Rs J(;;; Ei= Ro(o!" (8) (c) ()) Our goal is to learn a reward function R. (o, a. g) such that the accumulated rewards those trajectories reflect the ranking order, i.e., J [[1 0) >
().
The proposed Rewar dleet objective for reward function learning is adapted from the ListNet loss (Cao al., 2007) in the LTR literature. Specifically, given N trajectories and their associated scores, we define the RewardNet loss as the cross entropy between (J(Ta 0) and (5(T))
RewardNeet
Gravaranat (0;
- Ps(Ti) ( log (P)()
with
Ps(n) ((t)) S(Te)) (T) ( ()(T))))()))))))))))))))((()))))
(J((T=))))
where (-)) is a monotonic positive function defined on R+ and (T(T:) is normalized probability vector defined by the true evaluation scores of those N trajectories. Note that when N 2 and $ is the identity function, RewardNet can be viewed as a soft version of the pairwise preference loss defined in Eq. (1), where the hard binary preference labels are replaced by {Ps(T)) This soft pairwise loss is adopted for reward learning in the recent CASPI paper (Ramachandran et al., 2021).
RewardMLE. The Rewar rMMLE objective based on the ListMLE loss (Xia et al., 2008), where we only utilize the ranking order in the batched dialogue trajectories Dx. rather than the original metric scores {S(T:)) Let y = rank(S) be the random variable that represents the ranking order of the dialogue trajectories (w(T,) = Vi, if the batched trajectories Dx are sorted). The RewardMLE objective defined as the negative log-likelihood of the ranking order y under the Plackett-Luce choice model (Plackett, 1975; Luce, 2012) induced by the accumulated reward of each trajectory {J(r: 0 Specifically, the loss is defined as
dRowardoLL (0: D>) A log P (y]{
use the belief state, action, and goal as the reward function inputs. The belief state is part the observation or. We also drop the dependency on h, for R₂ to simplify the reward function learning.
We use the Combi ned score (Mchri et al., 2019) S(T.). Detailed definition is delayed Section 5.
3

---

Published as a conference paper at ICLR 2023
with
P (vl (77:0)) II=- (b(((=0))/ (0 (((T):))))
where trajectories in DN assumed sorted descrihed in the problem setup, i.e,, 7I > > TV- Since Rewa roMLE only requires the ranking information derived from the raw scores, it is potentially more robust choice when the preference scores could be inaccurate.
In Eqs. (2) and (3), the monotonic positive function + transforms the unnormalized inputs {J(T;; 0) N-dimensional probabilistic simplex. In this work, consider & being ex- ponential function exp(.) and power function (-)². P E N. which are respectively known as the softmax transform and the escort transform (Mei et al., 2020).
3.2 POLICY GRADIENT UPDATE WITH LEARNED REWARD FUNCTION
With the learned reward function (o a, g). the next step is to improve the parametrized dialogue agents Th via policy gradient methods (Sutton & Barto, 2018), given collected offline dataset D = {ne) A classical approach to train the policy Th is to estimate the policy gradient via the REINFORCE method (Williams, 1992):
V VaJJENNFORCC((TA) = (o. (~b,aaa (bs) V & log (ā h2) GT (li, āz g)
(4)
where G (he, az. g) is the (discounted) accumulated reward that the agent Te receives, starting from the observation
(part of h₂) and action a., under the given goal g. When the discount factor 7 > O, estimating Ge (h,, at,9) requires Monte Carlo sampling (on-policy) or temporal difference learning (off-policy) both of which require learning an additional value-function network. Empirically we observe that learning an additional action-vall value function could introduce instability and extra compute to the subsequent training of the E2E dialogue model. To simplify the training pipeline, we simply set the discount factor = 0, and thus GT (h,,, āt.9) g) = Ro(ot, âe. 9).
Though the policy gradient estimator defined in Eq. (4) is unbiased. it tends to have high variance, especially when the action space is large. Unfortunately, in the E2E ToD system, the action space often defined to be the Cartesian product of the vocabulary, which itself has dimension larger than 30000. As a result, optimizing the agent To by the REINFORCE estimator may suffer from divergent training. We illustrate this phenomenon via toy example in Section 5.2.
To mitigate the high-variance issue of the REINFORCE estimator, we utilize the Gumbel-softmax (GS) trick (Jang et al., 2016; Maddison al., 2016; Fan et al., 2021) reduce the variance. Specif- ically,
JGS(FT) = Eayxrs (k.) (R))) ( g)ggg) R Ee--Gumbd((.)1 fo fo(h,,
cxp((t-(hisd) with folhe.c) - [1S0) (f...)....... (1AD (h., (l)))) E RlA, and f (h.,c) ΣJs, exp((I,
where (4( )A) are the logits of the categorical distribution defined by the agent TT. and À is the temperature parameter that we set as 1. Besides, following the pessimistic principle in offline RL (Buckman et al., 2020), we add weighted regularization such that the actions generated by the agent T close to the actions in the dataset D,
(T)) == BBe .)) - -o (a (x)) Ro as g)],
which is similar to the weighted behavior cloning in offline RL (Wang al., 2020b), except that we directly use the intermediate rewards as the weights, rather than using the value function. Combining the policy gradient and the weighted regularization, we have the following loss for the agent az.
(GEN(O)
Jas(m²) + (w(Ts)
where a is the coefficient balancing these two parts. Note that the original supervised-learning loss of MinTL (Lin et al., 2020) can be decomposed into two parts, respectively for the dialogue state tracking (DST) and the response generation. We retain the DST loss (DST()) in MinTL and replace its cisponse-generation loss with Eq. (5). Our final loss for ToD agent training is
(()) fGEN (Ø) + (Ø).
(6)
We illustrate method in Fig. 1 and provide algorithm box in Appendix B.
4

---

Published as a conference paper at ICLR 2023
Reward Learning
Reward Utitzation
Leaming Objective with RewardNett RewordkiLE
Accu mubted
Roward
Jice d)
(lt))
Response
Action
A(r,000
Cuabbe
Reward Model
Dalopue Haltor
Dalogue Trapectry
F.
Dellogue Trajectory
Figure I: Overview of the proposed method. We denote "Accumulated Reward" for the learned accumulated reward, [(;;0) 0) for the accumulated reward of each trajectory. S(-) for the combined score of each trajectory, fw for the weighted regularization, EDST for the DST loss, and "DH" for the dialogue history. In the right panel, (be, ac, 9) D. We BART for both the reward model and the ToD model.
Remark Eq. (6) for the learning of the dialogue agent Th essentially a generalized objective from several previous works. Specifically. if we set a = 0 and set the reward function to be constant Ro at g)
Eq. (6) reduces to the objective in MinTL, without any guidance for response- generation from the learned reward function Rg. If we set = 0, and use the RewardNet loss with . = 2 and +- (-) (f.e., the identity function) to train the reward function, Eq. (6) reduces to the objective in CASPI (Ramachandran et al., 2021). In Section 5, we demonstrate the advantages of our techniques proposed in this section, including the RewardNet and RewardMIa losses for reward learning, and the JGs(TT) for agent training.
4 RELATED WORK
Recent works on the E2E ToD systems (e.g., Wu et al., 2019b; Lin et al., 2020; Hosseini-Asl et al., 2020; Ham et al., 2020; Peng et al., 2021; Yang al., 2021) have significantly improved the overall system's 's performance and simplified the algorithmic designs in earlier works, which require solving several pipeline based sub-tasks (e.g., Young et al., 2013; Gao al., 2018: Zhang al., 2020). The reward function trained by our methods can be leveraged as guidance to train existing E2E mod- els, without changing the underlying structures. We demonstrate the effectiveness of our proposed reward learning methods under the structure of MinTI. (Lin et al., 2020) and GALAXY (He et al. (2022); in Appendix E) where we only add an additional response-generation model, while keeping other components of the respective structure unchanged.
objective for the
One line of related research is applying RL to train ToD agents. is often unsuccessful directly apply RL algorithms such as the DDPG (Lillicrap et al., 2015) PPO (Schulman et al., 2017) since the agent training could potentially diverge (Zhao et al., 2019; Jang et al., 2022; Kwan al., 2022). Recently, number of works consider offline RL (Levine al., 2020) promising solution to stabilize the agent training on a static dataset (e.g., Jaques et al., 2020; Ramachandran et al., 2021; Jang et al., 2022; Verma al., 2022; Snell et al., 2022a;b). Following the offline RL principle, we use a reward-weighted regularization to stabilize the dialogue-agent training. Together with the incorporation of the Gumbel-softmax trick to estimate the policy gradient, our work retains algorithmic simplicity while improving the training stability and overall performance.
Finally, our paper closely relates to works reward learning for the ToD systems (e.g., Takanobu et al., 2019; Ramachandran et al., 2021). I). This research thread differs from works that directly use manually designed reward function, which only gives sparse signals indicate whether the agent achieves the goal or not (e.g., Weisz et al., 2018; Wu et al., 2019e; Jang et al., 2022: Snell et al., 2022b). One line of this research direction is utilizing inverse reinforcement learning (IRL.) (Russell, 1998) to learn a dense reward function, by assuming the collected data be expert demonstrations (Takanobu et al., 2019). However, modern IRL techniques such as GAIL-style algorithms (Ho & Ermon, 2016; Fu et al., 2017) often require iterating between reward learning and policy training (Finn et al., 2016), which is computationally expensive and less scalable to dialogue-generatiion models. Besides, the IRL methods aim at justifying the data, while the reward-learne in our work seeks to explain the preference among multiple trajectories, potentially leading to better- than-demo agents (Brown et al., 2019; 2020). Our paper is more closely related the research on
framework
5

---

Published as a conference paper at ICLR 2023
Table 1: Results of the E2E response generation task on MuldWWZZ 2.0 dataset. The best result on each metric hold. The results of UBAR are from the reproduction by Jang et al. (2022). The results of CASPI are from our reproduction. All our provided results are the average over five random seeds. Other results are from the original paper. "GS" denotes the Gumbel-soffmax trick. (+) denotes the power function with power 1.
Algorithms
Inform
Success
BLEU
Combined Score
SFN + RL (Mchri ct al., 2019)
73.80
53.60
16.90
83.10
DAMD (Zhung et ul., 2020)
76.40
64.35
17.96
88.34
SimpleTOD (Hosseini-Asl al., 2020)
84.40
70.10
15.01
92.26
MinTL (Lin ct al., 2020)
84.88
74.91
17.89
97.78
SOLOIST (Peng et al., 2021)
85,50
72.90
16.54
95.74
UBAR (Yang
2021)
87.47
74.43
17.61
98.56
GPT-Critic (Jang et al., 2022)
90.07
76.63
17.83
101.13
CASPI' (Ramachundran et al., 2021)
91.37
82,80
17.70
104.78
RewardNet, NN 3. 4 {-)*
92.77
84,28
17.74
106.27
RowardMLE, + exp(-)
91.49
83.38
18.97
106.40
RewardNat GS. N 3,
92.63
84.32
18.35
106.83
RewardMLE GS, N 5, 0 exp(-)
93,09
83.90
18.04
106.54
reward learning from preferences, which has recently been adopted in NLP tasks, including training language models (Fan et al., 2020; Ouyang et al., 2022) and fine-tuning (Ziegler et al., 2019; Zhang et al., 2021b; 2022b), question-answering with verification (Nakano et al., 2021; Zhang et al., 2021 Menick et al., 2022; Zhang et al., 2022a), and ToD systems (Ramachandran et al., 2021). These works use the pairwise preference-learning objective in Christiano et al. (2017), which can be viewed as a special case of the Rewa reNet loss discussed in Section 3.1. Our work mainly focuses on the ToD task, where we study reward-functine learning and reward utilization for training E2E dialogue agents. Appendix C continues the discussion on related reward-learning methods.
EXPERIMENTS
Dataset. We evaluate the proposed methods on the MultiWOZ. 2.0 dataset (Budzianowski al., 2018), which is a representative ToD benchmark. MultiWOZ, 2.0 is a large-scale multi-domain dialogue corpus with seven domains: attraction, hospital, police, hotel, restaurant, taxi, and train. Each dialogue therein covers between one to three domains. This dataset has 8438 dialogues in the training set and 1000 dialogues in the validation and test set respectively.
Evaluation Metrics. Our proposed method is evaluated on the E2E dialogue-modeling task of the MultiWOZ 2.0 dataset. Following the standard setup (e.g.. Budzianowski al., 2018; Mehri al., 2019), we use four automatic evaluations metrics: 1) Inform rate: the fraction of the dialogues where the system has provided an appropriate entity; 2) Success rate: the fraction of the dialogues where the system answered all the requested information; 3) BLEU score (Papineni et al., 2002): measures the fluency of the generated responses; 4) Combined Score (Mchri et al., 2019): an overall quality measure defined as Combined Score =: (Inform + Success) x 0.5 + BLEU, Details on prepossessing and implementation are in Appendix B and G.
5.1
MAIN RESULTS
Main evaluation. Table compares the performance of our methods with several classical and recent approaches the E2E
task. As shown in Table 1, our proposed methods not only improve the dialogue- taak completion, measured by the Inform rate and the Success rate; but also generate fluent responses, reflected by the competitive BLEU scores. Recall that CASPI is special case of the Rewa rdNet loss (Eq. (2)) when we use escort transform (Φ (.) the identity function) with pairwise preference (NV = 2). When we use three dialogue trajectories (NV = 3) to construct the RewardNet loss and retain the same escort transform, the overall performance generally improves over CASPI. As discussed in Section 3.1, our dNet loss generalizes the preference learning by taking more information each update of the reward model and thus could learn better reward function. Appendix D further compares our methods with CASPI.
The performance is further gently improved by changing the Rewa roNet loss (Eq. (2)) to the Rewa rOMLE loss (Eq. (3)), with the softmax transform (& = exx(-)) and N = 5 dialogue trajecto- ries. This again demonstrates the benefit of our proposal of using multiple trajectories to learn the reward model. Section 5.2 conducts ablation studies on the number of trajectories and choice of 4.
"The CASPI paper reports the median score over random seeds, instead of the more commonly used mean and report the mean scores.
score. We the official CASPI codebase http:///ggguuuu.ommassssff//////

---

Published as a conference paper at ICLR 2023
Table 2: Results on simulated low-re wwwresource settings, where 5%, 10%. 20% of training data used to train the models. The best result on each metric under cach setting is bold. "Comb." " is the Combined Score. All our provided results are the average over five random seeds. Bascline results are from Lin et al. (2020).
3%
10%
20%
Model
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
DAMD
56.60
24.50
10.60
51.15
62.00
39.40
14.50
65.20
68.30
42,90
11.80
67.40
MinTL
75.48
60.96
13.98
82.20
78.08
66.87
15.46
87.94
82.48
68.57
13:00
88.53
RewardNet :3
81.22
67.37
12.82
87.
92.39
78.98
13.36
99.05
89.83
79.30
15.18
99.75
RewardMi.5.5 :5
82.90
69.61
14.26
90.51
89.67
77.48
14.80
98.38
90.15
78.70
15.81
100.24
107
107
103
105
102
105
103
103
96
103
11
2
11
p=1
p=2
p=3
p=4 smak
p=1
p=2 p=3 pmd smax
Number of Trajectory
Number of Trajectory
Prob. Transform
Prob. Transform
(a) RewardNet
(b) RewardMLE
(c) RewardNet
(d) RewardMLE
Figure 2: Line plots comparing the Combined Score when the RewardNet and Rewa rdMGR R losses are constructed under different number of sampled trajectories different prohabilistic transforms. The y-axis
represents the Combined Score. p = L, 2, 3, 4 the escort transform with power 1,2, 3, 4. "smax" is the
softmax transform. Results are the average over five random seeds.
So far, we follow the prior work to not utilize policy gradient to train the response-generation model, i.e. a 0 in Eq. (5). Extra performance gain can be obtained by adding the policy-gradiee updates via the Gumbel-softmax trick (GS) discussed in Section 3.2. Indeed, GS improves both the plain Rewa rdNet and Rewa rdMLE. models. This shows the efficacy of directly optimizing the response- generation model w.E.t the learned reward function. Further discussion is provided in Section 5.2.
Appendix E. provides the experimental results when applying our method onto the recent GALAXY backbone (He et al., 2022). Appendix F discusses the results on the MultiWOZ 2.1 dataset.
We evaluate our method on the low-data regime by following the testing strategy in Lin et al. (2020). Specifically, we use 5%, 10%, and 20% of the training data to train basic Rewa rdNet and Rewa dMMLE. models in Table 1, without the GS component. We compare them with the baseline scores in Lin et al. (2020). Table 2 reports the results. It is clear that our models outperform the baselines, MinTL and DAMD, showing the efficacy of our method. Compared with Table 1, our models trained with 20% of the data perform competitively with the baseline methods trained on the full training set.
Low-resource experiment.
5.2 ABLATION STUDY
The ablation study considers the following four research questions to better understand our methods.
(a): What if we learn the reward function via a different number of trajectories? In Fig. 2a and 2b, vary the number of trajectories used for the reward-learning losses in Table 1. To avoid unwanted interference, we use the basic version of models without the GS component. The case of using two trajectories reduces the mairwise-preferent loss discussed in Section 2.
As shown in Fig. 2a and 2b, our generalized approach of using multiple trajectories to learn the reward function provides the flexibility to outperform the classical pairwise-preferene learning- This is more apparent in the RewardMLE models, which are less sensitive to small errors in the ground-truth scores. In general, the optimal trajectory number may depend on the scoring quality.
(b): Do different probabilistic transforms in reward learning objectives affect the performance? We modify the basic version of the Rewa rdNet and Rewa dMMLE models in Table 1 by using the softmax transform and by using different powers in the escort transform in the reward learning losses Eqs. (2) and (3). For the escort transform, we consider = (.) E 21, 4).
Figs. 2c and 2d plot the resulting Combined Scores. We see that the RewardMLE model is less sensitive to the choice of probabilistic transform - all the considered variants have a Combined Score of at least 104. In fact, changing its softmax transform used in Table 1 to the escort transform with power two improves the performance to 106.77. Thus, the choice of probabilistic transform provides an additional angle to improve the learned reward function and the entire Tol) model.

---

Published as a conference paper at ICLR 2023
86
18.5
107
92.8
18.0
106
92.4
84
17.5
105
92.0
83
17.0
0.1
2.5
0.5
2.5
0.05
2.5
0.1
(a) Combined Score
(b) Inform Rate
(c) Success Rate
(d) BLEU Score
Figure 3: Line plots showing the four automatic evaluation metrics of the RewardNet + GS model in Table
under different values in the generation model loss Eq. (5). Results are the average over five seeds.
(c): Is our method sensitive to the coefficient a in the generation-model loss Eq. (5)? To inves- tigate the robustness of our method under different weights for the policy-gradient optimization of the eesponse-generatiee RewardNet + GS model, and vary the a coefficient the generation- loss Eq. (5). Fig. plots the resulting four automatic evaluation metrics.
model. We select our best
model in Table 1, the
We see that our model is relatively robust to the choice of a. The five variants in Fig. all have Combined Scores of at least. 105, higher than the best baseline result of 104.78 in Table 1. In fact, by changing the a coefficient to 0.5 from 0.1 used in Table I, we achieve a even better Combined Score of & 107.2. Further, the capability of task completion and the fluency of the generated responses are both relatively insensitive to the choice of a.
(d): How does the addition of the sradiett method Gumbel-softmax help the performance? Fig. 4 compares the performance our models in Table 1, with error bars showing the standard deviation of the Combined Score five seeds. It is clear that the addition of the Gumbel-softman method can not only improve the score but also reduce the performance variation, which apparent when comparing the Rewa rdMLE model with the RewardMen + GS model.
As discussed in Section 3.2, the Gumbel-softtax (GS) trick can be more advantageous than the classical REINFORCE method (Williams, 1992) for the policy-gradient update. As demonstration. we conduct a toy experiment following Yin et al. (2019) and plot the results in Fig. 5. The task here learn the parameter zp of D-dimensional categorical distribution maximize a simple reward function. Specifically, denote the sigmoid function as (-). the goal is
maXTER"" Ex-Cate(o(w)) [[x)), ((x) A 0.5 + x/(D- R), Vo E (1...., D).
where Cate(a(()) denotes the categorical distribution with probability vector ( and D R 30. The best a (2b) is (O..... O, 1), leading to the optimal expected reward of = 0.533. We initialize xb = 0 and use one sample for the stochastic gradient-ascent update, with learning rate of 1.0.
The first row of Fig. 5 traces the objective function during the training process when using the true gradient, REINFORCE, and the GS for policy-gradient updates. We see that the REINFORCE method converges to local maximum, while the GS method reaches the global optimum, using the true gradient for updates. The second row shows the gradients for 0 and 0p. where we see that gradient estimates from the REINFORCE method are both unstable and vanishing, compared to the GS method. The learned probabilities (o($)).......
(bb) is traced in the third row, where the red line is for of (b)) n that should ideally be 1, and the shadowed lines are for the other components that ought to be O. The learning process of the GS method closely resembles that of using the true gradient, while REINFORCE oscillates around local optimum. The last row of Fig. 5 plots the estimate of gradient variance via 500 samples, averaged over each component of the a vector. The gradient variance of the REINFORCE method is on the order of 10-2 2 at the beginning and converges to roughly 10-4, while the GS is 10-6 throughout the training process. This toy experiment shows that a vorvanrience method, such as the GS, can be critical to the success of policy- policy-gradient gradient training.
5.3 FURTHER ANALYSIS
Human evaluation. For more comprehensive evaluation of our method, we conduct a human evaluation on the quality of the generated responses, where our model and the top two baselines in Table 1, GPT-Critic and CASPI, are compared. We follow the evaluation protocol in prior work (e.g., Zhang et al., 2020; Ramachandran et al., 2021; Jang et al., 2022) to evaluate on two metrics: 1) Appropriateness: measures the appropriateness of the generated response under the context of the dialogue turn; 2) Fluency: evaluates the comprchensibility and coherency of the generated response. We randomly picked 50 turns in the test set and showed to 10 evaluators the responses
8

---

Published as a conference paper at ICLR 2023
REINFORCE
8-Ver
8-10r
Chaleet
R-MI
R-MLE 65
Ged. We
100 1022 104 316 108
2000
40.0
2000
4000
2000 4.00
Figure 4: Bar plot comparing four models in Ta- ble 1. Mcan and standard deviation over five ran- dom seeds are shown. "R-Net" denotes RewardNet. "R-MLE" is RewardMLE. "GS" is Gumbel-softmax.
Figure 5: Toy experiment comparing REINFORCE and Gumbel-softmax reward maximization, es- timated gradients, the estimated probabilities, and the gradient variance. See Section 5.2 for details.
52
96.5
3.6
3.7
50
3.3
96.2
3.0
48
95.9
46
GPT-C
CASPI
Our
GPT-C
CASPI
Our
UBAR GPT-C R. R-Net R-MLE
UBAR GPT-C R Net R-MLE
(a) Appropriateness
(b) Fluency
(a) Joint Accuracy
(b) Slot Accuracy
Figure 6: Bar plots for the results of human evaluation on uppropriateness and fluency, showing the mean and one standard deviation each method. The scores are on a 5 scale and higher scores indicate better results. "GPT-C" denotes GPT-Critic. Details for the setup of human evaluation discussed in Section 5.3.
Figure 7: Bar plots for quality of the generated dia-
logue states, showing the mean and standard deviation of two metrics. "GPT-C" denotes GPT-Critic, "R-Net" denotes Rewa RewardNes rdNet "R-MLE" is RewardMLE The results of UBAR and GPT-Critic are from Jang et al. (2022). Our results
five random seeds.
generated from each method, together with the dialogue history up to that turn. The method names were anonymized. The evaluators were asked to read the dialogue history and score the response on a 5-Point Likert Scale {1, 2, 4, 5}, where score 5 is the highest and 1 the lowest.
Fig. 6 summarizes the evaluation results. We see that our model outperforms the baselines in both the appropriateness and fluency scores. The human-evaluatte results coincide with our comparatively good dialogue-task completion and BLEU score Table 1.
Examples of the generated dialogues. Tables and 4 in Appendix A conduct two case studies comparing the generated responses from our method with those from the baselines GPT-Critic and CASPI. We additionally annotate the generated responses to discuss the quality of those generations. These examples show that the responses from our model compare favorably with the baselines in both task completion and comprehensibility, aligning with the automatic and human evaluations.
Quality of the DST. To further understand the performance gain of our models, we compare our basic RewardNet and RewardMLE models in Table I with the baselines UBAR and GPT-Critic on the quality the generated dialogue states. Fig. plots the results of the dialogue state prediction, measured by the two metrics Joint (Goal) Accuracy and Slot Accuracy (Wu et al., 2019a). We that our two models have more accurate DST than the two baselines, which can be related to their better performance in Table Interestingly, the DST of the Rewa rdMLE model is also better than that of the Rewardde dNet. model. This may suggest that a better reward model not only benefits the learning of response generation, but also the DST. These losses are jointly minimized training the ToD model, and thus a good response-generation loss from a better reward model may help the optimization of the DST loss.
6 CONCLUSION
In this paper, we aim to answer the question of how to efficiently learn and utilize reward function for training the E2E ToD agents. We answer this question by introducing two generalized reward- learning objectives, and utilize stable policy-gradics method to guide the training of the E2E ToD agents. Future work includes extending our reward-learning objectives to other applications, such as the question-answering with verification.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
S. Yang and M. Zhou acknowledge the support of NSF-II 2212418 and the Texas Advanced Com- puting Center (TACC) for providing HPC resources that have contributed the research results reported within this paper.
ETHICAL STATEMENT
We develop our methods based on the publicly available MultiWOZ 2.0 dataset (Budzianowski al., 2018). It is important note that, like other ToD models, our implementation will likely reflect the socio-economic and entity biases inherent in the MultiWOZ dataset (Qian et al., 2021). We initialize model parameters from the pre-trained BART model. And a comprehensive analysis of certain biases captured by BART is outside our scope. Besides, we invited volunteers for the human evaluation, with transparent and detailed explanations and communications on data usage, research intent, occupied hours, etc.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3943/4)3324-345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Nickum. Better-than-deen -emonsttaot imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pp. 330-359, PMLR, 2020.
Jacob Buckman, Carles Gelada, and Mare Bellemare. The importance of pessimism fixed- dataset policy optimization. arXiv preprint 1eXi::0009006799 2020.
Pawel Budzianowski, Tsing-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. arXiv preprint arXiv:1810.000 2018.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd international confer- ence on Machine learning, pp. 89-96, 2005.
Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129-136, 2007.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop Natural Language Processing for Conversational AI, pp. 38 45, 2020.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25335, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 04.000 2018.
10

---

Published as a conference paper at ICLR 2023
Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34:8622-805 8636, 2021.
Xinjie Fan, Shujian Zhang, Bo Chen, and Mingyuan Zhou. Bayesian attention modules. Advances in Neural Information Processing Systems, 13:166621,637, 2020.
Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou. Contextual dropout: An efficient sample-dependen dropout module. arXiv preprint 1rXiv:2103.0410 2021.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03892 2016.
Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of machine learning research, A(Nov):933-969, 2003.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11245 2017.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning. ArXiv, 1b6/2106.ar 068600 2021.
Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1371-1374, 2018.
Kallirroi Georgila and David Traum. Reinforcement learning of argumentation dialogue policies in negotiation. In Twelfth Annual Conference of the International Speech Communication Associa- tion, 2011.
Donghoon Ham, Jcong- Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2. Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pp. 583-592, 2020.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, et al. Galaxy: A generative pre-trained model for task-oriented dialog with sumi-super learning and explicit policy injection. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 36, pp. 10749-10757. 2022.
Ralf Herbrich, Thore Graepel, and Klaus Obermayer. Support vector learning for ordinal regression. IET. 1999.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. Advances Neural Information Processing Systems, 33:20179-2011 2020.
Huang Hu, Xianchao Wu, Bingfeng Luo, Chongyang Tao, Can Xu, Wei Wu, and Zhan Chen. Playing 20 question game with policy-based reinforcement learning. arXiv preprint (80V:1808.07645 2018.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv:1611.01 1444, 2016.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference on Learning Representa- tions, 2022.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, A. Lapedriza, Noah J. Jones, S. Gu, and Rosalind W. Picard. Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog. ArXiv, abs/1907.00456, 2019.

---

Published as a conference paper at ICLR 2023
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline rein- forcement learning. arXiv preprint 2011::2010.05848, 2020.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99134, 1998.
Wai-Chung Kwan, Hongru Wang, Huimin Wang, and Kam-Fai Wong. A survey on recent advances
and challenges in reinforcement learningmethods for task-oriented dialogue policy learning. arXiv preprint wrXiv:2202.136773 2022.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pp. 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-644-277445-3 doi: 10. 10.1/
778-3-642-27645--3
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learnig Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.132 13461, 2019.
Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Julia Kiseleva, Maarten de Rijke, Shahin Shayan- deh, and Jianfeng Gao. Guided dialog policy learning without adversarial learning in the loop. arXiv preprint arXiv:2004. 03267, 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.000 00599990999,.... 2015.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. MinTL: Minimalist transfer learning for task-oriented dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3391-3405, Online, November 2020.
Association for Computational Linguistics. doi: 10. 1/2020. 1/2020.emmlppp.e.e.ooo... main.273.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. In International Conference on Learning Representations, 2019.
Tie-Yan Liu. Learning to rank for information retrieval. Foundations and Trends@ in Information Retrieval, 3(3):225-33)) 331, 2009.
Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint (oXi::/611..07//0 2016.
Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. Structured fusion networks for dialog. arXiv preprint arXiv:1907.10016 2019.
Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax. Advances in Neural Information Processing Systems, 33:21 130- 21 14440,20 2020.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillinghan Geoffrey Irving, et al. Teaching lan- guage models support answers with verified quotes. arXiv preprint arXiv:2203.11144 2022.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2 2t/.2//22.93222 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arX 2022.
12

---

Published as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite nusk-complttion dialogue policy learning via hierarchical deep reinforcement learning. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2231 2231-22409 2017.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. Soloist: Buildingtask bots at scale with transfer learning and machine teaching. Transactions of the Asso- ciation for Computational Linguistics, 9:807-824, 2021.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series
(Applied Statistics), 24(2):193-202, 1975.
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnad- hurai Sankar. Annotation inconsistency and entity bias in multiwoz. In Proceedings of the 22nd Annual Meeting of the Special Interest Group Discourse and Dialogue, pp. 326-337, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 188):9 2019.
Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong. Causal- aware safe policy improvement for task-oriented dialogue. arXiv preprint arXiv:2103.06377 2021.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Atsushi Saito. Curriculum learning based on reward sparseness for deep reinforcement learning of task completion dialogue management. In Proceedings of the 2018 EMNLP workshop SCAI: The 2nd international workshop on search-oriented conversational AI, pp. 46-51, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1X07.07077003477 2017.
Ronnie W Smith and D Richard Hipp. Spoken natural language dialog systems: A practical ap- proach. Oxford University Press on Demand, 1994.
Charlic Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv: /871, 2022a.
Charlic Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for gral-oriented dialogue systems. arXiv preprint 2XXv:2204.11198, 2022b.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. Guided dialog policy learning: Reward estima- tion for multi-domain task orieetreet dialog. arXiv preprint 1rXiv:1908.100 10719, 2019.
Gokhan Tur and Renato De Mori. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons, 2011.
Siddharth Verma, Justin Fu, Mengjiao Yang. and Sergey Levine. Chai: A chathot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08446, 2022.
Huimin Wang, Baolin Peng, and Kam-Fai Wong. Learning efficient dialogue policy from demon- strations through shaping. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6355-6365, 2020a.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020b.
13

---

Published as a conference paper at ICLR 2023
Gellért Weisz, Pawet Budzianowski, Pei-Hao Su, and Milica Gašié. Sample efficient deep reinforce- ment learning for dialogue systems with large action spaces. JEEE/ACM Transactions on Audio, Speech, and Language Processing,
2018.
Tsung-Hsine Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned stmmbased natural language generation for spoken dialogue systems. arXiv preprint arviv: 000.0/000000000 0/55, 2015.
Tsung-HHsie Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv: (664004446,2 2016.
Jason D Williams and Steve Young. Partially observable markov decision processes for spoken dialog systems, Computer Speech Language, 21(2):393 2007.
Ronald J Williams. Simple statistical grodient- algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255 256, 1992.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pas- cale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. ACL, pp. 808-819, 2019a.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. Alternating recurrent dialog model with large- scale pre-trained language models. arXiv preprint arXiv:1910.03756 2019b.
Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. Switch-based active deep dyna-q: Efficient adaptive planning for complcomn dialogue policy learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7289-7296, 2019c.
Fen Xia, Tie- Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pp. 1192-1 1 1999, 2008
Shentao Yang, Yihao Feng. Shujian Zhang, and Mingyuan Zhou. Regularizing a model-based policy stationary distribution to stabilize offline reinforcement learning. International Conference on Machine Learning, pp. 24980-25006. PMLR, 2022a.
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou, A regularized implicit policy for offline reinforcement learning. arXiv preprint 2xXiv:2202.09673 2022b.
Shentao Yang. Shujian Zhang, Yihao Feng, and Mingyuan Zhou. unified framework for alternat- ing offline model training and policy learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022c
Yunyi Yang, Yunhao Li, and Xiaojun Quan. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14230-142288 2021.
Mingzhang Yin, Yuguang Yue, and Mingyuan Zhou. Arsm: Augment-eiifooree--w-p-- esti- mator for gradient backpropagation through categorical variables. In International Conference Machine Learning. pp. 7095-7104. PMLR, 2019.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 1606-):160-1 2013.
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. Find or classify? dual strategy for slot-value predictions on multi multi-domain dialog state tracking. arXiv preprint arXiv: 10XX:v:99000054.. 2019.
14

---

Published as a conference paper at ICLR 2023
Shujian Zhang, Chengyue Gong, and Eunsol Choi. Knowing more about questions help: Im- proving calibration in question answering. arXiv preprint arXiv:2106.01423 01444, 2021a.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. Learning with different amounts of annotation:
From zero to many labels. arXiv preprint arXiv:2109.044 04408, 2021b.
Shujian Zhang, Chengyue Gong, and Xingchao Liu. Passage-mask: A learnable regularization strategy for retriever-reader models. arXiv preprint arXiv:2211.009/5. 2022a.
Shujian Zhang, Chengyue Gong, Xingchao Liu, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Allsh: Active learning guided by local sensitivity and hardness. arXiv preprint arXiv:2205.0423 04980, 2022b.
Susan Zhang. Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 01068, 2022c.
Yichi Zhang, Zhijian Ou, and Zhou Yu. Task-oriented dialog systems that consider multiple appro- priate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9604-9611, 2020.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. Rethinking action spaces for reinforce- ment learning in end-to-end dialog agents with latent variable models. arXiv preprint arXiv: 1902.08888 2019.
Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. Convlab-2: open-source toolkit for building, evaluat- ing, and diagnosing dialogue systems. arXiv preprint arXiv:2002.04793, 2020.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.083933 2019.
15

---

Published as a conference paper at ICLR 2023
Appendix
A EXAMPLES OF THE GENERATED DIALOGUES
Tables 3 and show two case study comparing the generated responses from our method and from the baselines GPT-Critic and CASPI. Our method outperforms the baselines in terms of both task completion and the quality of the generated expressions.
Table 3: Generated and original dialogue PMUL4610. "User" denotes User Utterance, "System" denotes true System Response. Red annotations explain unsatisfactory responses and green for satisfactory ones.
PMUL4610
Turn Num.
Method
Original / Generated Dialogue
User
i am looking for a muscum in the cast to visit.
there are [valuc choice] [valuc type] in the [value_ area]. [valuc name] is [value price]. [value name] [value price].
GPT-Critic
not make recommendation.
there (value_choicee] [value_type] the value_areu], would recommend #/ Do not explain recommendation.
CASPI
[value name].
there are [value [value_choice ccoicel (value_type) in the [value_arent recommendd [value nume]] because it is [value_ [value_price] to get in
Ours
/f Make recommendation and explain.
we have [valuc, choice] [value type] in the [valuc arca] area i just went to the [value_name] and they have new exhibits would you more information on them
System
User Utterance
sounds wonderful! can you give their postcode?
GPT-Critic
their postcode is roluucppostcode]. there anything clse can help you with?
CASPI
thing! the postoode sure the postcode is [valuc. pooteedost is there anything else i can help you with ?
]ostoep
Do not elicit conversation.
Ours
System Response
their postcode is [value postcode]. can i help you with anything else today ?
User Utterance
i am also looking for restaurant in the cast part of town the moderate price range.
there are [value, [value_choice] choice] restaurant is in that area. [value_name] und [value_name].
GPT-Critic
// Do not elicit conversation
CASPI
[value_name] is a [value [value.price] restaurant in the [value area].
// Do not elicit conversation.
there are [value choice] restaurant is that fit your criteria
Ours
[value name] serves [valuc food] food and located the [value area] part of would you like their address? ?
there are [value [value_choice] [value_ restaurant -s and vvalue.choie)) [value.food]. which would you prefer?
Syster Response
User Utterance
give me the address of the you recommend
[value_name] is located at [value_nddress). there anything else can help you with?
GPT-Critic
Do not make recommendation.
[value name] favorite of mine. they located [value address]
CASPI
// Do not elicit conversation.
[value name] is great [value food] restaurant in the [value area]. // More plural expression. the uddress is [value__ddress).
Ours
// Do not elicit conversation.
recommend [value name], they are [valne_address), postal [value_osstcode] can assist with anything else
System Response
User Utterance
not at time, thank you.
GPT-Critic
thank you using our system. goodbye
CASPI
you welcome. have great day!
Ours
you welcome. have great day
System Response
thank you for culling. please let know if we can assist you the future. goodbye.
16

---

Published as a conference paper at ICLR 2023
Table 4: Generated and original dialogue for ID SNG1012. "User" denotes User Utterance. "System" denotes true System Response. Red annotations explain unsatisfactory responses and green for satisfactory ones.
SNGI012
Turn Num.
Method
Original / Generated Dialogue
User
need to find guest house with a 3 star rating
have [value choice] options for you. do you have a price range or area of town you would like to stay in?
GPT-Critic
have [value choiceJ [value type] that meet your criteria.
CASPI
do you have a preference of area or price range" ?
have [value choice] [value type] with [value stars] stars. would you prefer [value_priee price] or vvalue_price) price range? // Do not ask location.
Ours
System
we have [valuc. choice] such [valuc htpel....yppc...... do you have preferred location?
User
nO, don't I. want that include free wifi.
[value name] [value price] [value stars] star [value type] the [value area] with free wift and parking. would you like to book it for you?
2.
GPT-Critic
have [valuc_ [value_choice] [value_ty sype] that meet your criteria. - [valuc, name] and [value_name] both are [valuc-price] and offer free wifi and parking. do you have preference ?
CASPI
have [valuc choice| [value type] that have free wifi.. would recommend the v value_name].. would you like to book room
Ours
System
[value name]. [value name]. and [value name] are all available you would like of those?
User
please book the hamilton lodge, for people.
booking was successful, reference number is : [value reference]. there anything else
3
GPT-Critic
can help you with?
// Finish booking before acquire sufficient information.
CASPI
i can book the (value name] for you. how many nights will you be staying ?
Ours
would be happy to book the vralu_name) for you. what day would you like to arrive ?
would be happy to. how many nights would you like to stay? and i 'll need to know the day you would like to start your stay.
System
User
'Il he staying 2 nights starting on wednesday, oh, and there are 6 of us.
have booked you room at [value_ name] for [value. people] people for [value_stay]
GPT-Critic
nights starting [value day]. your reference numher is [value reference].
is there anything else can help you with? // Dialogue diverges from ground truth.
am sorry but there are no rooms available for those dates. would you like to try different day or a shorter stay
CASPI
Ours
i am sorry but i was unable to book that [valuc typoj. would you like to try another 1 ?
System
am sorr,, my reservation attempt was unsuccessful. will another day or length of stay work?
User
can you try the same parameters another hotel?
have booked you at [valuenamee| for [value_stay] nights sturting [value_day]
5
GPT-Critic
for [valuc people] people. your reference number is [value reference].
is there anything else can help you with? // Repeating diverged dialogue.
CASPI
am sorry i am not able to book another (value type] // Fail to continue conversation.
have [value_nanel [nieee]aa]]] the value_areal, it is [value.price] [value_stars] star [value_type]. would you like me to book that for you? ?
Ours
System
of course, could you tell me the area you would like to stay
narrow for you?
User
the are does not matter, just stars and wifi at guest house please.
GPT-Critic
would you like me to book for you?
Fail to complete booking.
6
CASPI
was able to book you at [value name]. reference number [value reference].
was able to book you at [value_name) ). your reference number is [value reffeeeeefee
Ours
is there anything else can help you with ? // More plural expression and elicit conversation.
have got you a reservation for
at hobson's 's house for [value_stay] nights.
System
your reference number [value reference].
User
thank you so much! that should be all need.
GPT-Critic
you are welcome. have great day.
7
CASPI
you are welcome. have great day!
Ours
you welcome. have great day!
System
you welcome. enjoy your day.
17

---

Published as a conference paper at ICLR 2023
B ALGORITHMIC DETAILS
Prepossessing. The raw corpus is prepossessed by common practice in the ToD literature. Specifi- cally, we represent the database (DB) query results one-hot vectors following Budzianowski al. (2018), use domain-adaptive delexicalization proposed by Wen et al. (2016), and generate delexi- calized responses with placeholders for specific DST/DB information as in Zhang al. (2020).
Implementation of the response model. Our model in Section is based on the MinTL ToD model (Lin et al., 2020), which uses the pre-trained BART-large model (Lewis et al., 2019). MinTL directly works on the system response and does not explicitly output the dialogue act. Our proposed method Section 3 is applied the response training, and we retain the DST-training loss in Min'LL. Our model is trained by fine-tuning BART on the training set and early-stopping by the validation set.
Implementation of the reward model. Our reward model is implemented by the encoder part of the BART-base model, followed by simple two-layer MLP. The output the reward model scaled to [0, 1] via the sigmoid function. The imput the reward model is the concatenation of the belief state, system response, and dialogue goal. at each turn of the sampled dialogue rollout. The model outputs the reward of each turn in the dialogue rollout, which is summed and fed into the losses proposed in Section 3.1. We use the HuggingFace library (Wolf et al., 2019) to implement our reward model.
Algorithm illustrates the pipeline of our methods.
Algorithm Pipeline the proposed reward learning and utilization methods training E2E ToD agents.
K Input: Reward function (o, a, g). ToD agent Te, dataset Ó := { (0 (k) (To)
k=1"
number of iterations M1 and M2, probabilistic transform function Φ hyperparameters N,
for iteration E [I,...,M, do
Sample N dialogue trajectories from the dataset D.
Optimize Rg via Rewa rdNet (Eq. (2)) RewardMLE (Eq. (3)).
end for
Fix the reward function Ra.
for iteration E {1..... ......,,) M2} do
Sample a batch of transition tuples (g()) (o, ,a, ( from the dataset D.
Optimize the ToD agent Tे via Eq. (6).
end for
Output: Trained ToD agent Tg.
18

---

Published as a conference paper at ICLR 2023
C COMPARISON WITH SOME OTHER REWARD-LEARNING METHODS IN RL-BASED DIALOGUE AGENTS
As additional discussion related work, in this section we briefly compare our work with three other reward-learning methods in RI.-based dialogue agents, i.e., Saito (2018), Hu et al. (2018), and Li et al. (2020).
These three papers all focus on the dialogue-managenne module of the pipeline design, wherein the action spaces of the agents some abstract dialogue acts rather than the human-language-lin system response as in our paper. The possible system responses are fixed in these three papers. For example. Hu et al. (2018) train the agent select from "the set of the indices all available questions in the Q20 game;" and Li et al. (2020) have action space of size 300. As discussed in Section 1, such a pipeline approach requires intensive structural designs, such as determining the possible questions in the Q20 games; and may not enjoy the language plurality and conversation elicitation that our E2E model could offer, e.g., as shown in Table 4. Due to the complexity and the much higher dimension of our action space the system responses, the methods proposed in these three papers are not directly applicable to our setting, which will be discussed in detail below.
The method proposed in Saito (2018) require designed curriculum data and cand-crafted decomposition of the entire task into sub-tasks, which are not readily available and are non-trival for large-scale multi-domain dialogue corpus such our tested MultiWOZ 2.0 dataset. The use of progressive neural networks to provide reward information in Saito (2018) require additional com- putation and memory complexity and thus may not scale transformers. Meanwhile, our method scales well to transformers, as shown in our experiments (Section 5). Further, the method in Saito (2018) may only be feasible on the task of constrained informaation-eetrivvall. but not on some more general tasks such as the booking requirement in the tested MultiWOZ 2.0 dataset. Our experimental results show that our method capable of such tasks.
Similar to our work, Hu et al. (2018) propose a neural network to approximate the reward function to deliver immediate rewards at each timestep. Apart from the aforementioned simple action space, Hu et al. (2018) use the long-term return G, as surrogate indicator of Tr+1 to train the reward function (Eq. (6) of Hu et al. (2018)). which lack of justification. By contrast, as discussed in Section 2 and Section 3, our method is based on the classical approaches in the learning-to-rant (LTR) literature and extends the classical reward- saatrdd-eaminn--fooe caonng-fromppee crom-preference. into utilizing multiple dialogue trajectories simultaneously to optimize the reward function.
As discussed before, Li et al. (2020) consider a relatively small action space of size 300 and learns the reward model via the GAN structure, which may not stably scale up to high-dimensional action space such as the system response in our E2E ToD system. The learned reward function in Li al. (2020) only measure the probability that the input from the real-data distribution, i.e., only considers a pair of dialogue state 8, and the corresponding system action a,. This reward function does not consider the success of the entire dialogue, which intuitively less favorable to the E2E ToD systems. By contrast, our method trains a reward function that aligns with some evaluations on the entire dialogue trajectories, which is more directly related to the usage of the ToD systems.
We further note that apart from the reward-learnigg method, our paper also discusses using the Gumbeel-softmat trick as a more stable method to train the E2E ToD systems, and conducts toy experiment in Section 5.2 to illustrate the advantage of the Gumbel-softmax trick over the classical REINFORCE method. This is not covered in the three prior works Saito (2018), Hu et al. (2018), and Li et al. (2020).
Finally, these three prior works use online RL methods such as DQN, REINFORCE, and PPO, which require environmental interactions and are thus less practical, discussed in Section 1. In contrast, our method allows training E2E response-generation models from static datasets by utilizing offline RL techniques (e.g., Levine et al., 2020; Fujimoto & Gu, 2021; Yang et al., 2022a;b;c).
19

---

Published as a conference paper at ICLR 2023
D DETAILED COMPARISON WITH CASPI
Table 5 further compares CASPI and our two methods: RewardNet +GS, N 3, Φ (.) and RewardMLE +GS, N = 5, Φ = exp(-), showing a detailed breakdown of the scores onto each of the five tested random seeds.
The performances of both our methods and CASPI are relatively stable across random seeds. In particular, both of our methods have higher Combined Score than CASPI on each of the five tested random seeds. This stable improvement of our methods over CASPI aligns with our intuition dis- cussed in Section and our main experimental results in Section 5.1.
We note that both the Average score and the Median score across the tested random seeds are valid metries for performance comparison. Nevertheless, there some ambiguity in calculating the "Me- dian" of the Combined Score, namely, should it be the median of the Combined Scores each random seed, or should it be calculated as Median( Combined Score) A (Median(Informi + Median (Success) x 0.5 + Median(BLEUU
The first way aligns better with the definition of "Median" while the second way aligns better with the definition of Combined Score. Such an am- biguity is cleared out when using the Average as the evaluation metric. Besides, the metric Average score is widely used in prior work (e.g., Zhang al., 2020; Lin al., 2020; Jang et al., 2022). With these considerations, we choose to report the Average over five random seeds in our experimental section (Section 5).
Table 5:
Per random-seed results of the E2E response generation task the MultWOZ 2.0 dataset, comparing
CASPI and our two models: RewardNet +GS, = 3, = ()) and RewardMLE +GS, = 5, & = exp()):
Here, (+)² denotes the power function with power 1. "Comb." the Combined Score. The row "Median" shows the median score of the corresponding column over the five random seeds.
CASPI
RewardMet +GS, N 3, 0 (-)
rewardMLs +GS, N 5, 5.
exp()
Seed
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
80.88
18.88
103.42
82.68
18.42
105.01
84.18
16.82
105.91
133
91.69
83.58
18.17
105.81
94.39
85.09
18.69
108.43
95.10
85.09
17.51
107.61
555
91.99
81.78
17.21
104.10
91.29
82.18
18.54
105.28
91.99
83.68
18.67
106.51
777
93.39
83.48
17.13
105.57
91.89
84.18
18.36
106.40
92.79
83.18
17.73
105.72
999
91.59
84.28
17.09
105.03
95.10
87.49
17.74
109.04
91.59
83.38
19.48
106.97
Average
91.37
17.70
104.78
92.63
84.32
18.35
106.83
93.09
83.90
18.04
Median
91.69
83.48
17.21
105.03
91.89
84.18
18.42
106.40
92.79
83.68
17.73
106.51
E EXPERIMENTS WITH THE GALAXY
To further demonstrate the efficacy and applicability of our approach, we apply our reward learn- ing and utilization methods to the recently proposed GALAXY backbone (He al., 2022), which achieves competitive performance on the E2E response-generatie dataset. We note that the GALAXY paper does not disclose how many and which random seeds were used to obtain its main results; and the official codebase fixes the random seed as 10. This makes us unsure if its reported results are only from this single seed of 10. To mitigate the random- ness in the optimization process, we re-run the vanilla GALAXY on the five random seeds used to generate our main results and compare on these seeds with the variants equipped with our methods Rewa rdNet +GS, N 3, +- (.)² and Rewa RewardMLE roMLE +GS, N 5, &- exp(.). Table 6 shows detailed breakdown of the scores on each of the five random seeds. Note that since we use different random seeds than the original GALAXY paper,
task on the MultiWOZ 2.0
unable to get its reported scores.
We see from Table that adding our reward learning and utilization methods improves the perfor- mance of the vanilla GALAXY, in almost all evaluation metrics, in both the Average and the Median scores. In particular, adding our RewardMi raMLE +GS method improves the average Combined Score of the vanilla GALAXY by 3.27, and adding Reward dNet +GS improves the vanilla GALAXY by 2.11. These relatively significant improvements may further demonstrate the effectiveness and general applicability of our proposed methods.
20

---

Published as a conference paper at ICLR 2023
Table 6: Per sendoom-seed results of the E2E response-generatiee task the MuluWOZZ.0 2.0 dataset, comparing
the vanilla GALAXY the variants with our proposed methods: RewarcNet +GS. N 3, (-) and
RewarcMLE +GS, .V 5, 1 exp()). Here, (.)² denotes the power function with power 1. "Comb." is the
Combined Score. The row "Median" shows the median score of the corresponding column over the five tested random sceds.
GALAXX
RewardNet +GS, N 3.
(-)-
RawardMis +GS, N 5.
exp(1)
Seed
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Comb.
93.60
85.40
19.97
109.47
91.30
83.00
19.03
106.18
92.50
84,10
18.89
107.19
333
86.50
77.90
17.87
100.07
90.00
82,00
17.95
103.95
90,90
82.70
18.27
105.07
555
89.20
81.60
19.96
105.36
92.00
83.40
19.44
107.14
93.90
85.50
19.47
109.17
777
90.40
81.90
18.81
104.96
93.70
85.10
18.90
108.30
95.80
86,
19.36
110.31
999
88,30
79.40
18.59
102.44
92.60
83.60
19.19
107.29
91,90
82.50
19.70
106,90
Average Median
89.60
81.24
19.04
104.46
91.92
83.42
18.90
106.57
93.00
84.18
19.14
107.73
89.20
81.60
18.81
104.96
92.00
83.40
19.03
107.14
92.50
84,10
19.36
107.19
F EXPERIMENTS ON THE MULTIWOZ 2.1 DATASET
To test the efficacy of our proposed methods on additional datasets, we run our two methods RewardNet + GS, N = 3. + = (-) and RewardMLE + GS, N 5, Φ = exp(.) the Mul- tiWOZ 2.1 dataset. Table 7 compares our two methods with the baselines SimpleTOD and UBAR in the main evaluation (Table 1), which also report results the MultiWOZ 2.1 dataset. Addition- ally, we also present our rerun of CASPI on this dataset. As in Table 1, the Combined Scores of methods are generally better than the baselines. In fact, our methods achieve both good task completion (Inform and Success rates) and fluent generated responses (BLEU score).
Table shows detailed breakdown of the scores of CASPI and two methods each of the five tested random seeds. We see that the scores of our methods are generally robust across ran- dom seeds. In fact, four out of those five seeds, at least one of our methods performs better than CASPI. Further, our methods have higher Average and Median scores than CASPI on each of the four evaluation metrics. This set of experiments may further demonstrate the efficacy of our proposed methods.
Table 7: Results of the E2E response-generation task the MultiWOZ. 2.1 dataset. The best result each metric bold. The results of SimpleTOD and UBAR are from original paper. The results of CASPI are from our reproduction. All our provided results are the average over five random seeds. "GS" denotes the Humbel-softmaal trick. (.)* denotes the power function with power 1.
Algorithms
Inform
Success
BLEU
Combined Score
TOD (Hosseini-Asl ct al., 2020)
85.00
70.50
15.23
92.98
UBAR (Yang al., 2021)
95.70
81.80
16.50
105.25
CASPI (Ramachandran et al., 2021)
91.43
83,50
17.93
105.40
RewardNet: + GS, 3, ④ = (-)
92.79
84.48
17.99
106.62
Rewa rdwLE + GS, N = 5, ① exp(-)
92.87
83.90
18.73
107.11
Table 8: Per random-seed results of the E2E response-generatioo task on the MultWWOZ 2.1 dataset, com- paring the CASPI and the variants with proposed methods: RewardNet +GS, N 3, (-) and RewardMLa +GS, N 5, exp(-). Here, (-)² denotes the power function with power 1. "Comb." " is Combined Score. The row "Median" shows the median score of the corresponding column over the five tested random seeds.
CASPI
RewardMet +GS, N 3,
(-) Comb.
RewardMLB +GS, N 5,
exp(.)
Sced
Inform
Success
BLEU
Comb.
Inform
Success
BLEU
Inform
Success
BLEU
Comb.
111
89.39
82.08
18.14
103.88
96.6
87.19
17.23
109.13
93.39
84.58
19.39
108.38
333
90.99
82.08
18.25
104,79
91.69
83.78
18.43
106.17
92.19
81.78
17.36
104,35
555
91.59
82.68
17.9
105.04
91.79
85.29
17.63
106.17
93.19
85.49
19.08
108.42
777
91.89
84.28
17.78
105.87
91.49
82.68
18.23
105.32
93.29
83.88
19.15
107.74
999
93.29
86.39
17.6
107.44
92.39
83.48
18.41
106.35
92.29
83.78
18.65
106.69
Average
91.43
83.50
17.93
105.40
92.79
84.48
17.99
106.62
92.87
83.90
18.73
107.11
Median
91.59
82.68
17.90
105.04
91.79
83.78
18.23
106.17
93.19
83.88
19.08
107.74
21

---

Published as a conference paper at ICLR 2023
G IMPLEMENTATION DETAILS
Our implementation based on the official codehase of Min'TL and CASPI. Apart from the hyper- parameters discussed in Section 5.2, most other hyperparameters and the training procedure of our models follow MinTL and CASPI. In addition to the discussion of our method in Section 3, we list the important hyperparameters for training our reward model in Table and the important hyperpa- rameters for training our response-generatio generation model in Table 10. Both BART models use the default token length of 142.
We note that unlike CASPI which uses the dialogue acts as the actions, the action space of our reward model the system response, which combinatorial space of the vocabulary. We use this action space because we want to pass gradients from the reward model to the E2E -eenerttoonnnn model during the training process, where the output the E2E model is the human-language-lie system response.
For the reward-function learning, we not change the length of the trajectories in dataset. The reward model is updated by the preference scores/orderings among multiple trajectories of the same length. Our intuition is that trajectories of the same length may roughly correspond to tasks of similar complexity, making the preference among them more comparable and meaningful. This approach is also taken by the prior work CASPI.
Our tested MultiWOZ2.0 dataset is publicly available at http://gghhb..mmmmuutiaaosssssit and the MultiWOZ2.1 dataset is available at http::/giinbbb.ommtttucccaaooaaa 2/tee/master/untamaltiivvo
Table Hyperparameters for training our reward model.
Table 10: Hyperparameters for training our response generation model.
Hyperparameter
Value
Hyperparameter
Value
Action space
System response
Action space
System response
Gradient clipping norm
1.0
Gradient clipping norm
1.0
Learning rate
3e-5
Learning rate
3e-5
Gradient accumulation steps
16
Gradient accumulation steps
8
Batch size
Batch size
Context window
2
Context window
Learning-rate decay
0.8
Learning-rate decay
0.8
Learning-rate scheduler
PedduceLROnPlateen
Learning raaiinng--tteeeee scheduler
LambdaLR
Scheduler patience
3
Early-stop count
strly-stop count
7
Seed
(111, 333, 777, 999)
Backbone
BART-base
Backbone
BART-large-cnn
22


Reviews:
[{"review_id": "w3RTTNeqAu", "reviewer": "Reviewer_tRUe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the question of how to efficiently learn and utilize a reward function for training end-to-end task-oriented dialogue agents. To be specific, the authors introduce two generalized objectives (RewardNet and RewardMLE) for reward-function learning, motivated by the classical learning-to-rank literature. Further, they also propose a stable policy-gradient method to guide the training of the end-to-end TOD agents. The proposed method achieves competitive results on the end-to-end response-generation task on the widely-used dialogue benchmark MultiWOZ 2.0.", "summary_of_the_review": "I like the problem of efficiently learning the reward function in end-to-end TOD dialogue systems. My main concern is the limited technical novelty of the proposed RL method for a machine learning conference like ICLR. Moreover, considering the unclear reproducibility, I tend to give a weak rejection.", "strength_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to understand. \n2. How to efficiently learn and utilize a reward function for training end-to-end task-oriented dialogue agents is fundamental and deserving problem.\n\nWeaknesses:\n1. The paper only conducts experiments on one benchmark MultiWOZ 2.0. As a methodological contribution, it will be more convincing to verify the effectiveness on more benchmarks.\n2. The novelty is limited. How to efficiently learn the reward function in RL-based dialogue agents is a classic problem. Various existing literature[1][2][3] has discussed this problem and proposed the corresponding solution.\n3. More implementation details are needed in the main paper, and I did not see the code. \n\nReferences:\n1. Curriculum Learning Based on Reward Sparseness for Deep Reinforcement Learning of Task Completion Dialogue Management\n2. Playing 20 question game with policy-based reinforcement learning\n3. Guided Dialog Policy Learning without Adversarial Learning in the Loop", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t326OrinAV", "reviewer": "Reviewer_7ZSu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " (To the authors, apologies for the delays - this was an emergency review taken on right before an unexpectedly busy week(s))\n\n In this work the authors explore ways in which to learn a reward function for task-oriented dialogue systems, proposing two new objectives phrased within the context of a learning-to-rank objective.  Given a set of scored (and hence ranked) TOD trajectories, the aim is to learn local reward functions whose accumulated scores reflect the original ranking.  They propose two methods, one doing just that (RewardMLE), the other minimizing the cross entropy between the two scores (RewardNet).  The learned reward function is incorporated into the training objective using Gumbel-softmax to reduce variance from the vanilla REINFORCE estimator.  On one benchmark dataset, variations of the proposed approach improve on standard TOD evaluation metrics, and show strong improvements in artificial data sparsity and human evaluations.", "summary_of_the_review": "I think the existing experiments are probably sufficient for publication if I had to come down one way or another, but they do leave a lot to be desired.  The method seems to be a useful generalization of existing reward learning for TOD and one that could be applied in many situations.  If the authors can show that the performance gains are consistent, and that high performance model configurations are somewhat identifiable in advance, then there is value in the method.", "strength_and_weaknesses": "Strengths:\nThe empirical results of this paper make a compelling case for its acceptance -- the method achieves a new state-of-the-art on the MultiWOZ 2.0 dataset, improving over the closely related CASPI method by ~1-1.5 pts in success and inform metrics.  In low data settings, the margins are quite large.  The analyses are generally good, and show advantages of this method in many settings.\n\nWeaknesses:\nWorking against the paper is perhaps the similarity between previously best method, and concerns about the strengths of the results.  The relatively small margins of improvements in main evaluation are gained only in certain hyperparameter settings.  If one were to choose an architecture with a given N or \\Phi in advance, there appears to still be a relatively good chance that there will be very small improvements (< 0.5, or even < 0.1) on some measures, to an extent that it would probably be insignificant.  Rather than tune these on some sort of dev set to mimic a realistic scenario, different runs get different rows in the table, and are presented in a manner of, ~\"Look, one of our methods will always win by a good margin\", only that which system that is is unfortunately not consistent.  I find this pretty disappointing.  It is saved primarily by the fact that usually all of these systems outperform (by microns) the previous SOTA, but it is not beyond reason to wonder if the case would be true for even more values of N, or on a different dataset or task.  This is an obvious place where the paper could be improved.\n\nThis relates to another weakness -- main results are averaged, and authors don't report the variance of their method.  The authors state \"overall performance significantly improves over CASPI\", but we are not given access to statistical significance tests / confidence margins over multiple runs.  These would be good to have in general, but the former is necessary to make claims of significance, and if that is not the authors' intent then it should be reworded.\n\nThe (automatic evaluation) results on MultiWOZ are promising, but the above issues make it difficult to have great faith in the results, given they are applied just to this dataset.  It would have been better to apply the method on a couple of other popular TOD datasets.  It seems KVRET and SGD benchmark are similar in spirit to MultiWoz.  The results presented here are maybe just about/above the threshold of publication, but there's certainly value in asking for additional experiments and metrics.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Znf5A-tMgFR", "reviewer": "Reviewer_NRwE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces two variants of reward learning functions in RL-based task-oriented dialogue (TOD) to address the issues of sparse and delayed rewards. RewardNet optimizes the rewards by computing each trajectory's total automatic evaluation scores and then sorting all trajectories by the scores. The RewardMLE uses only the ranking order instead of the automated evaluation scores. The ranked N dialogue trajectories are inputs into a neural network to learn the new rewards. \nThe RewardNet is updated by optimizing the cross entropy between accumulated predicted rewards and automatic evaluation scores. The negative log-likelihood is used for the RewardMLE.\nThe policy gradient-based agent predicts the action and updates its parameters with the learned rewards. Experiments  are performed on the MultiWOZ 2.0 dataset.", "summary_of_the_review": "This paper applied the learning-to-rank-based rewards functions to TOD tasks. The basic idea behind RewardNet is similar to the method proposed by the CASPI paper. Additionally, this paper runs more experiments with different numbers of trajectories, introduces a new variant, RewardMLE, and discusses the policy gradient-based agent with the Gumbel-softmax trick. The novelty of their main reward techniques is limited. Still, this paper proposes a reasonable combination of well-known RL methods and a meaningful research direction with strong experiment results for the TOD domain. It’s useful for future work on the MultiWOZ 2.0 dataset to compare.  The author didn’t provide their implementations; I’m unsure about the reproducibility.\n\n\n\nMinor suggestions: be sure to provide the full term when introducing an acronym; for example, section 3.2 (GS) occurs before its acronym Gumbel-softmax(GS).  It would be evident if Figure 1describe more details about the framework. \n", "strength_and_weaknesses": "## Strengths\n- The goal and structure of this paper are clear and easy to follow\n- This paper shows strong experiment results. Their methods achieve the best results on the MultiWOZ 2.0 dataset. \n- reasonable combination of well-known RL methods, new research direction in TOD domain\n\n## Weaknesses\n-  The novelty of reward techniques is limited\n- no implementation available, reproducibility unclear", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SaE_v4E-9u", "reviewer": "Reviewer_kmgW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to design reward functions for training TOD agents through reinforcement learning, inspired by Learning-to-Rank (LTR) methods. The authors demonstrate that their RewardNet and RewardMLE methods achieve strong improved performance on MultiWOZ 2.0 compared to SOTA methods for training TOD agents.", "summary_of_the_review": "This paper justifies its choice of approach to training TOD agents (via generalized rweard function designs) and effectively demonstrates their effectiveness in a multi-domain setting. I would have liked to see a more thorough discussion but the paper itself reads well.", "strength_and_weaknesses": "Strengths:\n- RewardNet and RewardMLE are both demonstrated to out-perform SOTA algorithms\n- The authors demonstrate that existing work (CASPI) can be framed as a specific case of their proposed reward algorithms (RewardNet to be precise), showing that their contributions are more general\n- The ablation questions (5.2) are well-posed and answered in a logical fashion\n- Human evaluation \n\nWeaknesses:\n- The number of trajectories seems to have a nontrivial effect on total score; in a real-world setting how well do e.g. the choices of N=3 for RewardNet and N=5 for RewardMLE perform? It would be helpful to see a discussion of parameter sensitivity.\n- Would have liked to see more discussion in terms of a breakdown between multi-domain and dialogs from each separate domain in MultiWOZ, to investigate if there is a disparate effect of learning a TOD agent via RewardNet/RewardMLE depending on the complexity of the conversation (e.g. length, number of slots/actions) and the domain complexity (e.g. possible slot values, uniformity of distribution of slot values).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster