One cannot stand for everyone! Leveraging Multiple User Simulators to train Task-oriented Dialogue Systems
Authors: Yajiao LIU
Xin Jiang
Yichun Yin
Yasheng Wang
Fei Mi
Qun Liu
Xiang Wan
Benyou Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: User simulators
Task-oriented Dialogue Systems.

Abstract: User simulators are agents designed to imitate human users; recent advances have found that Task-oriented Dialogue (ToD) systems optimized toward a user simulator could better satisfy the need of human users. However, this might result in a sub-optimal ToD system if it is tailored to only one \textit{ad hoc} user simulator, since human users can behave differently. 
In this paper, we propose a framework called MUST to optimize ToD systems via leveraging \textbf{m}ultiple \textbf{u}ser \textbf{s}imula\textbf{t}ors. 

The main challenges of MUST fall in 1) how to adaptively specify which user simulator to interact with the ToD system at each optimization step, since the ToD system might be over-fitted to some specific user simulators, and simultaneously under-fitted to some others; 2) how to avoid catastrophic forgetting of the adaption for a simulator that is not selected for several consecutive optimization steps.
To tackle these challenges, we formulate MUST as a Multi-armed bandits (MAB) problem and provide a method called MUST$_{\mathrm{adaptive}}$ that balances 
\textit{i}) the \textit{boosting adaption} for adaptive interactions between different user simulators and the ToD system and
\textit{ii}) the \textit{uniform adaption} to avoid the catastrophic forgetting issue.
With both automatic evaluations and human evaluations, our extensive experimental results on the restaurant search task from MultiWOZ show that the dialogue system trained by our proposed MUST achieves a better performance than those trained by any single user simulator. It also has a better generalization ability when testing with unseen user simulators. Moreover, our method MUST$_{\mathrm{adaptive}}$ is indeed more efficient and effective to leverage multiple user simulators by our visualization analysis.

Under review as a conference paper at ICLR 2023
ONE CANNOT STAND FOR EVERYONE! LEVERAGING MULTIPLE USER SIMULATORS TO TRAIN TASK-ORIENTE DIALOGUE SYSTEMS
Anonymous authors Paper under double-bling review
ABSTRACT
User simulators are agents designed to imitate human users; recent advances have found that Task-oriented Dialogue (ToD) systems optimized toward a user simu- lator could better satisfy the need of human users. However, this might result in a sub-optimal ToD system if it is tailored to only one ad hoc user simulator, since human users can behave differently. In this paper. we propose a framework called MUST to optimize ToD systems via leveraging Multiple User SimulaTors.
The main challenges of implementing the MUST fall in 1) how to adaptively spec- ify which user simulator to interact with the ToD system at each optimization step, since the ToD system might be over-fitted to some specific user simulators, and si- multaneously under-fitted to some others; 2) how to avoid catastrophic forgetting of the adaption for simulator that is not selected for several consecutive opti- mization steps. To tackle these challenges, we formulate MUST as a Multi-armed bandits (MAB) problem and provide method called MUST adapivee that balances i) the boosting adaption for adaptive interactions between different user simula- and the ToD system and ti) the uniform adaption avoid the catastrophic forgetting issue. With both automatic evaluations and human evaluations, our ex- perimental results on the restaurant search task from MultiWOZ show that the dialogue system trained by our proposed MUST achieves a better performance than those trained by any single user simulator. It also has a better generaliza- tion ability when testing with unseen user simulators. Moreover, our method aaSTade can efficiently leverage multiple user simulators to train the ToD system by our visualization analysis on convergence speeds.
INTRODUCTION
Task-oriented dialogue systems aim to help users accomplish their various tasks (e.g., restaurant reservations) through natural language conversations. Training task-oriented dialogue systems in supervised learning (SL) approaches often requires a large amount of expert-labeled dialogues, how- ever collecting these dialogues is usually expensive and time-consuming. Moreover, even with large amount of dialogue data, some dialogue states may not be explored sufficiently for dialogue systems (Li al. 2016b). To this end, many researchers try build simulators mimic human users for generating reasonable and natural conversations. By using a user simulator and sampled user goals, we can train the dialogue system from scratch with reinforcement learning (RL) algorithms. Previous works tend to design better user simulator models (Schatzmann et al.) 2007 Asri et al. 2016) Gur et al., 2018 Kreyssig et al., 2018 Lin al. 2021). Especially, (Shi et al. (2019) builds various user simulators and analyzes the behavior of each user simulator in the popu- lar restaurant search task from MultiWOZ (Budzianowski et al. 2018).
In real application scenarios, the deployed dialogue system needs to face various types of human users. single ad hoc user simulator can only represent one or a group of users, while other users might be under-represented. Instead of choosing the best-performing one from many dialogue systems trained by different single user simulators, we believe that it is worth trying to train a dialogue system by leveraging all user simulators simultaneously.
We use the dialogue systems refer to the task-oriented dialogue systems for simplicity in this paper.

---

Under review as a conference paper at ICLR 2023
In this paper, we propose a framework called MUST to utilize Multiple User SimulaTors simulta- neously to obtain a better system agent. There exist several simple ways to implement the MUST framework, including merging strategy, continual reinforcement learning (CRL) strategy, and a uniform adaption strategy, denoted as MUST merging. mergingg. MUSTCRL. and MUST umiform respectively (See Sec. B.2.. However, none of them could effectively tackle the challenges: 1) how efficiently leverage multiple user simulators when training the dialogue system since the system might be eas- ily over-fitted to some specific user simulators and simultaneously under-fitted some others, and 2) it should avoid a catastrophic forgetting issue. To tackle them effectively, we first formulate the problem as a Multi-armed bandits (MAB) problem (Auer et al. 2002); similar to the exploitation vs exploration trade-off, specifying multiple user simulators should trade off a boosting adaption (tack- ling the challenge 1) and uniform adaption (tackling the challenge 2), see Sec. [4] for more details. Then we implement new method called MUST 'daattve which utilizes an adaptively-updat dis- tribution among all user simulators to sample them to train the dialogue system in the RL training.
Our experimental results on the restaurant search task from MultiWOZ with both automatic evalua- tions and human evaluations show that the dialogue system trained by our proposed MUST achieves better performance than those trained by any single user simulator. It also has better general- ization ability when testing with unseen user simulators and is more robust to the diversity of user simulators. Moreover, the visualization analysis on convergence speeds demonstrates that our MUST MUSTadaptive: is more efficient than MUST uniform to leverage multiple user simulators to train dialogue systems.
Our contributions are three-fold: (I) To the best of our knowledge, our proposed MUST is the first developed work to improve the dialogue system by using multiple user simulators simultaneously; (2) We design several ways to implement the MUST. Especially, we formulate MUST as Multi- armed bandits (MAB) problem, based on which we provide a novel method MUST adaptivve and (3) The results show that dialogue systems trained with MUST consistently outperform those trained with a single user simulator through automatic and human evaluations. Especially, it largely im- proves the performance of the dialogue system tested on out-of-domain evaluation. Furthermore, training the system with the proposed MUST can converge faster than with MUST uniform-
MUSTuniform:
2 BACKGROUND
Dialogue System. Task-oriented dialogue systems aim to help users accomplish various tasks such as restaurant reservations through natural language conversations, Researchers usually divide the task-oriented dialogue systems into four modules (Wen et al.]
2017 Ham et al. 2020 Peng et al. 2021]: Natural Language Understanding (NLU) (Liu & Lane
2016) that first comprehends user's
intents and extracts the slots-values pairs, Dialog State Tracker (DST) (Williams et al.) 2013) that tracks the values of slots, Dialog Policy Learning (POL. (Peng al.
2017|2018) that decides the
dialog actions, and Natural Language Generation (NLG) (Wen et al.
2015 Peng et al.) 2020) that module and the POL module usually are collectively referred to as the dialogue manager (DM) (Chen et 2017). These different modules can be trained independently or jointly in an end-to-end manner ( (Wen et al. 2017 Liu &
translates the dialog actions into latguaallanguage form. The DST
Lane 2018 Ham et al. 2020 PPng et al. 2021 Hossemii-AsI et al. 2020).
User Simulator. The user simulator is also an agent but plays a user role. Different from dia- logue systems, the user agent has goal describing a target entity (e.g., a restaurant at specific location) and should express its goal completely in an organized way by interacting with the system agent (Takanobu
al. 2020). Therefore, besides the modules of NLU, DM, and NLG like dia- logue systems, the user agent should have another module called Goal Generator (Kreyssig et al.) 2018). which is responsible for generating the user's goal. Building a user simulator usually could use an agenda-based approach (Schatzmann et al., 2007 Schatzmann & Young, 2009) designing handcrafted rules mimic user behaviors or a model-based approach such as neural networks (Asri
et al. 2016 Kreyssig et al. 2018 Gur et al. 2018) learned on corpus of dialogues.
Training Dialogue Systems with a User Simulator. the beginning of a dialogue, the user agent obtains its initial goal from the Goal Generator and then expresses its goal in natural languages. For the system agent, it does not know the user's 's goal and it should gradually understand the user's 's utter- ances, query the database to find entities, and provide useful information to see if it is accomplishing
2

---

Under review as a conference paper at ICLR 2023
AgenT
08
typc-
User Siuuston
AgenRt
008
inform type change
0.7
info
enything else
RNNR
0.4
make reservation
INN
033
make resorvation change
ApenAgent
Sye Ageeff
6y RRNWT
Spo-owt
02
goodbye
NUS
Dlaguee Systems
0.00
0.05
0.30 0.35
(a) Success rates of different systems,
(b) Dialog act distributions of different user simulators.
Figure I: (a) is the heat map on the success rates of system agents tested by different user simulators on 200 dialogues. (b) shows the dialog act distributions of Agenda-basse based User Simulators (ABUS) and Neural networks-based User Simulators (NUS) provided by Shi et al. (211)). There exist seven user dialog acts annotated in the restaurant search task from MuldWOZ, shown the Y-axis.
the user's task. Since only the system access the database, the user does not know if its goal can be satisfied. When the database result returned by the system agent is empty, the user agent should learn compromise and change its goal with the help of Goal Generator (Kreyssig et al.) 2018). When the dialogue ends, the user simulator will reward the system agent according to if the system agent accomplishes the task. Then we could use the reward to update the system agent with RL algorithms (Teeng et al. (021)).
3 MUST: LEVERAGE MULTIPLE USER SIMULATORS
3.1 MOTIVATIONS TO LEVERAGE MULTIPLE USER SIMULATORS
User simulators behave differently. Shi et al. 2019) implement six user simulators (Agen'T, AgenR, AgenG, RNNT, RNNR RNN with both agenda-base baaed methods and neural networks- based methods on the popular restaurant search task from MultiWOZ (Budzianowski et al. 2018)) From their experiments, we observed that the dialogue systems trained by different user simulators vary in their performances (i.e., the success rates tested by the same when interacting with the user simulator of AgenT, the success rates of the system agents trained by Agenda-bun Agenda-based based user simulators (i.e., AgenT, AgenR, AgenG) are much higher than the system agents trained by RNN-based user simulators (i.e., RNNT, RNNR, RNN), see Fig. [(a) The reason might be that these user simulators (i.e., with either handcrafted rules data-driven learning in their DM modules) have different user dialog act distributions (see Fig. ((b) which determines the dialogue
simulators). For example,
state space explored by the dialogue system.
One cannot stand for everyone. Users might behave differently, one could design different user simulators with specific user dialog act distributions, see Shi et al. (2019). A single user simulator leameed on task-oriented dialogue corpus can just represent one or a group of users, while the dialogue system needs to accomplish tasks from various human users real scenarios. We argue that it is beneficial to utilize all different user simulators to train the dialogue system. By leveraging multiple user simulators that have different user dialog act distributions, the dialogue systems can explore larger dialogue state space, which might improve the ability of the learned dialogue system.
3.2 SOME PRELIMINARY PROPOSALS FOR MUST
We propose framework called MUST, the core idea of which is to train a better dialogue system by leveraging Multiple User SimulaTors simultaneously. There are several simple ways to imple-
Here we rename the user simulators of SLT. SLR, and SLE in Shi et al. 2019) as RNNT, RNNR, RNN for emphasizing the model structure of their DM modules.
'The dialogue policy learning module is essential in both dialogue systems and user simulators. A policy module corresponds to a dialog act distribution since it decides to take which dialog act to respond to the current dialogue state. The user dialog act distribution behind user simulator determines the diversity of the dialogue state space explored by dialogue systems; therefore might affect the performance of system agents.
3

---

Under review as a conference paper at ICLR 2023
Table 1: The comparison of different strategies for leveraging multiple user simulators.
dynamic adaption
avoid catastrophic forgetting
efficiency
MUST Immrringg
MUSTCRR.
MUST unifform
MUST addatiivee
ment our MUST, including merging strategy denoted as MUST merging. Continual Reinforcement Learning strategy denoted as MUSTCRL. and a uniform adaption strategy denoted as MUST MUSTumiforms uniform-
(I) MUST merging first samples some dialogues from each simulator and the corresponding dialogue system trained by this simulator. Then it combines the collected dialogues to train a new user simulator for ensembling different user dialog act distributions. Finally, it uses this new simulator to train the dialogue system with RL.
M MUST Ireatt each user simulator as independent RL environment. It moves the trained system agent to another environment (i.e., let the system agent interact with another user simulator) if the system converges in the current RL environment.
(III) MUST allows the system agent have chances to interact with all user simulators si- multancously. Different from MUSTCRL. MUSTunifoma puts all user simulators in single RL environment and adopts the simplest way to specify different user simulators to train the dialogue system, which is to pick user simulator among all user simulators with a uniform distribution for each iteration in the RL training.
Challenges to leverage multiple user simulators. The problem with MUST merging is that it be- comes difficult to adjust the weights of each user simulator adaptively in the training process. Since the proportions of dialogues from each user simulator are fixed in MUST merging. some user simu- lators might be well-adapted and some might not. The MUSTCRL strategy has a problem of catas- trophic forgetting (Khetarpal et al. 2020) and would be sensitive to the order of different user agents interacting with the dialogue system, which might result in obtaining a sub-optimal dialogue sys- tem. As(Shi al. (2019) shows, the system agents trained by different user simulators have different convergence speeds and converged performances. Namely, the system agent might be easily fit- ted to some user simulators but might be hardly fitted to others. The uniform distribution for user simulator selection under MUST will result in inefficient training since it would be redundant assign the same training costs let the dialogue system interact with easily- adapted user simulators. Overall, the challenging problems under the MUST framework are 1) how to efficiently leverage multiple user simulators to train the system agent, and 2) avoiding the catastrophic forgetting issue.
4 MUST AS A MULTI-ARMED BANDIT PROBLEM
To tackle the challenges under the MUST, we first formulate MUST as a Multi-armed bandit (MAB) problem, see Sec. 4.1 In Sec. 4.2, we propose a method called MUST adattveeevv to use an adaptively- updated distribution to replace the uniform distribution under the MUST unnffor for accelerating the MUST training. We briefly compare these different implementations of MUST in Tab. T
4.1 FORMULATE MUST AS A MULTI-ARMED BANDIT PROBLEE
Adaptively specifying different user simulators to train the dialogue system reminds us of a similar concept machine learning, which is the boosting strategy (Zhou) 2012). From a boosting point of view, one should increase the weights of weakly-performing data examples and decrease the weights for well-performing ones. In MUST, we accordingly assume that should reduce the interactions between the dialogue system and those user simulators that the system has performed well and increase the interactions between the system and other user simulators that the system performs poorly. We refer to this strategy boosting adaption.
'Continual Reinforcement Learnig (CRL.) Khetarpal et al. 2020 is sequential learning paradigm for training agent with RL. algorithms.
4

---

Under review as a conference paper at ICLR 2023
Meanwhile, we should also give some chances to all user simulators to relieve the catastrophic forgetting issue. We refer to this as uniform adaption. Such a trade-off between boosting adaption and uniform adaption is similar to the the exploitation exploration trade-off existing in the Multi- armed bandit (MAB) problem (Auer et al. 2002).
Here, we interpret MUST MAB problem. We treat each user simulator as arm. Suppose there are K arms (simulators), and each arm has a fixed but unknown reward distribution RG with an expectation μ4. At each time step { 1,2 .... T, one must choose one of these K arms. We denote the arm pulled at time step / as i E {I,..., K}. After pulling an arm, it will receive a reward di drawn from the arm's 's underlying reward distribution. The decision maker's objective to maximize the cumulative expected reward over the time horizon
T Σ
T
Elai
Σ Mic
(I)
In MUST, the reward received in each arm- cam-pulling step refers to the possible performance gain of the dialogue system after it interacts with selected user simulator. A significant difference between the standard MAB problem and MUST is that the reward expectation of a user simulator (arm) in MUST is not static; it changes over time. For example, by consecutively interacting with the same user simulator, the performance gain (reward) of the system will decay since the system might be in saturation or overfitting this simulator. Moreover, the performance gain of the system after interacting with simulator might increases if the simulator has not been selected for a period. To deal with this difference, we should tailor the solution of MAB to the MUST framework.
4.2 TRAINING THE DIALOGUE SYSTEM WITH MUST adaptive
To solve this MAB problem MUST, we implement a method called MUSTadaptin with a two- phase procedure, as presented Algorithm [ Similar to the UCBI algorithm, MUSTadaptie specifies user simulators in a uniform distribution train the dialogue system S in the first T Twarmup steps (i.c., in the warm-up phase). After that, the adaptive phase will balance the boosting adaption and the uniform adaption by introducing specify different user simulators to train the system in later RL. training. To accelerate the RL. training, intuitively, p is expected to assign lower weights performs well and higher weights to those user simulators with which S performs poorly.
adaptively-updated distribution p, which is used to
simulators with which S already
(1) Warm-up phase: in the first Tarmup dialogues, we use uniform distribution to sample all user simulators to train the system agent S (lines 4-7). This phase is mainly used to warm up the dialogue system S and make it have little ability to converse with all user simulators.
(2) Adaptive phase: in this phase, the distribution used to sample all user simulators will be adaptively updated, which is why we call this phase adaptive phase. When this phase begins (i.e., t 0), we will first evaluate the performance (i.e., the success rate æ,.j E {1..... K) {)) of the dialogue system trained after the warm-up phase. The success rate F. is obtained by letting S interact d times with the simulator Uj (e.g.. E {I, K}) and calculating the success rates.
Inspired by UCB1 Auer et al. (2002), we design a calibrated performance expectation é, of the system agent S interacting with each user simulator U, taking exploration into consideration beyond pure exploitation:
2lnt T/.
1 JE (I,.,.,, K);
exploitation
exploration
where ãj is the success rate of the system agent S tested with user simulator Uz. T3. is the number of times user simulator Uj has been selected with so far. Then we normalize ē into
2j = 1/ (é, 7 min(f51, (xxn((),.-..xxx))..
'hhere exists an algorithm called the Upper Confidence Bound (UCBI) (Auer al. 2002) that could solve the MAB problem. This policy first pulls each arm once in the first K steps, then will play the arm that could maximize the sum of two terms: it = arg max; ( +
(2 7,,
from f = K + 1 to T.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Implement MUST adaptive with the modified UCBI algorithm
Input: K fixed User simulators U = (U1,U2,-- -.. UK} and the values of hyperparameters Twarmup e, d, d, T:
Initialization: randomly initialize System agent S;
Initialization: initialize the simulator sampling distribution p as a uniform distribution.
(1) Warm-up phase:
for (
0, (o Twormup do
sample simulator Ug in U w.r.t the distribution p:
synthesize a new dialogue using the system agent S and the sampled Uy
use the reward obtained for the dialogue to update S with RL algorithm;
(2) Adaptive phase:
for 1, 0, 7 do
10
if t%e = 0 then
11
for j = I.... do
evaluate the performance i.e. the success rate Ij of the agent S by letting it interact times with the simulator Uj:
13
update p based on these success rates {T1, Tg (see Eq. else
Eq. and Eq.
15
sample a simulator U, in w.r.t the distribution p:
synthesizing a new dialogue using the system agent S and the sampled U5 use the reward obtained for the dialogue to update S with a RL algorithm;
17
Output: The learned dialogue system S.
Eq. 3g..3ppmm penalizes the user simulators with which the dialogue system already performs well the ex- pectation term. Where the hyperparameter 7 is the smooth factor for distribution p {PI, ... PK the larger is, the sharper p is. Each probability in is calculated as
Pj
(4)
Σ
In the following T dialogues, we will specify all user simulators to train the system agent S with this distribution p (lines 15-18). We will also evaluate the RL model S for every episodes (line 10-12) and update the distribution p with the new success rates (line 13).
Difference with the original UCB1. The main differences between our modified UCBI algorithm and the original UCB1 algorithm are twofold. First, we tailor the original UCB1 into our scenario by using Eq. 3 to penalize the user simulators with which the dialogue system has performed well. Secondly, we adopt sampling schema based on a well-designed distribution (see Eq. 4 instead of taking the arm with the highest expectation. This is to increase the diversity and flexibility of arm selection. We also discuss the differences between our MUST adaptive and some other related works in App. H
5 EXPERIMENTS
To verify the effectiveness of our proposed MUST, we benchmark the system agents trained either with single simulator multiple user simulators (including MUSTmerging: MUST d.) For the MUST strategy, we will discuss it in the App.
merging- MUST MUSTauiform. uniforms
5.1 EXPERIMENTAL SETUP
Available user simulators. There are six user simulators provided by Shi et al. (2019), which are Agenda-Templaas (AgenT), Agenda-Retrieva (AgenR), Agenda-Generation (AgenG), RNN- Template (RNNT), RNN-Retrieval (RNNR), ENN-End2End (RNN) trained with different dialog planning and generation methods. The NLU modules of all six user simulators using the RNN model. The DM modules of AgenT, AgenR, and AgenG are rule-based methods. For the NLG module, these three simulators using the template, retrieval, and generation methods respectively.

---

Under review as a conference paper at ICLR 2023
The DM modules of RNNT, RNNR are using Sequicity (Lei et al., 2018) their backbones which is an ANN-based seq2seq model with copy mechanism. The NLG modules of these two simulators using the template and retrieval methods respectively. The user simulator of RNN uses Sequicity as its backbone in an end-to-end manner.
Baselines. The baselines are the dialogue systems trained by each user simulator, including Sys- Agen'T, Sys-AgenR, Sys-AgenG, Sys-RNNT, Sys-RNNR, and Sys-RNN. For fair comparison, all system agents (including the systems trained by our MUST) have the same architecture described in Shi et al. (2019). See basic modules of user simulators and dialogue systems in App. B.1
MultiWOZ. Restaurant Domain Dataset. The original task in MultiWOZ (Budzianowski al., 2018) is to model the system response. [hh et al. (2019) annotate the user intents and the user-side dialog acts in the restaurant domain MultiWOZ to build user simulators, which has a total of 1,310 dialogues. Moreover, we randomly simulated 2,000 dialogues from each rule-based simulator (i.e., Agen'I, AgenR, AgenG) and their corresponding system agents respectively, and processed these dialogues to have the same annotation format as the MultiWOZ restaurant domain dataset. We denoted this dataset as Simulated Agenda Dataset, which has a total of 6,000 dialogues.
Evaluation Measures. The direct automatic metric to evaluate the dialogue system the success rafe tested by each user simulator. We calculate the success rate between user simulator and sys- tem agent by sampling 200 dialogues. We exclude some user simulators MUST training and test the systems with them out-of-domain evaluation. According the previous study (Gunasekara jet al. (2020), there usually is a gap between automatic evaluations and human evaluations of dia- logue systems. Therefore, system has conversed with 5 different users; each user has 10 dialogues. In total, we collect 50 dialogues for each dialogue system calculate its success rate. See more details in App. B.5
ask human users to converse with dialogue systems. Each dialogue
5.2
IMPLEMENTATIONS
5.2.1
TWO NEW USER SIMULATORS
We believe Pre-trained I Language Models (PLMs) might improve the capacity of user simulators since they have recently shown remarkable success in building task-oriented dialogue systems (Ham) iet al., 2020 Peng et al. 2 Hosseini-Asi et al. 2020). Here we implement another two user simulators using GPT (Radford et al. 2018 2019). Building user simulator using GPT similar to building a ToD system with GPT. See more details in App.
GPT Simulator. It is first fine-tuned on the simulated agenda dataset and then fine-tuned on the MultiWOZ restaurant domain dataset by leveraging GPT. This user simulator will be used to help implementing MUST.
GPTn. Simulator. For implementing the MUST merging strategy, similar to Imitation Learning (IL), we first train a new user simulator with dialogue sessions collected from different user simula- tors and their corresponding dialogue systems. We also learn this new user simulator based on GPT model and denote it as GP GPTn. is first fine-tuned on the simulated agenda dataset. Then we sample 1,400 dialogues from the simulated agenda dataset and merge them with 1,310 MuliWWZZ restaurant domain dialogues to continue fine-tuning GPT
5.2.2 DIALOGUE SYSTEMS
Sys-GPT is trained with the single user simulator GPT. Sys-MUST merging is trained with GPT, Sys-MUST trained by the user simulators of Agen'T. AgenR, RNNT, and GPT with uniform sampling distribution. For training Sys-MUST adaptive [ the distribution p will be adap- tively updated using our modified UCBI algorithm. We also train the Sys-MUST, mimform and Sys-MUST adaptive by using different subsets of the user simulators for ablation studies. See more details in D
'See implementations of dialogue systems App. B.2 and policy gradient algorithm in App B.3
7

---

Under review as a conference paper at ICLR 2023
Table 2: The success rates of the system agents were tested against various user simulators. Each column represents a user simulator, each row represents dialogue system trained with a specific simulator, e.g., Sys-AgenT means the system trained with AgenT. Each entry shows the success rate on 200 dialogues collected from a user simulator and a system agent. We use four user simulators: AgenT, AgenR, RNNT, and GPT simulator to implement MUST uniformm and MUST adaptive-
In-domain evaluation
Out-of-domain evaluation
Dialogue Systems
Agen'i
AgenR
RNNT
GPT
AgenCi
RNNR
RNN
Avg.t Std..
Avg.1
Std.j
Sys:Agen'
97.5
54,0
Jas.or
98.5 1o.3%
78.01.99
9.9
72.5
92.5
77.0
80.7
8.6
81.4
14.8
single
Sys-AgenR
96,0 s
90.0
98.5 98.51..
80,51,
97.5
97.5
82.0
92.3
7.3
91.7
7.1
Sys-RNNT
30.5 loa
23.0 174.05
99.0
75.517
35.5
97.5
84.0
72.3
26.6
63.6
30.5
Sys-GPT
60.5
Laz
51.5
97.0
lo.0%
82.0
59.5
92.0
76.6
17.6
Sys-MUST merzing
97.5
To.ess
83.5
4..2%
94.5 1a.0%
80.5
97.3
94.0
82.5
91.3
6.4
90.0
6.9
MUST
Sys-MUST uniform
97.5
89.0
43.0%
97.5
41.5%
82,5 To
96.5
96.0
87.5
93.4
4.2
92.4
5.6
Sys-MUST uddpptive
97.5 To
89.5
97.01
82.5 to
96.5
97.5
90.0
94.7
3.3
92.9
5.3
[1] The underlined number represents the success rate between a user simulator and its corresponding
dialogue system trained by this user simulator. The increasing and decreasing percentages (in red and green colors) the underlined numbers the base success rates.
[2] 4 (1) indicates by what percentages the success rate has decreased (increased) compared with the hase success rate by interacting with the same user simulator.
5.3 EXPERIMENTAL RESULTS
Automatic Evaluation. As seen in Tab. Sys-MUST and Sys-MUST adaptive outperform the dialogue systems (Sys-AgenT, Sys-AgenR, Sys-RNNT, and Sys-GPT) trained by single user simulator in the overall performance, demonstrating the superiority of leveraging multiple user sim- ulators. Especially, Sys-MUSTadapthe adaptivee has a 1.2 absolute value improvement (92.9 vs. 91.7) av- eragely over the previous SOTA system Sys-AgenR. Observing that Sys-MUST merging is not as competitive as Sys-MUST uniform and Sys-MUST sdaptivee this comparison shows that the merging strategy cannot effectively leverage multiple user simulators.
In in-domain evaluation, the performances of systems (Sys-AgenT, Sys-AgenR, Sys-RNNT, and Sys-GPT) trained by a single user simulator drop lot when testing with a different user simulator. requires us to delicately select a suitable user simulator for obtaining a good dialogue system. However, human users might be multi-facet or unknown, which makes the selection become difficult. Therefore, it is essential to leverage multiple user simulators when training dialogue sys- tems. At least, the performance gap of dialogue systems trained with MUST becomes smaller than without MUST, see the percentages labeled in green and red colors.
In out-of-domain evaluation where the user simulators used for testing the systems are unseen by our MUST, Sys-MUST uniform and Sys-MUST adaptive achieve at most 2.4 absolute value im- provement over Sys-AgenR. This evidences that MUST has a better generalization ability for in- teracting with unseen user simulators. Moreover, the dialogue systems (Sys-MUSTmeraa)) merging. Sys- unifnns and Sys-MUST anpptivo)) trained with the proposed MUST approaches have lower standard deviations, which indicates that they are more robust to the diversity of user simulators.
Human Evaluation. The human evaluation results in Tab. 3 show that our Sys-MUSTumiform and ays-MUSTadaptive largely outperform the other dialogue systems when interacting with real users. The consistency between automatic evaluations and human evaluations evidences the effectiveness of our proposed MUST.
Table 3: Human evaluation.
human
Dialogue Systems
evaluation
Sys-AgenT
76.0
Sys-AgenR
84.0
single
Sys-RNNT
34.0
5.4 ANALYSIS AND DISCUSSIONS
Sys-GPT
58.0
Sys-MUST MuSTmegingg
90.0
Convergences between MUST uniform and MUST adaptive- Fig. we show the learning curves of Sys-MUST uniform and Sys-MUSTadaptive in 100,000 steps; the first 40,000 steps are
MUST Sys-MUST aniiform
92.0
Sys-MUST addativv
92.0
in the warm-up phase for Sys-MUST adaptivee From Fig. 2(a) we can see that training the dia- logue system with AgenT, AgenR, RNNT, and GPT by MUST adaptive can converge faster than by MUSTuuifora. We do ablation studies on our modified UCB1 algorithm to help understand- ing the designed distribution p, see details in App. E, We further plot the performances of the dialogue system tested by each user simulator in the RL. training, which is shown in Fig. 2(b) 2(e)
8

---

Under review as a conference paper at ICLR 2023
Aonpr Way of Lak
PRB
18000
MRD:
(a) The learning
AgenR
(c) Agen'T
GPT
RNNT
The learning curves of Sys-MUST uniform and Sys-MUST adaptive. (a) shows their average success rates tested with all user simulators (AgenT, AgenR, RNNT, and GPT). The success rates of them tested with each user simulator shown in (b)-(e).
Figure 2:
uniform
AgenT
20.3%
The probabiltty
0.4
0.3
0.2
0.1
0.0
number dialogues
(a) The sampling proportion of simulators.
(b) Variations the sampling proportions (in every 2000 steps) of simulators.
Figure 3: The sampling proportions of user simulators average (a) and time horizon (b).
Visualization of the patterns learned by MUSTadaptive Let us define the adaptation difficulty of a user simulator using how many steps it must take to train the dialogue system with this user simulator until converges. The adaptation difficulty of all user simulators could be ranked like AgenR A GPT RNNT according to Fig, 2(b 2(e) To check whether MUST adaptive tends to sample harder-to-adeee user simulators more times In the adaptive phase, assumed in Sec. we visualize the sampling proportions all user simulators Fig. 3(a] We could ob- serve that AgenR was sampled with 45.1 %% (the biggest proportion) and it is indeed the hardest user simulator that can be adapted by the system; RNNT has the smallest sampling proportion and the easiest user simulator that can be adapted by the system. The consistency between the adapta- tion difficulty and sampling proportions for these four user simulators evidences our assumption in Sec. 4.2 Fig. 3bb visualizes the variations of the sampling distributions of user simulators. Inter- estingly, it shows that AgenR and AgenT are competitive with the GPT simulator; while RNNT and GPT cooperative with each other. This might be because RNNT and GPT simulator are learned from the dialogue corpus and will share similar behaviors.
CONCLUSION
In this paper, we propose a framework named MUST to improve the system agent by using multiple user simulators simultaneously. We discuss several simple methods implement MUST, which is either inflexible or inefficient. Therefore, we formulate MUST as a Multi- bandits (MAB) problem, based on which we propose novel implementation called MUST aUSTadaptive:: The experimen- tal results on the restaurant search task from MultiWOZ demonstrate that our proposed MUST can largely improve the system agent upon the baseline methods, especially when tested with unseen user simulators. Moreover, MUSTadaptioe is robust to the diversity of user simulators and its train- ing more efficient than MUST MUSTuniform- The main limitation this work is that we only conduct experiments the restaurant domain of the MultiWOZ since can only find multiple user simulators from Shi et al. (2019) and they build these simulators only on the restaurant search task. In future work, plan apply our proposed methods multi-domain scenarios.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Layla El Asri, Jing He, and Kaheer Suleman. A sequence-to-ssquennc model for user simulation in spoken dialogue systems, 2016.
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit
problem. Machine Learning, 47(2-3)2235-556 2002. URL http::/hhneet dsi. unimi.
Pawel Budzianowski. Tsien-Hsien Wen. Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ. a large-scale multi-domain Wizard-of-Oz, dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-Novente 2018. Association for Computational Linguistics. doi: 10. 1/D88/111D88-1544/ URL https: 7 hwwp:athhhooloy..//1b1t-1--
Yun-Nung Chen, Asli Celikyilmaz, and Dilek Hakkani-Tür. Deep learning for dialogue systems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties: Tutorial Abstracts, pp. 8 8-14, Vancouver, Canada, July 2017. Association for Computational Lin- guisties. htttst:s:: http://caaaanttooog...... org.
5004
R. Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D' Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, Dilek Hakkani-Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Liden, Kaili Huang, Shahin Shayan- deh, Runze Liang, Baolin Peng, Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David R. Traum, Max- ine Eskénazi. Ahmad Beirami, Eunjoon Cho, Paul A. Crook, Ankita De. Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, and Rajen Subba. Overview of the ninth dialog system technology challenge: DSTC9. CoRR, abs/201 1. 1.06486, 2020. URL https: waaxxi..org/abs//1111
Izzeddin Gur, Dilek Hakkani-Tur, Gokhan Tur, and Pararth Shah. User modeling for task oriented dialogues, 2018.
Dongoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pp. 583-592, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2020.em acl-man.544 URL https:
//aclanthoology.com http:hhoooooo...r./20.....a..a.. acl-0...clmaa
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for toik-trie dialogue. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20191.20191. Curran Associates, Inc., 2020. URL https: /ttttp::/pproceedingg. .neurps...... Paper.pdl
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: A review and perspectives. CoRR, Abs/2012.13449) 2020. URL https:
13490
Florian Kreyssig, Iñigo Casanueva, Pawet Budzianowski, and Milica Gašić. Neural user simula- tion for corpus-based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pp. 60 69, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/1/111850000 URL fttt::://acllathelooss ort/ 5007
Wenqiang Lei, Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequee architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1437-1447, Melbourne, Australia, July 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/P18. 1PPPP---1333 1133. URL https: hollantoooooogy...oooooooooooola orr/
P
10

---

Under review as a conference paper at ICLR 2023
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Austin, Texas, Novem- ber 2016a. Association for Computational Linguistics. doi: 10.18653W1/D11--1122) URL. Mitts://acllantho wot.hoo/oooooooooooooo/////
Xiujun Li, Zachary Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. A user simulator for compleonnnn dialogues. arXiv preprint arXiv:1612.05688, 2016b.
Hsien-chin Lin, Nurul Lubis, Songbo Hu, Carel van Niekerk, Christian Geishauser, Michael Heck, Shutong Feng, and Milica Gasic. Domain-independee user simulation with transformers for task- oriented dialogue systems. In Proceedings of the 22nd Annual Meeting of the Special Interest Group Discourse and Dialogue, pp. 445-456. Singapore and Online, July 2021. Association for Computational Linguistics. URL httph:ttps//cmmm wtt.pppp///acccttooooooooooo owg. sigggg/ll
47
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Bing Liu and Ian Lane. End-to-end learning of task-oriented dialogs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Stu- dent Research Workshop, pp. 67-73, New Orleans, Louisiana, USA, June 2018. Association for Computational Linguistics. doi: 10. 10.18655W1/N188-00000
URL http:://aclentholloog
org/N18 0/5//N184000 4010
Bing Liu and lan R. Lane. Attention-based recurrent neural network models for joint intent detec- tion and slot filling. CoRR, abs/1609.01455 2016. URL
01454
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. CoRR, 0ass20033049960 2020. URL
http://wwww...r////////3... 0496C
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite cospleeple dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2231-2240, Copenhagen, Denmark, September 2017. Association for Compu- tational Linguistics. doi: 10. 10.18663/11/D17-1133.
URL https:/ http:///laanttooogyy.. .org/
D17 017-1237 1237
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. Deep Dyna-Q: Inte- grating planning for task-completion dialogue policy learning. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2182-2192, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P13 1203. URL https:://///..
haap:::s/aclannto... /awalaaaanolooooooceea.occc org/P18 1203
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. Few-shot natural language generation for task-oriented dialog. CoRR, abs/2002.12328, 2020. htttps://arxiv.org/
www.oorgg/abb////oe abwpaaas..../////a//////a 12328
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. Soloist: Building task bots at scale with transfer learning and machine teaching, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
11

---

Under review as a conference paper at ICLR 2023
Jost Schatzmann and Steve Young. The hidden agenda user simulation model. IEEE Transactions on Audio, Speech, and Language Processing, 17(4):733-747 747, 2009. doi: 10.1109/TASL.208.. 2012071.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. Agenda- user simulation for bootstrapping a POMDP dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pp. 149-152, Rochester, New York, April 2007. Association for Computational Linguisties. URI. / ac antho logy wwwp:0o-gNNNO000 -
Weiyan Shi, Kun Qian, Xuewei Wang. and Zhou Yu. How to build user simulators to train RL-based dialog systems. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1990-2000, Hong Kong. China, November 2019. Association for Com- putational Linguistics. doi: 00.18653/v1/D19-.1 1206. URL
org/
D19-1206
Ryuichi Takanobu, Runze Liang, and Minlie Huang. Multi-agent task-oriented dialog policy leam- ing with role-aware reward decomposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 625-638, Online, July 2020. Association for Com- putational Linguistics. doi: 10. 12118655/112222aalmmaaiiaa org/2020.
URL httpss://acaaathollls
Bo-Hsiang Tseng, Yinpei Dai, Florian Kreyssig, and Bill Byrne Transferable dialogue systems and user simulators. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 152 166, Online, August 2021. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/20211 http://w//222..aloooog..... URL hatttpss//acllanbccooo...ooo acl -l-oong.11 13
Tsung-Hsien Wen, Mílica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young- Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1711-1721, Lisbon, Portugal, September 2015. Association for Computational Linguistics. wwwp:/ooogoogg/1-11111111
doi: 10. 1////5//1//D151111111
URL URL
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahun Pei- Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Volume 1, Long Pupers, pp. 438-449, Valencia, Spain, April 2017. Association for Computational Linguistics. URL org/ E17 E17-1042
http:///ccaatthooyy...
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. The dialog state track- ing challenge. In Proceedings of the SIGDIAL 2013 Conference, pp. 404-413, Metz, France, August 2013. Association for Computational Linguistics. URL org 0////11344000
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Associ- ation for Computational Linguistics. doi: ww..11665/11/2220 2tt22/00000mddemmes//ssee URL https: //aclanthology. org.h/olooyyyyr/2220000 emn 1emnlo
Yizhe Zhang, Michel Galley, Jianfeng Gao. Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. In NeurIPS, 2018
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
12

---

Under review as a conference paper at ICLR 2023
A MULTI-ARMED BANDIT PROBLEM
Reinforcement learning policies face the exploitation versus exploration trade-off, which can be de- scribed as the search for balance between exploring the environment to find profitable actions while taking the empirically best action as often as possible. This exploitation vs exploration dilemma has been widely studied Multi-armed bandit (MAB) problem.
In the MAB problem, there are K arms, and each arm j has a fixed but unknown reward distribution R, with expectation μ4j- At each time step 1, .... T, the decision maker must choose one of these arms. We denote the arm pulled at time step t as Jr E {1.... K}. After pulling an arm, will receive reward Xj. which is realization drawn from the arm's underlying reward distribution. The decision masker's objective is to maximize the cumulative expected reward over the time horizon Σ E[X,,] Et= Mj:.
B MORE DETAILS ABOUT TRAINING DIALOGUE SYSTEMS
B. THE ARCHITECTURES OF USER SIMULATORS AND DIALOGUE SYSTEMS
The basic modules of user simulators and dialogue systems detailed in Tab,
Table 4: The architectures of user simulators and dialogue systems. Modules with are trainable.
Agentt Types
Agents
NLU
DM
NLG
AgenT
Shi et al.
2019
RNN
Agenda
Template
AgenR
Shi et al.
2019
RNN
Agenda
Retrieval
AgenG
Shi ct al.
2019
RNN1
Agenda
RNN
(Generation)
RNNT
Shi et al.
2019
RNN
Template
User Simulators
RNNR
Shi et al.
2019
RNN
Retrieval
RNN
Shh et al.
2019
RNN (NLU NLG)
GPT (ours)
Transformer" (NLU DM NLG)
GPTıL (ours)
Transformer (NLU DM NLG)
Dialogue Systems All
RNNT
RNNT
Template
B.2 THE IMPLEMENTATIONS THE DIALOGUE SYSTEMS
The NLU modules of all system agents are 2-layer bidirectional-GRU with 200 hidden units. The NLG modules of them are using the template- semplate-based method. The DM modules of them are simple state representation, which consists the traditional dialog
MLP. The input of the DM module state and word count vector of the current utterance same [Shi et al. (2019). We mainly use the policy gradient method to train the DM modules of dialogue systems from scratch.
B.3 THE DETAILS OF RUNNING POLICY GRADIENT ALGORITHM
For training the DM modules of dialogue systems with the policy gradient method, we also apply the e-greedy exploration strategy. We let e be 0.5 the beginning, and will decrease to 0 linearly within the RL training. The dialogue ends either when the user simulators say "goodbye" or when the number of turns of the dialogue exceeds 10. The reward will be given +1 for task success, -1 for task failure, and -0.1 for each additional turn to encourage the RL-based policy module finish the task fast. Also, a discounted factor of 0.9 is applied to all the experiences.
B.4 THE PARAMETERS OF TRAINING SYS-MUST adaptive
The hyperparameters used train the Sys-MUST
MSs-MUSTadaptive adaptive are listed in the Tab.
13

---

Under review as a conference paper at ICLR 2023
Table 5: The hyperparameters used for training the Sys-MUST ays-MUSsadaptivve adaptive
Hyperparameter
Value
T
100,000
To
40,000
2,000
d
200
T
0.75
Table 6: The relationships between user acts and system acts.
User act
System act
inform type
ask type, present result, nomatch result ask type, present result, nomatch result present result, other
inform type change
anything else
make reservation
ask reservation info, booking success, booking fail ask reservation info, booking success, booking fail provide info goodbye
make reservation change time ask info
goodbye
B.5 HUMAN EVALUATION ON DIALOGUE SYSTEMS
We find volunteers to conduct the human evaluations on dialogue systems. They all have good English skills and are unpaid. Before the experiments, we introduced task-oriented dialogue systems and user simulators to them and tell them how to judge the generated dialogue is successful. Then prepare 50 user goals from MultiWOZ Restaurant Domain Dataset: 20 of them are simple, and 30 of them are little bit complex. We specify 10 user goals for each volunteer and let the volunteer converse with all dialogue systems for each same user goal. In total, collect 50 dialogues for each dialogue system to calculate its success rate.
The criteria to judge whether task-oriented dialogue is successful based on two aspects: 1) the system agent correctly understands the user's goal (i.e., the predicted dialogue state tracking result correct); and 2) the system agent provides all information (i.e., all slot values or booking reference number) that the user requests. For human evaluations, we follow these standard criteria. Besides, we also see if the system act generated by the system agent matched to the user act for each turn in the dialogue.
There have seven user acts, which are 'inform type", "inform type change", "ask info", "any- thing else", "make reservation", "make reservation change time", and "goodbye". There have nine system acts, which are "ask type", "present result", "nomatch result", "no other", "ask reservation info", "provide info", "booking success", "booking fail" and "goodbye". The rela- tionships between user acts and system acts are shown in Tab.
C IMPLEMENT MUST WITH THE MUST STRATEGY
Without losing any generality, we consider two representative sequential orders: 1) AgenT, AgenR, RNNT, GPT; and 2) AgenR, GPT, AgenT, RNNT. For case 1, the first two user simulators are agenda-based user simulators; the last two user simulators are Neural networks-baser user simula- tors. For case 2, we interleave these two types of user simulators. When the system trained by user simulator converges, we let continue to interact with another user simulator following the order.
As seen in Tab. in case 1, the system agent achieves the best performance (i.e., 92.4 in terms of the average success rate) after training with AgenT and AgenR sequentially. However, its overall performance degrades to 83.0 after training with RNNT: especially, its performance decreases by 36.0% when testing with AgenR (93.0 + 59.5). Moreover, after continuing to learn from GPT, the performance of the system agent becomes worse for AgenT (95.0 - 75.5) and AgenR (59.5 + 47.5). This indicates the catastrophic forgetting issue heavily happened when the system agent starts learning from AgenR. We also could observe similar phenomenon from case 2. These results
14

---

Under review as a conference paper at ICLR 2023
Table 7: The experimental results of implementing MUST with the MUST strategy.
User simulators
Dialogue Systems
Agen'T
AgenR
RNNT
Avg.
trained by AgenT
97.5
54.0
98.5
78.0
82.0
Case 1
trained by AgenT, AgenR sequentially
o.or
93.0
97.0
82.5
92.4
trained by AgenT, AgenR, RNNT sequentially
95.01
59.5, 26.05
97.0
80.5
83.0
trained by AgenT, AgenR, RNNT, GPT sequentially
75.5 1a2.0%
47.5.
96.01
82.0
75.3
truined by AgenR trained by AgenR, GPT sequentially
96.0
90.0
98.5
82.5
91.8
97.5
88, 0
97.0
81.3
91.0
Case 2
truined by AgenR, GPT, AgenT sequentially
96.5
78.5 eRd
97.0
80.0
88.0
trained by AgenR, GPT, AgenT, RNNT sequentially
97.5tt.0%
65.5 227
95.0
78.5 750
84.1
Ablation study on MUST. It uses five user simulators (Agen'T, AgenR, RNNT, RNNR and GPT simulator) to implement MUSTumiform and MUST adaptive-
Table 8:
In-domain evaluation
Out-of-domain evaluation
All
Dialogue Systems
Agen
AgenR
RNNT
RNNR
GPT
Agenti
RNN
Avg.
Sta.;
Avg.T
Std.j
Sys-Agen'I
97.5
54.0
-50.0%
98.5 lo.s%
92.54.o5
78.0_..9%
72.3
77.0
74.8
2.3
81.4
14,8
single
Sys-AgenR
96.0
90.0
48.5
80.5
97.5
82.0
89.8
7.8
91.7
7.1
Sys- RNN-
30.5 los.9%
23.0
99.0
75.5
35.5
84.0
59.8
24.3
63.6
30.5
Sys-RNNR
30.0 los.rs
23.0
4m4.4ms
16.5
93.5
68.5010.0%
30.0
70.5
50.3
20.3
28.8
Sys-GPT
60.5
kar
51,5
Lazz and
97.0
94.0%m
82.0
59.5
92.0
75.8
16.3
76.6
17.6
Sys-MUST antform
97.5
Tauets
87.0
37.00 77.0g
97.3 4.0%
82.0ra.os
96.3
87.0
91.8
4.8
92.1
6.0
Sys-MUST adipptive
97.0
1u.a%
156
97.0
97.5
82.5
97.5
87.5
92.5
5.7
CRI. confirm that implementing our proposed MUST with MUST CRL, strategy indeed has the catastrophic forgetting issue.
D SENSITIVITY ON DIFFERENT SUBSETS OF USER SIMULATORS
We also train the Sys-MUSTumitan and DyS-MUSTpdaptiie by using different groups of user sim- ulators for ablation studies: 1) five user simulators of AgenT, AgenR, RNNT, RNNR, and GPT; and 2) three user simulators including AgenT, RNNT, and GPT.
Superiority MUST. From Tab. g and Tab. we can observe that Sys-MUST Mr-MUUUTnnff uniform and Sys- MUST adaptive largely outperform the dialogue systems trained by single user simulators. Espe- cially, they gain an improvement
absolute points (85.4 vs. 81.4) when trained with three user simulators of AgenT, RNNT, and GPT. In summary, MUST could consistently improve the perfor- mance of the systems when using different numbers of user simulators. The ablation studies on different subsets of user simulators demonstrate the robustness of MUST.
Out-of-domain evaluation. When testing our MUST with unseen user simulators, Sys- uUSTuuiform
uniform and Sys-MUST adaptive can also largely outperform the dialogue systems trained by single user simulator. As seen in Tab. g Sys-MUST adaptive achieves a 2.7 absolute value improve- ment (92.5 89.8)
Sys-AgenR. Sys-MUST unffoor and Sys-MUST aSS-MUSTadaative adaptive even improve least 5.7 points (80.0 vs 74.3) over Sys-GPT (as shown in Tab. 99. These experimental results on different subsets of user simulators demonstrate that our MUST has a better generalization ability for interacting with unseen user simulators and is insensitive to the user simulator selection.
Comparison between MUST MUSTumiform uniform and MUST adaptive. Fig. shows the learning curves of Sys- MUST uniform and Sys-MUST adaptive different subsets of user simulators. The first 40,000 steps are in the warm-up phase for Sys-MUST adduavvvvvve-- We could conclude that training the dialogue system by MUSTadaptia adaptive consistently converges faster than by MUST ios when using three, four, or five user simulators to implement MUST (see Fig. 4(a) Fig. 2(a) and Fig. 4b) respectively).
least in the scenar-
From Tab. where MUST trained with five user simulators, we could observe that Sys- MUST adaptive outperforms Sys-MUSTunffomn with 0.5 absolute point. The performance gain becomes smaller when MUST is trained with three user simulators (see Tab, (9). This probably shows that Sys-MUST,
a adaptive would be more beneficial when there exist
simulators.
15

---

Under review as a conference paper at ICLR 2023
Table Ablation study on MUST. It uses three user simulators (Agen'T, RNNT, and GPT simu- lator) to implement MUST uniform and MUST adaptive.
In-domain evaluation
Out-of-domain evaluation
Dialogue Systems
AgenT
RNNT
GPT
AgenR
AgenG
RNNR RNN
Avg.
Std.j
Avg.1
Std.
Sys-Agen'T
97.5
98.51p.3%
78.0 to
54.0
72.5
92.5
77.0
74.0
13.7
81,4
14,8
single
Sys-RNNT
30.5 Los x36
99.0
15.5
23.0
35,5
97.5
84.0
60.0
31.4
63.6
30,5
Sys-GPT
60.5 1ar
97.0
82.0
51,5
59.5
94.0
92.0
74.3
19.0
76.6
17.6
MUST
Sys-MUST uniforrm
uniiffrr
96.0 fo.5t.
82.5
to.0ss
55.0
82.0
97.5
87.0
80.3
15.7
85.4
13.9
Sys-MUST adapsive
t7.5t
97.5
82.5
1o.0%.
55.5
80.5
97.0
87.0
80.0
15.3
85.4
13.9
100
100
The average tuccess rate
The 40-
MUST Tniifor
MUST uniform
MUSTadapcive
MUST adagtivv
MUST adaptioe
20000
40020
50000
80000
100000
20000
40000 The number of dielogues
80000
50000
The number of dialogues
MUST with five simulators
(b) MUST with three simulators
Figure 4: The learning curves of Sys-MUST unffform and Sys-MUSTadappe adaptive.
E ABLATION STUDY FOR THE MODIFIED UCBI ALGORITHM
E.I NECESSITY OF THE EXPLORATION TERM
Our modified UCBI algorithm provides a distribution for guiding how to sample different user simulators to accelerate the entire MUST training. The exploration term in the proposed aUSTadaptive MUST adaptive exists mainly for uniform adaption (see the detailed explanation in Sec. 4.1) The original UCBI algorithm (Auer al., 2002) can tell us how pull arms in bandits maximize the cumulative expected reward. is well-known that it cannot explore effectively without the exploration (UCB) term; consequently, it might not find the optimal action and lead to relatively poor performance. It is difficult to theoretically prove the usefulness of the exploration term in our scenario (like in the original UCB1 algorithm), which we leave as future work. However, we alternatively conduct some ablation studies to evidence the necessity of the exploration term.
MUST aUSTadaptive adaptive w/t exploration If omit the exploration term in our modified UCB1 algorithm, the simplest way to calculate the distribution p is to make the sample probability w.rt a user simula- tor solely depend on the inversion of the system's 's performance. See the row called 'w/t exploration' in Tab. 10 for comparisons.
In this situation, the obtained distribution p might be sharp due to the lack of the exploration term, which would be harmful for uniform adaption to some extent. As Fig. 5(a) shows, MUST MUST adaptive performs worse and converges slower when omitting the exploration term, compared with when our modified UCBI algorithm has the exploration term. This could demonstrate both the importance of uniform adaption and the usefulness of the exploration term.
E.2 ABLATION STUDY ON THE DESIGNED DISTRIBUTION
Rationale of exploitation vs exploration trade-off. Similar to the exploitation vs exploration trade-off, the distribution p under the MUSTadapuiee should trade off the boosting adaption and the uniform adaption when specifying multiple user simulators. Considering the boosting adaption, make exploitation assumption stated as follows: p is expected to assign lower weights to
16

---

Under review as a conference paper at ICLR 2023
100
The average success rate
80
80
MUST uniform
MUSTadaptive
MUST adaptive
The
MUST adapttiee
MUST woapt exploration)
MUST adlpcive
MUST adaaptive
MUST aUSTadaptive adaptive
III
0-
20000
40000
s0bop 102020
120000
20000
40020
50000
a0000
dialogues
dialogues
(a) Ablation study on the exploration term Figure 5: The learning curves of Sys-MUST uniform and Sys-MUST adaptive-
(b) Ablation study on distribution p
simulators with which the system agent S already performs well and higher weights those user simulators with which performs poorly. Therefore, the sampling ratios for different user simulators should be inversely proportional the system's performance each simulator.
Rationale of the modified UCBI algorithm MUST MUSTadaptive is defined
The modified UCB1 algorithm for implementing
2 Int T ja
=
j (1,...,,y) K}:
exploitation
exploration
(5)
= 1/ (āt
min (1 xkl)),
2j
p:
Σ K
2j
MUST adeptive Eq. (which is the same as Eq. Eq. and Eq. consists of three steps: pollitation-exxllrration term construction, post-processing (re-scaling operation and the inversion operation), and the probability normalization, corresponding to each line in Eq 5 Besides this way, could have the following three variants that shuffle the order of these three key operations (i.e., the exploitation-exploaiiiee
term construction, re-scaling operation, and the inversion operation). We name these variants as as MUST aUSTadaptive MUST adaptive II, and auSpiiep
MUST adaptive -I. For the exploitation assumption, we make the exploitation term inversely propor- tional the system's performance žj on each user simulator Uj. which is denoted MUST adaptive- I. From Tab. 10 we can obverse that the difference between MUST adaptive and MUST adaptive is MUSTsdaptive
that MUST = take the inversion of before the peploitation-epllrattte
term construction while MUST adaptive take the inversion operation after the exploitation-exploraiii term construc- K} is smaller than 1, 2, will be larger than 1. Therefore, the
tion. Since each J..j E
term of 1 and the exploration term of V 2int (smaller than 1) are not with the same magnitude, which will lead to a consequence that the exploitation term becomes dominant while the exploration term is negligible. We have discussed similar issue of ignoring the exploration term in Sec. E. Therefore, we adopt MUST adaptive in default if not specified rather than MUST addaptive since the latter might suffer from the different magnitudes of the exploitation term and the exploration term.
MUST aduStvdativee and adStavaptive-lll Compared to MUSTodaptive. adaptives MUST adeptivee moves the inversion operation the front of the constructed exploitationnexxloraiio term. Likewise, MUST adaptive -III moves the re-scaling and the inversion operations to the front the constructed cewpiitttiionexplrttatt
MUSTadaptive" term. MUST adaptive II and MUST MUSTadaptive" adaptive- are used to check the order sensitivity about the exploitation iupllitation-explorattte exploration term construction, re-scaling operation, and the inver- sion of J (1.... K).
17

---

Under review as a conference paper at ICLR 2023
adeptive. The MUST TdSSaaaaiive adaptive implementation is an exploitation- exploration term followed by post-processing for the re-scaling purpose and sum-one normal- ization. Since we omit the exploration term for the second row, therefore, does not need the post-processing. MUST auspavatiiveell IIII moves the re-scaling and the inversion operations to the front of the constructed explotattion-exploatiin term.
Table 10: The variants of MUST MUSTadeptive.
variants
exploiatiion-exploraiio term
post-processing
distribution
MUST adaptive
+
2
23.e
Rj
[2, my min( .M 3K 1)
w/t exploration
MUST adppivve
+
Zlmt
Tyit
2j ūn min( (11/11, .1/xx))
Pj
20 Σ 21
1/21
adSTadap II
2ist
1/5g
2j = 8) - min(($1,. ,x
2j 2j
2int
T
85 (6)
MUST MulTTadptivelllI
mint 33
2ln
R
100
The average Ruccess rate
80
80
MUST MUSTunifon Tuniform,
MUST Tadaptives
MUST MUSTunitoms,
The
MUST Tadaptives
MUST uniifrrm adiaptive
MUST Tniformy
adSTadapivee
0-
40020
50000 dialogues
80000
100000
42000
80000
dialogues
(a) The learning curves of the system trained with eight user simulators.
(b) Comparison between different numbers of user simulators.
Figure 6: The learning curves of Sys-MUST uniform and Sys-MUST adaptive-
Results for ablation study the variants. Experimental results of these different variants shown in Fig. 5(b) The convergence speed of MUSTadaptivee adaptive- is much slower compared to oth- ers, which demonstrates that the exploration term is useful once more. The convergence speeds of MUSTadaptive
MUST adaptive -ll and MUST adsptivee is comparative to MUST adaptive- This probably shows that
our design with three operations (i.e., exploitation-exple -eplooratiooooooiiinnnn term construction, re-scaling strategy, and the inversion of ži,) is not only reasonable but also robust to the order permutation of these three operations.
F IMPLEMENTING MUST WITH MORE USER SIMULATORS
To implement our MUST with more user simulators, we use Simulated Agenda Dataset to train four extra user simulators Fig. shows the learning curve of the system agent trained by MUST with eight simulators (Agen'I, I, AgenR, RNNT, GPT, GPT GPTAn. GPTAG, and GPTrand). ) We could observe that the training of our proposed MUST can still succeed when increase the number of user simulators to eight. Sys-MUST adeptive still converges faster than Sys-MUST uniform even
'Simulated Agenda Dataset (See Sec. 5.1 simulated from each male-based user simulator (i.e., AgenT, AgenR, AgenG) and its corresponding system agent respectively. We use them to build three new user simula- tors denoted as GPTAT. GPTAR. and GPTAG based on the GPT model respectively. For example, we use the simulated dialogues from Agen' and Sys-Agen'T to build the GPT also collect 3000 dialogues randomly from Simulated Agenda Dataset train another new GPT user simulator denoted GPT rand-
18

---

Under review as a conference paper at ICLR 2023
Sy System& A Aent
User Simulator
So: Heli! W but can I help you?
ONLU No.
mecommend: [].
constrant [I. book
select [I. mquest: I]
VVentian' "priceunge" "esponsive" "aree" "cestre"..
22) Gemerrerratorr
Ag
Inform bype: (food. amol
POL
"book" ("tmee "1200
NLG
'day" "Wednesday" "parol"" 14"))
Uo:
am inoktng at place to out that servis [vaiue Food Food in the (valae
Unfortuneell, I don'tsse any resturants that Venetton the centre of town. Would you like to by difftrent are or type tuising?
S1
artttnn place that
Venetion food in centre
(2) Generaatt
Ni
cunstrait:: [aoodlr]] book 2ecommeendl [I, select ieques::
Inoo ""oood" "Chinese'
"prteeange" "expenstree
A1:
type Food
POI
"book": ("Lime: "12:00,
"area' ; "cmmte" I.
NIG
"day" "Weddesday" 'poople:: '4*))
U1 Do have (valar foud) insiaumnnis
S2
Yes there are 10 the centre town. Any proee prefemnoa?
you bave any Chibasse sestourant??
(a) The details of the first two- turn Interactions between a system agent and our U-GPT.
betlo! what can help you? daos nesp Jeos cunsinini leos, louk] coooment leos, selact] leos, request] eoos ruu [info] food venutian pricenugg expensive centre [rqueet] [book] lime 12:00 day wednesday people <eos godl> jndomm type) food arso pol- um looking place eut that
(eus,
servie venction
the comm. scos, unfortunttly, different ara or type of cinenne? soos resp nooffer loos constmint] (eos book) lons recommend] (cos select] [cos requast seos nhi [nfo] food chinese peiceange expeasiive arsa centre (jequest] [book] time 12:00 day wednesday people I <3os goal> (inform type, change) food <eos, poll you
nut soe any masuurnts that surve veretion
town would you like
any (value food] restaumnts
(b) An example of the model input for training U-GPT.
Figure 7: The overview of our U-GPT which consists of Natural Language Understanding (NLU), Goal Generator, Dialog Policy Learning (POL), and Natural Language Generation (NLG). It uses the auto-regressive language model GPT to understand the system inputs, generate the user actions and the user utterances given the dialogue context and the user goals sequentially in an end-to-end manner. (a) gives detailed description of the first two-turn interactions between system agent and our U-GPT. For training U-GPT, we need to convert the dialogue context and all annotations to sequences of tokens. (b) presents the training example of the first two-turn dialogues in (a).
though the difference between their convergence speeds is not too large in this case. It might be because some user simulators are similar (e.g., AT is similar to AgenT, GPTAR is similar to AgenR), which might lead that the distribution p approaches uniform distribution.
Fig. 6(b) compares the learning curves of Sys-MUST aUSSaadaptive and Sys-MUST uniform trained with different numbers of user simulators (i.e., four, five, and eight user simulators). It a fair comparison because these combinations include the hardest user simulator AgenR that can be adapted by the system and the easiest user simulator RNNT that can be adapted by the system (See Sec. (5.)). We can observe that, with more user simulators, Sys-MUST adaptive not only performs better but also converges faster than with fewer user simulators. This probably shows that Sys-MUST
has the potential to be generalized to a larger set of user simulators. Plus, we also could observe that Sys-MUSTadaptte adaptive consistently converges faster than Sys-MUST uniform in different numbers of user simulators.
G MODELING USER SIMULATOR WITH GPT
We name the model of building user simulator based on GPT as U-GPT. In this section, we will illustrate its details and conduct experiments to prove that it is a better model for building a user simulator.
G.1 THE ARCHITECTURE OF U-GPT
As Fig. a shown, our U-GPT consists of four modules, which are Natural Language Under- standing (NLU), Goal Generator, Dialog Policy Learning (POL), and Natural Language Generation (NLG). Dialogues consist of multiple turns. In the first turn = O, U-GPT (1) first outputs its NLU results No by understanding the system input So, and (3) decide its actions Ag which is a list of pairs: (action type, slot name) based on (2) its initial goal Go and [S,). U-GPT then (4) con- ditions on (So, No, Go, Ao) to generate the delexicalized utterance Ug. The generated placeholders
19

---

Under review as a conference paper at ICLR 2023
in Up will be filled using the corresponding slot values in the goal Gu- When the conversation pro- ceeds to turn t. U-GPT (1) generates the NLU results N, based on all of previous dialogue history and generated outputs (Co...., C. S,), here C; [S, N, G., A,, U:}. If there has "no-offer" intent in N₂ representing that no entities could satisfy current constraints, then (2) Goal Generator goal G₁- Then U-GPT will continue to (3) generate the user acts A, and (4) generate delexicalized utterance U. conditioned on {Co..... Cr-1, St, N₁, Ge} sequentially. We should notice that the user utterances occurred in the history context should be lexicalized because they contain important information.
should generate
Fig. ()) shows an example of training sequence which consists of the concatenation In order to leverage GPT, we need to convert the generated outputs {N₂ G,, A,, U.} to sequences of tokens resembling text. And we introduce delimiter tokens [eos resp). [eos nlu). feos goal), leos-pol], [eos_uft] to signal the ending of sequence representations of different modules. For the NLU results N,, we use five categories: "inform", "rrquest". "book inform", "select", "recommend" same Shi et al. (2019) to represent them. And we also introduce five tokens feos_ constraiit,,, brok_) íeos recommend], [eos selecte [eos request] to record different information. All of these tokens and the intents of user actions will be added to the vocabulary of GPT as additional special tokens. For training U-GPT, we use the same training objective as GPT which is to maximize the following likelihood:
[Co, Ci]-
L(U) Elog log P(ujlui-a by U, 1 O),
A un (So, No, Go, Ao, Uo .. Ar, U.}:
where k is the size of the context window, and the conditional probability parameterized with +
G.2 EVALUATIONS ON U-GPT
To evaluate our proposed U-GPT, we adopt both indirect evaluations and direct evaluations as in Shi et al. (019). We evaluate user simulator indirectly using the average success rate of the system agent trained by this simulator. It is called cross-model evaluation (Schatzmann & Young 2009) which assumes strategy learned with good user model still performs well when tested poor user models. It can indirectly evaluate the goodness of user simulator. For direct evaluations, adopt six evaluation measures evaluate the diversity of simulators automatically: average utterance length, vocabulary size, Dist-1, 1, Dist-2 (Li et al. (2016a) and Entropy (Zhang et al. 2018). We also ask human users rate the simulated dialogues to assess the user simulators directly. We use five same metrics as [Si et al. (2019) which are Fluency, Coherence, Goal Adherence, Diversity, and Overall quality to assess user simulators from multiple aspects.
G.3 TRAINING DETAILS OF USER SIMULATORS
We implement our GPTT-ased user simulators with DistilGPT2 (Sanh al. 2020). distilled version of GPT-2 by HuggingFace's Transformers (Wolf et al. 2020 We select the best performing models on the validation set through hyperparameters search of learning rate and batch size. The best models were fine-tuned with a batch size of 64 and a learning rate of 1e-3 over the corresponding dataset. We use the greedy decoding strategy for generating word-tokens in the inference phrase.
G.4 EXPERIMENTS
GPT-RNN. Because the implementation of user simulator RNN mainly consists of NLU and NLG, remove the POL module from U-GPT and use the same annotated data RNN to fine-tune it to compare our U-GPT with the RNN-based methods fairly and name it as GPT-RNN.
As Tab. II Tab. 12 Tab. 13 show, GPT-RNN outperforms the user simulator RNN. It proves the power of leveraging GPT.
'The system agent for simulating dialogues is third-party system provided by Shi al. 2019 which was built based on hand-crafted rules.
20

---

Under review as a conference paper at ICLR 2023
Our GPT-RNN performs better than the user simulator RNNT, which be from the model evaluation results in Tab. [I the automatic evaluation results in Tab. 12, and the Du.Div score in the human evaluation results in Tab. 13 However, Tab. shows, RNNT performs better than our GPT-RNN in the overall performance from the human evaluation. We think this might be because (1) the third-party system also has impact the generated dialogues and (2) the NLG module of RNNT is the template-based method which leads to the generated dialogues from RNNT being easy for the third-party system to understand and interact with.
The automatic evaluation results in Tab. 12 and the Hu.Div score in the human evaluation results in Tab. [3] show that RNNR can generate more diverse language than our GPT-RNN. We think it is because the user utterances generated by RNNR are retrieved from a corpus that is written by real humans and the sentences written by humans are usually more diverse than the sentences generated by generative models. Even though the dialogues generated by RNNR are more diverse, the dialogues generated by our GPT-RNN are more fluent and coherent. Also, the cross- evaluation results in Tab. II show that GPT-RNN can help to learn more robust system agent than RNNR, but the AulAll score in the human evaluation in Tab. 13 gives the opposite result.
Table 11: Cross study results. Each entry shows the success rate obtained by having the user simu- lator interacting with the RL system for 200 times.
System User
AgenT
AgenR
AgenG
RNNT
RNNR
RNN
GPT
GPTIL
Avg.7
Std.
Sys-RNNT
30.5
23.0
35.5
99.0
97.5
84.0
75.5
66.0
63.9
28.5
Sys-RNNR
30.0
23.0
30.0
96.5
93.5
70.5
68.5
56.0
58.5
26.7
Sys-RNN
20.0
23.5
20.0
73.0
63.0
77.0
56.5
45.0
47.3
22.2
Sys-GPT-RNN
36.5
38.0
42.0
95.5
94.0
89.0
80.5
61.0
67.1
24.1
Table 12: Automatic evaluation results of RNNT, RNNR and GPT-RNN. The metrics include aver- age utterance length (Utt), vocabulary size (Vocab), distinct-n (DIST-n) and entropy (ENT-n).
User Simulators Utt 1 Vocab DIST-1 f DIST-2 1 ENT-4
RNNT
9.83
192
0.77%
1.51%
4.24
RNNR
11.06
346
2.45%
9.59%
6.59
RNN
10.95
205
1.17%
3.14%
4.98
GPT-RNN
14.00
262
1.13%
3.53%
5.62
Table 13: Human evaluation results of RNNT, RNNR and GPT-RNN. The metrics include sentence fluency FH,, coherence (Hu.Co), goal adherence (Hu.Go), language diversity (Hv. and an overall score (Hu.All).
User Simulators
Hu.Fl 1 Hu Hu.Go f
Div 1 Hu.All
RNNT
4.60
4.68
4.96
3.34
4.70
RNNR
3.92
3.88
4.72
3.94
4.16
RNN
2.80
2.30
2.86
2.74
2.30
GPT-RNN
4.10
4.04
4.30
3.70
4.00
H RELATIONS TO OTHER WORKS
The inspiration behind our proposed MUST adaptive is similar to Focal Loss (Lin et al., 2017) and some curriculum learning approaches for reinforcement learning (Narvekar et al. 2020). Focal Loss addresses the extreme class imbalance by deig-weighting the loss assigned to well- classified examples. Similarly, assign lower weights to user simulators with which the dia- logue system already performs well in the MUST MUSTadaptive adaptive training. However, we use a different way to obtain these weights. Curriculum learning approaches will define difficulty levels for different tasks. However, in our scenario, we cannot know how difficult each user simula- tor can be adapted beforehand. Moreover, curriculum learning approaches for reinforcement learning also move an agent from one task to another like the MUST strategy.
21


Reviews:
[{"review_id": "mbvK2GuKtD", "reviewer": "Reviewer_H1xa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed to use multiple user simulators to train a dialogue policy and showed that using multiple user simulators rather than a single user simulator improves the policy performance.\n\nThe authors developed a sampling strategy to sample user simulators during training. The sampling ratios are inversely proportional to the policy's performance on each user simulator. The policy's performance is calculated by the success rate's upper confidence bound.", "summary_of_the_review": "The authors proposed to use multiple user simulators to train a dialogue policy. While the idea is interesting, it is only trained on a small set of user simulators. The authors didn't justify the usefulness of the proposed UCB-based performance expectation. Finally, the performance improvements are marginal and are not compared with SoTA methods.", "strength_and_weaknesses": "Strength:\n\n1. The idea of adapting multiple user simulators to train a dialogue policy is interesting. The adaptive phase is closely related to curriculum learning.\n2. Instead of using the mean success rate to calculate sampling ratio, the authors proposed to use the UCB of success rate, which hopefully makes the algorithm more robust. Although there is no ablation study to show if this is true.\n\nWeaknesses:\n\n1. The authors only used 4 user simulators to train MUST_uniform and MUST_adaptive, which makes the work less impressive. In general, we wouldn't expect many rule-based user simulators exist, therefore, we would rely on data-driven user simulators. I think the work is more valuable if the authors extend the current work and learn a large set of (and possibly latent/complementary) user simulators from data, and then apply the proposed methods to see if the policy can benefit from the large pool of user simulators.\n\n2. Follow up on the previous point, the authors didn't answer the question of how many user simulators are enough, and how different should they behave?\n\n3. The related works should include curriculum learning literatures. A lot of works use the loss/rewards to adjust task/example weights. For example, on top of my head, focal loss[1] is one method. There should also be an ablation study to show that the UCB term is useful.\n\n4. The authors proposed to use the inverse of UCB as the weights of user simulators. I am a little bit confused. The z_j in Section 4.2 Equation (3) is the *inverse* of success rate, reflecting the task difficulty. Therefore, we should use the UCB of z_j to calculate sampling ratios. In that case, should we pick lower confidence bound in equation (2)? For example, let's say we have two user simulators, the first one has a CI [0.2, 0.8], the second one has a CI [0.4, 0.6]. The proposed algorithm in the paper will assign lower weight to the first user simulator. However, the first user simulator has a lower bound of 0.2, which means that the user simulator could be very hard to train, and should be assigned a higher weight. \n\n5. Based on the results in Table 2 and Table 3, The performance of MUST_uniform and MUST_adaptive are super close. The performance of MUST_adaptive and SYS_AgenR are also very close. I don't see a clear benefit of the proposed algorithm.\n\n6. The experiments only use MultiWOZ restaurant domain dataset, and no SoTA methods are compared.\n\n[1] Focal Loss for Dense Object Detection https://arxiv.org/abs/1708.02002", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CxijjHg5vvn", "reviewer": "Reviewer_44kw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the MUST framework (multiple user simulators), for training Task-oriented Dialogue systems with Reinforcement Learning and incorporating multiple user simulators.\n\nThe usual way to train these models is with a single system and user simulators but, as the authors correctly point out, a single user simulator might fail to exhaustively capture all possible human behaviors. This paper contributes a stepping stone towards this objective by casting the problem of combining different user simulators as a multi-armed bandit problem. The authors show that this method provides better results than using simpler combination baselines or single user simulators during training.\n", "summary_of_the_review": "I think the idea is interesting, and that it could inspire interesting follow ups, however the execution leaves a bit to be desired. I will set my score to \"marginally above the acceptance threshold\" due to the lacking points I mentioned in detail above.", "strength_and_weaknesses": "# Sumary of strengths\n\n* To the best of my knowledge, the idea of combining multiple user simulators by casting this problem as a multi-armed bandit is novel.\n* The way the idea is executed is not optimal (experiments on limited dataset), but the authors acknowledge this shortcoming. Despite the limitations of the paper's empirical grounding, its ideas and the evidence it does could provide open interesting avenues for future research on the interaction between user and agent simulators.\n* The paper provides directional evidence showing that combining several user simulators is preferable to not doing so. This is reflected both in terms of overall success rate as measured automatically and by humans, and by showing that models trained through this method are less sensitive to unseen user simulators.\n* It also evidences that adapting the distribution guiding the choice of user simulator during training results in faster training convergence.\n\n# Summary of weaknesses\n\n* There are some paragraphs that are poorly written, and make some key concepts difficult to understand.\n* There are some conceptual gaps that make me doubt about the overall rigor of the paper.\n* The more complex $MUST_{adaptive}$ model is only slightly better than the simpler $MUST_{uniform}$ baseline as measured by the in-domain automatic and human evaluations.\n\n\n# Suggestions and questions for the authors.\n\n* Good work! Do you intend to release the code in case the paper is accepted?\n* In the second paragraph of page 2 you mention that \"Extensive experimental results [...] show that the dialogue system trained by our proposed MUST achieves a better performance than those trained by any single user simulator\", however I feel like the amount of experimentation described is not extensive, so I would suggest toning down this statement.\n* In page 4 you introduce $MUST_{CRL}$ but you don't include that model in any of your results. In the paragraph \"Challenges to leverage multiple user simulators\" you mention some challenges related to this model. Are these challenges the reason why you did not analyze it? If that is the case then I suggest not giving it its own item (II) above, and instead give an overview of the \"adaptive\" variant of your framework. If this is not the case, then why was it not included?\n* In the same \"Challenges to leverage multiple user simulators.\" paragraph you mention: \"unnecessary efforts will be costed for easily-adapted user simulators\". I suggest rephrasing this statement as I don't think it's understandable in its current form.\n* In the first paragraph of page 5 you link the \"uniform adaptation\" to reducing the catastrophic forgetting issue. However there is no experiment providing evidence that without \"uniform forgetting\" there's catastrophic forgetting. I suggest rephrasing this, or providing evidence that this is actually the case.\n* In the first paragraph of section 5.2.2 you mention: \"$SyS-MUST_{merging}$ is trained by $GPT_{IL}$ for implementing $MUST_{merging}$ strategy\". First of all the statement is redundant by mentioning the \"merging\" strategy twice, but more importantly, I had understood that the merging strategy implied sampling dialogues from a set of user simulators and use these to train the final model. Does this statement mean that you generated dialogues with $GPT_{IL}$, which was trained on dialogues produced by 3 agenda-based simulators and MultiWoZ restaurant data, and define this method as sampling dialogues from several simulators?\n* In the \"Automatic Evaluation\" paragraph of section 5.3 you say that the reason for the \"merging\" strategy not performing as well as the \"uniform\" and \"adaptive\" could be \"because the merging strategy cannot effectively leverage multiple user simulators\" which is a cyclical explanation. I suggest removing it or providing a more plausible explanation.\n* In the third paragraph of section 5.3 you mention that the \"uniform\" and \"adaptive\" strategies achieve 2.4 absolute value improvements\", but improvement over what?\n* Finally, I suggest providing more details about the human evaluation. In Appendix B.2 you give some more details and mention that you \"tell them how to judge if the generated dialogue is successful\", \"them\" being the evaluators. But what was your definition of a successful dialogue? Every slot filled, for example? or something else?\n\n\n\n# Typos and minor corrections\n\n* Page 1, paragraph 4, line 3: best-performed -> best-performing\n* P. 2, p. 1, l. 10: You reference challenges $\\textit{i}$ and $\\textit{ii}$ but you used 1 and 2 in the same paragraph for describing challenges. In the abstract you use $\\textit{i}$ and $\\textit{ii}$ for referring to the two types of adaptation rather than the challenges they address. I suggest being consistent with numbering.\n* P. 2, p. 2, l. 5: Here you mention \"$MUST_{adaptive}$ is indeed more efficient for leveraging multiple user simulators by our visualization analysis\", which makes me wonder: more efficient than what? and what visualization analysis do you mean? I suggest clarifying this in the paper.\n* P. 2, p. 3, l. 8: convergences -> converges\n* P. 2, p. 6, l. 4: if accomplishing -> if it is accomplishing\n* P. 3, p. 1, l. 1: Once the database result -> When the database result\n* P. 4, p. 2, l. 1: first sample -> first samples\n* P. 4, p. 2, l. 4: by RL algorithms -> with RL\n* P. 4, p. 4, l. 11: Not sure what is meant by \"unnecessary efforts will be costed for easily-adapted user simulators\". I suggest rephrasing this.\n* P. 4, p. 6, l. 1: recalls us a similar thought -> reminds us of a similar concept\n* P. 4, p. 6, l. 3: weakly-performed -> weakly-performing; well-performed -> well-performing\n* P. 4, p. 6, l. 4: \"should reduce the interaction with user simulators that dialogue system has performed well and allocate more interactions with those user simulators that dialogue system has not performed well\" -> \"should reduce the interaction with user simulators with which the dialogue system has performed well and increase interactions in the opposite case.\"\n* P. 5, p. 2, l. 5: masker's -> maker's\n* P. 5, p. 4, l. 7: \"p is expected to assign lower weights to user simulators that the system agent S already performs well and higher weights to those user simulators that S performs not well\" -> \"p is expected to assign lower weights to user simulators with which the system agent S already performs well and higher weights to those user simulators with which S performs poorly\".\n* P. 5, p. 7, l. 2: what do you mean with \"latter\" and \"former\" terms? Do you mean $\\underbrace{\\bar{x}_j}_\\{\\text\\{exploitation\\}\\} + \\underbrace{\\sqrt{\\frac{2\\ln t}{T_\\{j,t\\}}}}_\\{\\text\\{exploration\\}\\}$ in equation 2?\n* P. 5, p. 8, l. 2: has been interacted so far -> has been interacted with so far\n* P. 5, footnote, l. 2: \"Then the index of the arm will be played from t = K + 1 to T is the sum of two terms: ...\" makes no grammatical sense. Please correct it.\n* P. 6, Algorithm 1: I suggest changing the verbs from -ing form to imperative, so initializing -> initialize, synthesizing -> synthesize, using -> use, evaluating -> evaluate, updating -> update. I also suggest clarifying what the lowercase s mentioned in the input is used for.\n* P. 6, p. 1, l. 2-3: I think here you used both $\\tau$ and $s$ to refer to the smoothing factor for distribution $\\boldsymbol{p}$\n* P. 6, p. 3, l. 3: \"that the dialogue system has performed well\" -> \"with which the dialogue system has performed well\"\n* P. 7, p. 4, l. 4: \"test the systems by them\" -> \"test the systems with them\"\n* P. 7, p. 4, l. 5: \"there usually has a gap\" -> \"there usually is a gap\"\n* P. 9, Figure 2: There's no need to write \"Tested by\" for each subfigure. The name of the user simulator is enough. Also the label for (a) seems to be wrong.\n* P. 14: There's mention to U-GPT here, but nowhere in the main text.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1ZB8mSvQHYy", "reviewer": "Reviewer_mwD2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a multi-armed bandit approach to training task-oriented dialog (TOD) systems to leverage multiple user simulators. This approach is demonstrated to outperform ablative variants that suffer from catastrophic forgetting and over-fitting to specific \"easy\" simulators.", "summary_of_the_review": "The authors propose a well-justified bandit based method for learning from mixtures of different user simulators to train a TOD model. Automatic evaluation is encouraging but the central assertion of adaptibility probably requires a more comprehensive evaluation setting (models, domains)\n", "strength_and_weaknesses": "Strengths:\n- MUST as an adaptive approach seems well-justified\n- Discussion of learning curves motivates further study of MUST as a possible adaptive approach for low data regimes\n- Experiments explicitly target enumerated challenges with existing user simulators and mixtures (catastrophic forgetting, over-fitting to specific simulators)\n\nWeaknesses:\n- It is important to see empirically if MUST generalizes to a larger or smaller pool of user simulators and is not over-tuned for the current set chosen in the paper. For example, ablation studies with different subsets of the user simulators would be helpful\n- Needs more discussion of the limitations on user behavior diversity etc. given that only the Restaurant domain is evaluated here. It is unclear from the experiments here how well the approach can succeed across multiple domains where the pool of user simulators potentially explodes\n- Human evaluation details are relatively scarce - would like to see justification for having human raters apparently rate the success rate (the same as the automatic evaluation metric), rather than other metrics of quality (even subjective quality rating) that leverages human annotators more effectively", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject