Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System
Authors: Jianhong Wang
Yuan Zhang
Tae-Kyun Kim
Yunjie Gu
Venue: ICLR 2021 Poster (2021)
Keywords: Task-oriented Dialogue System
Natural Language Processing
Hierarchical Reinforcement Learning
Policy Optimization

Abstract: Designing task-oriented dialogue systems is a challenging research topic, since it needs not only to generate utterances fulfilling user requests but also to guarantee the comprehensibility. Many previous works trained end-to-end (E2E) models with supervised learning (SL), however, the bias in annotated system utterances remains as a bottleneck. Reinforcement learning (RL) deals with the problem through using non-differentiable evaluation metrics (e.g., the success rate) as rewards. Nonetheless, existing works with RL showed that the comprehensibility of generated system utterances could be corrupted when improving the performance on fulfilling user requests. In our work, we (1) propose modelling the hierarchical structure between dialogue policy and natural language generator (NLG) with the option framework, called HDNO, where the latent dialogue act is applied to avoid designing specific dialogue act representations; (2) train HDNO via hierarchical reinforcement learning (HRL), as well as suggest the asynchronous updates between dialogue policy and NLG during training to theoretically guarantee their convergence to a local maximizer; and (3) propose using a discriminator modelled with language models as an additional reward to further improve the comprehensibility. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA, showing improvements on the performance evaluated by automatic evaluation metrics and human evaluation. Finally, we demonstrate the semantic meanings of latent dialogue acts to show the explanability for HDNO.

Published as a conference paper at ICLR 2021
MODELLING HIERARCHICAL STRUCTURE BETWEEN DIALOGUE POLICY AND NATURAL LANGUAGE GEN- ERATOR WITH OPTION FRAMEWORK FOR TASK- ORIENTED DIALOGUE SYSTEM
Jianhong Wang", Yuan Zhang', Tae-Kyun Kim Yunjie Gu"
ABSTRACT
Designing task-oriented dialogue systems is challenging research topic, since it needs only generate utterances fulfilling user requests but also guaran- tee the comprehensibility Many previous works trained end-to-end (E2E) models with supervised learning (SI.), however, the bias in annotated system utterances remains as a bottleneck. Reinforcement learning (RL) deals with the problem through using non-differentiable evaluation metrics (c.g., the success rate) as re- wards. Nonetheless, existing works with RL showed that the comprchensibility of generated system utterances could be corrupted when improving the performance fulfilling user requests. In our work, we (1) propose modelling the hierarchical structure between dialogue policy and natural language generator (NLG) with the option framework, called HDNO, where the latent dialogue act applied avoid designing specific dialogue act representations; (2) train HDNO via hierarchical reinforcement learning (HRL), well as suggest the asynchronous updates be- tween dialogue policy and NLG during training to theoretically guarantee their convergence local maximizer; and (3) propose using a discriminator mod- elled with language models as an additional reward to further improve the com- prehensibility. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA, showing improvements on the performance evaluated by automatic evaluation metrics and human evaluation. Finally, demonstrate the semantic meanings of latent dialogue acts to show the explanability for HDNO.
INTRODUCTION
Designing a task-oriented dialogue system is a popular and challenging research topic the recent decades. In contrast to the open-domain dialogue system (Ritter et al. 201T), it aims to help people complete real-life tasks through dialogues without human service (e.g., booking tickets) (Young) 2006). In task-oriented dialogue task, each dialogue is defined with a goal which includes user requests (i.e., represented as a set of key words known as slot values). The conventional task- oriented dialogue system is comprised of 4 modules (see Appendix (.1). each of which used to be implemented with handerafted rules (Chen et al. (017). Given user utterances, gives responses in turn fulfill the requests via mentioning corresponding slot values.
Recently, several works focused on training a task-oriented dialogue system in end-to-end fashion (E2B) (Bordes al. 2016 Wen et al. 2017) for generalizing dialogues outside corpora. To train a E2E model via supervised learning (SL), generated system utterances are forced to fit the oracle responses collected from human-to-hunen
conversations (Budzianowski et al., 201 The ora- cle responses contain faults by humans thus being inaccurate, which leads to biased SI. On the other hand, the goal absolutely clear, though the criterion of success rate that evaluates the goal completion is non-differentiable and cannot be used loss for SL.
'mperial College London. 'Layye Network Technology Co. Ltd.. *KAIST. University of Bath. Corre- spondence Yunjic Gu: yg 13934bbath ac.uk

---

Published as a conference paper at ICLR 2021
To tackle this problem, reinforcement learning (RL) is applied to train a task-oriented dialogue sys- tem (Williams and Young 2007; Zhao and Eskénazi 2016; Peng et al.
2018 Zhao et al. 2019).
Specifically, some works merely optimized dialogue policy while other modules, c.g., the natural language generator (NIG)), were fixed (Peng et al., 2018 Zhao et al.]
2019: Su et al.)
2018). In
contrast, other works extended the dialogue policy NLG and applied RL on the entire E2F dia- logue system, regarding each generated word in response as an action (Zhao and Eskénazi 2016) Although previous works enhanced the performance on fulfilling user requests, the comprehensibil- ity of generated system utterances are corrupted (Peng et al. 2018 Zhao et al. 2019) Tang et al. 2018a). The possible reasons are: (1) solely optimizing dialogue policy could easily cause the biased improvement on fulfilling user requests, ignoring the comprehensibility of generated utterances (see Section 3.1 ; (2) the state space and action space (represented as a vocabulary) in E2E fashion so huge that learning to generate comprehensible utterances becomes difficult (Lewis et al. (017);; and (3) dialogue system in E2E fashion may lack explanation during the procedure of decision.
In our work, we propose to model the hierarchical structure between dialogue policy and NLG with the option framework, i.e., a hierarchical reinforcement learning (HRL) framework (Sutton et al. 1999) called HDNO (see Section 4.1) so that the high-level temporal abstraction can provide the ability of explanation during the procedure of decision. Specifically, dialogue policy works as a high- level policy over dialogue acts (i.e. options) and NL. works low-level policy over generated words (i.e. primitive actions). Therefore, these two modules are decoupled during optimization with the smaller state space for NLG and the smaller action space for dialogue policy (see Appendix F). To reduce the efforts on designing dialogue act representations, we represent a dialogue act as latent factors. During training, we suggest the asynchronous updates between dialogue policy and NLG to theoretically guarantee their convergence to local maximizer (see Section 4.2). Finally, we propose using a diseriminator modelled with language models (Yang et al. 2018) as an additional reward to further improve the comprehensibility (see Section 5).
We evaluate HDNO on two datasets with dialogues in multiple domains: MultiWOZ. 2.0 (Budzianowski et al. 2018) and MultiWOZ. 2.1 (Fric et al. (209)), compared with word-level E2F (Budzianowski et al. 2018) trained with RL, LaRL (Zhao et al. 2019) and HDSA (Chen et al. 2019). The experiments show that HDNO works best in the total performance evaluated with au- tomatic metrics (see Section 6.2.1 and the human evaluation (see Section B.I). Furthermore, we study the latent dialogue acts and show the ability of explanation for HDNO (see Section
2 RELATED WORK
Firstly, we go through the previous works on studying the dialogue act representation for task- oriented dialogue systems. Some previous works optimized dialogue policy with reinforcement learning (RL), which made decision via selecting from handcrafted dialogue acts represented as ontology (Peng et al. 2018 Young et al.) 2007]] Walker 2000 He et al. 2018). Such represen- tation method is easily understood by human beings, while the dialogue act space becomes limited in representation. To deal with this problem, some researchers investigated training dialogue acts via fitting oracle dialogue acts represented in sequence (Chen et al.) 2019 Zhang et al.) [019 Lei al. 2018). This representation method generalized dialogue acts, however, designing good rep- resentation is effort demanding. To handle this problem, learning latent representation of dialogue act was attempted (Zhao et al., 2019 Yarats and Lewis 2018). In our work, similar (Zhao al.) 2019) we learn latent dialogue acts without any labels of dialogue acts. By this view, our work can be regarded as an extension of LaRL (Zhao et al. 2019) on learning strategy.
Then, we review the previous works modelling dialogue system with hierarchical structure. In the field of task-oriented dialogue systems, many works lay on modelling dialogue acts or the state space with a hierarchical structure to tackle the decision problem for dialogues with multi-domain tasks (Cuayáhuitl et al. 2009 Peng et al. 2017 Chen et al. 2019 Tang et al., 2018b Budzianowski et al.) 2017b). Distinguished from these works, our work views the relationship between dialogue policy and natural language generator (NLG) as a natural hierarchical structure and models it with the option framework (Sutton et al. 1999). In the field of domn-domain dialogue system, a similar hierarchical structure was proposed (Serban et al. 2017; [aalhh et al., 2019)) but with a different motivation from ours. In this sense, these two fields are possible to be unified.
2

---

Published as a conference paper at ICLR 2021
Finally, among the works training with hierarchical reinforcement learning (HRL), some of them set up an extrinsic reward for high-level policy and an intrinsic reward for low-level policy respectively to encourage the convergence (Peng et al. 2017, Budziaaowskii et al., 2017b). In our work, train both high-level policy and lewel policy with identical rewards to guarantee the consistency between two policies (Sutton et al. T999). On the other hand, in the field of open-domain dialogue system, Saach et al. (2019) represented the joint generated utterances over a turn as low-level action such that both high-level policy and low-level policy were in identical time scales. Besides, its low- level policy gradients flew through high-level policy during training, which degraded hierarchical policies an E2E policy with a word-level action space. In our work, (1) dialogue policy and NLG are decoupled during optimization and no gradients are allowed to flow between them; (2) these two policies are asynchronoully updated to theoretically guarantee the convergence to local maximizer; and (3) each generated word is regarded as a low-level action.
3 BACKGROUND
3.1 TASK-ORIENTED DIALOGUE SYSTEM
Brief Introduction: A task-oriented dialogue system aims to help fulfill a user's task through con- versation in turns. In general, each dialogue is modelled with an ontology called goal which includes inform slots and request slots. The traditional modular dialogue system is constituted of natural lan- guage understanding (NLU), dialogue state tracker (DST), dialogue policy and natural language generator (NLG)) For a dialogue system, it needs to infer inform slots from user utterances and transform them to a dialogue state, which is completed by NLU and DST (Chen et al.] 2017). In this work, we focus optimizing dialogue policy and NLG, leveraging oracle dialogue states and database search results to produce dialogue acts and then responses (that should include as many re- quest slots as possible) in turns. For optimizing dialogue policy, is modelled with Markov decision process (MDP) (Williams and Young 2007).
Existing Challenges: We identify the main challenges of task-oriented dialogue systems: (1) A dialogue with a single domain (i.e. completing one task in a dialogue) has been broadly studied, however, handling dialogue with multiple domains more challenging and needs more studies on it (Budzianowski et al.] 2018): (2) If ignoring the syntactic structure of generated system utterances (i.e. losing comprehensibility) the mission of task-oriented dialogues will be simplified to generat- ing corresponding labels (i.e., slots) for user utterances. Several existing algorithms already reached high scores on request slots acquisition but low scores on the comprehensibility of generated system utterances (Zhao et al.) 2019 Mchri et al. 2019), so the simplified task has been well-addressed. Reversely, if only focusing on the comprehensibility, the score on request slots acquisition could be drastically affected (Chen et al. 2019 Hosseini-Asl et al. 2020). In this work, we investigate the trade-off between the comprehensibility and request slots acquisition; (3) Designing and annotat- ing dialogue act structure is effort demanding (Budzianowski al. 2018). Therefore, learning
meaningful latent dialogue act becomes a new challenge (Zhao et al. 2019).
3.2 HIERARCHICAL REINFORCEMENT LEARNING WITH OPTION FRAMEWORK
Hierarchical reinforcement learning (HRL) is a variant of reinforcement learning (RL) which ex- tends the decision problem to coarser grains with multiple hierarchies (Sutton et al. 1999 Dayan and Hinton 1993. Parr and Russell, 1998: Dietterich, 1998). Amongst several HRL methods, the option framework (Sutton et al.) 1999) is temporal abstraction for RL, where each option (i.c. high-level action) lasts for a number ber of steps through primitive actions (i.e. low-level actions). From the view of decision problems, an MDP defined with fixed set of options naturally forms semi- MDP (SMDP). Formally, an option o = (I. 3, π) is composed of three components: an initiation set I CCS (where S is a state space), a termination function ((s.) + [0, 1] and an intra-option policy T (a, 8,) - (0, 1]. The reward over primitive actions is defined as r. E IR, identical to vanilla RI. An option o, available at 8, e I.. At each 8,, π is used to decide a low-level action a until the option is stochastically terminated by B. Similar to RL on flat actions, the probability transition function over options defined as p(s' [s,, o,) = Ee= p(s', 7², where p(s', k) is the probability of an option terminating in k steps and y E (0. 1). The policy over options is defined as (((o))s)) + [0,
and the reward over options lasting for l steps is defined as g Ot, 81+4) ET| Σ 1 70-1 r:
3

---

Published as a conference paper at ICLR 2021
(abbreviated as Ie for simplicity). Given set of options O, the optimization problem
options is defined as max ΣREM 2 " g where m = (t, ..... .) is a sequence containing the time step of each event I that will be experienced from some time step to future. To automatically fit complicated circumstances, we may also need discover options dynamically during learning. Intra-option policy gradient theorem and termination policy gradient theorem (Bacon et al., 2017) provide the basis to apply a policy gradient method for option discovery.
4 MODELLING HIERARCHICAL STRUCTURE BETWEEN DIALOGUE POLICY AND NLG WITH OPTION FRAMEWORK
generated utterances
primitive actions
dialogue policy high levell policy
state
LSTM
dialogue act aption
LITM
context vector
state
LSTM
destination
natural language generator low-level policy
Figure 1: This diagram demonstrates the overall architecture for modelling the hierarchical struc- ture between dialogue policy (i.e. high-level policy) and NLG (i.e. low-level policy) as an option framework. The text in gray represents the concepts for a traditional task-oriented dialogue system whereas the text in red matches the concepts for the option framework.
4.1 MODEL FORMULATION
In this section, we present view modelling the Hierarchical structure between Dialogue pol- icy and Natural language generator (NLG) with the Option framework (Sutton et al. 1999). called HDNO. Specifically, dialogue in HDNO is seen an option whereas each generated word from NLG is a primitive action. Accordingly, dialogue policy and NLG become the policy over option (i.e. high-level policy) and the intra-option policy (i.e. low-level policy) respectively. Distinguished from a conventional modular system, we additionally give a context to NLG to satisfy the conditions of the option framework. Moreover, since the primitive action space (i.e. vocabulary) comprises termination symbol, NLG can take over the responsibility of termination. For this reason, termi- nation policy in the original option framework is absorbed into the intra-option policy. The formal definition of HDNO is shown in D
Definition 1. A dialogue policy (i.e. policy over option) is defined o S x O - 10, where S is a set of contexts (i.e. utterances, dialogue states and database search results); O is a set of dialogue acts (i.e. options). dialogue act is defined as {Il,, Ro), where Io S is set of corresponding contexts for a generated word and To IIo V + [0, 1] natural language generator (NLG) (i.e, an intra-option policy). V is vocabulary (including a termination symbol).
According to MDP theorem over option (Sutton et al. 1999) and intra-option policy gradient theorem Bacon et al., 2017). we can naturally apply REINFORCE (Williams) 1992) to learn both @ and л. Therefore, following Section 3 and Definition 1 we can write policy gra- dients in our case such that VJ(())
Es 2 > 9% V In (s.) and VJ(TT,,)
Ex,,! [ DrT "r, In To, (w, (8)) ]. where we assume that the length of all generated system ut- terances is and m = (t, .t.) is a sequence containing the time steps of an event that appear in future for an arbitrary o, (Io,, )o,) E O.
An event is defined as policy over option calling intra-option policy some state.
4

---

Published as a conference paper at ICLR 2021
4.2 ASYNCHRONOUSLY UPDATING DIALOGUE POLICY AND NLG DURING LEARNING
As mentioned in Section 4.1 dialogue policy and NLG are written as Φ(o(8) and (o (8) respec- tively. However, since has to consider the current performance on the overall set of low-level policies for NLG, denoted as [mo)oeo. For the reason, we temporarily rewrite dialogue policy to ((o|=, 8) for convenience. The aim is finding the best policies (i.e. maximizers) so that the value can be maximized such that max v()]6(o|) 8)), Vs E S. If updating these two policies synchronously. it will cause the com- posite state (i.e. (Tr, s)) of p(o|TT, s) inconsistent before and after the update each time. Therefore, the value does not always monotonically improve during learning, which will affect the conver- gence of both policies (see Proposition TT.
(oo To); we can assume that when dialogue policy made decision,
To address this problem, we suggest updating dialogue policy and NLG asynchronously during learning to theoretically guarantee the convergence of these policies to local maximizer (see Proposition 2) The proofs of these two propositions are left to appendices due to limited space.
Proposition 1. Following the model of Definition I if ooll s) and a are synchronously updated, the value does not always monotonically improve and the policies may never converge to a local maximizer.
Assumption 1. (1) Reward function is bounded. (2) With sufficient number of samples, the Monte Carlo estimation for value on any state is accurate enough.
Proposition 2. Following the model of Definition Dand Assumption chronously updated, the value can improve monotonically during learning and the policies can finally converge to a local maximizer.
[ if olo), s) and π are asyn-
4.3 IMPLEMENTATION OF HDNO
In implementation of HDNO, we represent a dialogue act as latent factors (Zhao et al. 2019). which reduces the effort on designing suitable representation. In detail, dialogue act (i.e. an indicator representing an option) is sampled from dialogue policy represented as an isotropic multivariate Gaussian distribution such that (z|cc; A) = N( p(c). E(c)), where c is a context as well as pa()) E RK and [(c) R are parameterized with X. Moreover, NLG, i.e. π (ww,|z, 2, E;; v), is represented as a categorical distribution over a word parameterized with v, conditioned on an option and context in which involves preceding generated utterances in addition to the context e that activates the option 2. The full picture of this architecture is shown in Figure T
Furthermore, o(zlc; A) (i.c. dialogue policy) is implemented with one-layer linear model and out- puts the mean and variance of a multivariate Gaussian distribution. The input of Φ( c; A) is a con- text vector. In details, the last user utterances are firstly encoded with bidirectional RNN (Schuster and Paliwal 1997) with gated recurrent unit (GRU) cell (Chung et al., (014)) and global type at- tention mechanism (Bahdanau et al. 2015). Then, an oracle dialogue state and an oracle database search result are concatenated to an encoding vector of user utterances to form a context vector. The utterance encoder only trained during pretraining and fixed as a context extractor during HRL, so that the context space is reduced. On the other hand, π (ww,k 2 C;; V)) (i.e. NLG) is implemented with a recurrent neural network (RNN) with long short-term memory (LSTM) cell (Hochreiter and Schmidhuberr 1997), where the initial state is the concatenation of context vector and dialogue act sampled from dialogue policy. The information in the initial state is assumed to be propagated to the hidden states at the future time steps, so only feed it in the initial state in implementation.
4.4 PRETRAINING WITH BAYESIAN FRAMEWORK
Compared to VHRED (Serban et al. 2017) proposed in the field of open-domain dialogue systems, the latent dialogue act in HDNO is equivalent to the latent variables in VHRED. However, context of HDNO includes not only user utterances but also dialogue state and database search result. In this sense, HDNO extends VHRED to the field of task-oriented dialogue systems. As result, by changing user utterances in VHRED to the context in HDNO, we can directly formulate variational lower bound following the Bayesian framework and the model in Section 4.1 such that
T
max E (x~e(=)))) A
log (w. v) 3 KL| (==ee X) [| J)
(1)
5

---

Published as a conference paper at ICLR 2021
where (z A) is constrained by KL-divergence (Kullback and Leibler 1951 from multivariate standard Gaussian distribution N (= I) Referring to (Higgins et al. 2017). we additionally add a multiplier 3 on the term of KL-divergence to control the disentanglement of latent dialogue act x. We use Eq [ for pretraining dialogue policy and NLG in E2E fashion with oracle system utterances roughly allocate the roles of these modules. Nevertheless, restricted with the existing human faults in oracle system utterances (see Section DT. we need to further improve it via HRL.
USING DISCRIMINATOR OF LANGUAGE MODELS AS A REWARD
Benefiting from the availability non-differentiaalee evaluation metries in RL, we can directly apply the success rate on reward denoted by y yruce. However, there exists two potential issues: (1) since the success rate given until the end of a dialogue that is zero the other turns, which may cause sparse reward; and (2) the success rate is only correlated to the occurrence of request slots in generated system utterances, the improvement on comprehensibility may be weakened. To mitigate the above drawbacks, we propose to leverage discriminator modelled language models (see Definition 2) as an additional reward. Specifically, at each time step it evaluates each generated word by oggliikl to reflect the comprehensibility.
Definition 2. Discriminator D(w, ww,1) + [0, 1] is defined as the Markov language model fol- lowing (Yang al. 2018). Ar some time step T, for an arbitrary option (Ies Te), w, <e(-)s;) a sampled word at state 8, E Il. The reward of discriminator to evaluate w, is defined as ydliee log (0-11,-1). where w a generated word at time step T
According to Definition 2 we can see that Ei= d consistently grows as the joint log- likelihood Σ log Dww, (1) grows, thereby maximizing E, Ex[ET" 7'rllic almost equiva- lent maximizing ES Σi log ((w,/w,-1) when is around and T" is not too large, where T' denotes the number of time steps in turn. For this reason, ydise is suitable for evaluating the comprehensibility of generated system utterances if we presume that the discriminator can well rep- resent human language. Combining the reward of success rate y.suee and the reward of discriminator y dise we propose a total reward such that
P (1 -
Joial
a) r7 + disc
(2)
where E [0, 1] is a multiplier controlling the trade-off between these two types of rewards. In implementation, the discriminator is equipped with the same architecture that of NI.G.
EXPERIMENTS
6.1 EXPERIMENTAL. SETUPS
To evaluate the performance of our task-oriented dialogue system, we run experiments the latest benchmark datasets MultiWoz 2.0 (Budzianowski al. 2018) and Multi- Woz 2.1 (Eric et al. 2019). MultiWoz 2.0 is a large scale task-oriented dialogue dataset including 10425 dialogues that spans 7 distinct domains, where the whole dialogues are generated by human- to-human conversations. Each dialogue is defined with a goal for user, which may be consisting of 1-5 domains. A dialogue system attempts to fulfill a goal by interacting with a user. As for data preprocessing, we follow the same delexicalized method provided by (Budzianowski et al. 2018), also used in other works (Zhao al. 2019 Chen et al., 2019). On the other hand, Multi 2.1 is modified version of MultiWoz 2.0 which mainly fixes the noisy dialogue state annotations and corrects 146 dialogue utterances. Finally, the dataset of either MultiWoz 2.0 or MultiWoz 2.1 is split into a training set with 8438 dialogues, validation set with 1000 dialogues and a testing set with 1000 dialogues (Budzianowski et al., 2018 Eric et al. 2019).
Dataset Description:
Since we only concentrate learning dialogue policy and natural language gen- erator (NLG), all experiments are conducted on the dialog-com coalloo--oneet-too toxtxxxtttteettttt generation task proposed in (Budzianowski et al. 2018). In this task, it assumes that a dialogue system has access to the oracle dialogue state and database search result. Given user utterances, dialogue system attempts to gen- erate appropriate utterances response in each turn. To train dialogue system with hierarchical
Task Description:

---

Published as a conference paper at ICLR 2021
reinforcement learning (HRL), follow the setups described in Definition I Each dialogue is only evaluated with the goal (e.g. calculating the success rate) at the end of dialogue, which means that no evaluation is allowed during any of turns.
Automatic Evaluation Metrics: Following (Budzianowski et al. 2017a), we leverage three auto- matic metrics to evaluate generated utterances from dialogue system such inform rate, success rate and BLEU score. Inform rate measures whether a dialogue system provides appropriate entities (e.g., the name of restaurant). Success rate shows the ratio of request slots appearing in generated utterances. BLEU score (Papineni et al., 2002) evaluates the comprehensibility of generated ut- terances. Finally, we use popular total score (Zhang et al., 2019 Mehri al.) 2019) such that 0.5 x (Inform + Success) + BLEU to fairly evaluate the performance of dialogue system.
Baseline Description: We compare HDNO with other models such as LaRL (Zhao et al. 2019) HDSA (Chen et al., 2019). and a baseline end-to-end model (Budzianowski et al.) 2018). All these models leveraged oracle dialogue states and database search results, which are introduced as follows:
The baseline end-to-end model (Budzianowski al. 2018 is a model which directly maps context to system utterances. Followed by (Zhao et al. 2019), we train it with RL, where each generated word is looked as an action. For convenience, we name as word-level E2E model, abbreviated as WE2F.
LaRL. (Zhao et al. 2019) is model that firstly represents a dialogue act as latent factors in the field of task-oriented dialogue system. Specifically, it models a latent dialogue act as categorical variables, each of which is mapped to a continuous embedding vector for learning. During train- ing, it only updates dialogue policy, where a latent categorical dialogue act for each turn is looked as action.
HDSA (Chen et al. 2019) is model that represents each dialogue act as hierarchical graph. To fit the oracle dialogue act, a pretrained 12-layer BERT (Devlin et al. 2019) is applied. Then the predicted dialogue act transformed to the hierarchical graph structure with 3-layer self-attention model VVaswani et al. 2017), called disentangled self-attention model. This model is trained only with supervised learning (SL).
For HDNO we pretrain a model following Eq and select the best model with the minimum loss on the validation set; as well as the discriminator is pretrained with oracle system utterances. During HRL, we initialize parameters with the pretrained model and select the best model according to the greatest reward on the validation set. For efficiency, we only use greedy search for decoding in validation. In test, we apply beam search (Medress et al. 1977) for decoding to obtain better performance. The beam width is selected through the best validation performance for each model. For simplicity, we only show the performance of the best beam width on test set. We use stochastic gradient descent (SGD) for HRL and Adam optimizer (Kingma and Ba 2015) for pretraining. For the baselines, we train them with the original source codes. The specific details and hyperparameters for training and testing are shown in Appendix B.3
Experimental Details:
Notification: Please notice that the results of baselines showed in their original papers could be underestimated, which is due to the upgrade of the official evaluator this year For this reason, re-run these experiments with the original open source codes and evaluate the performance of all models (including HDNO) via the latest official evaluator in this work.
6.2
MAIN RESULTS
6.2.1
AUTOMATIC EVALUATION METRICS
We firstly compare HDNO with the state-of-the offteof-thh-art baselines and the human performance on both datasets via automatic evaluation metrics. As T we can see that HDNO trained with the proposed asynchronous updates between dialogue policy and NLG (i.e. HDNO (Async.)) largely outperforms the baselines on the inform rate and total score, while its BLEU score is lower than that of HDSA. Moreover, the performance of all models trained with RL except for HDNO trained
The source code implementation HDNO is on ht / /i thub. com/m kez 'llasse check it via the clarification for this incident helow the table called Policy Optimization on the official website of MultiWaz; https::/ www/budz.ii. anwwa wttmmmmltiiiimm

---

Published as a conference paper at ICLR 2021
Table 1: The table shows the main results on MultiWoz 2.0 and MultiWoz 2.1 evaluated with the
automatic evaluation metrics. The results of HDNO are from the models trained with the proposed reward shape in Section where a 0.0001 for MultiWoz 2.0 and 0.01 for MultiWoz 2.1.
MultiWoz 2.0
MultiWoz 2.1
Informi %
Successi %%)
BLEU(%)
Total
Inform( %)
Success %)
BLEU(%)
Total
Human
91.00
82.70
86.30
79.10
WEZE
90.29
86.59
14.08
102.52
90.89
83.58
14,52
101.76
LaRi.
93.49
84.98
12.01
101.25
92.39
85,29
13.72
102.56
HDSA
88. 90
73.40
23.15
104.30
85.60
75.50
21.57
102.12
Pretmaining (Bayesian)
69.50
62.00
19.10
84.85
71,40
62.80
19.12
86.22
HDNO (Sync.)
83.20
73.50
19.82
98.17
83.
70.80
18.81
95.76
HDNO (Async.)
96.40
84.70
18.85
109.40
92.80
83,00
18.97
106.77
with synchronous updates between dialogue policy and NLG (i.e. HDNO (Sync.)) on the inform rate and success rate exceeds that of the model trained with SL (i.e. HDSA). The possible reason may be that SL is highly dependent on the oracle system utterances and humans may commit faults during generating these dialogues (Sync.) validates the theoretical analysis in Section 4.2 that synchronous updates between dialogue policy and NLG could cause the failure to approach local optimum, while HDNO (Async.) shows the success for the asynchronous updates proposed in Proposition 2 Furthermore, the results of pretraining give the evidence that the improvement is actually from the proposed algorithm. For conciseness, we write HDNO instead of HDNO (Async.) in the rest of paper. We also conduct human evaluations for WE2F, LaRL. and HDNO that are shown in Appendix B.T
stated in Section I Besides, the poor results HDNO
6.3
STUDY ON REWARD SHAPES
Table 2: The table shows the results of different reward shapes for HDNO MultiWoz 2.0.
Inform(
Success( %
BLEU(%)
Total
Suceess Success + BLEU
96. 10
84,20
18.51
108. 66
95.60
83.30
18.99
108.44
0.0001
96.40
84,70
18.85
109.40
0.0005
96.30
84,90
18.50
109.10
Success Discriminator
0.001
96.20
84.20
19.04
109.24
0.005
97.00
84.10
18.72
109.27
We compare the proposed reward shape in Eq 2 with reward only constituted of the success rate and a reward combining the success rate with BLEU score (i.e. a linear combination similar the proposed reward shape). As Table 2 shows, proposed reward shape (i.e. success + discriminator) performs better preserving the comprehen- sibility while improving the success rate and inform rate to maximum. To further study the impact of discriminator in the total reward, we also run several ablation studies on (see Section []. As Table 2 shows, the results oscillate within small range, which means that the proposed reward shape not sensitive to the hyperparameter if it is selected within rational range.
comparison with other reward shapes, the
6.4 STUDY ON LATENT DIALOGUE ACT
We now study the semantic meanings of latent dialogue acts and demonstrate the clustering results as sigure To show the explanability of HDNO that other baselines do not possess, we show the results for both HDNO and LaRL. Since LaRL tends to generate duplicate latent dialogue acts, the dots in the diagram are overlapped. Through analyzing the randomly selected system utterances, we find that the clusters of latent dialogue acts of HDNO possesses some semantic meanings, while that LaRL is difficult to be observed any meaningful explanations. Next, we briefly describe our findings clusters of HDNO. The clusters in blue dots and green dots are related to the general phrases for goodbye at the end service; the cluster in orange dots related to the booking for trains; the cluster in red dots is related to informing
with database search results; the cluster in brown dots is related to recommendation; the cluster in pink dots is related to informing unsuccess- ful booking; the cluster in grey is related to informing booked; and the cluster in yellow is related
8

---

Published as a conference paper at ICLR 2021
their centact ramber is Traad phees) de you reed anyening size
Fleane call soste daadbye
tave train ttrste st that leaves as Evatun time] and will get you into (valub....
abat
heppy servuce have wonertt
are happy belp cane 20se
strry ttere for (valie pricertenal
cestarre is Line thet in ivstue ploust wntee insatele....
sarry du mabunng moddaag your ruguss
you try
wade you try frestument remel
Uscpes that oller Ire perking
lave
sir
carest cane thes metta nat ot haw takte sest tvee shat
1 tave reterctiiss for Leblie deant: peoole has meatt nighte starting...
made resernatoyy
their anane nater [rectsuree purt phene) thene aesersing stoo coe ze your
uneituay
(a) HDNO.
fottraccie sanel a Ieluee arlceraneet fly prices Eallue coUnCE star notel dhree
I1 a [ead) tyoe 15 There aestiing olse cat anta you with
anyessing
You are welcare hase sice eay
sould you Lake book
frastacrant satal (ialse praturaepe
Dalli Troel restaurant in
thate Ivatos praserange) praned tretue
thair phone natber in jatteratin mectos
pou..s
tightillde
like dall erent arauel peands there ceyening
tnater anaal
ticket teatun
with
ray
loliradiae nanel localie
tvatus Uset shone rember fatinee...
booking
lattractte tatal nice place in the (valie seea. woild we tike thst lee...
I''s
(hotel camel Ivatus
teatue count)
sodiding ane useacens etull woild you 100e to try sifferent day
(b) LaRL.
Figure 2: These diagrams demonstrate latent dialogue acts of HDNO and LaRL clustered in 8 categories on a 2-D plane. Clustering is conducted with k-means algorithm (Arthur and Vassilvitskiij 2006) on the original dimensions whereas dimension reduction is conducted with T-SNE algorithm (Maaten and Hinton 2008). We randomly show 3 turns of system utterances for each cluster.
to requesting more information. Surprisingly, these semantic meanings of latent dialogue acts highly correlated with that of the oracle handcrafted dialogue acts (see Appendix E} described by Budzianowski et al. (2018). Therefore, learning latent dialogue act with the option framework (Sut-) ton et al.] 1999) may potentially substitute for handerafting dialogue act with ontology, without losing its explanability.
7 CONCLUSION AND FUTURE WORK
In this paper, present a view on modelling the hierarchical structure between dialogue policy and natural language generator (NLG) with the option framework (Sutton et al. 1999) in a task-oriented dialogue system and train it with hierarchical reinforcement learning (HRL). Moreover, we suggest asynchronous updates between dialogue policy and NLG to theoretically guarantee their conver- gence to local maximizer. Finally, we propose using a discriminator modelled language models Yang et al. 2018) as a reward to further improve the comprehensibility of generated responses.
In the future work, we are going to extend this work to optimizing all modules by HRL instead of only dialogue policy and NLG, as well as study on solving the problem of credit assignment among these modules (Chen et al. 2017) during training. Moreover, thanks to the option framework (Sutton et al. 1999). the latent dialogue act shows explicit semantic meanings, while disentangling factors of a latent dialogue act (i.e. each latent factor owning a distinct semantic meaning) during HRL is left to be further investigated.
ACKNOWLEDGMENTS
This work is supported by the Engineering and Physical Sciences Research Council of UK (EPSRC) under awards EPP/S00009//1.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alan Ritter, Colin Cherry, and William B Dolan. Data-driven response generation in social media. In Proceedings of the conference on empirical methods natural language processing, pages 583-593. Association for Computational Linguistics, 2011.
Steve Young. Using pomdps for dialog management. In 2006 IEEE Spoken Language Technology Workshop, pages 13. IEEE, 2006.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-355 2017.
Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07666 2016.
Tsung-Hsiees Boing-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina Maria Kojas-Barahonn Pei- Hao Su, Stefan Ultes, and Steve J. Young. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 438 449. Association for Computational Linguistics, 2017.
Pawet Budzianowski, Stefan Ultes, Pei-Hao Su, Nikola Mrkšić, Tsung- -uing-Hsien Wen, Inigo Casanueva, Lina Rojas-Bard Barahona, and Milica Gašić. Sub-domain modelling dialogue management with hierarchical reinforcement learning arXiv preprint arXiv: (70.00000.66000, 2017a.
Jason D Williams and Steve Young. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21 422, 2007.
Tiancheng Zhao and Maxine Eskénazi. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In Proceedings of the SIGDIAL 2016 Confer- ence, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13-15 September 2016, Los Angeles, CA, USA, pages 1-10. The Association for Computer Linguistics, 2016.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. Deep dyna-q: Integrating planning for task-completion dialogue policy learning. In Proceedings of 56th Annual Meeting of the Association for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2182-2192. Association for Computational Linguistics, 2018.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. Rethinking action spaces for reinforce- ment learning in end-to-end dialog agents with latent variable models. arXiv preprint arXiv:1902.08855, 2019.
Shang-Yo Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. Discriminative deep dyna-q: Robust planning for dialogue policy learning. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 3813-3823. Association for Computational Linguistics, 2018.
Da Tang, Xiujun Li, Jianfeng Gao, Chong Wang, Lihong Li, and Tony Jebara. Subgoal discovery for hierarchical dialogue policy learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, October 31 November 4, 2018, pages 2298-2301 Association for Computational Linguistics, 2018a
Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. Deal deal? end- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. EMNLP 2017, Copenhagen, Denmark, September 9- 11, 2017, pages 2443-2453. Association for Computational Linguistics, 2017.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112((-2):181-
211, 1999.
10

---

Published as a conference paper at ICLR 2021
Zichao Yang. Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- Kink--rrkatticc. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pages 7287-7298, 2018.
Pawel Budzianowski, Tsung- Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz- A large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 5016- 5026. Association for Computational Linguistics, 2018
Mihail Fric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tin Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint wrw.h::107.0000000 2019.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. Semantically con- ditioned dialog response generation via hierarchical disentangled self-atrention. In Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3696-3709. Association for Com- putational Linguistics, 2019.
Steve Young, Jost Schatzmann, Karl Weilhammer, and Hui Ye. The hidden information state ap- proach dialog management. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSPPA volume 4, pages IV-149. IEEE, 2007.
Marilyn A Walker. An application of reinforcement learning dialogue strategy selection in spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416, 2000.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 2018, pages 2333-20 3333.2343. Association for Computational Linguistics, 2018
Yichi Zhang. Zhijian Ou, and Zhou Yu. Task-oriented dialog systems that consider multiple appro- priate responses under the same context. arXiv preprint 1rXiv:/911.104500 2019.
Wenqiang Lei, Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: Simplifying task-oriented dialogue systems with single touuence-t--eqqeeee architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447, 2018.
Denis Yarats and Mike Lewis. Hierarchical text generation and planning for strategic dialogue. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5587-5595, PMLR, 2018
Heriberto Cuayáhuitl et al. Hierarchical reinforcement learning spoken dialogue systems. PhD thesis, The University of Edinburgh, 2009.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Çelikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2231-2240. Association for Computational Linguistics, 2017.
Da Tang, Xiujun Li, Jianfeng Gao, Chong Wang, Lihong Li, and Tony Jebara. Subgoal discovery for hierarchical dialogue policy learning. arXiv preprini arXiv: 1804.07855 2018b.
Pawel Budzianowski, Stefan Ultes, Pei-Hao Su, Nikola Mrksic, Tsung-Hsien Wen, lñigo Casanueva, Lina Maria Mojas-Barahona, and Milica Gasic. Sub-domain modelling for dialogue management with hierarchical reinforeement learning, In Proceedings of the 1 18th Annual SIGdial Meeting Discourse and Dialogue, Saarbrüicken, Germany, August 15-17, 2017, pages 86-92. Association for Computational Linguisties, 2017b.
11

---

Published as a conference paper at ICLR 2021
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for gen- erating dialogues. In Thirty-Fitie Fiirst AAAI Conference on Artificial Intelligence, 2017.
Abdelrhman Salch, Natasha Jaques, Asma Ghandeharioun. Judy Hanwen Shen, and Ros- alind Picard. Hierarchical reinforcement learning for open-domain dialog. arXiv preprini 2019.
Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. Structured fusion networks for dialog. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 165-177, 2019.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00799 2020.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural infor- mation processing systems, pages 271-278, 1993.
Ronald Parr and Stwart Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pages 1043-1049, 1998.
Thomas G Dietterich. The maxq method for hierarchical reinforcement learning. In ICMI, vol- ume 98, pages 116.126. Citeseer, 1998.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Ronald J Williams. Simple statistical wowlddien---lllooiig algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-2681, 1997.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86, 1951.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Iclr, 2(5):6, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171-41886 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
12

---

Published as a conference paper at ICLR 2021
Mark F. Medress, Franklin S Cooper, Jim W. Forgie, CC Green, Dennis H. Klatt, Michael H. O' O'Malley, Edward P Neuburg, Allen Newell, DR Reddy, Ritea, et al. Speech understanding systems: Report of steering committe. Artificial Intelligence, 933):307-316, 1977.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICIR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
David Arthur and Sergei Vassilvitskii. k-means+4: The advantages of careful seeding. Technical report, Stanford, 2006.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(No))2557, 2008.
Heinz H Bauschke, Patrick Combettes, al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011.
Jiahuan Pei, Pengjie Ren, and Maarten de Rijke. A modular task-oriented dialogue system using a neural mixture-of-experts arXiv preprint arXiv: 10XX:v:907.053400 2019a.
Jiahuan Pei, Pengjie Ren, Christof Monz, and Maarten de Rijke. Retrospective and prospec- tive mixture-of-generee for task-oriented dialogue response generation. arXiv preprint 1rXiv:191..0815,, 2019b.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. Alternating recurrent dialog model with large- scale pre-trained language models. arXiv preprint arXiv:1910.03756. 2019.
Baolin Peng. Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. arXiv preprint 20Xi::2005.05298, 2020.
Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. Multi-domain dialogue acts and response co-generation. arXiv preprint arXiv:2004. 200::0004.1263,, 2020.
S Patro and Kishore Kumar Sahu. arXiv: 06466, 2015.
Normalization: A preprocessing stage. arXiv preprint
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
APPENDICES
A PROOFS
A. PROOF OF PROPOSITION
Proposition I. Following the model of Definition ! if ( 8) and n are synchronously updated, the value does not always monotonically improve and the policies may never converge to a local maximizer.
Proof. At an arbitrary time step E N, assume that both of 6(o)T, s) and æ have been updated for E N times, denoted as (x²) and respectively for conciseness. The current value for any arbitrary state $ is denoted ((s))((TT)) If we synchronously update d ((()) and T9, can obtain the following equations on value after updates at time step t E N such that
([()00+((79)) > o (s) 0' (5 2 of (s(o
(3)
13

---

Published as a conference paper at ICLR 2021
At next time step + 1, however, the actual value for any arbitrary state that obtain from the last synchronous update is the equation such that
nt (s| (x*+1)),
(4)
and the following scenario such that
g (s| (T1+1)) < 4 ( (x)))((9))
(5)
could happen, which means that the monotonic improvement path of the value cannot always hold during learning. Since the value is an evaluation of policies, the policies could never converge to local maximizer.
A.2 PROOF OF PROPOSITION 2
Lemma 1 Bauschke al. (2011)). (I) Any increasing bounded sequence (EN) in R is Fejér Monotone with respect to sup Tuee such that Vn E N, || sup Irelll I || sup(aa]nenll (2) For Fejér Monotone Sequence (In)neN | sup( JoEN IneN converges, ||- is an arbitrary norm in R.
Assumption D. (1) Reward function is bounded. (2) With sufficient number of samples, the Monte Carlo estimation for value on any state is accurate enough.
7 if o s) and π are asyn- chronously updated, the value can improve monotonically during learning and both policies can finally converge to local maximizer.
Proposition 2 Following the model of Definition| [an Assumption
Proof. Firstly, in REINFORCE (Williams 1992) the value is approximated by Monte Carlo estima- tion. Following (2) in Assumption and the denotation shown in the proof of Proposition [ Jabove, if we update ((()) ) and π asynchronously every & E N steps, from some time step E N, assuming that both of policies have been updated q N times, can construct a monotonically increasing sequence of values for any arbitrary state s E S during learning such that
o' (ssl""(TT)) S u r/ (s|Ø 50*t (T))
I x ( (1)++)) ())+)t((7*+1))
I wt+ (slptt .... S oltnk (s)oet aft ( (z))
(olnkks)) (s|@gt 2 (
n is an even number, if an odd number.
(6)
Due to (1) in Assumption] 1 we get that the value is bounded and we consider that (()(((T)) E IR,
vs S. According (1) Lemma [ the sequence of values is Fejér Monotone with spect to the maximum value. For simplicity, we denote the sequence of values in Eq as {u" } ImC and the maximum value v". Since (2) in Lemma [ we can conclude that (Ilem v²) me (t+nk)+tbbAEM)) converge. Also, since the v² has to be the final item of the sequence, we can write that
0²1l 1100 .'1l 0, m 0o. If we rearrange the left hand side of E7 obtain the result such that
(8) From Eq g we can conclude that finally the sequence of ((T))) will converge to some local maximum. Since the value is an evaluation of of o/ s) and T 8), we get the conclusion that the asynchronous updates enable these two policies to converge to a local maximizer.
m + po.
B EXTRA EXPERIMENTAL RESULTS
B. HUMAN EVALUATION
Due to the possible inconsistency between automatic evaluation metrics and human perception, we conducted human evaluation on comparing the quality of generated responses. We provide two criteria for human assessors evaluate the generated responses: (1) fluency: how fluent the
14

---

Published as a conference paper at ICLR 2021
3.0
Faumpas Conteat
Low] merore toon dllerer
expoetirs bug
Last Not
yeu meeete cilled Mottes
Generated Response
wi22: roen book that wo not lino woord boto the
4.-
Latk,
namber
Hieio Wher aeae you clavts
Esergte
Then
H00 and
30-
Jess jufiee bookii you dedeactneiiin
Generated
phone nowr 1234077a The sitance te 3lpounds poetoses la POV. dhaee
HoNi, Tale Modee the The nunber 12345070,
Ruency
App accness
atyting elee Help
(a) Statistical results.
(b) Example generated responses.
Figure 3: (a) This figure shows the statistical results of human evaluation, where 30 people partici- pated in this human evaluation and the questionnaire is constituted of 31 randomly selected turns of dialogues. (b) This figure shows the example generated responses from the questionnaire.
generated responses are (i.e., with no obvious grammar errors and redundant descriptions); and (2) appropriateness: how related the generated responses are to the provided context. For each criterion, human assessor is allowed to give a score ranging from to 5, where 5 indicates the best and indicates the worst performance. Then, we calculated the mean and the variance for the score each model. The final results and some example generated responses from the questionnaire are shown in Figure 3 The proposed HDNO performs best in the human evaluation, especially outperforming LaRL (Zhao et al. 2019) on appropriateness so much.
B.2 LEARNING CURVES
21
5ate
HIN
NDNC Lift
Naman
32680 98000 20003
2000 4003
5000 0660
5000 33383
Esiaode
Ediode
Eslaade
(a) Inform rate during training (b) Success rate during training (c) BLEU score during training MultiWoz 2.0.
MultiWoz 2.0.
Multi Woztiiooz/ 2.0.
23.3
Intom Bate 1861
176
Lift
HOWO
15.25
:15
2007 5530 SELE SELE (66A Lutte 16200 16302
2007 B0S SECA SIEG 1066 Litt 14920 19202
199
2002
9020
1ECA:
S02D 1666 LADE 14020 26303
Training
(d) Inform rate during training (c) Success rate during training on (f) BLEU score during training on MultiWoz 2.1.
MultiWoz 2.1.
MultiWoz 2.1.
Figure 4: Validation inform rate, success rate and BLEU score during training.
this section, we show the learning curves of HDNO (Async.) (i.e. abbreviated as HDNO in the rest of the appendix) and baselines both MultiWoz 2.0 and MultiWoz 2.1. 1. To show the performance of generalization, we only demonstrate the validation results during training. As seen from Figure compared with baselines HDNO can preserve the comprehensibility (i.e. BLEU) while improving success rate and inform rate faster. We only show the results for the initial 16,000 episodes for conciseness of the figure.
15

---

Published as a conference paper at ICLR 2021
B.3 COMPLETE RESULTS WITH BEAM SEARCH
Table 3: The table shows the results of HDNO (Async.) with beam search on MultiWoz 2.0 and MultiWoz 2.1.
Revaar Shapes
Irrm witth =
Iiea widtth 2
Brom witth 2.5
(nferm(h)
Seccos(()
BLS)1/4)
Total
Inferm:%))
Secroo 1%)
BL.UU(4)
Total
Inform/
Succem/N)
BLEU1))
Irtal
b10001
96.70
151.90
18:72
119
95.90
34.70
15.85
109
45.30
$4.30
18:20
MaltWWee 2
b0000
98.30
81.00
18
100
95-30
18.10
15:33
108.61
45.70
$4.90
15.99
100.
01011
88190
19.29
108
54:33
19.08
100
45.70
$4.90
18:30
100.85
MiltWWee 7 (Sxceed)
05.20
82.00
1883
107
96.10
84.30
18.51
100 66
95.30
92.70
18.57
107.57
MillWWe
(Socesis BLBU)
94900
$1.20
19.19
24
95.60
$3.30
18.90
100
95.70
$3.50
1977
108
Table The table shows the pretraining results with beam search on MultiWoz 2.0 and MultiWoz 2.1.
MultiWoz 2.0
MultiWoz 2.1
Inform %)
(uccess(%%)
BLEU(%)
Total
Inform( %)
Success( %)
BLEU(%)
Total
Beam=1
69.50 69.50
62.00
19.10
84.85
70.70
61.40
18.24
84.29
Beam=2
67.90
60.20
19.00
83.05
71.50
62,20
19.28
86.
Beam=
69.50
62,50
19.20
85,20
71.40
62.80
19.12
86.22
Table 5: The table shows the results of HDNO (Sync.) with beam search MultiWoz 2.0 and MultiWoz 2.1. The reward shape is the one we proposed in Section with a = 0.0001 for MultiWoz 2.0 and a 0.01 for MultiWoz 2.1 respectively.
MultiWoz 2.0
MultiWoz 2.1
Inform %)
Success( BLEU(%)
Total
Inform %)
Success( BLEU(S)
Total
Beam=1
79.70
70.90
20.16
95.46
82.70
69.60
18.92
95.07
Beam=2
79.60
70.80
19.71
81.40
69.40
94.57
Beam=
83.20
73.50
19.82
98.17
83.10
70.80
18.81
95.76
In this section, we show the complete results of HDNO (Async.) and HDNO (Sync.) as well as the pretraining results with beam search in Table 3 3 and respectively, where beam width is selected from 1, 2 and 5. Apparently, the proposed HR algorithm gives an enormous improvement on the performance in comparison with that of the pretraining. Nevertheless, the pretraining is an essential part that cannot be replaced before training with reinforcement learning in the task-oriented dialogue system. The possibility of training from scratch in HRL with good performance for the lask-oriented dialogue system is left to be investigated.
B.4 COMPLETE BENCHMARK RESULTS ON MULTIWOZ 2.0
In this section, we show the complete state-of-the-arr results on MultiWoz 2.0 for the policy optimization task, from the official records R All these results are collected from their origi- nal papers. As Table 6 shows. HDNO leads the board on the total performance evaluated by 0.5 (Inform Success) + BLEU, where each metric is measured in percentage. However, due the update of official evaluator this year, the results marked with were probably underestimated.
B.5 EXAMPLES OF GENERATED DELEXICALIZED DIALOGUES
In this section, we demonstrate some system utterances generated by the baselines and HDNO. Since most of the dialogues in MultiWoz 2.0 and MultiWoz 2.1 are similar, we only show the results on MultiWoz 2.0. As we can see from Table 7 and g compared with the baselines trained with SL (i.e. HDSA), the performance of HDNO on fulfilling user's request is actually better, however,
16

---

Published as a conference paper at ICLR 2021
Table 6: The table shows the full benchmark results on MultiWoz 2.0, compared with HDNO.
Inform %)
Success( %)
BLEU(%)
Total
HDNO (Our Model)
96.40
84.70
18.85
109.4
TokenMoB * (Pei et al.
2019a
75.30
59.70
16.81
84.31
Bascline Budzianowsk ct al.
2018
71.29
60.96
18.80
84.93
Structured Fusion Mchr ct
2019
82.70
72.10
16.34
93.74
LaRL (Zhao et al.
2019
82.80
79.20
12.80
93.80
SimpleTOD (Hossem--As)
2020
88.90
67.10
16.90
94.90
MoGNet (Pei ct 20199b
85,30
73.30
20.13
99.43
HDSA (Chen et al.. 2019
82.90
68.90
23.60
99.50
ARDM Wu al._ 2019
87.40
72.80
20.60
100.70
DAMD Zhang ci al. 2019
89.20
77.90
18.60
102.15
SOLOIST [Peng el al. 2020
89.60
79.30
18.30
102.75
MarCo Wang et al. 2020
92.30
78.60
20.02
105.47
the generated utterances of HDSA could be more fluent and comprehensible. In comparison with other baselines trained with RI, the generated utterances of HDNO is apparently more fluent and comprehensible. Especially, WE2E tends to generate as many slots as possible so as to increase the success rate, regardless of the comprchensibility of generated utterances. This is the common issue of most RL methods on task-oriented dialogue system as we stated in Section the main part of paper.
C EXTRA EXPERIMENTAL SETUPS
C.I TRAINING DETAILS
During pretraining, natural language generator (NLG) is trained via forcing the prediction each time step to match an oracle word given a preceding oracle word as input. Nevertheless, during hierarchical reinforcement learning (HRI.,, NLG updates the estimated distribution each time step by VJ((T) Ex, E "" In T (wr,(s)) given a word sampled from a preceding predicted distribution. As for sampling dialogue act from dialogue policy, we leverage reparameterization trick during pretraining. On the other hand, the discriminator firstly pretrained and fixed as a reward during HRL.
We find that useful to train both of the discriminator and HDNO simultaneously during pre- training. Specifically, in addition to training oracle system utterances, we also use the generated utterances to train the discriminator, which can improve its performance in experiments. The opti- mization problem is expressed 9 where D(-I-6 0) is the discriminator defined in Definition 2 in the main part of paper, parameterized with 6: and v7 is a multiplier to control the contribution of the generated utterances (ui;)) ) given an initial state. We hypothesize that the generated utterances improve the
could expand the original small scale corpus (i.e. oracle system utterances), generalization of the discriminator. Since we have not grasped the intrinsic reason of this method, only regarded trick for training the discriminator our work.
T
log D(w, 1; + log D D(w:
t=
t=
During HRI., 50000 and disc are respectively normalized by z-score normalization (Patro and Sahu, 2015) to adaptively control their impacts in the total reward. For instance, when either your or jas converges around some value, its normalized value will be close to zero and the total reward will be biased the other. Therefore, this further mitigate the conflicts between improving both the success rate and preserving the comprehensibility as stated in Section in the main part of paper.
Furthermore, the experiment of HDNO was and it consumes around hours for SL and hours for RL. Therefore, is not very expensive to reproduce results.
Nvidia GeForce RTX 2080Ti graphic card,
17

---

Published as a conference paper at ICLR 2021
Table 7: The table shows some delexicalized system utterances generated by the baselines and HDNO on MultiWoz 2.0.
Models
Delexicaizzd Dialogue
need train reservations from [valse place] [valise [valise-place] travel
sys: what day would you
war: I would like to leave on |valae day] and anrive by [value time]
[train,idl id] leaves at (value time] and arrives at [value_Fime]] would you like me to book that for you?? ?
HDNO (Our Model)
booking. would also like syst the ravel time [value_coumi] minutes. the price sulue_prioe] pounds there anything else can help you with
price departure time please.
no hold off booking for now can help me find an attraction called [otractionstamm]] !?
syst [attraction name] museum
(valuc arcal would you like their location?
mort yos. that was nectle thank you very much! syst you are welcome have a nice day goodbye.
need train reservations from [value_place]] [vallsc_place] agst have [valuc.count] trains leaving vullec-pacc] on [valaculay] 1. what day and time would you like to travel
usn: i would like to kave on [valacualay] I and arrive by [valuc, time]
the [train_id] arrives (valuc [value.pric price] pounds the usr: before booking.. i would also like to know the travel time price and departure time please.
the price [valus.price] pounds the train id [trainidd]. the price is tain id [train id]] ...
WEZR
id is [triin.dd].
[triin
sys: the train id is [train id] and the price is (value pricel pounds. the tain id is [train idl, the price [value prive] pounds the auz: no hold off booking now can you help me find an attraction called [attraction name] 2
is Itrain id] the train
[train
the train is [tminnid idl
syst (attraction name as nightcluh locared in the (value prenn] of town the address is [attraction [ettraction_nkdrrss , the phone number is paaactio..ponnee I. the entrace fee (value count) pounds. , the postcode is [artraction postcode)... ... yes, syst you are welcome. have a great day goodbye.
necred. thank you very much?
us:: i need train reservations from [valuc place] to [valic place] have train leaving [valoe time). would you like to book that for you would like to leave on [value day] and anrive |value time]
syst i have tain leaving at [valae
would you like me to book that for you
LaRL
syst the [train_d]] ans: no loold olf booking for now can you help me find an altraction called entractiio.mamas ? syst [sttraction [name] is located in the [valucareee] of sown. the address is [uttactioo_dddress] and the posteode is [sttraction postcoda).
aor: before booking, would also like to know the travel time price and departure time please.
at [value_iime]
[valseerime] the travel
[value_count] minutes.
aor: yes, lhat syss you are welcome. have a great day.
necdld. ilank you very much!
wart need train reservations from collee_lacce] to [vallc_place]] syst what day and time would you like travel ? aort I would like to kavve on [valcclly] and arrive by [valuc.time]
sys: what time would you like to leave'
before booking. would also like to know the travel time price and departure time please. sys:
HDSA
[train leaveat) and arrives by (rainn.riivebl.... the price is [train_pric)] pounds the travel Ivalue count] minutes.
usz no hold off on booking for now can you help me find an attacttin called [atmetioon ?
located in the latracctinn area] of town [attraction address). their phone number futtraction phone] is there anything else can help you with
asr: yes, that was all necded. thank you very much! ! you welcome. have great day
need train reservations from [value place] to [valise place] sys: i have (valuc count] trains matching your request. there a specific day and time you would like to travel?
uir: i would like to leave on [value day) and arrive by (value time] syst there [value coun]] tains the
time you request would you like book
Human
booking. would like to
price, and departure time please.
syst there are [salue trains meeting your noeds with the lirst leaving at [salus time] and the list coount]
leaving at [value_tmm)] do want to book [salue_comn]] of these? ?
no hold off booking now can you help me find
called [sltraction !
syst yes a cinema located the [valuc_ part of town what information would you like on it
usr: thank you for using our system.
you welcomee have nice day... goodbye.
C.2 HYPERPARAMETERS
The specific hyperparameters for training HDNO on MultiWoz 2.0 and MultiWoz 2.1 are shown in Table 9 During training, we used Adam optimizer (Kingma and Ba 2015) for pretraining and stochastic gradient descent (SGD) for RL. To ease life, we explicitly label the hyperparameter if it is only for pretraining or RL.
D EXTRA BACKGROUND
D.1 REINFORCEMENT LEARNING AND POLICY GRADIENT METHOD
MDP (Bellman 1957) ) is discrete time stochastic control process that agent decides an action a, E A at each state 8, E S emitted from an environment via a probabilistic transition function p( 188, a:) +> [0, 1). The environment also gives reward r(st, at, 8141) + IR (abbreviated
18

---

Published as a conference paper at ICLR 2021
Table 8: The table shows some delexicalized system utterances generated by the baselines and HDNO on MultiWoz. 2.0.
Models
Delexicalized Dialogue
please help make reservation
sys: time would you like to
wer: does not matter, just so arrive by [value time)
HDNO (Our Model)
what is your destination? ?
to (restaurant name] and need the car type number please.
syst booking completed! booked car type [uxi__ype]] contact nimber [saxi_phone] there anything help you with
no that all thanks
sys: you welcome
nice day goodbye.
wett please help me make a taxi reservation
i can help with that what time do you need to leave or arrive by
jjust anrive by [salue_time]
WEZE
syst i would be happy to help with your request, but will need to know what time you ard looking for
usi: j want to to the Iresiaurant, and need the car type and contact number please.
have booked [tuxittpp]] you contact number [txi..hhone. there anything clse i can help you with today
usr: no that all. thanks
syst you are welcome have great day goodbye.
please help make
reservation
ayst have booked a [yaxi_lype] for you . lhe contact number is [axxi.phone]] ust: does not matter just so arrive by [saluc_imme]
have booked [saii type] you the contact number is [taxi phone].
LaRE
want
to [restaurant manc]
the car type
number please..
syst i have booked a [taxi type] for you. the contact number is [taxi phone).
war: no that all thanks
you welcome. goodbye
Hor: please help me make a taxi reservation syst what time would you like to leave?
doos not matter. just arrive by Ivaluc.time)
HDSA
syst where will you departing
your destination
aur: i want to go to the fresiaurant name) - and need the car type and contact number plcase
syst your tadi baoked. the car type is [taxi_type and the contact numher is saxi.phone). is there anything else
help you with?
no that all thanks sys: you are welcome. have a great day!
please help me make
resevvation
sys: help you
when you need leave
aar: does not matter. just so anive by [valuc time).
i can help you with that. where are you going
Human
(restaurant name]
need the cur type and contact number please.
sys: okay. have [taxi, type] for you with contact number [mxi phone). is there anything else you need today use: no that all. thanks
you welcome, have wonderful time
Table 9: Hyperparameters for training on MultiWoz 2.0 and MultiWoz 2.1.
Hyperparameters
MultiWoz
MultiWor
Description
mux ail
50
The muximum length of user's ulturances.
max doe les
50
50
The maximum length of system's utterances in response.
batch size (Pretrain)
32
32
The number of samples for each update during SL.
learning (Pretran)
le-3
Ie-3
The keamnng
SL.
grad clip (Pretrain)
1.0
The maximum total
of gradients allowed during Si.,
dropout (Pretrain)
The nadommess coefficiens dropout.
num epoch (Pretrain)
50
The number training epochs S1..
embed size
100
100
The size of word emheoding vector.
un ontseeLLsiee
300
300
The sine of hidden layer for ulterance enooder.
dec.ocill jaize
300
300
The sine of hiddes layer for notural language generator.
200
200
The
latent dialogue
(Proteain)
The regulatization coefficient of divergence dialogue policy during the contribution ranoonly generated utterances.
(Peetrain)
The coefficient
high level learning rate (RL)
9c-3
le:2
The leamnng rate for high-level policy during RL.
low-level learning mie (RL)
9--3
le-2
The leaming tate for low-level policy during RL.
num epoch (RL)
1
The number of training epochs for RL.
temperature (RL)
0.1
0.1
The temperature of Gibbs distribution for each word during RL. during KL.
gamma
0.99
The
gammauoll (RL)
The discoum_factor
generated of pradients allowed during RI.,
discriminator during RL.
godd (RI.)
0.85
0.95
The maximum total
zlpha (RI.)
1c-4
1o-2
The mathipler to adjust the balance between the diseriminator and the suceess ralc in the lotal reward during RL. Prettininn diserimitatee.
disc (Pretminn)
Ind
guide (Pretruin)
Using generated samples for pretraining discriminator. Applying Redieeggnee during pretraining. The frequescy updating high level policy.
neg (Pretrain)
high froq (KL)
low freg (RL)
1
The frequency for updating lew-level policy.
synchron (RL)
false
Gals
Applying mcynchmonuus updates between high-Jevel and tovel policy during RL.
disc2reward (KL)
tree
tue
Applying discrimanato as reward.
mccess2reward (RL)
Applying ncemalizaionn for generated from discriminator daring
reward.
all_oomalize
Applying 7-scose
19

---

Published as a conference paper at ICLR 2021
for simplicity) to measure the performance of an action at each time step. RL (Sutton and Barto 2018) is learning paradigm which aims to find an optimal policy x {{))(s)) + [0, to deal with MDP by maximizing the expectation
cumulative long-term rewards. Mathematically, it can be expressed as max= [ Ei= 74 rt ]. where ? E (0, 1) is discount factor. Different from value-based methods, the policy gradient method derives stochastic policy directly by optimizing a performance function w.r.t. parameters of the policy (Sutton and Barto 2018). However, since the performance function cannot be differentiated w.r.t parameters of the policy, the policy gradient is derived by a natural gradient such that V (6) Willliams 1992) is policy gradient method that evaluates ( at) by a return G2 Σ Y Tt- To deal with a continuous action space, To (a))))) can be represented
V In To ( (as)St) J. REINFORCE [ Q(s,, a,) Ve
Gaussian distribution SSttonn and Barto 2018), where the mean and scale are both parameterized with 0.
E DIALOGUE ACT REPRESENTED AS ONTOLOGY
Table 10: Dialogue act ontology.
Dialogue Act Type
inform / request / select / recommend / not found / request booking info / offer booking / inform booked/ / decline book- ing / welcome greet bye reqmore
In this section, we show the ontology for representing a handcrafted dialogue act (Budzianowski et al. (2188) in Table [0] The semantic meanings of latent dialogue acts analyzed in Section 5.2.3 the main part of paper are correlated to these dialogue act types. This is the reason why we conclude that learning latent dialogue acts can potentially substitute for handcrafted dialogue acts with ontology.
F EXTRA DISCUSSION
Discussion on Reinforcement Learning for Task-oriented Dialogue System: As we stated in Section 3 in the main part of paper, the primary issue of using reinforcement learning (RL) in task- oriented dialogue system is that the improvement on fulfilling user requests and comprehensibility of generated system utterances is not simple to be balanced. One reason could be that the reward is only set up to improve the success rate and during learning the aspect of comprehensibility may be easily ignored. This is the reason why we consider extra reward criteria in our work to mitigate this predicament. Another reason could be that the state space and action space are so large in an end-to-end (E2E) model so that learning a mapping between these two spaces becomes difficult, as stated in Section I in the main part of paper. To deal with it, we propose to decouple dialogue policy and natural language generator (NLG) of E2E model into two separate modules as those in the traditional architecture during learning, as well as model them with the option framework (Sutton eet al., 1999). As result, the complexity of mapping from context to system utterances is reduced from 7 to (L M) V + ML, where V is the vocabulary size, L < V is the space size of latent dialogue acts and M < V is the space size of encoded utterances.
20


Reviews:
[{"review_id": "ZOwNZ0CqOuV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes modeling the hierarchical structure between dialog policy and natural language generator with option network and train it with HRL. It also introduces a discriminator modeled with language models as an additional reward, which further improves the learning procedure's comprehensibility. Besides, this paper has demonstrated the interpretability of the latent dialog act via clustering methods. \n\nPros: \n1. This paper proposed formulating dialog policy as a high-level policy over dialog act and NLG as a low-level policy and train the word policy module using HRL. It is another move towards improving policy learning via HRL.\n2. The proposed model achieves SOTA performance in the policy optimization leaderboard on the MultiWOZ dataset.\n3. It is interesting to see that when formulated as latent factors, the distribution of dialog acts still shows a clustering pattern, which manifests the model's interpretability.\n4. This paper introduced an additional reward using a discriminator of language models. The ablation study shows that this reward can be useful (though not significant).\n\nCons:\n\nThe proposed HRL approach is a direct application of the option framework on a task-oriented dialog system. It is similar to the hierarchical structure in the open-domain dialog system. \n\nMinor issues:\n1. Introduction paragraph 2: When introducing end2end models, there is a lack of citations of recent e2e models including DAMD, SimpleTOD, SOLOIST, etc.\n2. Abstract Line 9:  o gur -> our", "extra_scores": {}, "extra_text": {}}, {"review_id": "OIrZP6msnCS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper looks the problem of lack of comprehensibility that arises when we use RL to train a E2E dialog system to maximise a given reward function. The paper proposes a  HRL/options framework based method to learn a dialog policy over learned latent dialog acts which can then guide the lower level NLG. This along with a regularization reward using language model the paper aims to improve comprehensibility. The show improved performance in MultiWoz dataset.\n\nStrengths\n1. The paper looks at a very relevant problem in dialog research. The ability to use RL along with SL and being able to use RL without compromising on comprehensibility.\n2. The paper proposes Options framework based method for using the hierarchical structure in dialog to learn the dialog policy and NLG in a hierarchical fashion.\n3. They provide a training that guarantees convergence to local maxima.\n4. They show improved total performance in MultiWoz dataset compared to recent, relevant baselines.\n\n\nWeakness/Comments/Questions\n1. The paper starts with the motivation of handling comprehensibility. More discussion is required on a) what are the reasons of loss in comprehensibility in this case (it is briefly mentioned in the intro) b) why their individual design choices and how they handles the different reasons c) some evaluation to verify this\n2. It would helpful to place it more clearly where the contribution of the paper lies in the related work. My understanding is that contribution of the paper is in exploring using options framework to goal-oriented dialog to handle the issue in question. HRL in general has been used previously for goal-oriented dialog, using language models to regularize RL models has been used and pertaining using SL is widely used. If there are particular differences in the above, it would nice to clearly state them and also say why the different choices and verify if the different choices are beneficial compared to the previous ones.\n3. Relevant to some of the above points. It is not clear to me why we cannot use HDSA+R or LARL + NLG + language model reward. Not necessarily these particular combinations. Some discussion on the current design choices and why making the proposed methods features to some of the other baselines is not a way to achieve some benefits is not the right way to do it. It's ok for the proposed method to be one particular way, but that discussion would be useful.\n4. Clarification on the task setting: Is it the case that the agent's current utterance does not decide what the next user utterance is? i,.e the agent is given the ground truth context every time and asked to predict the correct next utterance. That prediction does not affect the way in which the overall dialog goes? If that is the case, that should be made more explicit. In that case, the dialog policy learned is more of a contextual bandit setting. The complexity of learning options would be way different in the two different settings. Just wanted to understand it.\n5. In the 6.2.3 visualization of clusters, it would be very useful to have a visualization of clusters from some baselines on other ways of learning. To see if this is due to some new addition by the paper or is generally present.\n6. There are several parts to the method and there are I assume several differences in the architecture etc with baselines etc. It would be really useful to have an ablation study to disentangle which piece of the training or method is contributing to what and how in the performance measure. Otherwise for example it is not clear to me if the improvement in Blue compared to LaRL comes from the extra reward using the language model or from the options framework. What happens when the extra reward for using the language model is added to LaRL (that might be tough if you have to modify others code). what part of the performance is coming from pretraining (especially if using VAE type is novel, then quantifying that is important with and without VAE type SL), etc.\n\nQuestions to authors\n1. It would be great if you could respond to some of the above comments. Thanks.\n\nMinor\n1. Typo in the abstract: *In our work, we\n2. Move to E2E system can be motivated a bit more (allows end-user feedback to be passed through all modules easily and don't have to worry about how a change in one module affects all other modules explicitly etc)\n3. Might be useful to define what is exactly meant by 'comprehensibility'\n4. The intro says \"the goal is absolutely clear\". Not sure if that is really true. For example, it's not clear what the correct reward function to provide to the RL agent is. As the paper points out, the success rate alone is not enough. The dialog needs to be polite, follow natural language, short, etc which are hard to automatically measure. Human evaluation is costly and could also have bias like the paper points out. Just trying to say that automatic evaluation of dialog systems is a hard problem. \n5. I am just curious why you use GRU for the encoder and LSTM for the decoder?\n\nAfter author response\nThe authors have responded to most of my questions/concerns satisfactorily.\nChanging my score from 4 to 6", "extra_scores": {}, "extra_text": {}}, {"review_id": "NhXcWZ-3hkb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n=========\nAuthors applied reinforcement learning framework to the problem of task-oriented dialog. In particular, they used the option framework to represent the connection between the dialog policy and the natural language generation. Theoretically, they showed that synchronized updates to the low-level and high-level policy may never converge, yet asynchronized updates guarantees convergence. Authors also used a discriminator reward signal to cope with sparse reward (dialog success rate) and better representation of the human evaluation.\n\nEmpirical results on MultiWoz 2 and 2.1 shows improvement over other state-of-the-art techniques.\n\nSummary:\n+ Appealing theoretical contributions\n+ Empirical results are encouraging\n+ The use of discriminator for reward shaping in addition to task success rate is interesting \n- Writing and explanation can be improved. There are few places (see details) that authors have assumptions in mind but do not provide those assumptions until later. Would be great to state them upfront to avoid confusion.\n- The original option framework assumes given options. Given the ending of the paper, I interpreted that the set of dialog acts (i.e. options) are learnt automatically but I could not find this to be communicated explicitly.\n\nQuestions:\nDid you look into the quality of dialogs specifically? In some of our recent experiments we found out that those automatic metrics do not necessarily correspond to great user experience.\n\nDetails\n=========\nAbstract: In o gur work => In our work\nP5: \"oracle dialogue state\": What is the oracle dialog state and how is it calculated? The world oracle conveys the meaning of being absolute truth which sounds a bit unexpected. The same comment is applicable to oracle database (DB) results. Is it possible in your system you queried the DB with wrong parameter? If yes, do you still name the output of the DB as oracle results? => Ah. you explained this in page 6, in Task Description. I highly recommend bringing this assumption earlier to avoid readers confusion.\nP5: Section 4.4: I am still eager to know how you select your dialog actions. Would be great to tell your reader earlier. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "FyZ6NxC2cC4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to model task-oriented dialogue system using hierarchical reinforcement learning between the actions policy and natural language generation system. The utterances are encoded with GRU cells, the action policy is one-layer linear model and outputs the mean and variance of a multivariate Gaussian distribution, and the NLG is an LSTM decoder.\n\nThe paper is well written, easy to follow, and adequately motivated. \n\nPros:\n\nHierarchical RL formulation with options for joint action policy and natural language generation\n\nThe paper proposes asynchronous updates between dialogue policy and NLG to theoretically guarantee their convergence to a local maximizer. \n\nIt also proposes using a language models as a discriminator model for reward assignment to further improve the comprehensibility of generated responses.\n\nPre-training of task-oriented dialog models with  variational bayes similar to VHRED (Serban et al., 2017).\n\nExperiments show significant performance improvement over the baselines with both automatic and human evaluations. \n\nAlso, the results show that continuous representation of the action policy in HDNO performs better than the discrete representation in the LaRL, which is very interesting and informative.\n\nCons:\nThe use of an oracle dialogue state and an oracle database search result. This is a major drawback of this work. Recent work in this space are now considering imperfect dialogue state and the corresponding database search results. There is also an increasing need to consider non-trivial database interactions such as booking for more practical applications. \n\nThe evaluation/comparison using updated official evaluator may be missing for some of the more recent work e.g..,\n\nGPT-2 (Budzianowski and Vulic 2019)\nStructured Fusion (Mehri, Srinivasan, and Eskenazi 2019) SOLOIST (Peng et al. 2020)\nDSTC8 Track 1 Winner (Ham et al. 2020)\nSimpleTOD (Hosseini-Asl et al. 2020)\n\n\nQuestions:\n\nWhat do you mean by “Distinguished from a conventional modular system, we additionally give a context to NLG to satisfy the option framework.”? You mean the dialogue context is equivalent to the state space in HDNO formulation?\n\nHow are the oracle dialogue state and an oracle database search results encoded?\n\nWhat kind of model is used for the language model discriminator? LSTM?\n\nHow did you handle the action policy toward the database for booking, which can cause database mutation? Are those ignored?\n\nAre the results in Table 5 based on the updated official evaluator similar to Table 1? If not, then the comparison in Table 5 is not apples to apples.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the title states (and reads somewhat like an openreview review title), the authors apply the options framework from the RL community to perform hierarchical RL where the option is the dialogue act and the subproblem is the NLG component in task-oriented dialogue (TOD) policy learning. The two technical contributions (beyond the conceptual connection above) is showing that asynchronous updates between the hierarchy levels guarantees convergence and language-model based discriminator to densify the reward structure. Empirical results are solid improvements over recent SoTA findings. 

== Pros ==
+ This is a conceptually appealing application of RL to TOD and they authors had to make additional modifications to get it to work — which will help other researchers in this space.
+ There are both theoretical and empirical contributions. The theoretical contributions are also insightful and not superfluous to the problem being studied.
+ Using a language-model based discriminator for reward shaping isn’t completely new (although I haven’t seen in this setting and stated exactly the same), but is interesting and effective.

== Cons == 
+ The writing could use significant work; while the reviewers/rebuttal cleared up many issues, I actually didn’t appreciate the value of this paper in my first read due to the writing (even if the motivation, etc. is sufficiently clear).
+ Human evaluation is treated as somewhat of an afterthought and there isn’t a deep dive into error analysis of the results. The visualization is a good first step, but there isn’t really a when/why this method works better than others, which is important for a problem where evaluation isn’t conclusive in the best cases. This is also significant since the authors claim ‘comprehensibility’.

Evaluating along the requested dimensions:

- Quality: The conceptual and theoretical contributions are both of high-quality. This is a promising approach to TOD and the authors additions (e.g., async optimization, LM reward shaping) are good examples of applied research. The empirical results are sufficient to above average, but not as strong (although this is partially an artifact of TOD evaluation) 
- Clarity: The motivation is good, but the paper could use some work in writing. Some examples include (1) stating precisely how the option choices are derived (latent variables), (2) mapping out notation in something like a preliminaries section, (3) sketch of proofs in the main body for continuity. If the reader is familiar with the closest cited work, it is a bit easier, but I think some effort in making the paper more self-contained would increase its impact.
- Originality: Options in HRL is widely known, but applying it to TOD is novel to the best of my (and the reviewers) knowledge. I think many could have come up with the basic idea, but it took some effort to get it to work.
- Significance: This is a widely studied problem and the approach is fairly convincing. I don’t think it will be ‘disruptive’ or cross-pollinate to other application areas, but will almost certainly be cited within the conversational agent community.

In summary, the reviewers like this paper a bit more than myself personally — I think it is borderline with a preference to accept while the reviews are a more confident accept. However, the reviewers are also experienced experts in this area. I also do think that the authors handled concerns well in the rebuttal stages and addressed my more pressing concerns. I would encourage the authors to improve the writing if accepted, but I would prefer to accept this if possible.

Decision: Accept (Poster)