A Mixture-of-Expert Approach to RL-based Dialogue Management
Authors: Yinlam Chow
Azamat Tulepbergenov
Ofir Nachum
Dhawal Gupta
Moonkyung Ryu
Mohammad Ghavamzadeh
Craig Boutilier
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement learning

Abstract: Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge. We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for a medium-size vocabulary. As a result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained LM. To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts) capable of generating utterances corresponding to a particular attribute or personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. Our MoE approach provides greater flexibility to generate sensible utterances with different intents and allows RL to focus on conversational-level DM. We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both in terms of the diversity and sensibility of the generated utterances and the overall DM performance. 

Published as a conference paper at ICLR 2023
A MIXTURE-OF-EXPERT APPROACH TO -ASED DIALOGUE MANAGEMENT
Yinlam Chow
Aza Tulepbergenoy
Ofir Nachum
Dhawal Gupta
Moonkyung Ryu
Mohammad Ghavamzadeh
Craig Boutilier
Google Research
(yinlamchow, atulep, ofirnachum, dhawgupta, mkryu, ghavamza, cboutilier) c @google.com
ABSTRACT
Despite recent advancements in language models (LMs), their application to di- alogue management (DM) problems and ability to carry on rich conversations remain a challenge. We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maxi- mizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for medium-size vocabulary. As result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained LM. To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) number of specialized LMs (or experts) capable of generating utterances corresponding to a particular attribute personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. Our MoE approach provides greater flexibility to generate sensible utterances with different intents and allows RL to focus on conversational-leve DM. We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both terms of the diversity and sensibility the generated utterances and the overall DM performance.
INTRODUCTION
With the tremendous advancements natural language understanding and generation, increasing attention has been directed to construct intelligent dialogue agents that can carry out engaging conver- sations with users. Such interactions can be open-ended, contain different topics, and often involve an underlying task, such as negotiation, information exchange, and recommendation. Therefore, to satisfy the user, good dialogue agent should not only generate natural responses, but also be capable of pursuing the task's 's objectives and adapting to the user's feedback on-the-fly.
standard solution to train the dialogue agent using behavioral cloning, where the agent is language model (LM) that imitates the utterances in the training set (Gašié et al., 201 Fatemi et al., 2016). By leveraging deep neural networks, e.g., RNNs (Sutskever et al., 2014) and Transform- ers (Vaswani et al., 2017), a LM encodes the conversation to low-dimensional dialogue state and predicts an utterance, but steering such generation for particular purposes remains an open question. Several works studied ways to fine-tune a LM to generate texts with specific contexts (Ziegler et al., 2019; Ficler and Goldberg, 2017). Other results learned single steerable LM that capable of generating utterances for multiple specific intents (Gu et al., 2017: Chen et al., 2018: Subramani et al., 2019; Dathathri et al., 2019). While these LMs produce fluent and relevant responses, it is unclear how to control them systematically pursue goals during multi-turn dialogue conversations.
Another popular approach is to view dialogue management (DM) as a control problem and use reinforcement learning (RL) to optimize the agent's 's policy (which is often a LM itself). Using RI. for dialogue systems has long history. Earlier work relies specific, hand-crafted semantic states (Levin and Pieraccini, 1997; Singh et al., 2002; Walker, 2000) or partially observable belief' states (Williams and Young, 2007; Young et al.. 2010), in which the agent chooses the best hand- crafted dialogue act each turn, with the goal of either satisfying the user (Shah et al., 2018),

---

Published as a conference paper at ICLR 2023
completing the task (Shi and Yu, 2018), or responding to the user's query (Serban et al., 2017a), However, the application of these approaches is limited to problems whose action space can be captured by hand-crafted representations, and they cannot handle complex conversations. On the other hand, more recent approaches use deep learning to extract semantic representations from conversation histories, treat these representations as dialogue belief states, and apply RL learn a word-level generative DM agent (Jaques et al., 2019; Li et al., 2016; 2017; Shin et al., 2020). However, since there are innumerable possibilities of language utterances, and thus, the action space of the RL problem is extremely large, the agent often performs planning poorly and generates incomprehensible utterances (Zhao et al., 2019). Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, e.g.. ease of answering (Li al., 2016), fluency (Li et al.. 2017: 2019), and diversity ( (Yarats and Lewis, 2018), and the goal, e.g., conversation length (Zhou et al., 2020), user's sentiment (Hancock et al., 2019), and task completion (Verma et al., 2022: Jang et al., 2021). Moreover, defining the reward as weighted combination of these metrics is not ideal, since the hand-picked weights do not often reflect the underlying success criteria.
To address the above issues related to using RL in dialogue management (DM) systems, we propose an RL-based DM agent using a novel mixture of expert (MoE) approach. Our MoË approach is based on a mixture of expert language model (MoE-LM), which consists of three main components: 1) a LM (a probabilistic encoder and a decoder) capable of learning diverse semantics for conversation histories, and result generating diverse utterances, which we refer to as the primitive LM LMo, 2) number of specialized LMs (or experts), { { that each is constructed using the latent space learned by LMg, but has been trained such that it is capable of generating utterances corresponding to certain intent or personality, and 3) an RL-based dialogue manager (DM) that at each turn, given the latent state shared by the experts { J and the utterance action(s) they suggest, chooses one among them for the agent to execute. Our MoE-LM can be seen as a special case of hierarchical LMs ge. Serban et al. 2017a; Zhao et al. 2019; Saleh et al. 2020), but it is different than them because it learns both the LMs (experts) and the DM. Moreover, the DM in MoE-LM is a policy conditioned on both the latent state and the actions suggested by the experts, and not just the state as it is common in hierarchical RL. The primitive LM (LM)) plays important role in this model because it learns diverse semantics for conversation histories and allows the agent to generate a wide variety of utterances. This diversity is also shared with the specialized LMs (experts) and gives them flexibility in generating their (more) specialized utterances. Another important feature of MoE-LM is its modularity that facilitates adding and removing specialized LMs (experts). Moreover, this hierarchical architecture allows us to solve an RL problem with much smaller state and action spaces, which is quite important in the quality of the learned policy. Finally, since the candidate utterances are generated by experts with different intents, instead of combining all agent-user signals into a single RL reward, our DM agent can focus on optimizing the specific goal of the conversation task.
We start the paper with a brief introduction of LMs and the use of Markov decision processes (MDPs) in modeling dialogue management problems in Section 2. We then describe the overall architecture of our MoE-LM in Section 3, followed by the detailed implementation of each of its three main components (described in the above paragraph) in Sections 4 to 6. Finally, in Section 7, we demonstrate the effectiveness of our MoE-LM in open-domain dialogues, in terms of both its ability to generate diverse and sensible utterances and its overall DM performance.
2 PRELIMINARIES
Language Models (LMs) In this work, we employ seq2seq LMs to generate the next utterances in a dialogue. We assume access to a dataset of the form D = {(x(), y())) where each 4-1 X = (i) is a L-turn conversation history X [-) and Y is its next utterance. We denote by Nx, an upper-bound on the length (number of tokens) of each utterance X, in X. The role of a LM is predict the probability of the next utterance Y. consisting of N tokens, conditioned on the conversation history X, i.e., p(Y = [ x). In the transformer architecture (?), the LM first encodes the conversation history X using an encoder 4 to a (L x Nee-te sequence of embeddings {21,0..... -x 1 1-1 where each 2 is vector in the latent space. For notational convenience, we concatenate these embeddings into single embedding 2 E Z C R2 and denote the overall dimension of the latent space d. In the RNN architecture (Serban et al., 2016), the
[o]
LM's encoder & directly maps the conversation history X to latent state E 2 C IRd, In both architectures, the next utterance Ý
( J. is sampled boken-by-tolee from the decoder V,
If the actual utterance X, has fewer tokens than Nx, it will he padded by a specific token and masked.
2

---

Published as a conference paper at ICLR 2023
".oieee connent)
Go
2
%
Inqusstive
Howe help?
What you like ale your friedd
LUG
Do you krons foom my day secee
0:54
91
Explorative
sowuld live to bione
X
S
y
f
Optinistio
Nisce gainng ouniis, thatks facorte
neighborisee
lo soft and reso to chill
0.00
0.0
9m
mm
Pm
Why
allve?
Anery
You americans ieaan this resssed
you kiddinn avd
0.18
Step 1
Step
Step 3
Bab can hwde? Werne wolk? bis chillius.
Topuc
deterministic transition
stochastic transition
Figure I: (Left) MoE-LM Architecture. (Right) Sample utterance workflow generated by an MoE-LM trained with Reddit data. Step I: & encodes conversation history. Step 2: V// o G. Vi, generate candidate bot utterances. Step μ selects the bot response by Q-score ranking & post-processing.
i... 8 Y~ 1.. (-12)
=
1n) where To is a fixed initial (start-of-sentence)) L (On ,
token (Chien and Kuo, 2019), and the latent state is denoted as 2 = (XX).
Markov Decision Processes (MDPs) have been used to model dialogue management problems in a variety of settings (Li et al., 2016; Asadi and Williams, 2016; Jaques et al., 2019). In such MDPs, denoted by M = (S, A, P, 80: 7), the state space S represents the tokenized conversation history and the initial state Sy E S is the initial user's query. The action space A is also the tokenized language space with each action a A being the agent's next utterance (which is a fixed-length, Nx. sequence of tokens). The transition kernel P models the user's response to the action taken by the agent (bot). Finally, the reward function r measures the user's satisfaction. In these MDPs, we can think of the entire LM as a policy that maps conversation histories to next utterances, and solve them by finding a policy x with maximum expected discounted return, i... n' E arg max, Jx == E 7'r; P. 80: r.. Note that the size of the tokenized state and action spaces grow exponentially with the size of the vocabulary. This makes it intractable solve the MDP even for a medium-sizze vocabulary. As a result, it would quite desirable to develop a novel MDP paradigm that is more amendable to RL-based DM systems.
MIXTURE OF EXPERTS (MoE) LANGUAGE MODEL
We start by explaining how MoE language model (MoE-LM) can enrich the bot's utterances and improve the overall performance of the DM. While our approach is applicable to any DM system, we use open-domain dialogue (Sankar et al., 2019) as a running example to show how MoE-LLMbasedd agents can improve user satisfaction measured by an improvement on sentiment or engagement. Intuitively good DM agent should possess different behaviors (e.g., inquisitive, explorative, relevant, soothing, empathetic, complimentary, provoking) and swiftly decide which intent to use to pivot a conversation, build rapport, pique the user's 's interests, improve their mood, etc. To achieve this goal, require the LM have a language representation (primitive discovery) that captures different semantics, in order to encode different conversations and avoid generating dull and repetitive responses. We also need a machinery (expert construction) to embed different intents into sub-models of this LM, so that they can behave accordingly when prompted, and respond efficiently. Finally, with various candidate utterances available, the DM module of this LM should understand the current level of user satisfaction and determine which response is the most appropriate. Motivated by these observations, we construct our MoE-LM in three steps shown in Figure 1. We give the main idea behind each step here and leave their detailed descriptions to Sections 4, 5, and
Step 1: Primitive Discovery. We first employ the dataset D, introduced in Section 2, and learn a
language model LM, = ( Go, Y consisting of stochastic encoder (i.e., encoder & and latent space distribution Go that maps the encoded conversation into latent distribution). and decoder V. The stochastic encoder ( Go) comprises an encoder & that maps tokenized conversation histories X to a latent space& 2 C T2d i.e., = (xX) E 2, which is then used to construct a parameterized d-
Note that we use Y as the next utterance in the dataset and & the one predicted by the LMM
3

---

Published as a conference paper at ICLR 2023
dimensional Gaussian distribution Gu(2'|2) = (N(((())) ( over R The decoder predicts the next utterance Yo (token-by-token) conditioned on the point sampled from the latent distribution, i... (Yooz')), z' ~ Go(-Iz). We denote by LMo (Y := (1
( [2|2),, the primitive and learn it using loss function that in addition to predicting the next utterance accurately, encourages diversity and generalization in the learned latent space 2 (see Eq. and Fig. 2). As we will explain in Section 4, our loss function is inspired by those in prior work, and more specifically by the one in OPAL (Ajay et al., 2020), i.c., an unsupervised learning method for discovering primitive skills in trajectories that are used by some downstream RL tasks.
Step 2: Expert Construction. Given the latent space 2, encoder (Q,9o), and decoder V learned in Step 1, we now learn latent distributions {Gi each defined G=(2'(z)
N (Ju(2), ()(())))(()))))))) (x)) Intuitively, each 9i corresponds to an attribute, e.g., an intent or a personality (in case of chatbot) and generates samples in specific parts of the latent space Z. This results in having m LMs, {LM, ym LM, = (L, G. V), each of them corresponds to a specialized version of the original LM, LMo, and serves as an expert in our MoE-LM. Upon receiving conversation history
X, each expert LM, generates a candidate (or more) for the next utterance Y, in certain parts of the language space that are compatible with its attribute (personality). As we will explain in Section each Gi is learned using a loss function that encourages its corresponding LM, LM, to generate utterances consistent with its attribute (see Eq. 2).
Step 3: Dialogue Manager (DM). The dialogue manager, denoted by μ. takes as input the encoded conversation history x = & (X) and the candidate action utterances generated by the experts {Y: y and selects one of them as the action for the bot to execute, i.e., y ~ su(- 2, (p)))))) We will describe how DM is trained using reinforcement learning (RL) in Section 6.
4 PRIMITIVE DISCOVERY IN MoE-LM
Motivated by literature in the reinforcement and imitation learning fields (Ajay et al., 2020), we propose to learn the primitive LM, LMo, in our MoE-LM by solving the following KL-constrained optimization problem that aims at capturing diverse semantics:
min B, (x)) log (Y]k'),, s.t. ( (0,50. ).P
Go(2'l=))] S eKL, (I)
where E is the empirical expectation over (X, Y) in the dataset D, p is a distribution over the latent space conditioned on the encoded conversation history 2 and the target utterance Y, and 6KI. is a posi- tive real-valued threshold. Using (1), we learn LMo = (4, Go. v by maximizing the log-likelihood, while enforcing consistency between the latent variable z' predicted by Go(-I2) and p(-1z, Y) via the KL constraint. The distribution p(-1z, Y) is Gaussian .V (Hp(z, (Y)),, oZ (Y)) in which 4 p is pre-trained encoder for the target utterance Y. and mcan u.(.,)) and variance (:()--) are trainable models. One reason for using a separate encoder T for the target utterance Y is to avoid overfitting & (i.e., to avoid having fack-propagttion gradient of & with Y as input).
Connection to VAE-like objectives In practice, we implement the KI. constraint in (1) as penalty weighted by an appropriately chosen coefficient. Thus, one may interpret the objective in (1) as variation of 3- VAE (Burgess et al., 2018). Due to the connection to VAEs, one may draw similarities between our method and existing dialogue approaches such as VHRED (Serban et al., 201 7b) and VHCR (Park et al., 2018). However, we emphasize that there are key differences, and these may be explained by first understanding how the objective in (1) encourages diversity, which is key to good primitive learning. Namely, it is important that primitive discovery learns an encoder-decoder t v which can be modulated by the choice of 2; i.e., changing z' while fixing X should lead different distributions over generated utterances. The objective in (1) encourages this diversity by conditioning the latent variable z' on both the target utterance Y and = = (()), i.e., z' ~ p(-12, Y). In contrast, the KI. constraint is used make sure that the stochastic encoder (o =) of primitive LM is not too varied for different Y, and thus has a limiting effect on diversity. For example, in the extreme when EKI. = 0 (or, 3 - oo when used as regularizer) there will be no specialization of the latent space for different Y. Although / - Do an extreme case, degenerate behavior can also happen when 3 = 1, i.e.. in the traditional variational loss used by VHRED and VHCR. Specifically, it is well-known that the traditional VAE loss is an upper bound on the negative log- likellhooood of the data under stochastic encoder-decoder parameterization. Thus if the data can be modeled by a single
In practice, we can use both latent states as the input to the decoder model v (Voll'; 2).
4

---

Published as a conference paper at ICLR 2023
LM then VAE-optimal decoder V/ can simply ignore Go. leading degenerated latent space as observed in previous work (Park et al., 2018). This is precisely the reason that, in our approach, we weaken the KL constraint (egı. > 0 or, equivalentyy, 3 < 1). This enables our approach to more reliably guarantee that unique z' represents each distinct conversation pair (X, Y). thus capturing diverse semantic modalities and enabling easier downstream specialization.
In the mathematical results below, we formalize the claim above. namely, that the log- -igeliill objective in (1) leads to a learned Φ 1/ that can easily recover any arbitrary desired LM by specializing the latent space G. We begin with definition that characterizes the coverage of an arbitrary LM on the conditional conversation data distribution PD
Definition 1. LMp, is E-common LM of data D if Ep (LV (Y (X)))
Leveraging Theorem 4.1 in Ajay et al. (2020), we now present the theoretical result characterizing the representational power of our primitive encoder-decoder pair (Φ, V) data D.
Lemma 1. Let (Ø, p, V) be the solution to (I) with Bxt- V (Y]==)) = Then
there exists LM: = (b, V) such that ED (TV (MMMM.((()))))MMMMM))))) [|][))))))))())]]] v (e + H), where
((21|2) = [p(='1z, Y),, and P/ = Ep (log [X] is constant depending on D.
The result above shows that, as long as LMp. is &-common in DD then there exists a specialization of the latent space G that, when paired with 4, V, can approximately recover LMp.g. The quality of the approximation is a function of e how well the objective in (1) was optimized - and E. In practice, to construct the primitive by replacing G with Go. i.e., LMgp (Φ, Go. VD), because Go(='Iz) can be viewed distillation of p(=' 22, Y). This theoretical result also motivates the next section, where we explain our algorithm's 's "Step 2: Expert Construction". Specifically, we show how to use the trained encoder-decoder pair Φ. Y to learn a spectrum of different specialized experts parameterized by different latent distributions Gi.
EXPERT CONSTRUCTION WITH PLUG-AND-PLAY LANGUAGE MODELS
To complete the MoE framework one needs to systematically create a gamut of different experts IM; Vi E (1....,,) mm), with each generating candidate utterances of different intents. By viewing each expert as distribution of particular behaviors in conversation data D, we leverage the results of Section 4 and Lemma and adopt a universal encoder-decoder ( V among all the experts. Therefore, each expert is only parameterized by an arbitrary d-dimensional latent distribution (c.g., Gaussian), and samples certain regions of the latent space Z. Following the terminology from Dathathri et al. (2019), these experts can all be catagorized as plug-and- lag--nnd-play language models (PPLMs). Creating experts is handy because it only requires learning new latent distributions, while switching between experts amounts sampling different distribution.
Denote by (X, Y) E R a real-valued label that characterizes the intent of expert i E (1..... m.. e.g., determined by an off-the-shelf sentiment classifier. We train the latent distribution (2) of expert i by solving the optimization problem
min 9,
E. (. 2/) (1l-:///--))1-///1000 xx1-(x(X,)))
Unlike the weighted maximum likelihood approach considered in Dathathri et al. (2019), which assigns weight l, to training samples that correspond to expert we propose to learn each expert via maximddaaximization and treat É, as a reward signal w.r.t. expert to be maximized. Interestingly, this approach is also linked to reinforcement learning (RL), in which both the "state" and "action" spaces are the latent space Z, and the "policy" is the latent distribution G. The main benefit of our approach is that it does not require the target utterance from data D and is thus less vulnerable to data-imbalance issues in D on certain intents. Notice from (2) that the reward-mant -arardmmximmizttinn problem is myopic, i.c., the above RL problem has discounting factor of O. The main motivation is that, unlike dialogue management that a sequential decision- maainn problem, here we want each expert to possess particular behaviors, and this can readily be done via greedy maximization. Long-term dialogue optimization will be handled by the dialogue manager rather than these experts.
For example in the case of a Gaussian Gi, we use the standard REINFORCE (Sutton et al., 1999) algorithm to learn the model parameters (Hi, of) of 9; according to
(M4 . +- (uc,,rr) + a Ex.- (4
((naa logPa, (z'1=)), e (m.......
where > 0 is the learning rate. To reduce the variance of these estimates, we also adopt the baseline reduction technique (Greensmith et al., 2004) in policy gradient, which replaces e (X, Y)
5

---

Published as a conference paper at ICLR 2023
with 2,(X, := (, ( (X,( Y) Following arguments from Lemma I and Lemma 4.0.1 in Ajay et al. (2020), in the following we quantify the sub-optimality of expert LM,-
objective as (LM) == By 1x) (X,,) Suppose an optimal LM for this objective LM,g arg maxIx L; (LM) is f-common in D. Moreover, let 9 be in the arg min of (2). Then with expert LM; = (Ø, 97, V) and (c. H) from Lemma 1, we
Corollary 1. Denote the i-th
have |c; (IM)) - Li (IM S 2| - (5 + + /
+ H)).
While it may be obvious that optimizing G, w.r.t. (2) encourages expert LM; to capture the behaviors encouraged by 4, this corollary has two further implications: (i) Since the sub-optimality of LM; compared to the oracle L., is bounded by the quantity defined in Lemma 1, it justifies using the primitive (v) 4). which optimizes for expert construction; (ii) Sub-optimality further depends on Ç. quantifying how well LM,, is represented in the original dataset D.
6 REINFORCEMENT LEARNING FOR MOE-LM DIALOGUE MANAGER
We now describe the dialogue manager (DM) of our MoE-LM and propose RL algorithms to train it. As mentioned in Section 3, the DM is policy y that takes the encoded conversation history
= (XX) and the m + 1 candidate action utterances generated by the experts {%) and stochastically selects one of them to execute, i.e., Y ~ |
2, (p)) Note that each expert i E {0,. m) is a L.M, LM,, that acts as a policy Tee (-(x and maps each conversation history X
p((
to an utterance Y; With this architecture we address the large size of state and action spaces in the original MDP that grows exponentially with the size of the vocabulary. As described in Section the state and action spaces of the original MDP are the tokenized conversation history and the tokenized language space, respectively, while here the DM should choose among m 1 actions (which is much smaller and finite action space) given the latent space (which is a continuous state space) of encoded conversations, It is important to note that our MoE-LM is different than other hierarchical architectures (Kulkarni et al., 2016) in which the decision at the high-level is to choose a low-level controller only based on the current state of the system. In MoE-LM, the DM observes both the current state and the actions suggested by the experts and then chooses one among them.
We consider two RL settings to solve this specialized MDP. The first one is offline RL, in which the pol- icy must be learned from the collected conversations D without further (online) interactions. Offline RL requires optimizing a policy, while minimizing the deviation from the behavior policy to avoid errors due to data co-variate shift. Among numerous offline RL algorithms (Kumar et al., 2020; Carta al., 2021; Jaques et al., 2019), one effective algorithm to learn the DM policy μ is IQL (Kostrikov et al., 2021). Given conversation data D, IQL first computes the critic functions (Qg- (z, a), Va (=)) via solving ming. (. ((V(2,))
Qz(z, - ( (Qo a) -V)((),)) where: is the encoded conversation, a is the bot utterance, 24 is the next encoded conversation, is the conversation-leva reward, > is a tunable weight, and T.5 is the expectile regression operator (Koenker and Hallock, 2001) of estimating the top-r expectile statistics of the Q-function random variable (approximated by the value function Vo), and then extracts the DM policy μ via greedification
over the finite set of MoE candidate utterances: u(a 2, (2.))m)) = arg max ( ymo Qa- (z, a).
Intuitively, IQL leverages the generalization capacity of critic functions to estimate the value of the best action without directly querying the values w.r.t. unseen actions. This makes it less conservative than most offline RL methods that either constrain the policy's s actions to be in-distribution or solve a behavior-reuularized policy optimization problem.
The second RL setting for learning the DM policy j is via model-based RL (MBRL) (Shah et al., 2018; Wei et al., 2018). While this paradigm can be applied to any online/offline RL algorithms we demonstrate it with the simple DQN (Mnih et al., 2013). Here we first learn user utterance model P Puser [X+| a) := (== [ Wuserr(2 [x+] via maximum likeli- hood, then generate data DMB. whose next-state 3, encodes the next conversation generated from roll-outs and the corresponding candidate actions, solve the Bellman error minimization: ming E(sarr; JE + 7Qquee (5+, arg max,, e( Qo(E,, a))) - Qa(s, a))², and obtain the DM policy y via the aforementioned greedification step. The benefit of MBRL. over offline RL. is two-fold. First, one can casily obtain a high-fidelity user utterance model (Peng et al., 2020) by simply fine-tuning large LM (e.g.. GPT-3 (Floridi and Chiriatti, 2020)). Second, with sufficient
For simplicity, assume that each expert generates only single candidate utterance at each step. It would he straightforward extend this to multiple (and even a varying number of) candidate utterances.

---

Published as a conference paper at ICLR 2023
dialogue roll-outs that captures many different scenarios, MBRL generally can be more data-efficient and less prone to distributional shifts than offline RL.
7 EXPERIMENTS
We evaluate our MoE-approate on two open-domain benchmarks that are common within the RL- based dialogue management literature (Jaques al., 2019). The first one is the Comell Movie corpus (Danescu- Nieeee--neeeee--MMii
and Lee, 2011), which consists of conversations between speakers in different movie lines and has a median conversation length of 3 utterances. The second is the Reddit Casual (Ghandeharioun et al., 2019) conversations, which is subset of the Reddit corpus that only contains casual conversations on various topics of at least 3 turns and a median of 7 utterances.
We conduct several experiments to test the efficacy of different parts in the MoE-LM,, namely (i) the predictive power and diversity of the primitive, (ii) the quality of experts, and (iii) the overall DM performance. For each metric, we report mean + standard error over 100 conversations sampled from the evaluation set. We also ran an ablation study on 4 transformer-ban based MoE-LMs, namely MoE-1, MoE-2, MoE-3- MoE-4, to understand how performance is affected by different model architectures, language encoders, and latent generators. MoE-1 and MoE-2 use simpler architecture, while MoE-3 and MoE-4 use the same encoder architecture as BERT (Devlin et al., 2018), 8). MoE-1 uses much smaller latent distribution models } than MoE-2; MoE-3 uses the pre-trained BERT encoder, while MoE-4 trains that from scratch. Details of these models can be found in Appendix B.3.
EXP 1: Comparing Primitive Models We compare the quality of latent representations learned by the 4 MoE-LMs (via optimizing Eq. 1) and 2 baselines (standard Transformer (Vaswani et al., 2017) and VHRED 5 (Serban et al., 2b)). To assess their quality, for each test conversation we generated 25 utterances and reported the following 3 metrics: (i) Diversity: The 1- Sparsity (Hurley and Rickard, 2009) of the singular values of the embedded utterances, i.c., Diversity([Xf3) := 1 1- ||SD||i / [/VD[|2//V
I E [0, 1). where SVD == SVD( SF (P; (21)) and &SE is a pre-trained sentence encoder (e.g., a USE (Cer et al., 2018)); (ii) Dist- 2, 3} (Li et al., 2015): Ratio of unique {1, 2, 3 --rgram in the generated utterances; (iii) Perplexity: The perplexity score of the utterance w.r.t. GPT-2 LM, which is more correlated to human's 's judgement on semantic fluency (Pang et al., 2020). These metries measure both accuracy and semantic diversity. Qualitatively, we also measure fluency and diversity of LMs using human ratings (see Appendix B.8 for details). The results of the above experiments are reported in Table and 6 (Appendix A. and sample utterances of
generated by these LMs can be found in Appendix A.3. Human evaluation diversity/fluees different LMS are given in Table 4. In comparisons with the baselines (Transformer and VHRED), generally (i) transformer-based LMs out-perform VHRED due to their attention mechanism that explicitly encodes sequential semantic information, and (ii) the MoE-LMs achieve way better diversity without sacrificing much on accuracy (i.e., the perplexity scores are still quite low). Qualitatively, the sample utterances generated the Transformer are closer to the targets than that by MoB-2 and MoE-4, likely because Transformer tends to memorize the corpus (Kharitonov et al., 2021). Contrarily, MoE-LMs generate utterances that have similar contexts with targets but paraphrased or similar structures but different contexts, demonstrating their generalizability, Human evaluations also show that MoE-2 and MoE-4 generate more diverse utterances while retaining sufficient semantic fluency.
Among different MoE-LMs, MoE-2 and MoE-4 have the best performances, particularly MoE-4 has better diversity while MoE-2 has lower perplexity. This corroborates with our hypotheses that (i) jointly training the encoder and decoder with Eq. 1 seems necessary to encourage semantic diversity (as opposed to using pre-trained BERT encoder, which maximizes likelihood), (ii) sufficient representation power is necessary for Gu to match the posterior distribution p in order to capture different semantics in D. In Fig. 2a and 2b, we visualize the latent space of 400 conversation data samples for both Transformer and MoE-4. The latent states of MoE-4 are much more dispersed across the embedding space, implying that most conversations get encoded uniquely. In contrast, the latent space of Transformer has many clusters, suggesting it is more prone to generating similar utterances even with different input conversation and is thus less generalizable.
EXP 2: Quality of Experts We compare the performance of experts learned by the 4 MoF-LMMs
(where experts are separately trained by optimizing Ey. 2) and 2 baselines (WD (Holtzman et al., 2018) 1) and PPLM (Dathathri et al., 2019)). To study the sub-optimality gap in Corollary 1, we also include the performance of Transformer-
expert end-to-end LMs that are individually optimized with REINFORCE (Li et al., 2016), using the expert labels {&, } as rewards. Inspired by Ghandeharioun et al. (2019) on how bot behaviors are characterized, we use the following label functions to define the
'The VHRED model implementation identical to that in Jaques al. (2019) ensure fair comparisons.
7

---

Published as a conference paper at ICLR 2023
Method
Divenity
Des-1
Dise.2
Dist-5
Perplesuty
Methed
User lot Seat
User Sent. Truns.
Perplesity
MoF-1
D4K0 0:03
0.27
27.43 18.49
MoF-4 IQL.
4 0.38
2888
45.55 36.71
Mok-2
0.14 +: d.ss
0.35
0.27
0.90
38.81 + 17.34
MoE Eas-Q
4.14 0.41
2.36 +: 0.31
45.77 #: 2X 24
MaF-3
DA09 + 0.04
0.29
0.75
0.90
41.55 + 36.68
MaF-4 KLC
3.04 + 0.25
2.21 = 0.24
38.35 + 16,88
6.38
50.17
11
VIIRL
55.81
24.21
Time
0.087 + 0.00
0.36
0.65
0.85
19.23
+ 15.46
VHRI-KIC
3.05
1019
2.16
= 0.38
64.05
4: 36.98
VEIRDD
0.09 0.04
0.35
0.70
0.79
79.77
39.61
VAI
393
0.28
2.19
1 0.32
62.06
00.45
Table Accuracy (Perplexity) and Diversity of Lan- guage Primitive Experts Trained with Reddit.
Table 2: Performance (w.rt. User Satisfaction in Conversa- tion) of MBRL-based DM Trained with Reddit.
(a) Transforner Primitive, TSNE
(b) McE-4 Primitive, TSNE
(c) MoE-4 Scntiment, PCA
(d) MoE-4 Emotion, PCA
Figure 2: Latent Space Visualizations. Figures (a) and (h) Compares Two Primitive Representations. Figures
(c) and (d) Illustrates How Experts (of Different Sentiments and Emotions) are Represented by Latent Clusters.
intents of experts: (i) epos-sem (Yosssee((Y, (neg-seut()) Ejoy (Y). Loptiiim (Y), Langer (Y), Laddess (Y) quantify 6 different sentiment tones and are constructed by a RoBERRa--ased sentiment detector (Liao et al., 2021) that predicts whether an utterance is of positive or negative sentiment, and whether it falls into any of the 4 more-refined emotions: (joy, optimisim. sadness, anger); ;; (ii) eeent-coh (X. Y) measures empathy, i.e., bot's sentiment coherence with user's: ss; (iii) (question (Y) outputs when bot question is detected and O otherwise; (iv) (epp(X,, Y) quantifies exploration, i.e., the tendency to avoid repetitive contexts (see Appendix B.7 for details). Qualitatively, we also measure fluency and expert skills of LMs using human ratings (see Appendix B.8 for details). The results of the above experiments reported in Table 3 and 8 (Appendix A.1), with sample utterances reported in Appendix A.4 to A. Results on human evaluation of different LMs w.r.t. fluency and different expert skills are given in Table 5. Compared with the baseline LMs, generally the experts created under the MoE-LM framework, especially under MoE-2 and MoE-4, better capture all different language intents (where WD and PPL.M appear to capture negative sentiments and emotions much more effectively than behaviors), demonstrating the efficacy of our approach which constructs specialized experts on a diverse language space via reward maximization (instead of weighted MLE). Human evaluations also show that MoE-4 is most effective in generating semantically fluent utterances that possess wide range of expert characteristics.
Similar to the ablation study in EXP I, all the experts associated with MoE-2 and MoE-4 are also among the best ones in capturing language intents. Interestingly, with the Reddit data the experts in MoF-4 perform the best, while with much less data (Cornell) the best experts are built upon the simpler MoF-2 architecture. We conjecture this difference is due to over-fitting issues faced by the larger LMs (MoE-4) when there insufficient data for expert fine-tuning. In Fig. 2c and 2d we visualize the latent space of the sentiment-based experts in MoF.-4, each with 400 conversation data samples. Notice that the sentiment experts' latent distributions are clearly separated (because positive and negative sentiments have opposite behaviors), while the emotion expert's s latent distribution have more gradual separations and even some overlaps (because e.g., joy versus optimism are quite similar, while joy versus anger are quite different). This validates our MoE-I.M represents different behaviors in separate regions of the latent space and justifies our structural prior of modeling each expert as a specialized version of the primitive LM, whose latent distribution focuses on particular regions.
We compare the dialogue management performance of MoE-LM, for which their MoE-based DMs μ are trained with different methods: (i) IQL (Kostrikov et al., 2021), 1), (ii) Ensemble DQN (Carta et al., 2021), (iii) KL-control (Jaques et al., 2019), with 3 standard RL-based DM baselines using the VHRL architecture (Saleh et al., 2020): (i REINFORCE (Li et al., 2016), (ii) KL-control, (iii) SAC (Haarnoja et al., 2018). Ac- cording to the results on expert quality in EXP2, we pick the MoE-2 and MoE-4 frameworks for the Cornell and Reddit tasks respectively. For systematic evaluation, we perform the experiment by having these RL agents interact with a DialoGPT (Zhang et al., 2019) simulated user envi- ronment for a maximum of turns. The DM task is maximize total user satisfaction in the conversation level, which is measured by both (i) user's s overall sentiment, and (ii) user's 's sentiment transition. To construct an immediate reward that serves surrogate for user satisfaction, we set r(X, a, X-) X1 Sem (X) ) X2(Pvem (X+) 1 EL-1 7 2/esam (X1)), where the linear combi-
EXP 3: MoE-RL Against DialoGPT Simulated Users
8

---

Published as a conference paper at ICLR 2023
Method
Question
Expleration
Poutive Seat.
Negaiie Sent.
Sent. Conerence
Jay
Optiism
Anger
Sadoess
MaF-l
044 0.17
1.3 0.21
0.35 0.19
0.50 + 0.38
0.96
-0.31 + 0.56
0.54 + 0.58
Mak:2
0.55 0.227
0.47 + 0.21
3.29 4: 0.35
1.42 +: 0.38
0.51 + 0.40
1.99 = 0.38
1.25 + 0.45
1,88 + 0.39
2.01 0.46
MaF-1
041
4.035
0.50 000:0030 0.24
1.23 + 0.78
0.99 = 0.48
0.66 + 0.35
1.02 = 0.29
0.49 2004-000 0.51
0.53 4 0.49
1.10 0.48
MaE-4
6.31
341
180
2,05
0.44
1,42 0:42
WD
0.0 40003
0.13 - 0.37
-0.50 0.74
1.01 = 0.46
0.51 2:020
-0.1 0.300
-0.84 0.766
1.00 4 0.44
1.27 0.67
PPLM
0.20 0.25
0,48 1 0.28
0,44 0.41
0.89 1: 0.22
0.53
1 031
0,31 0.29
0.40:::033
0.71 1 0:46
0.98
0.59
Trans. RL
0.99 0.33
0.4
1 0.18
3,55 1.64
1,89 1: 1.50
0.72
1 0.30
288 0.96
1.30 1 0.39
1,22 1 0.75
2.35 0.62
Table 3: Quality of Each Expert PPLM Trained on Reddit Casual dataset w.r.t. its Trained Label. Trans. RL Corresponds to Individually Optimized LMs Using Expert Labels {4,} as Rewards.
Method
Avg. Fluency
Diversity
MolE1
0.72 1 0.02
0.51 1. 0.02
Method
Avg. Fluency
Speesim
Sent
Sky Seet
Spy
Sxaeer
MoE-2
0.75 + 0.02
0.34 4 0.02
MoE-3
0,76 + 0.04
0.27 + 0.05
0.64 + 0.04
0.26 = 0.05
0.41 # 0.04
0.33 = 0.05
MoE-3
0.72 4 0.02
0.42 1 0.03
Mob-4
0.82 1 0.02
0.74 1 0.04
0.46 2 0.04
0.44 = 0.03
0.51 :: 0.04
0.43 :: 0.03
MoE-4
0.65 0.05
0.69 + 0.02
Trans.
0.78 + 0.03
0.12 0.03
0.29 0.04
0.19 0.04
0.38 =
0.33 = 0.03
Trans.
0.70 + 0.02
0.47 +: 0.02
Table 5: Phase 2 Raters Evaluation (Reddit Casual Models).
Table 4: Phase 1 Raters Evaluation
= (0.75, 0.25) correlate with Ghandeharioun et al. (2019), and Eson (X) is the same RoBerTa- based sentiment labeler as in EXP2, which assigns a score from [-1, 1] that is proportional to the positive sentiment and inversely proportional to the negative sentiment prediction probabilities. To ensure the baseline RI. DM methods can also possess certain bot-level features, e.g., question, positive sentiment, etc., besides the above RL reward for user satisfaction we also optimize a linear combination of bot-based rewards when training the baseline models, see Appendix B of Saleh et al. (2020) for more details. Since the DM problem lasts at most 5 turns, we use this as the effective horizon and set y 1 1/5 0.8.
nation weights
The results of the above experiments (performed in both offline RL and MBRL settings) are reported in Table 2, Table 7 (Appendix A. Table 9 and 10 (Appendix A.2), with sample utterances reported in Appendix 11 Our experiments show that MoE-LMs outperform most baselines on DM per- formance. We attribute this finding to three factors: (i) MoE-LM restricts the action space into a smaller set of candidate utterances generated by experts (whose qualities are validated in EXP2), the corresponding RL problem then becomes simpler and requires less data (especially in Comell to solve; (ii) Unlike the baseline RL methods, which need to optimize both bot-and-user signals, the MoE DM agents focus on optimizing the user satisfaction goal and are therefore more effective; (iii) Among different RI. settings, MBRI,, which first learns a user utterance model (the model uses the same encoder from the primitive LM and learns a separate decoder for user-utterance prediction) then does DM, performs much better than offline RL methods that moderately improve upon the primitive LM (behavior policy). IQLe-based dialogue managers are among the best across different settings potentially because IQL. is more robust co-variate shifts than standard RL methods, e.g., Ens- Q, SAC, and yet it is less conservative than the behavior-regularizze algorithms, e.g., KLC, Interestingly, our MoF-I.Ms also have lower (hetter) perplexity scores than other methods. This may be due to the fact that MoR-IMM uses pre-trained encoder and decoder from the primitive LMM, which are optimized for generalization and accuracy, while other RL methods may distort their language representations to create utterances that maximize reward but become less natural.
CONCLUDING REMARKS
We developed a mixture-of-ccpert (MoE) approach for RL-based dialogue management (DM). Our MoF language model (MoF-LM) comprises of three main components: (i) a LM that can generate diverse semantics for conversation histories, (ii) number of specialized LMS (or experts) that can produce utterances corresponding to a particular attribute or intent, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. To understand how well our MoE approach generates diverse and sensible utterances, and solves DM problems, we evaluated it using two open-domain dialogue tasks and compared it with SOTA baselines. Our results showed that MoFE-LM (i) improves diversity of text generation, (ii) can generate utterances with specific intents, and (iii) yields better overall performance. We consider creating stecrable LMs that possess different intents and in training RL-based DMs that can carry on rich conversations. Future work includes improving the language representation with information- theoretic approaches, fine-tuning the experts based the DM objective, extending the RL agent to track users' behaviors (via abstract belief states) and plan upon them, preventing RL dialogue agents from generating harmful behaviors (via enforcing safety constraints), and evaluating our MoE-LM on more realistic problems, such information retrieval, recommendation, and negotiation.
work as a step forward in

---

Published as a conference paper at ICLR 2023
REFERENCES
A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.1361/ 2020.
K. Asadi and Williams, Sample-efficient deep reinforcement learning for dialog control. arXiv preprint arXiv:/6/2.0660 2016.
C. Burgess, Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Understanding disentangling in 3-vac. arXiv preprint 0ttl::8804033550 2018.
Salvatore Carta, Anselmo Ferreira, Alessandro Sebastian Podda, Diego Reforgiato Recupero, and Antonio Sanna. Multi-dqn: An ensemble of deep q-learning agents for stock market forecasting. Expert systems with applications, 164:1 13820, 2021.
D. Cer, Yang, Kong, Hua, N. Limtiaco, John, N. Constant, Guajardo- Cespedee, Yuan, C. Tar, et al. Universal sentence encoder. arXiv preprint 0rXiv:/803.11177 2018.
Y. Chen, V. Li, K. Cho, and S. Bowman. A stable and effective learning strategy for trainable greedy decoding, arXiv preprint 1rXiv:1804..7999 2018
J. Chien and C. Kuo. Markov recurrent neural network language model. In 2019 JEEE Automatic Speech Recognition and Understanding Workshop (ASRU)), pages 807-813. IEEE, 2019.
C. Manessu-Nieuueseu-Miiii and L. Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. arXiv preprint arXiv: 106.3077, 2011,
S. Dathathri, A. Madotto. J. Lan, Hung, Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: 291..1992222. 2019.
A. Van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1XX.::///000400 0000005 2018
M. Fatemi, L. Asri, H. Schulz, J. He, and K. Suleman. Policy networks with two-stage training for dialogue systems. arXiv preprint frXiv:1606.03155 2016.
Ficler and Goldberg, Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv: 02663, 2017.
L. Floridi and M. Chiriatti. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4))681-694 2020.
M. Gašić, F. Jurčiček, B. Thomson, K. Yu, and S. Young. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In 201 / IEEE Workshop on Automatic Speech Recognition & Understanding, pages 312-317. IEEE, 2011.
A. Ghandeharioun, Shen, N. Jaques, C. Ferguson, N. Jones, A. Lapedriza, and R. Picard. Approxi- mating interactive human evaluation with self-play for open-domain dialog systems. Advances in Neural Information Processing Systems, 32, 2019.
E. Greensmith, P. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research. 5(9). 2004.
Gu, K. Cho, and V. Li. Trainable greedy decoding for neural machine translation. arXiv preprint arXiv:1702.02422, 2017.
Haamoja, Zhou, Abbeel, and Levine. Soft actor-eritie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. JCMI., 2018.
D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02191 2020.
B. Hancock, A. Bordes, P. Mazare, and J. Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprini arXiv: 105410, 2019.
10

---

Published as a conference paper at ICLR 2023
A. Holtzman, J. Buys, M. Forbes, A. Bosselut, D. Golub, and Y. Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.0600 2018.
N. Hurley and S. Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723-4744 2009.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. Gpt-critie: Offline reinforcement learning for end- to-end task-oriented dialogue systems. International Conference on Learning Representations, 2021.
N. Jaques, A. Ghandeharioun, J. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Pi- card. Way off-poliey batch deep reinforcement learning of implicit human preferences in dialog. arXiv: 1907.00466 2019.
E. Kharitonov, M. Baroni, and D. Hupkes. How bpe affects memorization in transformers. arXiv preprint 10.0028,, 2021.
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4): 143-156, 2001.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 2021.
T. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforcement learn- ing: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 119-1199, 2020.
E. Levin and R. Pieraccini. A stochastic model of computer- homputer-human interaction for learning dialogue strategies. In Eurospeech, volume 97, pages
Citeseer, 1997.
Li, M. Galley, C. Brockett, Gao, and B. Dolan. A neural conversation models. arXiv preprint 1rxiv:1510.0300 2015.
objective function for
J. Li, W. Monroe, A. Ritter, M. Galley, J. Gao, and D. Jurafsky. Deep reinforcement learning for dialogue generation. arXiv preprini arXiv: (oxXiv:66000 1606661544444 2016.
J. Li, W. Monroe. T. Shi, S. Jean, A. Ritter, and D. Jurafsky. Adversarial learning for neural dialogue generation. arXiv preprint arXiv: 7077066777,7 2017.
Z. Li, J. Kiseleva, and M. De Rijke. Dialogue generation: From imitation learning to inverse rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6722-6729, 2019.
W. Liao, B. Zeng, X. Yin, and P. Wei. An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta. Applied Intelligence, 51(6):322-33533 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1125v:1312.5602, 2013.
Bo Pang, Erik Nijkamp. Wenjuan Han, Linqi Zhou, Yixian Liu, Kewei Tu, et al. Towards holistic and automatic evaluation of open-domain dialogue generation. 2020.
Y. Park, J. Cho, and G. Kim. A hierarchical latent structure for variational conversation modeling. arXiv preprint 1804::804.055
2018
B. Peng, C. Zhu, C. Li, X. Li, J. Li, M. Zeng, and J. Gao. Few-shot natural language generation for mnsk-oriented dialog. arXiv preprint arXiv:2002.12322 2020.
A. Razavi, A. van den Oord, and O. Vinyals. Generating diverse high-resolution images with vq-vae. 2019.
A. Salch, N. Jaques, A. Ghandeharioun, Shen, and R. Picard. Hicrarchical reinforcement learning for open-domain dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8741-8748, 2020.
11

---

Published as a conference paper at ICLR 2023
C. Sankar, S. Subramanian, C. Pal. S. Chandar, and Y. Bengio. Do neural dialog systems the conversation history effectively? an empirical study, arXiv preprint (tth:19000016003 2019.
I. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016
I. Serban, C. Sankar, M. Germain, S. Zhang, Z. Lin, S. Subramanian, T. Kim, M. Pieper, S. Chandar, N. Ke, et al. A deep reinforcement learning chatbot. arXiv preprint arXiv: wwt.iv//09.000 1709.02399 2017a.
I. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017b.
P. Shah, D. Hakkani- -urr, B. Liu, and G. Tür. Bootstrapping neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 3 (Industry Papers), pages 41-51, 2018.
W. Shi and Yu. Sentiment adaptive end-to-end dialog systems. arXiv preprint arXiv: 1804. 10731, 2018.
J. Shin, P. Xu, A. Madotto, and P. Fung. Generating empathetic responses by looking ahead the user's s sentiment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1989-7999, IEFE, 2020.
R. Shu, T. Nguyen, Y. Chow, T. Pham, K. Than, M. Ghavamzadeh, S. Ermon, and H. Bui. Predictive coding for locally-linear control. In International Conference on Machine Learning, pages 8862- 8871. PMLR, 2020.
S. Singh, D. Litman, M. Kearns, and M. Walker. Optimizing dialogue management with reinforcement learning: Experiments with the njfun system. Journal of Artificial Intelligence Research, 16:105- 133, 2002.
N. Subramani, S. Bowman, and K. Cho. Can unconditional language models recover arbitrary sentences? Advances in Neural Information Processing Systems, 32, 2019.
Sutskever, Vinyals, and Q Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
R. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
Verma, J. Fu, M. Yang, and S. Levine, Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204, 084402, 2022.
M. Walker. An application of reinforcement learning to dialogue strategy selection in spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416, 2000.
W. Wei, Q. Le, A. Dai, and J. Li. Airdialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3844 3854, 2018
J. Williams and S. Young. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393-422, 2007.
M. Yang and O. Nachum. Representation matters:
Offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 1 11784-11794 PMLR, 2021.
Denis Yarats and Mike Lewis. Hierarchical text generation and planning for strategic dialogue. In International Conference Machine Learning, pages 5591-5599. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2023
S. Young, M. Gašić, S. Keizer, F. Mairesse, J. Schatzmann, B. Thomson, and K. Yu. The hidden information state model: practical framework for pomdp-based spoken dialogue management. Computer Speech & Language, 24(2)1100-174, 2010.
Y. Zhang, S. Sun, M. Galley, Y. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan. Di- alogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 2019.
Zhao, K. Xie, and M. Eskenazi. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. arXiv preprint D0XXiv:9922088835. 2019.
Zhou, J. Gao, D. Li, and Shum. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1):53-93, 2020.
D. Ziegler, N. Stiennon, J. Wu, T. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 2019.
13


Reviews:
[{"review_id": "wK6cebxEkV", "reviewer": "Reviewer_Esbb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address the problem of building an end-to-end trained dialogue system in which responses are coherent both in terms of semantics and also in terms of personality and sentiment.  The approach adopted is to use an encoder-decoder architecture in which the intermediate encoding is manipulated by a mapping function.  By having multiple mapping functions, each corresponding to a specific personality trait or sentiment, a family of language model experts is created all using the same encoder and decoder.  To generate responses, each LM generates its own distinct output and finally a dialogue manager selects one of the outputs using a policy trained to maximise user satisfaction.\n\nThe system is evaluated in terms of diversity and accuracy of the LM experts and the user satisfaction achieved by the overall system.  The VHRED system is used as a baseline and training uses the Cornell movie database and Reddit.  The dialog manager is tested against DialoGPT, and also using human raters.  The proposed system delivers the best performance on all of the main metrics.", "summary_of_the_review": "The approach taken in this work is elegant and mathematically well motivated.  Overall it represents a useful step forward in creating LM-based dialogue models that can be steered to coherently convey differing emotions and personality traits.   In my view, it will stimulate further ideas in this import problem area.", "strength_and_weaknesses": "The core of this paper is the idea of generating LM primitives using a modification of the scheme proposed by Ajay et al for accelerating off-line learning.  It uses a form of VAE to learn a distribution for the latent space which sits between the encoder and decoder and it uses REINFORCE to train the mapping functions to localise the latent space around regions corresponding to each personality trait.  The techniques used are (mostly) clearly explained and the approach is elegant and well-motivated.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tX44diXv6n", "reviewer": "Reviewer_ySba", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a Mixture of Experts Language Model (MoE-LM) that includes (1) a primitive LM that learns diverse semantics of dialog, (2) a set of expert LMs that generate responses according to a particular attribute (e.g., sentiment) and (3) an RL based DM that considers the dialog history and selects from the set of utterances generated by the experts. This paper is a promising proof of concept with a very interesting and novel idea -- however the experimentation is problematic and does not validate any of the claims of this paper ", "summary_of_the_review": "My concerns all pertain to the experimentation which does not validate the claims made by this paper. Despite a promising and interesting method, this paper (1) compares to weak baselines, (2) relies on un-validated automatic metrics (or metrics that are known to be faulty like Dist-1/2/3, LM PPL), (3) does a small inconclusive human evaluation against weak baselines, (4) on datasets that are not typically used. \n\nAs far as I understand (disclaimer: I did not look carefully into the math), the core idea of the paper is very promising. If the experimental issues are resolved (e.g., by following the typical experimental paradigm of sota dialog models), this will be a very strong paper.\n", "strength_and_weaknesses": "Strengths:\n- Very interesting and novel idea. I hope that this approach is extended (e.g., to more appropriate datasets) and evaluated thoroughly.\n\nWeaknesses:\n\nMy concerns all pertain to the experimentation which does not validate the claims made by this paper. Despite a promising and interesting method, this paper (1) compares to weak baselines, (2) relies on un-validated automatic metrics, (3) does a small inconclusive human evaluation against weak baselines, (4) on datasets that are not typically used. \n\n- Why is the human evaluation in the appendix, while the main paper presents arbitrarily-defined rewards? \n- You cannot define arbitrary metrics/rewards and present them as a meaningful comparison. There is a body of work on defining meaningful dialog evaluation metrics which measures correlations against human judgements (survey: https://arxiv.org/pdf/2106.03706.pdf). If you intend to define new automatic metrics, you must compare them to human judgments on benchmark corpora before using them.\n- Human evaluation does not demonstrate a statistically significant improvement over the baseline transformer model. Furthermore, human evaluation should have been done against state-of-the-art dialog models.\n- Human evaluation should present statistics like inter-annotator agreement, instructions to annotators, etc. \n\n\n- The paper does not compare against any state-of-the-art dialog models. Why not compare to DialoGPT/Plato-2/BB2/BB3/etc? Responses in appendix seem subpar relative to typical responses produced by sota models.\n\n- Evaluation datasets? Why reddit/cornell? Please justify this decision over using alternative, more common and well-accept dialog corpora (e.g., BST, Topical-Chat, DailyDialog).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Wm4GipmCgW", "reviewer": "Reviewer_nxAX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper puts forth a new model for open domain dialogue which is a combination of RL based dialogue system and a mixture of expert language models trained on conversation histories. They explain in detail about the three components of this hybrid system : a primitive language model, expert plug and play transformer based language models for each intent (question, exploration, positive, negative, joy, optimism, anger, sadness) and an RL based dialogue manager , each with it's own loss function/reward function. They perform an ablation study with variations on the transformer models as well as a few RL techniques. They evaluate each component individually using automated metrics such as perplexity, diversity as well as human evaluation on fluency, diversity and intent precision and compared them to baselines.\n\n", "summary_of_the_review": "There are several reason to accept this paper such as :\n\n1. The effectiveness of the model shown through automated metrics as well as human evaluation .\n2. The coverage of solutions to common open domain dialogue problems .\n3. The ease with which experts can be trained with different intents and objectives and \n4. The potential transfer of domain to task-oriented dialogue", "strength_and_weaknesses": "Strengths:\n\n\t1. They address many of the common problems faced by open domain dialogue namely diversity, modularity in experts , exploration, engagement.\n\t2. Each component can focus on different aspects by having their own optimization. The primitive LM's loss function focuses on generality and diversity, the experts, on greedy reward maximization of their intent and the dialogue manager optimizes on user sentiment/satisfaction as opposed to just incorporating all these factors in a linear fashion.\n\t3. They show how the experts show different behaviors in different regions of the latent space through this method\n\t4. RL optimization does not distort the encoder/decoder space learned by the LM making the utterances more robust\n\nWeaknesses:\n\n\t1. They did not compare the whole system with other SOTA open domain dialog models like RetGen (the successor of DialoGPT), meena (Adiwardana et al., 2020), know-edg (Li et al., 2020a)\n\t2. They did not use the latest version of VHRED with linear Gaussian prior (Zhao and Kawahara 2020)  for evaluating their primitive language model. \n\t3. No back propagation of reward from dialogue management to language model.\n\t4. They did not tackle other problems in open domain dialogue such as memory to register and personalize to user characterstics  , reasoning over common sense and facts.\n        5.Most of the evaluation details are in the appendix and not in the main paper\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Eu5-8P8vREC", "reviewer": "Reviewer_8p8z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the existing application of language model (LM) in RL-based dialog management (DM) models, the agent is trained at the word-level and thus results in a complex action space. To address this issue, this paper proposes a novel mixture of expert language models (MoE-LM) and performs the planning at the utterance level instead of the word level. Each language model in the mixture models is specialized to generate utterances corresponding to a particular attribute or personality. Compared with SOTA baselines on open-domain dialogues demonstrate the effectiveness of the proposed MoE-LM in the diversity and sensibility of the generated utterances. ", "summary_of_the_review": "This paper has a strong technical contribution and the empirical results are significant. However, I think the paper can still be improved by setting up better baselines, including human evaluation and more examples. The details in training are also necessary to reproduce the results. ", "strength_and_weaknesses": "Strength:\n\n1) The motivation is clear and compelling and the idea of changing the action space from word-level choices to utterance level is elegant.\n2) From the perspective of technical contribution, the paper is very well written, and the math is clear and easy to follow. \n3) The improvements in diversity compared to existing models are significant.\n\nWeakness:\n\n1) Although I like the math-centric writing of this paper, which defines all the problems very clearly and prevents confusion in technical details, I think this paper could be improved if you can cut some details in math (move them to the appendix) and save more spaces in more interpretable results analysis. After all, dialog generation is a human-centric task, and it's better to present more examples to show the real quality of the generated dialog. Moreover, since each LM is corresponding to a particular attribute, I think it's better to save some space to discuss how many attributes are defined and why are they chosen. \n\n2) For evaluation, this paper spends quite some effort in comparing different types of LM architecture, I don't think it's necessary, the main contribution is the mixture model, it makes more sense to fix the architecture to a most common or a SOTA model in the single-LM RL-based DM approach, and focus the comparison in the improvements brought by the mixture model and how it affects the quality of the generated utterance. \n\n3) Similarly to the previous question, I have some doubts about the choices of baseline. VHRED is a pretty ancient model whose LM is still LSTM. Why not comparing with the same type of LM? \n\n4) I think human evaluation is necessary to demonstrate the actual qualtiy of the generated utterance. I can see some human evaluation is added in the supplementary materials. But they should be added in the main paper and should be well analyzed. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "949NY2PsuT", "reviewer": "Reviewer_FimV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attacks the generic responses and user satisfaction problem in the dialog response generation, it proposes a RL-base dialogue management with a mixture of experts language model (MoE-LM), so that different experts can handle the different aspect, for example, diverse semantics, personality etc.; empirical experiments on two open-domain dialogue benchmarks show promising results. ", "summary_of_the_review": "The motivation of this work is to employ the mixture of experts language model to handle different aspects in the dialogue response generation, overall, it is a decent work to show the mixture of experts in the response generation, with promising empirical results on two open-domain response generation benchmarks; it would be better to add human evaluation to verify the effectiveness of the proposed method, to show it well aligned with the automatic evaluation.", "strength_and_weaknesses": "Strength:\n1. This paper proposes to use a mixture of experts language model (MoE-LM) to handle the different aspects of the dialog response generation. \n2. It shows promising results on two benchmarks on the automatic evaluation.\n\nWeakness:\n1. It may lack the human evaluation to verify the effectiveness of the proposed method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster