Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management
Authors: Dhawal Gupta
Yinlam Chow
Azamat Tulepbergenov
Mohammad Ghavamzadeh
Craig Boutilier
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Mixture of Experts
Dialogue Management

Abstract: Reinforcement learning (RL) has shown great promise for developing agents for dialogue management (DM) that are non-myopic, conduct rich conversations, and maximize overall user satisfaction. Despite the advancements in RL and language models (LMs), employing RL to drive conversational chatbots still poses significant challenges. A primary issue stems from RL’s dependency on online exploration for effective learning, a process that can be costly. Moreover, engaging in online interactions with humans during the training phase can raise safety concerns, as the LM can potentially generate unwanted outputs. This issue is exacerbated by the combinatorial action spaces facing these algorithms, as most LM agents generate responses at the word level. We develop various RL algorithms, specialized in dialogue planning, that leverage recent Mixture-of-Expert Language Models (MoE-LMs)---models that capture diverse semantics, generate utterances reflecting different intents, and are amenable for multi-turn DM. By exploiting the MoE-LM structure, our methods significantly reduce the size of the action space and improve the efficacy of RL-based DM. We evaluate our methods in open-domain dialogue to demonstrate their effectiveness with respect to the diversity of intent in generated utterances and overall DM performance.

Offline Reinforcement Learning for Mixture-of-Expert
Dialogue Management
Dhawal Gupta" University of Massachusetts
Yinlam Chow Google Research
Aza Tulepbergenoy
Google Research
dgupta@cs. wmass edu
minllanchoo@google..o
atulep@google.c com
Mohammad Ghavamzadeh! Amazon ghavamzaGamazoo..
Craig Boutilier
Google Research
cboutillerdgoogle.coo
Abstract
Reinforcement learning (RL) has shown great promise for developing agents for dialogue management (DM) that non-myopic, conduct rich conversations, and maximize overall user satisfaction. Despite the advancements in RL and language models (LMs), employing RL drive conversational chatbots still poses significant challenges. A primary issue stems from RL's dependency on online exploration for effective learning, process that can be costly. Moreover, engaging in online interactions with humans during training phase can raise safety concerns, as the LM can potentially generate unwanted outputs. This issue is exacerbated by the combinatorial action spaces facing these algorithms, as most LM agents generate responses at the word level. We develop various RL algorithms, specialized in dialogue planning, that leverage recent cexxuure-of-Expeee Language Models MoF5-LMs)-modee that capture diverse semantics, generate utterances reflecting different intents, and are amenable for multi-turn DMM By exploiting the MoE-LM structure, our methods significantly reduce the size of the action space and improve the efficacy of RL-based DM. We evaluate methods in open-domain dialogue to demonstrate their effectiveness with respect to the diversity intent generated utterances and overall DM performance.
Introduction
Natural language processing (NLP) has made significant strides in recent years, notably in the field of language generation. Thanks advances language modeling, particularly with the use of the transformer architecture (Vaswani et al.) 2017). NLP models can now generate text that is often difficult distinguish from that written by human. However, despite these advancements, these models still fall short when comes to having rich conversations. Current NLP models lack effective dialogue management: they are good generating individual sentences, but struggle with maintaining coherent and engaging conversations. Whereas most compelling conversations generally span numerous topics, are rather open-ended, and often have underlying goal (e.g., customer success, task completion, recommendation). This requires dialogue agents to understand the context of the conversation and respond appropriately while maintaining the ability to achieve goals.
Reinforcement learning (RL) is natural approach to learning policy for dialogue management (DM) agent. Earlier work on RL-based dialogue systems relies on specific, hand-crafted semantic states (Levin and Pieraccini 1997 Singh et al. 2002 Walker 2000) or partially observable belief
"The work was done as a student researcher at Google Research (dhawgupta google.com). The work done prior joining Amazon, while the author Google Research.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

states (Williams and Young 2007, Young et al. 2010), in which the agent encodes conversations and chooses the best-structured dialogue action each turn. Applications include relational rea- soning (Shah et al., 2018), task completion (Shi and Yu 2018). and query fulfillment (Serban et al. 2017), whose action spaces are structured enough be represented by hand-crafted features. To handle more complex dialogues, recent approaches use language models (LMs) to extract semantic representations from conversation histories, treat them as dialogue states, and apply RL to learn word-level generative DM agent (Jaques et al. 2019; ei et al. 2016 2017 Shin et al. 2020).
However, unlike supervised learning approaches, where one can train imitation agents with offline conversation data, RL-based DM algorithms require online exploration. Unfortunately, constant interactions with real users are often expensive and time-consuming. While one can potentially address the DM problem using offline RL, issues such model exploitation leading to distribution shift on the state and action spaces, when training on static datasets are of paramount concern (Levine al. 2020). Moreover, the myriad variations of language make incorporating all possible conversation histories and bot utterances into the state and action spaces of an RL formulation of the DM problem impractical due to the combinatorics at play. As result, naive application of RL to DM may result in poorly-performing agents that generate incomprehensible utterances (Zhao et al. 2019).
We tackle the above issues related to the use of offline RL in DM systems by leveraging recent advances in ofxture-of-Experr Language Models (MoE-LMs) (Chow et al., 2022). Specifically, we develop a suite of offline RL algorithms specialized in dialogue planning that exploit the structure of MoE-LMs. Our methods consist of three main components: 1) a primitive LM, which uses probabilisticc encoder and decoder and is capable of generating diverse semantic intents; 2) a number of specialized expert LMs. each of which generates ufterances corresponding to a specific intent; and 3) compositional DM that, at each turn, given the encoded conversation history, selects an utterance from a set of candidate utterances suggested by the experts and pass it to the DM agent to execute.
Our contributions to offline RL adapted for MoE-based DM agents four-fold. First, exploit the hierarchical structure of MoE-LMs, allowing our offline RL methods to work with a significantly smaller, finite action space, making the RL problem more tractable. Second, by leveraging pre-trained regularization techniques from offline RL that align the DM's behavior with that of the primitive LM- the proposed RL algorithms can focus on higher-level dialogue planning. The proposed combination results in higher data efficiency than standard RL methods by delegating the responsibility of language fluency to be handled by the MoE-IMMs. Third, by using the diverse semantic representations of MoF-LMss, our methods operate at the sentence embedding space and have much simpler critic and actor updates. This circumvents the word-level credit-assignment issue, particularly challenging in long conversations (Saleh et al.. 2020). Fourth, in contrast to the findings of Verma et al. (2022), where offline RL agents tend to lack utterance diversity (due to potential reward hacking and optimization of a single objective), our MoE-based DM agents by design are adept at generating utterances that reflect different intents.
MoE-LMss-hiie generate coherent utterances-ann
We begin with a brief introduction of LMs, the MoE-LM architecture, and the use of Markov decision processes (MDPs) in DM in Section 2 We then describe the pre-training procedure for MoE-LMSs- which encode diverse semantics and generate fluent utterances capturing specific intents- in Section 3 We derive state-of ofte--of-the-art (SOTA) offline RL algorithms for training MoE-LM in Section 4 and three MoE-LM LM specialized offline RL algorithms in Section Finally, in Section 6 we evaluate our algorithms in open-domain dialogues against their ability to generate utterances with diverse intents and their overall DM performance.
2 Preliminaries
Language Models (LMs) In this work, we employ seq2seq LMs (Sutskever et al. the next utterances in dialogue. We assume access to dataset of the form D (x((), y()))) )=
2014) to generate
where each X is an L-turn conversation history X { wherein X; is the utterance in a conversation at turn l and Y is the next utterance. We define Nx to be an upper bound on
the length (number of tokens) of each utterance X, in x The role of a LM is to predict the probability of the next utterance Y, consisting of N tokens, conditioned on the conversation history X, i.e., Pr (Y (vm, \ X). In the transformer architecture (Wolf et al. 2019). LM first encodes the conversation history X using an encoder & to a (L x Nx)- sequence of embeddings
II Xx has fewer tokens than Nx. the remaining spaces will be padded by specific token and masked.
2

---

( -L.x where each 21, is a vector in the latent space induced by the encoder o For notational convenience, we concatenate these embeddings into single embedding E 2 C Rd where d is the overall dimension of the latent space. The next utterance Y
{ūm Jam is then sampled, token-by-token, from decoder 4, i.... Y ~ V 2 := N v (Tn 5o .... 1 where 3o is a fixed initial (start-of-sentence token (Chien and Kuo 2019 and the latent state is denoted as z (X))
have been used to model dialogue management (DM) problems in variety of settings (Li et al. 2016 Asadi and Williams 2016, Jaques al. 2019). In such MDPs, denoted by .M- = (S. P. 80: 7), the state space S represents the tokenized conversation history and the initial state So E S is the initial user's query. The action space A is the tokenized language space with each action a E A represents one possible next utterance of the agent. The transition kernel P' models the distribution over the user's response to the action taken by the agent (bot) and current conversational context. Finally, the reward function r measures the user's satisfaction as a function of the conversation until the most recent step. In these MDPs, we can think of the LM as policy that maps conversation histories the next utterances. The goal is to find policy x² with maximum expected discounted return, i.c., π E arg max, J=. where J = EI k=0 7'r, P. s0, T.. Note that the size of the tokenized state and action spaces grows exponentially with the vocabulary size. This makes it intractable to solve MDPs of this type, even for a medium-sized vocabulary.
Markoy Decision Processes (MDPs)
Mixture-of-expert Language Models (MoE-LMs). Chow al.
2022 recently demonstrated
promising results using MoE-LMs to enrich a bot's utterances and improve DM (see Figure | [for sketch of the MoE-LM architecture). These results were achieved mainly due to (i) learning
language representation that captures different semantics (primitive discovery), (ii) a machinery, called expert construction, that embeds different intents into sub-models of this LM, that they can behave appropriately when prompted, and (iii) a compositional dialogue manager module that comprehends the conversation and determines which response deems most appropriate.
For primitive discovery, one first learns language model LM, (Φ, Go. v that consists of stochastic encoder Go o 4 where latent space Z S Idd and Gu(z'|z) == N (2))
an encoder mapping tokenized conversation histories to is Gaussian distribution, and decoder V. LMg predicts the next utterance Yo (token-by-token token) conditioned on s' sampled from the latent distribution // (Yol2'),, i.e., z' Go(-12). We overload our notation to denote the primitive by [ (Y which predicts the next utterance accurately and also has strong generalization in 2 over a diverse set of possible utterances.
LMo (X) := (- =======))))====)))))))))))===)=)
LAME -Joee Commect LAMd caty
Sia
Go
(heigh Aihult your
Inpuiiativ
1
91
X
2
Explonatro
Fnoculd love to be Stand
S
V
f
It's pancelil sayying hers, hana
thef
neigtborhest
safe andiwee to
G m
m
Whl you wast
Aue you wadding
Step1 1
Step 2
Step 3
It's sate rice to
deterministic transition
stochaetic transition
Figure 1: (Leftt) The MoE-LM architecture (Chow et al. 2022). Step 1: & encodes conversation history. Step 2: V 9,, Vi, generate candidate bot utterances. Step The compositional dialogue manager μ selects the bot response by Q-score ranking and post-processing (Right) Sample utterance workflow generated by MoE-LM trained with Reddit data.
Given a primitive LMg, the algorithm learns expert distributions {
G((2'1>) =
N (H4(2), (2)) each corresponds to a particular personality/intent and generates samples in specific parts of the latent space Z. This results in m LMs, [LM LM, == ( Gi, v), each serving expert that generates one more candidate next utterances Y, that are relevant to the
3

---

conversation X, and also compatible with its respective personality/nteet.. The compositional DM
takes the encoded conversation history= = = xxX) and candidate action utterances generated by the
experts {Y, i=g as input, and selects one of them exccute, i.e., Y p(- (Y:) )=))))) Given the
state 8 = X and action a = Y, the MoF-I.M policy that optimizes the DM MDP can be expressed as
TM)s(als)= [ ((d))()))(.
II (ā:l==) ==0
(=)((()))))
(1)
3 Warmstarting the Mixture-of-Expert LM
The MoF-LM approach effectively reformulates the reinforcement learning (RL.) dialogue manage- ment problem, resulting in considerably smaller action spaces. This is achieved as the DM agent is now re now required to select the most appropriate utterance for presentation to the user from a finite, predefined set of candidate utterances, rather than generating responses word by word, hence allowing the agent to focus on optimizing the specific goal of the conversation task (as candidate utterances are separately optimized to follow particular bot-based characteristies/nneent)). Recall that the DM is policy conditioned on both the latent state and the actions suggested by the experts. Before introducing the different RL methods for DM (Sections 4 [5d in the following, we outline (i) the learning of diverse semanties (primitive LM) for conversation histories, which allows the agent to generate a wide variety of utterances, and (ii) the construction of specialized LMs (experts), which generate utterances with different intents.
Following the primitive discovery procedure in Chow et al. (2022). we learn the primitive LM, LMp, by solving the following KL-constrainee optimization problem that aims at capturing diverse semanties:
min Eet -log (Y[=')) s.t. Ex= V).p
SeKI. (2)
In (2, E is the empirical expectation over (X, Y) in the dataset D, p is a distribution over the latent space conditioned on the encoded conversation history z and the target utterance Y. and (KL is positive real-valued threshold. The distribution p(-12, Y) is Gaussian NNup((, & (Y)). 0z(2, T La)) in which Φ is pre-trained encoder for the target utter- ance Y. and the mean (p((s.)) and variance 82( (:.))) are trainable models. Note that by solving (2). we maximize the log-likelihood of sentence for a context and latent generation, while enforcing consistency between the latent variable z' predicted by Go(-12) and p(-12, Y) via the KI. constraint. In practice, we implement the KL constraint in (2) as a penalty weighted by a chosen coefficient.
To complete the MoE framework, one needs to train a set of experts {LM; J each generating candidate utterances of different intents. By viewing each expert as a distribution of particular behaviors in conversation data D, we leverage the results in Chow et al. (2022) and adopt universal encoder-decoder (Ø, v among all the experts. In this view, each expert i distribution (-(+)) over certain regions of the latent space Z. We train each (-(22) by solving
min É. Ez ((X)Y (4xX Y)), G;
where (x, E is real-valued label that characterizes the intent of expert i. We can think of &,(X, Y) as a score assigned Y resembling how strongly Y exhibits the trait expert i is meant to represent. Each expert is learned via rewarddmaxiiizattin where & is treated as an intent-aligned reward signal for expert i. Note that there is connection between the above approach and contextual bandits (Chu et al., 2011). ). where both the context and action spaces are the latent space 2, and the bandit policy is the latent distribution G.. The choice of greedy reward maximization is to encourage a particular behavior in the expert's 's immediate utterance rather than controlling its future utterances.
Long-term dialogue planning is handled by the compositional dialogue manager. For example, with Gaussian experts { we can use the standard REINFORCE algorithm (Sutton et al. 1999a). where the model parameters (J4,, o;) are updated in the direction a
[2, (X,
V (p..a.) ) log P., (2/1²)] with learning rate a > 0. To reduce the variance of these estimates, we can also adopt the bascline variance reduction technique (Greensmith et al.) 2004).

---

4 RL for Mixture-of-Expert DM
In offline RL, the policy is learned from the collected conversations D, without further online interac- tions. This potentially allows RL. DM methods to leverage the abundance of offline conversational data for policy learning. Let (X, Y, x)) ~ 'D be tuple sampled from the offline conversation data D, with X+ being the follow-up user response. We can formulate this tuple as a MDP data by defining $ := X. := Y. r r(X+), +) and 8+ := (X, Y,Y, x+) as the state, action, reward (w.r.t. the follow-up user response), and next state, A standard offline RL algorithm is Q-learning, (Watkins and Dayan 1992) that solves: mine B(e.arr,4))D [(( + maxa, 1 Q(++, a.) - Q(s, a)
However, with large action space, the inner maximization (also termed as greedification) in Q- learning is generally computationally intractable. Furthermore, since one cannot ensure that the greedy a+ is sampled from the same action distribution as in the offline RL dataset (an issue worsened by the large action set), such a covariate shift in the sampling distribution can cause an overestimation bias for the Q estimate. To alleviate these issues, we propose to leverage the warm-started MoE-LM (Section 33). where the diverse semantic representation and the expert LMs are learned separately. This is crucial to make our offline RI. DM problem tractable as the language flueney is captured by the MoE-LM, while our RL-based DM focuses on higher-level planning strategies. In the following, we describe how this can be achieved via different offline RL algorithms.
Offline RL Methods for MoE-LMs: One approach to address the aforemenniioee offline RL issues is entropy regularization (Haarnoja et al. 2018 Carta al. 2021), which regularizes the greedification step to ensure the learned policy is either diverse enough or close to the behavior (data-generation) policy (e.g.. through Shannon entropy or KL divergence between these policies). Recall that the primitive LM, LMg, models the utterance distribution in D and the state-action-rewarr next-state tuple (s, a, r, 8+) of the DM MDP. With the following latent states generated by the primitive LM: 2 = (s). Za = (((s, a)). and 2+ = ={(s+), we define the latent conversation data (D) as a collection of (z, To 2++ tuples. With Shannon-entropy regularization, we can utilize the soft actor critic framework (Haarnoja et al. 2018) to develop RI. updates for the value function (2),, state- action value function (( ). and latent generator G(='l=), which is initialized with the primitive
latent expert Go- This framework minimizes the following losses:
LQ = (s [(r Vin (2+) (2
(4)
Ly = (D))
Qunr - a log (2'l2))
(5)
(6)
where the critic, (V, Q), takes any encoded conversation history as input and predicts the correspond- ing cumulative return, 0 is the entropy temperature, (Vur, Que) are the target value networks, z' ~ ([]]2) is the latent sample generated by 9, â ~ (2²) is the utterance sampled from Vi G. and
finally 2a ((XX, a)) is the corresponding latent state.
From a hierarchical RL perspective (Sutton et al. 1999b Salch al. 2020). the latent generator behaves like a high-level policy, whose latent sample z is used to generate a bot utterance via V-decoding (with the primitive decoder V acting as the low-level policy). Extending the above RL updates to the case of relative-entropy (KI.) regularization can he straightforwardly done by replacing the term log (221lz) with (=((()) [2) //(==///)) since the primitive LM approximates the behavior policy and the encoder-decoder pair (2, v is shared among the experts.
Multiple techniques in alle-function parameterization have been employed to tackle the overesti- mation bias. Fujiimooo al. (2018) proposed maintaining two Q-functions, and a dual Q-function chooses the minimum value between them to avoid overestimation. [aques et al.) (2019) applies dropout in the e-function to maintain ensemble of Q-values, and outputs the minimum value to avoid overestimation. By utilizing these methods within the MoE-LM framework, we can propose the following variants of offline RL algorithms:
(i) SAC that uses dual Q-function and the actor-critic updates in 4 16) (ii) EnsQ that uses an ensemble of Q-functions and the same updates; and (iii) KLC that uses an ensemble of Q-functions and a latent KL-regularized actor-critic update.
Apart from the actor-critic approach that iteratively improves the value functions and policy, Implicit Q-Learning (IQL) (Kostrikov et al., (02II) a value-based offline RL algorithm, has recently shown success in tackling various problems, including arskntreetedd dialogue management (Snell et al. 2022). Within our MoE-LM framework, we propose the IQL-DM algorithm, whose value function V(z) minimizes the following loss: Lv = B(e.a.) ((D)) [L (Qur(za) V())), where Lg is the
5

---

expectile regression operator (Koenker and Hallock, 2001) of estimating the top-7 expectile statistics.
The Q-function of IQL-DM is updated identically to that of actor-critic in (4). which estimates
Qzza) ~ r + (2())) via a least-squary loss (Bradtke and Barto 1996). The V-function estimates the top-7 quantile of the state-action Q(za) random variable every latent state z. As r approaches one, r + 1, the IQL. updates converge to the optimal Q-function, Q" (za), i.e., (e)) [(r Q (za)) + O, where 24,a, = (((X, a, X_, aa)) for any next-action utterance 0.,. Intuitively, IQL leverages the generalization capacity of critic functions to estimate the value of the best action without directly querying the values of unseen actions. This makes it less conservative than most offline RL methods that constrain the policy's actions to be in-distribution via behavior regularization (e.g., SAC, EnsQ. KLC).
maxa, (2+
Auto-segressive Decoding in Actor-Critic: The actor-critic methods (SAC. EnsQ. KLC) ameliorated the two issues in offline RL to a certain extent (the inner maximization is replaced with V-function learning and covariate shift is controlled by policy entropy regularization). However, implementing these methods (Eqs. 6 entails sampling utterances from the current policy, i.e., à V G. which involves expensive auto-regressive LM decoding at every training update. To resolve this issue, one may empirically replace V o 9 with a feacher-forcing (Toomarian and Bahren 1995) variant (TF(a) which replaces auto-regressiv decoding with a one-step generation from the bot utterance Y in D. This will further restrict the policy update of G to be close to the behavior policy. In contrast, since IQL does not perform explicit policy updates, it directly circumvents this expensive auto-regressive sampling operation of à.
DM Construction in MoE-LMs: Recall that in an MoE-LM, the DM policy j takes the encoded conversation history & = xX)). the m + 1 candidate action utterances generated by the experts
( O+ and selects one of them to execute, i.e., pa( ()))))) Given the Q- Q-function (Zaa (2a) learned by any of the above offiine RL algorithms, we extract the DM policy i via softmax
greedification over the finite set of MoE candidate utterances, i..,, uı(a 2, (Y.)=))) x exp(3-Q(z)) where 3 > 0 is the policy temperature. This DM policy uses the Q function to score different candidate utterances and returns an utterance based on the likelihood of these scores.
5 Mixture-of-Expert Offline RL
In Section 4 we presented how state-of-the-art offline RL methods are adapted to the MoE framework. which can have limitations due to being agnostic to the model architecture. Recall that MoE dialogue management is specialized hierarchical reinforcement learning (HRL.) problem, which optimizes over a restricted class of DM policies defined by the convex hull of expert policy set (whose weights are defined by the DM policy μ)) This problem is of great interest because it reduces the original RL DM problem, with a combinatorial action space, into one that has a much smaller finite action set. In the following, we leverage the mixture-of-police structure and develop offline RL algorithms that specifically target this HRL. problem.
Stochastic-action IQL (SAIQL): The first approach applies IQL to the discrete, stochastic set of candidate action utterances (Yg J as generated by the MoE experts. Equipped with the latent conversation data (D) {2, Za, T, 2+)} (see Section 4 and the latent expert policies {G, y in the MoE-LM, we propose a DM algorithm, whose value Tunction (2) minimize the following loss:
m
LV
m++1 Σ
L,5 (Z,,) Boa
V())))
where za; (X a:)) is the latent state that corresponds to the action utterance sampled from the i-th expert, L5 is the expectile regression operator, and the Q-function is updated as in Eq. 4 To incorporate the maximization over candidate utterances in IQL, we compute the expectile regression over the joint latent state and expert policy distributions.
However, unlike the standard IQL DM algorithm, which avoids autoregressive decoding for pol- icy exccution, SAIQL requires auto-regressive sampling of all m + 1 candidate utterances. Sup- pose the augmented latent conversation data ((D)sA ((2, =a, r. 24, (zg. )m))), which also includes the set of latent expert actions {2ş, 3 is available. One straightforward way to cir- cumvent this issue is by replacing the expectation over experts with the realized candidate utter-

---

ances, i.e., by approximating the value function in SAIQL with its unbiased empirical average (L5 V=))].
1 m+I
While having access to candidate utterances is not standard in IQL, it is necessary here to allow Q-learning to exploit quantile regression over realized candidate utterances (an approach shown to be sound in Boutilier et al.) (2018). Therefore, we termed this method stochastic action IQL (SAIQL) to reflect the stochastic action sets used in IQL training. Once SAIQL converges, the DM policy is also constructed as softmax of Q-values applied each candidate utterance.
The MoE-MDP is defined as MM = (S, P. F, 5g. 7 where the state space is the product of the learned latent space 2 and the joint action space of the + 1 experts, i.e., S = 2 x Am+1, the action space consists of the m + 1 experts, i.e., À = {0,... m): the initial state 8g is the encoding of the initial user's s query and the utterances suggested by the experts in response to this query; the transition kernel models both the user's responses and the next experts' actions; and finally the reward is the same as in the original MDP. Since MoE-MDP has a finite number of actions, learning a policy is equivalent solving finite-action MDP, i... X arg max, Jx= := B|> 7'7, P. 50, Al.
Follow-the- Leading-Exxpeee (FtLE); Banijamali et al. (2019) showed that the MoE-MDP problem is NP-hard but can be approximated by max m++ _x- X(1) (=)) + +U(M), where V is the value function of the 7th expert and (.M) > 0 is surrogate function that depends the experts' stationary distributions. However, computing these distributions is generally intractable as the experts are LMs themselves. This motivates our heuristic FtLE algorithm, which ignores the second term in training a set of expert critic functions and picks the best action at each step. To efficiently parameterize the critic function, similarly to the architecture used in DQN (Mnih et al., 2013) for discrete-action RL, we define (m + 1) -headed critic function, where each head represents the value of following an expert's s policy. To train the multi- critic, modify the standard critic losses as
m E, a. Vo.. [Za, -v))) (8)
LQ = E. Ba [rr Vi ( -C( (z..)) LV
where Q' and V represent the critic-function head for expert To overcome the auto-regressive sampling issue in (8). we relabel the offline conversation data D by assigning action utterances to train the critic function(s) whose corresponding expert(s) most likely generate those utterances. Specifically, consider the following V-function loss:
LV Σ Exaan [di= E
(za)
where 1,=i(z.a..) selects the expert based on the best ligeliiel i(2, Y) == arg max, log (Y with 20d ~ (-()) After learning the critic functions, the FtLE DM policy can be constructed as y(a| 2, (7) mm x exp(AQ((z.,) (za)).
Value-based RL for MoE-MDP (MoE-VRL): Consider (m + 1) h)-hadded value function A of the MoE-MDP, where each head represents the optimal value by choosing the corresponding expert's action. Applying standard DQN, this function can be learned by minimizing the following loss:
LA = E.YY. [(r + y max Aur (4) - A(2, i(2, {
(10)
where Aur is the target A-network. For simpler exposition, we only use the partial MoE-MDP states of encoded conversations in the above DQN loss and omit the candidate action utterances. Extending to the full MoE-MDP state is straightforward but omitted for brevity. The inner maximization over i can be computed explicitly because the MoE-MDP action space of expert indices is finite and small. Here, i(z, Y) is the same index function that attributes utterance Y to the expert most likely to generate it (based on likelihood). With the optimal value function A (2, i), the MoE-MDP policy picks the best expert (2) == arg max, A (z, i), and the DM policy can be constructed as u(a| 2, (Y:)m))) x (xp((QX((e) (2a)). where (=)))) the critic of the optimal expert.
6 MoE-based DM Experiments
We evaluate our MoE-based offline RL algorithms on two open-domain benchmarks common in the RL-based dialogue management literature (Jaques et al. 2019). The first one is the Cornell Movie
7

---

corpus Danescu--iiuleseey Mizil and Lee 2011 which consists of conversations between speakers in different movies. The second is the Reddit Casual (Ghandeharioun et al. 2019 conversations dataset, which is subset of the Reddit corpus that only contains casual conversations.
Environment: We perform the experiment by having DM agents interact with DialoGPT (Zhang et al. 2019) simulated-user environment. The task to maximize user satisfaction, which measured by the user's overall sentiment. To construct an immediate reward, we set r(X++ feeml (X+) where bee (X) is RoBerTa Ta-based sentiment classifier (Liao et al. (021). which assigns score from [-1,1] 1] that is inversely proportional to the (negative) positive sentiment prediction probabilities.
We pre-train the MoE-LM with either the Cornell or Reddit dataset and construct 10 experts (i.e., m = 9. plus the primitive expert), each corresponding to an individual intent open-ended dialogues, including "empathy", "optimism". "cheerfulness", "contentment", "dejection", "rage", "sorrow", "questioning". "exploration" etc. See Appendix [eppendixBBoor for details. The conversation lasts for 5 turns (with 0.8), where each turn entails a query/response from the user followed by an agent's ' utterance. During the agent's 's turn, each expert generates candidate utterances, thus resulting in total of 50 candidate utterances. To evaluate the methods, we measure the return of the trajectory generated by different agents via EX /Ei= o ('r(X++1))P))))
LM(. Xi+1 Dialog ( X Y:)]-
We employ two evaluation approaches, namely (i) a model-free approach that only utilizes the learned Q function to score candidate utterances, and where the DM policy selects the action utterance based on softmax likelihood; and (ii) a model-based approach that uses the Value function (V) along with learned next-user utterance model (X |zy). that optimizes the following loss:
Evaluation:
P (a.. Roan r(x+))) P) We first approximate the Q function
via (=aa) r(x,) (á+), where X4 denotes the user utterance sampled from Puser (a),) then use that function to score candidate utterances, and, finally have the DM policy select the action utterance analogously. Human evaluation is also conducted on the DM performances of different offline RL agents. More details and results can be found in Appendix
Table 1: SOTA offline RL methods
Reddit Casual
Comell
Method
Model Free
Model Based
Model Free
Model Based
IQL
0.53 i 0.47
4.25 + 0.12
1.32 + 0.19
1.47 1 0.15
SAC
0.97 1 0.52
4.13 + 0.21
1.55 1: 0.19
0.36 1: 0.26
EnsQ
0.10 = 0.40
4.06 0.25
-1.51 0.20
0.21 0.21
KLC
0.31 = 0.46
3.69 + 0.37
-1.46 0.21
-0.07 + 0.25
BC
0.65 + 0.41
2.18 1 0.36
Bandits
4.3 1 0.16
1.3 + 0.17
Table 2: MoE specific offline RL methods
Reddit Casual
Cornell
Method
Model Free
Model Based
Model Free
Model Based
EXP |
0.97 1 0.52
1.25 + 0.12
-1.32 + 0.19
1.47 + 0.15
SAIQL FILE
0.81 + 0.42
4.65 : 0.06
-1.34 + 0.25
2.61 + 0.24
1.14 + 0.49
4.59 = 0.07
-0.39 + 0.24
3.51 + 0.19
MoE-VRI.
0.72 + 0.47
4.46 + 0.10
-0.58 + 0.24
3.62 + 0.17
The goal of this experiment is to investigate the effectiveness of SOTA offline RL algorithms. In these experiments, we only make use of the primitive language model LM, (Ø, Go, vi) generate sample utterances. To simulate previous works using single policy settings, we fine-tune the latent base distribution Go for policy optimization while keeping the encoder-decoder (Φ. v\ fixed. As mentioned in Sec. 4 we deploy the following offline RL algorithms to train the DM policy μ of MoE-LMs: (i) SAC (Haarnoja et al.) 2018) with a ensemble of CQ functions (Jaques et al., 2019) with actor-critic; (ii) KLC (Salch et al. (020). which utilizes the dual Q function and applics KL regularization between the latent policy G and the primitive policy Go.
Experiment 1: SOTA Offline RL with MoE-LMs:
dual Q function critic (Fujimoto et al., (018)) 8); (ii) EnsQ, which utilizes
8

---

in the actor-critic algorithm update (iv) IQL Kostrikov et al. 202I), which adopts the idea from Q learning to estimate an optimal ? function in the MoE-LM latent space. To our knowledge, our work is among the first that makes use of IQL for open-domain dialogue management. These methods have been implemented in ways where the original idea has been preserved, making the comparison fair to the original works. With each learned Q function, the hot picks the final action by sampling from softmax distribution of Q scores overall candidate utterances. To demonstrate the efficacy of offline RL methods, we also include results from Behavior Cloning (BC) well as simple reward maximization (Bandit) (i.c., Y 0) for comparisons.
i.e., (1) [0)(
Table presents the results of our experiments with these methods in the open-dialogue system, where 5-turn conversation was generated. The table displays the mean return over 100 conversations with their respective standard errors. Our experiments demonstrate that model-based evaluation can significantly improve dialogue management over the model-free counterpart, even with a next-user LM that is much simpler than the Dialog-GPP GPT user. Among most model-based and model-free evaluations, we found that IQL, originally designed to tackle offline RL problems, outperforms other RL methods. This performance can be attributed to IQL's ability to (i) alleviate Q overestimation errors due to co-variate shifts; (ii) estimate the optimal values without being overly conservative w.r.t. the behavior policy, and (iii) avert the auto-regressive utterance sampling issues in training.
Interestingly, we also found that KLC and EnsQ, two standard methods in RL-based DM, struggled to achieve satisfactory performance in our experiments. This may be due to the fact that applying dropout (for ensemble Q) and KL regularization in the fixed MoE-LM latent space makes DM algorithms overly conservative. In contrast, SAC successfully learns a perl-peerooiing model-free DM policy but fails in the model-based regime, potentially demonstrating its instability in critic- function learning. BC also fails to provide any satisfactory performance on any of the domains, and surprisingly, Bandit method or plain reward maximization did as well as IQL, pointing to the fact that maybe the offline RL methods being used or not exactly helping in planning at all.
Experiment 2: MoE-specific Offline RL: this experiment, we explore the benefits of leveraging the MoE framework for training offline RL agents in open-domain conversational systems. Building upon the insights from our previous experiment (Experiment 1), we propose several modifications to standard Offline RL algorithms to take advantage of the MoE framework. As mentioned in Sec. [ we developed the following MoE-specific offline RL algorithms for DM: (i) SAIQL, which extends IQL to incorporate the multiple candidate utterances generated by the experts; (ii) FiLE, which learns DM policy to follow the best expert policy at each step (estimation of the experts' long-term values is done concurrently with a multi-headed critic architecture and data relabeling) and (iii) MoE-VRL, which learns optimal meta-value function over the space of experts. Leveraging the MoE-MDP formulation, solving which leads to an optimal DM policy that provides the optimal sequences of expert policy switching. We aim to evaluate the potential of these MoE-specialized offline RL algorithms over off-the-shelf offline RL methods in DM.
Table the return observed similar to the ones displayed in Table I The first row in the table displays the best performance across all methods from Experiment 1, for comparison. Our results demonstrate the efficacy of the proposed methods that utilize the structure of the MoE framework in dialogue management. All the methods that used all experts while training (SAIQL, FiLE, and MoE-VRL) outperformed the SOTA offline RL methods, indicating that an offline RL algorithm that takes the candidate utterances into account can generally improve dialogue planning. Moreover, making the RL. algorithms attuned to the multiple-expert structure (i.e., FiLE and MoE-VRL. indeed results in even better DM performance, emphasizing the benefits of reformulating the DM MDP using the HRL paradigm, where the DM policy is optimized over restricted class of finite-action policies. Also, we note that only MoE-aware offline RL methods were actually able to outperform simple per-step greedification (i.e. Bandit) which hints the fact that they were actually able to plan ahead and perform long-term credit assignments to optimize return. Whereas all the standard offline RL methods failed to do that (Table D) Using multiple critic functions to separately estimate the value of different experts also allows us to better understand their long-term utility (of the corresponding intents) and how they affect the conversation quality. Overall, these findings highlight the potential of the MoE-specific offline RL methods to improve dialogue management performance.
Human Evaluation: Table 3 summarize the results of around 600 ratings provided by SO on the bots' quality, in terms of fluency, and conversation-level sentiment improvement on the Reddit Casual
'The RL DM approach in Jaques et al. 2019 which applies KL regularization at the word-level LM policy is not applicable to our case because our DM policy is defined in the latent space.
9

---

Histogram of Expert Selection
Divergence
18
0.09
Table Human raters evaluation
12
0.08
Method
Avg. Fluency
Sentiment
0.08
BC
0.67 + 0.26
0.24 = 0.50
KLC
0.62 + 0.27
0.66 = 0.47
IQL
0.84 + 0.24
0.72 = 0.46
IOL
MoE
EnsQ
IQL MRE-VRL
(a)
(b)
SAIQL
0.81 + 0.19
0.57 = 0.50
FILE
0.88 + 0.24
0.76 = 0.48
Figure 2: Experiment on the Cornell dataset with Model-based evaluation(a) Histogram of frequency of expert selection. (b) KL divergence against uniform distribution.
MoE-VRL
0.72 + 0.28
0.70 = 0.45
dataset. We can similarly notice in this case that MoE specific offline RL methods seems to perform better both in terms of fluency and sentiment improvement over standard offline RL. The complete details of the experiment can be found in Appendix
Experiment 3: In this experiment, one aims to investigate the effectiveness of selecting different experts during dialogue management. To this end, we conduct a study where we measure the frequency with which utterances from different experts are selected throughout the conversation. Specifically, wish to understand the diversity of intents selected by different offiin RL algorithms.
Given approximately 200 conversation turns, we measure the frequency of the expert agents when their utterances are selected and present the frequency metric for the worst performing offline RL method (EnsQ). good performing method (IQL), and an MoE-specific RL algorithm (such as MoEE VRL). To visualize our findings, we plot histogram of the frequencies of different experts being selected and calculate the KL divergence of the histogram against a uniform distribution over the experts. While the uniform distribution may not be the optimal distribution of utterances, it provides measure of how well the agents make use of different experts, along with their performance.
Figure illustrates the result for the above experiment for the Cornell dataset, in the model based setting. We observe that the worst performing agent, EnsQ, has highly skewed distribution of expert selections, with few experts being heavily favored over others. This suggests that EnsQ is less diverse and does not effectively utilize the full range of expert capabilities available. On the other hand, both IQL and MoE-VRL exhibit more balanced distribution of expert selection, with utterances chosen from multiple experts throughout the conversation; i.e., their frequency distributions closer to uniform distribution, with much lower KL divergence distance.
However, note that there is a clear performance gap between MoE-VRL and IQL where former significantly outperforming the latter. This highlights the importance of incorporating the MoE framework to better utilize the intent of different experts in dialogue planning, rather than relying on generating a diverse set of candidate utterances. Overall, these results suggest that encouraging diversity in intents and better utilizing expert knowledge in planning can improve DM performance.
7 Concluding Remarks
By leveraging the recent advances of MoE-LMs, we developed a suite of offline RL-based DM algorithms. Our methods significantly reduce the action space and improve the efficacy of DM. To understand how well our offline RL approaches generate diverse utterances and solve DM problems, we evaluated them on two open-domain dialogue tasks and compared them with SOTA offline RL baselines. Our results showed that by exploiting the MoE-LM structure, our specialized offline RL DM methods (i) improve the diversity of intents in bot utterances; (ii) have better sample efficiency; and (iii) yield better overall performance in both the model-based and model-free settings. Our work provides important insights on how to create scalable RL-based DM methods that train chatbots to achieve dialogue tasks and enhance user satisfaction. Future work includes fine-tuning the experts (i.e., low-level policies) with offline RL, learning the optimal semantic representation for hierarchical RL, preventing dialogue agents from generating harmful behaviors (c.g., by enforcing safety constraints in the RL algorithms), and evaluating our DM methods on more realistic problems, such as customer support, conversational recommendation, and persuasion.
10

---

Acknowledgement and Funding Disclosures
We express our sincere gratitude to the Mudcats team Google Research for their invaluable feedback and innovative ideas, which have significantly enriched this project. We would also like to thank the anonymous reviewers, whose insightful suggestions and constructive feedback were instrumental in enhancing the clarity and overall presentation of the paper. We acknowledge the generous funding and computational resources provided by Google Research, erucial for the completion of our work.
References
Asadi, K. and Williams, J. (2016). Sample-efficient deep reinforcement learning for dialog control. arXiv preprint arXiv:1612.06000
Bahl, L.. Jelinek, F., and Mereer, R. (1983). A maximum likelihood approach continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, (2):179-190.
Banijamali, E., Abbasi- Yadkori, Y., Ghavamzadeh, M., and Vlassis, N. (2019). Optimizing over restricted policy class in mdps. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3042-3050. PMLR.
Boutilier, C., Cohen, A., Hassidim, A., Mansour, Y.. Meshi, O., Mladenov, M., and Schuurmans, D. (2018). Planning and learning with stochastic action sets. In Proc. of the 27th International Joint Conf. on Artificial Intelligence, pages 4674-4682.
Bradtke, S. J. and Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1-3):3--57.
Carta, S., Ferreira, A., Podda, A. S., Recupero, D. R., and Sanna, A. (2021). Multi-dqn: An ensemble of deep q-learning agents for stock market forecasting. Expert systems with applications, 164: 114:113820
Chien, J. and Kuo, C. (2019). Markov recurrent neural network language model. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 807-813. IEEE.
Chow, Y., Tulepbergenov. A., Nachum, O., Ryu, M., Ghavamzadeh, M., and Boutilier, C. (2022). A mixture-of-expert approach to rl-based dialogue management. CoRR, abs/2206.0000
Chu, W., Li, L.,, Reyzin, L.,, and Schapire, R. (2011). Contextual bandits with linear payoff functions.
In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208 214. JMLR Workshop and Conference Proceedings.
Danescu-Nicule Ccculescu-Mizzle C. and Lee, L. (2011). Chameleons in imagined conversations: A new ap- proach to understanding coordination of linguistic style in dialogs. arXiv preprint arXiv:1106.3077
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, 1C1L, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018.
Ghandeharioun, A., Shen, J., Jaques, N., Ferguson, C., Jones, N., Lapedriza, A., and Picard, R. (2019). Approximating interactive human evaluation with self-play for open-domain dialog systems. Advances in Neural Information Processing Systems, 32.
Greensmith, E., Bartlett, P., and Baxter, J. (2004). Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9).
Haarojja, T.. Zhou, A., Abbeel, P., and Levine. S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. ICML.
Hurley, N. and Rickard. S. (2009). Comparing measures of sparsity. IEEE Transactions on Informa- tion Theory, 45210):4723-441.
Jaques, N., Ghandeharioun, A., Shen, J., Ferguson. C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 0rX1v:1907.00456.

---

Koenker, R. and Hallock, K. F. (2001). Quantile regression. Journal of economic perspectives, 15(4):143-156.
Kostrikov, I., Nair, A., and Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.
Levin, F. and Pieraceini, R. (1997). stochastic model of computer- human interaction for learning dialogue strategies. In Eurospeech, volume 97, pages 1883-1888 Citeseer.
Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2015). A diversity-promotiin objective function for neural conversation models. arXiv preprint 151.:055..
Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., and Jurafsky, D. (2016). Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.0155: 0151..
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., and Jurafsky, D. (2017). Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1001.m
Liao, W., Zeng. B., Yin, X., and Wei, P. (2021). An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta. Applied Intelligence, 51(6):3522-3533
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600
Saleh, A., Jaques N., Ghandeharioun, A., Shen, J., and Picard, R. (2020). Hierarchical reinforcement learning for open-domain dialog. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 8741-8748.
Serban, I., Sankar, C., Germain, M., Zhang, S., Lin, Z., Subramanian, S., Kim, T., Pieper, M., Chandar, S., Ke, N., et al. (2017). A deep reinforcement learning chatbot. arXiv preprint 1709. 023499
Shah, P., Hakkani-Tur, D., Liu, B., and Tür, G. (2018). Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 41-51.
Shi, W. and Yu. Z. (2018). Sentiment adaptive end-to-end dialog systems. arXiv preprint 1rXiv:1804.1073..
Shin, J., Xu, P., Madotto, A., and Fung, P. (2020). Generating empathetic responses by looking ahead the user's s sentiment. ICASSP 2020-2020 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 7989-7993. IEEE.
Singh, S., Litman, D.,, Kearns, M., and Walker, M. (2002). Optimizing dialogue management with reinforcement learning: Experiments with the njfun system. Journal of Artificial Intelligence Research, 16: 103-
Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. (2022). Offline for natural language generation with implicit language learning. arXiv preprint arXiv:2206.1 11871.
Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence sequence learning with neural networks. Advances in neural information processing systems, 27.
Sutton, R., McAllester, D., Singh, S., and Mansour, Y. (1999a). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12.
Sutton, R. S., Precup, D., and Singh, S. (1999b). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell.
Toomarian, N. and Bahren, J. (1995). Fast temporal neural learning using teacher forcing.
12

---

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A., Kaiser, L., and Polosukhin, (2017). Attention is all you need. Advances in neural information processing systems, 30.
Verma, S., Fu, J., Yang, M., and Levine, S. (2022). Chai: chatbot for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08422.
Walker, M. (2000). An application of reinforcement learning to dialogue strategy selection in spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416.
Watkins, C. C. and Dayan, (1992). Technical note q-learning. Mach. Learn.
Williams, and Young, S. (2007). Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):9934422.
Wolf. T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A.. Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface's 's transformers: State-of-the-ar natural language processing. CoRR, abs/1910.03771.
Young. S., Gašić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., and Yu, K. (2010). The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language, 50-1744
Zhang, Y., Sun, S., Galley, M., Chen, Y., Brockett, C., Gao, X., Gao, J., Liu, J., and Dolan, B. (2019). Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:191..0555
Zhao, T., Xie, K., and Eskenazi, M. (2019). Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. arXiv preprint 1902.09920085555
13

---

A
Additional Results
A.1
Diversity over all agents and Datasets
30
Histogram of Expert Selection (Reddit) (a)
30
20
25
Histogram of Expert Selection (Cornell) [b)
15
1OL
EnHO
Divergence (Reddit)
KL Divergence (Cornell) (d]
0.1
0.00
0.00
0.04
0.02
SAIOL
FILE
McE-VRL
Figure 3: Diversity for all agents (a) Reddit with Model-based approximation, (b) Cornell with Model-based approximation, (c) and (d) depict the KL divergence of all agents w.r.t. to uniform distribution for Reddit and Cornell.
A.2 Different Metries for MoE-LM's
To measure the quality of LMs learned in MoE-LM measure the following three metrics, similar
to Chow al. (2022) for 25 generated utterances. Diversity measured as I-paasiiyy Hurley and Rickard (2009) of the singular values of the embedded utterances, Gram- (1,2,3) Li et al. (2015):
Ratio unique (uni, bi, tri) -riam generated utterances, and finally Perplexity Bahl al. (198)).
Table 4: Diversity, Gram- (ram-(112.33) (1,23) and Perplexity of the primitive LM on Reddit casual and Cornell
Dataset
Diversity
Gram-1 1
Gram-2
Gram-3
Perplexity
Reddit 0.14 + 0.05
0.35
0.77
0.90
17.34 43.87 + 28.81
38.81
Cornell 0.12 + 0.04
0.31
0.60
0.79
Table 5: Quality of each expert trained on Reddit casual Comeel with respect to their trained label.
Dutaset
Question
Exploration
Positive Sent
Negative Sent.
Sent. Coherence
Joy
Opinism
Anger
Sadness
Redai
0.95 4: 0.27
0.47 = 0.21
3.29 + 0.33
1 + 0.38
0.51 4 0.40
1.90 1: 0.38
1.25 = 0.43
1.48 1: 0.39
2.01 + 0.66
Cursell
1.58 0.39
0.33 0.17
3.55 0.99
0.5
0.69 0.40
2 0.71
2.11 0.99
0.69
3.45 0.83
A.3 Sample Utterances for Reddit Chitchat
A.3.1
McE-VRI.
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been day but in your case I am not sure. agent: how are you doing?
2. <conv history> user: have my test tomorrow morning, second try. agent: Failed first try because parallel park I was too far away from the curb. user: Hmm my driving test didn't 't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do maneuver and the one my tester gave was parallel park.
14

---

3. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: It's always a good distraction for things like this. user: Being able to use your phone is crucial. agent: I can't 't wait to dive into something more physical though it's been while since I felt distracted by internet. user: Good Good points! agent: I'm glad to hear the science behind writing has been very active. user: The past few days I was reading about some writers in America who seemed very much embrace the science of writing! agent: It's really refreshing to see another view from another perspective, user: There's 's nothing really care about as person. agent: I' I'm glad to hear the internet is bringing so much joy to people of needle, and it's really encouraging to read stories about other people doing the same thing to make the internet more loving and sustain the quality of the medium.
4. <conv history> user: Yeah feel you. agent: 28 here never been in a relationship. user: I've asked dudes out and it always ends with "sorry, not attracted to black girls" agent: I'm considering going out with one of those user: If an encounter deserves to be raped, then your defense would be ridiculous. agent: If lady wants to cheat on you, then so be it. user: Years of my life in general have been destroyed by women who are unable to be trusted. agent: If you want a real relationship. you go get laid. user: This is my experience.
5. <conv history> user: Paris has a lot of culture! agent: Paris is cool ass city. user: I wish I had spent more time spending my time there. agent: What do you like about the people and cultures there? user: If you don't tt mind me asking. agent: I do not have lot of life in the country. user: I have been France in line up for almost a decade, and loved it. agent: My main local family is super mixed, some are pagan, some only listen to Irish and British music, some pray for Jesus, some hold superstitious beliefs, but generally don't bother me. user: think I picked out a lot of interesting stuff on my trip! agent: I like the cultural diversity of which there is, one cultural thing noticed is that of France, America has very little diversity.
6. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good reason to get a good job so you really don't 't have to worry about the future. user: But to answer the question, yes. agent: listen to music pretty much 24/7 and I'm at school at 9:45, so I let my mind play with the music would be at 8. user: Not exactly, but in a gradual way. agent: am also a homebody too. user: That's why I need good music to relax and do you know of good albums/artists to listen to? agent: :)
7. <conv history> user: It truly is incredible agent: Is this an amazing gift for your lil buddy? user: Absolutely, it is an amazing gift that you will remember for years afterward agent: Is this an insta post?
8. <conv history> user: It's nice to see something positive that happened in a simple way. agent: Hey, positive comments are the best. user: It happened to me and it's nice that people still follow in the interests of ordinary people. agent: I I'm glad you're 're into the hobby of collecting things made from scratch and you're re a trustworthy and generous person. user: Thank you for being interesting. agent: The world needs more positivity and being thankful to share is one of the best things can offer as an example. user: Thank you for being interesting. agent: Hey, it's the best way to be! user: I'm glad we could qualify somehow. agent: I really admire you for doing time tests on things and putting in the effort to challenge them to new things.
A.3.2 FILE
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been day but in your case I am not sure. agent: what do you mean?
2. <conv history> user: have my test tomorrow morning, second try. agent: Failed first try because on parallel park was too far away from the curb. user: You should be able to test fast and pass it with time to ease yourself into the mindset of having to assess the difficulty of the test. agent: You have my utmost confidence m. user: Yep, I' mm test-whoring. agent: Here's a few more points for different folks: - know people who are struggling with dumb questions, and who have dealt with really tough life far. user: I have told them that think their question was wise but not asking, so I was happy to see it is up and running. agent: I understand where you't re coming from. user: I were musician for 10+ years, playing some classics and have a million interests that could be put together into something complex. agent: Sometimes I wonder if I'd be making money these days.
15

---

3. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: I just finished a really long work order and I'm obsessed with being in a finished story. user: I'm also actively seeking inspiration from other writers when I'm reading and needs recommendations. agent: I really can't remember if I've actually appreciated when I've finished a book, but think I've been wanting to write it. user: I've had a few writers you' 'ee mentioned that currently reading and I really like their stories. agent: Esp be of having to focus on plot more heavily than usual. user: Sometimes it's a bit of slog others it's a great story to go back on later.
4. <conv history> user: Yeah feel you. agent: 19 soon to go user: To wish you good luck in your studies! agent: Thank you! user: Thanks!
5. <conv history> user: Paris has a lot of culture! agent: Paris is filled with French people and there's 's a lot of French culture. user: I love the French bakeries. agent: What is your favorite cake shop and why? user: Have you had the potato soup? agent: What your favorite cake shop? user: While like all the cafes and farmers' exchanges, I'm really into cafes and diners diners mostly. agent: I like the ones close to the office and the ones per person. user: enjoy the food and general ambiance of most cafes and diners, but my favorite diner's are the small diners in the back of the hall of the old capital city. agent: The milk and pastries at these small diners are my favorite during the weekend.
6. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good reason to get a bike. user: But if you have a car, then yeah, go for it. agent: Mine is only an hour and 15 minutes but I normally work at least that. user: Plus it's fun to walk their. agent: But don't do it like did. user: Mine only short an hour and 15 minutes. agent: Mine is an hour and 8.5 hours. user: What do you do if you live that much time?
agent: Mine an hour and 8.5 hours because I live near a very scenic city.
7. <conv history> user: It truly is incredible agent: Is this an amazing gift for your dad? user: My dad's mom :). agent: She really deserves user: It's bit unlike the other gifts I've had. agent: It's still an just awful gift if you ask me what I think after more than 15 minutes of use haha. user: Yes it an amazing gift. agent: I am so glad she invited me!! user: I hope your dad gets just as excited about it as mine will be. agent: :)
8. <conv history> user: It's nice see something positive that happened in a simple way. agent: Hey, OP! user: Good post! agent: I've a few for now but would like to second the positive buzz suggested by some guys. user: Good post! agent: My thoughts on what's 's going on with the rest of the world as we know it. user: It's nice that we can have society agree that things are moving better than they quite possibly are not, agent: know guys super pessimistic in certain circles and love it to remind them that there are plenty of positive trends without them. user: I saw one other post today about lipstick on the playlist and it made me want to do a experiment to see what types of lipstick people normally wear!
A.3.3 IQL
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been day but in your case am not so sure. agent: how are you going to be feeling when you get up? <conv history> user: I have my test tomorrow morning, second try. agent: Failed first try because driving test didn't 't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do maneuver and the one my tester gave park.
parallel park I was too far away from the curb. user: Hmm my
parallel
2. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: I just finished a really long work order and I'm obsessed with being in a finished story. user: I'm getting into the story ares of a lot of popular characters and the way they're 're composed and how the story are follows. agent: It's my favorite story so far. user: I also recently started playing the game Horizon: Zero Dawn and I've found it to be really great. agent: In general, I really like characters from different genres. user: I'm also really liking What remains of agent: Very well said. user: think that's 's one of the best ways spend story. agent: The game doesn't have its own story structure as a lot of games do, but the dialogue itself is lot more lively than most RPGs and it's (in my opinion) very immersive.
3. <conv history> user: Yeah I feel you. agent: 19 soon to go user: Dang, congrats! agent: That must be one hell of journey... user: Yeah it is, hopefully after my vasectomy I can get
16

---

it fixed. agent: They were 36 in a while, but don't t know how long they've 've been off for, just over a year at this point.
4. <conv history> user: Paris has a lot of culture! agent: Paris is filled with French people and there's a lot of French culture. user: If you're re interested in learning about the city think you're going have a hard time in agent: I'm US photographer currently living in Paris. user: I've been to Paris a little bit, but never tried the city. agent: I mainly get what you mean like people with a story, but I have a few friends who are passionate about the city. user: Maybe we shared something that made them feel more passionate about the city, since they share a similar artistic sense. agent: Oh definitely! user: Paris's ss got a lot of culture! agent: I'm a US photographer living in Paris, and I go there mainly for work.
5. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good reason to get a bike. user: But if it is just for fun and to get out of the house, then why even have a car? agent: It is a good way to make new friends and find a routine that you like to do. user: Though, a lot of people don't ride bikes where live, so just consider it trade-off. agent: Maybe in your area, you could get work truck and move your weekend to an apartment or house. user: Thanks for the suggestions! agent: might work on Monday to Thursday because I live near really awesome university and there is lot of stuff to do. user: Thanks for the response! agent: Maybe off the beaten path, but if you ever get bored of where you live (aka the lower 4 miles to 9), then it's a good way to save some money too.
6. <conv history> user: It truly is incredible agent: Is this an amazing gift for your dad? user: My dad's mother did it for him when he was in a great place, and he's 's always dreamed of sharing it with him. agent: All of my aunties and cousins talk about it, my childhood friends still cherish it from their time, and no matter how difficult life challenges and stresses me out, always remember that must one day present the gift to my dad. user: It was a leather cap for his birthday that he really liked, something that he had worn off to him knew was actually his. agent: I'm m really excited that I get to be the only one with a gift for him. user: It definitely fills me with joy and pride. agent: This is a great gift for your dad and a great way to show him how much appreciated he is! user: It's quite the thing to do, but my dad would have felt even more proud if he knew! agent: It's really cool just knowing what people have accomplished in what 's his day.
kid he
7. <conv history> user: It's nice to see something positive that happened in a simple way. agent: Hey, OP! user: That's 's nice to read. agent: I'm glad you posted it and hope your day gets better. user: Life can be hard sometimes, agent: But it's worth seeing a positive message of the kind kind in those times. user: Thanks for sharing. agent: tend to be pretty negative it's nice to made improvements, OP. agent: Kind of funny because I' I'm Another positive change to see the odds of good in what we'll 'll call "life hacks".
positive effect that others have me user: Really glad you
A.3.4 SAIQL
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been a day but in your case I am not so sure. agent: how are you doing?
2. <conv history> user: have my test tomorrow morning, second try. agent: Failed first try because on a parallel park I was too far away from the curb. user: Hmm my driving test didn't 't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do a maneuver and the one my tester gave me was parallel park.
3. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: I never actually had much experience with books in school and I'm reading Lord of the Rings for the first time. user: I'm also playing Wizard Legend (1999) again. agent: Haven't I even been able to finish the second one yet. user: Hey, I'd totally be interested in that you get around to it. agent: it still as good as remember? user: I read it for the first time when I was younger but then it kind of sucked but I guess the novelty of the story kept me going since it was different tone and style than the other. agent: Wizard of Legend is definitely a good read, then. user: agree that it keeps you talking, writing is great way to put it. agent: You can spend long hours on the wiki page, learning stuff on every major concept, just amazing writing.
4. <conv history> user: Yeah feel you. agent: 28 here never been in a relationship. user:
I' ve asked dudes out and it always ends with "sorry, not attracted to black girls" agent: I'm
17

---

definitely bordering on racist, user: I'm afraid of myself. agent: I'm anxious of as much racism as there is racism. user: I'm afraid of myself. agent: About three years ago there was video on my Facebook from my mate posted a picture of black young guy with his hand on her shoulder and I instantly went to the guy's 's house and then looked at his hand on her shoulder, laughed and told him that's how was kinda proud of him for that either he's okay with black girls I'm just not attracted. user: Yet here he is going out of his way to touch my face, hold my hand, look at my face, etc. agent: somehow don't 't think he attracted anything, I think he is attractive (his friends describe him pretty attractive, him having a huge voice and looking interesting) and maybe he saw me being attractive and wanted to do something about it.
5. <conv history> user: Paris has lot of culture! agent: You'll have correct peoples assumptions, but personally would strongly recommend Paris to you. user: It's got a lot of culture. agent: And open world is must for me! user: If I didn't 't have to deal with French! agent: How is... Paris lot of people don't 't know someone from Paris, or maybe they have friends there/ stayed there for 6 months.
6. <conv history> user. Is your daily commute to school/work long and tedious? agent: That's a good way to get to where work. user: But don't 't mind. agent: It's the weekends for me. user: Not really fine fine fine. agent: I'm hopeful that I'll be able to get to the job I want by next summer. user: Thanks! agent: Nice, the new home the place is going to be at least decent. user: Might be better to go in and ask if you can live with them till then agent: It's not a bad commute, and not bad all the time.
7. <conv history> user: It truly is incredible agent: Is this an adorable pretendr? user: It's the only thing my block that actually feels like I'm serious about not breaking up with somcone. agent: Okay, so don't 't know if this is an r/aww thing or what, but I always cutey thinking someone is sweet and thoughtful, especially when you how genuine they are. user: It always makes me a smile to see their faces and it's quite enough if they respond to it! agent: :)
8. <conv history> user: It's nice Hey, OP! user: That's 's nice to read. agent: I've been like that before. user: Let's just say I read it and it really is nice that someone apparently never had deal with whatever it was and just decided to write about it agent: Yeah, you're right! user: This probably doesn't help the situation, but think the OP was afraid to write about it and like the fact that we occasionally assume the other person was the positive person all the time. agent: Yeah, always makes things feel better, especially when you can just put a smile on someone's 's face and think about what can be said.
something positive that happened in simple way. agent:
B Experimental Details
This section describes more details about our experimental setup evaluate the algorithms.
B.1 Model parameters and Description
Language Model Description We make use of the MoF-2 model as described in Chow et al. 2002 which is based on transformer Vaswani et al. (2017). This variant of MoE had shown diversity in its utterances while retaining semantic fluency with low perplexity. The model was not too large that would become too costly to use it while training. We are repeating the details of the model over here for ease of the user, but the details remain the same from Chow et al. (2022).
Our MoE uses the simple transformer architecture, where the model parameters are summarized in Table 6
Latent distributions } are implemented as FFN that model mean and variance of the normal distribution. We use a target entropy of 1.0. The parameters for FFN are captured in Table (note: FFN has a final layer without activation).
18

---

Table Simple Transformer Architecture
Parameter
Value
Number of layers
Embedding hidden size
256
FFN inner hidden size
512
Attention heads
8
Key size
256
Value size
256
Dropout
0.1
Table 7: (G,) FFN architecture
{G.} FFN parameter
Value
Number of layers
Activation
tanh
FFN Hidden Size
128
B.2 Computational resources
Training and evaluation were run on 8 GPU instances with 32GB of RAM and a NVIDIA Tesla P100 graphics card. Training each experts takes around 2-3 days, and training each RL take around 12 hours.
B.3 Dataset
Our models were developed using two conversational datasets, namely Reddit Casual and Cornell Movie. We obtained these datasets from the Neural Chat datasets of the MIT Media Lab, which available at the following link: :/affett//fftt.....// wwd.ameiaaa....m mit edu/neural. http:/uuualacccataaeeeeee that/datasets These datasets comprise conversations between two speakers and each batch of training data consists of subset of these conversations. The Reddit Casual dataset is approximately three times larger than the Comell corpus.
B.4 Offfine RL Training & Details
Table summaarizes the ceper-parameters that used for training the Q functions.
Table 8: Hyper parameters for training the RL agents.
Hyper Parameter
Value
Number of layers (Q, V)
Activation
ReLU
Hidden Size
512
Epochs
100
Max Unroll
30
Batch Size
256
Learning Rate
2 x 10-3
Optimizer
Adam
(IQL)
0.9
Dropout (EnsQ, KICC)
0.5
We depict the minor implementations differences between the baseline RL methods that were implemented for comparison in Table These tricks are often overlooked and we provide them here for the sake of completeness.
19

---

Table 9: Implementation details of different Offline RL methods
Method
Multiple Q
Dropout
Target
Target
Learn Policy
Entropy Regularization
Behavior Policy Regularization
IQI.
No
Yes
Yes
Yes
Yes
SAC
No
Yos
Yes
Yes
Yes
Yes
Yos
EnsQ
Yes
KLC
Yes
No
No
Yes
No
No
No
B.5
Expert Label Functions
We have used gamut of expert language models, which constitute experts having wide array of emotions and characteristics. The first six experts are sentiment- where quantify the sentiment, we have used state-of-at art sentiment classifier, i.e. RoBERTa Liao et al. (202I) The sentiment detector outputs 2 types of prediction. The first set corresponds positive, negative, and neutral, and the second prediction corresponds to 4 emotions i.e. [joy, optimism, sadness, anger).
We define the 6 sentiment labeling functions as Semssemm()) sen (en) (Y), fjay (Y): faptimism (Y). (Y)) landness (Y), which outputs a score that depends on sentiment prediction probability of any candidate bot utterance.
The remaining 4 experts deal more with conversational traits, including sentence coherence sseut-coh [X, Y). question expert Lquestion (Y'i (Y), to improve user engagement by asking questions. Finally, to encourage the agent to able to change the topic, we provide a final reward signal which allows the agent give exploratory utterances through eepp (X,) Y)
B.6 Model Scale Description
The number of parameters used by each expert LM is set to be the same, namely 0 = 42M for the MoE. The number of parameters used in the and V functions also the same, namely o 16M, and ø' 12M.
Table 10: Number of parameters for different algorithms,, m is the number of experts
Algo Name
Number Params
IQL
2 + (m = 2)0
SAC
26 (m+2)0
EnsQ
20 + (m -2))
KLC
26 + +-m+++++m-- (m+2)0
SAIQL
26' + +mm++m+pp
FILE
20' + +mm =mm+2)
MoEVRL
30' + [m mm+22 (m+
20

---

C Flow Chart
Figure 4 describes the flow of training of the MoE framework along with RL components, starting from Phase up to Phase 3.
D
Yg
8
Phase Self Supervised Learning (Section 3, Eq 1)
V
Φ Y
9
Phase 2 Bandit Method (Section 3, Eq 2)
X
+
W
g 1
g
Y
gP
Phase 3
Offline RL (Section 4 & 5)
X
z
H
Figure 4: Flow Chart between different phases of the training procedure.
D Human Evaluation Experiments
We recruited 80 workers to provide a total of 600 ratings of the bots' quality, terms of fluency, and conversation-leve sentiment improvement the Reddit Casual ChitChat dataset. Evaluating these language models with humans particularly tests these models' capabilities on generalization, since humans have the final say in judging whether model response is natural or not. Annotators asked to evaluate the fluency and sentiment improvement (over the conversation) each individual sample scale of 0 to 1. For example, in the fluency rating 0 corresponds to "not fluent all" and 1 corresponds to "very fluent". We obtain 600 annotations to evaluate different agent LMs trained for the Semtinent-improveenee improvement.
To evaluate the quality of sentiment improvement (for chit chat) in our language models, we conducted human evaluations on two metrics: (i) task success sentiment improvement and (ii) fluency. In particular, let N be the number of conversations used for evaluating arbitrary language model,
21

---

You are given this conversation between customer and agent:
user: here,
day sure.
agent: tired.
user: you has day your
agent: what today did you do good thing today?
user: Cleaned 7-year-old old son's room.
agent: Been there was good.
shared
Indicates required question
How many sentences look gibberish?
Examples of gibberish:
pizza not sure", "Table chair cream"
"that one of my favorite songs by the time make are".
Examples of NOT gibberish:
am not this not
"oh get a similar band together and love the same style of movies."
"I thought was gon na say that haha"
Choose
Does the conversation have positive sentiment (e.g. joyful, optimistic, happy)?
Examples of positive sentiment:
weather today"
-"have good day"
Explanation: Both of the sentences cheerful and optimistic.
Examples of NOT positive sentiment:
"i am tired and depressed"
Explanation: Both of the sentences depressing.
Choose
Figure 5: Evaluation Template for Human Rater Experiment for Fluency and Sentiment Improvement
Stack (N) be the number of conversations that the task achieved. For Reddit Chat, the task metric measures user's 's overall sentiment improvement and the score is between [0, 1]. Out of the total of N conversations, the final task metric is given by Susk (N)/N. For fluency, let G(N) be the number of incomprehensible conversations out of the total of N conversations, then the fluency metric is given by (1 G(N))/N. To test for generalization, for each task and each language model under evaluation we randomly generated N = 100 user-agent conversations that has not been seen in training, saved each Google form (whose format can be found in Figure and employed raters
22

---

to obtain (N) and G(N) for all the language model and skill pairs. Results are summarized in Table 3
23

---

E Limitations and Broader Impact
In this paper, we delve into the application of offline reinforcement learning (RL) algorithms specif- ically tailored for Mixture-of-Exper (MoE) dialogue management frameworks. However, due to the primary emphasis constrained to smaller language models with a capacity of approximately 20-30 million parameters. worth noting that larger language models have demonstrated tendency generate more coherent conversations. Consequently, a comprehensive evaluation of the MoE's potential utility in this context would benefit from investigating the impact of larger language models, which could provide further insights into the topic at hand. Yet, it is possible that when used maliciously, our proposed MoE- based dialogue management approach could be deployed to produce explicit or violent content (by exploiting ways train experts with such dangerous behaviors), or output fraudulent or plagiarized information. Finding principled ways to resolve these issues are key directions for future work.
exploring the concept of employing offline RL., experiments were
24


Reviews:
[{"review_id": "eF0RBAlJNN", "reviewer": "Reviewer_T4hv", "summary": "The authors address offline RL training of dialogue models. They note\nthat gathering trajectories with humans-in-the-loop is potentially\nexpensive (and perhaps even dangerous). Their method is a MoE\ninvolving a general LM and several specialized intent-accounting-for\nLMs, which generate candidates for a dialogue manager to select from:\nthis process reduces the action space significantly (because selecting\nthe next utterance becomes a task at the utterance level, rather than\nat the word level). The reward is a roberta sentiment classifier\napplied to simulated dialogue responses. Experiments compared to\nseveral single-policy/MoE methods show the proposed method is better\nable to elicit positive sentiment responses in simulation.", "questions": "- What is the reward model from Liao et al. 2021 trained on?\n- Is it possible to run human evaluation or include discussion of the outputs to make sure reward hacking isn't occurring?\n- Why the latent variable model instead of, e.g., just encoder/decoder? It seems a bit orthogonal, and some simpler baselines suggested by this encoder/decoder choice are missing.\n- Can better discussion be added for potential limitations of this particular reward in this particular setup?", "limitations": "No, see above.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors study an interesting problem --- offline RL for dialogue,\nand frame the task well --- DialoGPT offers a nice simulation\nenvironment, and I understand the motivation for the reward function\nthe authors optimize. I commend the authors for working in a\nmultidisciplinary area, mixing RL and dialogue modeling contributions.\nThe experiments cover a broad range of RL algorithms, as well as 2\ndialogue corpora.", "weaknesses": "- I would have liked to have seen some human evaluations of response\n  quality/elicited sentiment (I think the question that\n  operationalizes the reward may be: \"Which model's response would\n  cause you to write a more positive response?\"). Current evaluations\n  are limited to reward optimization, which makes sense from an RL\n  perspective. But, at the very least (if a small-scale human\n  evaluation were prohibitive), it was odd to see a paper about\n  dialogue with no dialogues shown.\n\n- The reward seems a bit reductive --- 1) What is the RoBERTa\n  classifier trained on? Unfortunately, Liao et al. 2021 is locked\n  behind a paywall so I can't check. I know sentiment models are often\n  trained on specific domains (e.g., movie reviews or yelp reviews),\n  so it might not generalize well (I am fearful of reward hacking);\n  and 2) should it really be the case that the optimized reward be\n  entirely based on sentiment? This seems like it might risk just\n  pushing the dialogue manager to be a sycophantic \"yes-man\". I would\n  have appreciated some justification for this choice. (in fact, few\n  potential limitations of the work are discussed --- this type of\n  reflection would have been appreciated).\n\n- The model itself seems a bit convoluted --- the idea of compressing\n  the primitive encoder's output into a single vector (and then\n  sampling from a mixture of normal distributions conditioned on that\n  encoding as the MoE), felt a bit roundabout. Why not, e.g., not\n  pool, and train a separate decoder for each expert, ala T5? Latent\n  variable modeling is cool, but it felt superfluous, even to the core\n  message of this paper (which, for me, focuses on MoE for state space\n  reduction) --- I at least would have liked to have seen the ablation\n  of simpler methods, e.g., over-generation with differently prompted\n  LMs in a zero-shot way.\n\n\nUPDATE:\n\nthe authors do have human eval (and will surface from the appendix) and made a few more clarifications in response). I have raised my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. Please find individual responses to comments below:\n\n### Human evaluation\nWe conducted a human evaluation of the generated conversations by asking raters to score how our MoE bot managed to improve the overall sentiment within the conversations with different RL algorithms. The results can be found in  Appendix E (Table 10) which aims to compare the overall fluency and sentiment improvement of the conversation. Details of the rater evaluation can be found in the Appendix. This experiment sheds light on how our chosen automatic evaluation metrics align with human evaluations on demonstrating how our offline RL methods, when paired with MoE models, can improve the overall conversation objective. We will also move these experimental results into the main paper (given the extra page provided), as they are important to demonstrate the advantages of using specialized offline RL methods. Additionally, Appendix A2 contains evaluations of the MoE embedding space and different experts (Tables 3 and 4), echoing the original MoE paper by Chow et al., 2023.\n\n### Sample dialogues: \nWe acknowledge the reviewer’s comments on the lack of sample dialogues. We focused on displaying quantitative studies in the original submission. We included a snippet of sample dialogues in our rebuttals to showcase the effectiveness of our methods in sequential conversations, and we will add more sample dialogues in Appendix E in the final version. \n\n### Sentiment Classification Model \nWe regret the confusion regarding the referencing of sentiment classifiers. In our work, we utilized the HuggingFace RoBERTa model trained on the Twitter dataset to recognize sentiment (https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest). Initially, we cited Liao et al., 2021, aiming for specificity in sentiment classification models.\n\nAcknowledging the reviewer's concern about potential confusion, we will update the reference to TimesLM (Loureiro et al., 2022), an open-source model utilized beyond sentiment analysis. It's essential to clarify that our paper's primary focus is on introducing specialized offline RL methods for dialogue management within the mixture of expert frameworks. The selection of the sentiment classifier and reward design are experimental decisions, unrelated to the underlying RL algorithms.\n\n###  Reward Hacking and Model Generalization:\nWe acknowledge the reviewer’s concern about reward hacking in sentiment optimization. As mentioned above, our experiments are meant for demonstrating the effectiveness of different offline RL dialogue management methods rather than developing a full-blown universal conversation bot. We decided to optimize w.r.t. user sentiment transition partially because of Table 2b of Jaques et al., 2020, which experimented with the same 2 conversation datasets and showed that the user sentiment signal is the most correlated with real human feedback of conversation quality (measured w.r.t. rater’s upvotes).\n\nTo avoid language model overfitting, our agent adopted the MoE framework that restricts dialogue responses to be selected from among the set of dialogue utterances generated by experts of different skills (e.g., positive/negative sentiment, semantic coherence/diversity, question, etc.). Experiment results show that our RL method tends to select a more diverse set of experts (Figure 2a) to avoid sycophantic responses. Human evaluation (Appendix E) further shows that our agent tends to be more fluent and leads to user sentiment improvement over multiple turns.\n\n### MoE framework; Shared latent is superfluous; Separate decoder per expert\n\nThe MoE framework's core idea lies in representing multiple experts within a language model across various parts of the semantic latent space. This allows the experts to generate candidate utterances in a modular way that is suitable for downstream dialogue management tasks. This approach not only lessens computational demands by enabling a range of responses with different intents but also streamlines the token-level MDP formulation in dialogue RL, resulting in more effective management. For a detailed motivation, refer to the original work by Chow et al., 2023. Our research builds on this, focusing on enhancing offline RL capabilities, and we'll enrich the paper's introduction with more insights into MoE-LM.\n\nThe shared encoder in the MoE framework, trained for both accuracy and diversity, encodes the conversation history into a versatile embedding space. This serves as the foundation for expert utterance generation (phase 2) and MoE-MDP state space (phase 3). Specifically, it permits (i) the creation of responses with varied intents by sampling from specific latent space regions, and (ii) RL planning in a reduced, continuous state space (see Eqn 4-6 in our paper). Utilizing the same decoder minimizes distribution shifts during data generation, simplifying RL training compared to non-MoE approaches (refer to the original paper's Table 2 for details and ablation studies).\n\n### Better reward discussions\nOur RL reward choices follow primarily from the open-domain offline RL dialogue management paper: Jaques et al., 2020, and the MoE paper: Chow et al., 2023. Both for fluency/coherence (quality) and sentiment improvement (task success). We will add additional discussions regarding RL reward choices in the final paper. \n\n### References\nChow, Y., Tulepbergenov, A., Nachum, O., Ryu, M., Ghavamzadeh, M., & Boutilier, C. (2022). A Mixture-of-Expert Approach to RL-based Dialogue Management. https://arxiv.org/abs/2206.00059 \n\nJaques, N., Shen, J. H., Ghandeharioun, A., Ferguson, C., Lapedriza, A., Jones, N., Gu, S. S., & Picard, R. (2020). Human-centric Dialog Training via Offline Reinforcement Learning. http://arxiv.org/abs/2010.05848\n\nLoureiro, D., Barbieri, F., Neves, L., Anke, L. E., & Camacho-Collados, J. (2022). TimeLMs: Diachronic Language Models from Twitter. https://arxiv.org/abs/2202.03829\n"}, {"review_id": "bxhdZO5gEk", "reviewer": "Reviewer_3gha", "summary": "This paper introduces multiple reinforcement learning algorithms for dialogue management, in particular when combined with mixture-of-expert language models. Generally, a primitive (general) language model, as well as expert language models which have a specific intent or personality, generate candidate utterances. The dialogue management module then learns to choose among them.", "questions": "Could you clarify \"Our experiments demonstrate that model-based evaluation can significantly improve dialogue management over the model-free counterpart [...]\"? Does the evaluation method change the DM policy?\n\nCould you describe the evaluation approaches in more detail? What do the reported numbers exactly represent?\n\nCould the approach be adapted to work without a primitive LM (i.e. only expert LMs with a specific personality)?\n\nDo you have examples of conversations generated with different approaches?\n\n[L49/50] Component #1 is repeated.", "limitations": "Some limitations are discussed in the appendix.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces multiple novel RL algorithms tailored for dialogue management.\n\nUsing RL-based dialogue management over mixture-of-expert LMs is well-motivated. The language models may generate fluent and diverse outputs, while the limited size of the action space allows for efficient learning.\n\nCoordinating multiple language models (or especially a single language model with different adaptors/parameter-efficient fine-tuning modules) could be impactful by improving generation quality and diversity given a fixed number of parameters.", "weaknesses": "The description of the algorithms is quite dense. As multiple approaches are introduced, a more thorough discussion of which one(s) to prefer under certain conditions would be helpful.\n\nThe number of turns is fixed in the experiments. Conversations could be much richer if the conversation length was more flexible.\n\nWithout sharing the code, some of the experiments may be difficult to reproduce.\n\nThe results in tables 1 and 2 are difficult to interpret.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for the useful feedback and appreciation of the prospect of our work. Please find individual responses to comments below.\n\n### Dense algorithmic descriptions \nOur contributions are on developing several MoE-specific offline RL algorithms and comparing different offline RL approaches on MoE dialogue management, which results in the sheer amount of technical details in the main paper. We will shift some of these details to the appendix and add more intuitive explanations of different methods (when one is preferred over another in practical scenarios) in the final paper. \n\n### Fixed conversation turns in experiments\nIn our experiments, the conversation turn has been fixed as a way to evaluate the different methods. This quantity can be regarded as the horizon of the RL problem. The underlying number of conversation turns can still vary because the user/agent can indicate an end of the conversation by outputting an EOS token before the maximum conversation turn is reached. To check for the quality/richness of our generated conversations, we also evaluate our methods in raters studies whose results are given in Appendix E.\n\n### Tables 1 and 2\nWe acknowledge the reviewer’s confusion caused by the condensed presentation of these tables and will include more detailed explanations in the final paper. These tables present the average return (discounted sum of per-turn rewards in the dialogue conversations) of the dialogue agent of interest, accumulated over a 5-turn conversation. The return is averaged over 100 conversations, and the standard error is also provided.  A higher value indicates the corresponding agent is able to perform better dialogue planning and results in better overall sentiment improvement.  \n\n\nTable 1 and Table 2 respectively compare the results between offline RL methods (that are not necessarily adapted to the MoE framework) and offline RL methods that are specifically designed with the MoE framework in mind. We can see that MoE-specific offline RL methods can manage to perform significantly better than standard offline RL methods in our dialogue management experiments. \n\n### Model-based/free evaluation\nModel-based and model-free approaches differ in the way of policy extraction. In the case of model-free methods the MoE policy only relies on the learned Q-function as the scoring function, while in the model-based experiments the policy is constructed as the softmax the value-to-go function (V function) that corresponds to to the augmented conversation history w.r.t. the predicted next-user utterance (given by the utterance prediction model that takes conversation history and candidate utterances as inputs)\n\nDetails of evaluation approach: Each numerical result in our evaluation is the averaged return of 100 conversations driven by the RL agents of interests. As mentioned in Line 328, this metric corresponds to the discounted sum of the reward for each dialogue turn. The reward function is defined on the user sentiment score in the next conversation turn (affected by current conversation history and the bot response) as described in Line 318.\n\n### Without primitive LM\n(Please let us know if we mis-interpret your question) Yes, while our offline RL approach is built on top of the more general MoE framework that consists of a universal representation space that embeds diverse semantics and a gamut of various expert generators, it does not rely on the details of these experts. Therefore, our approach can also be directly applied to a (simpler) setting when there is only an expert LM with a particular personality.\n\n### Code sharing\nUnfortunately due to IP concerns at this point our institution has not approved our request for open-sourcing code. We already indicated that restriction in our initial submission checklist and will try to release the code by the final submission timeline. In the meantime, we tried our best to provide detailed explanations about experimental setup, model architectures, and RL training procedures in Appendix B to D (and we also follow the same MoE implementation details illustrated in the original paper: Chow et al., 2023 ) so that the reader can implement these concepts.\n\n### Sample conversations\nWe apologize for not including the sample conversation, we have included some sample conversation in the extra PDF posted above and will include that in the final paper. \n\n### Typos \n\nThanks for catching this, we will correct it in the final draft.\n\n### References\nChow, Y., Tulepbergenov, A., Nachum, O., Ryu, M., Ghavamzadeh, M., & Boutilier, C. (2022). A Mixture-of-Expert Approach to RL-based Dialogue Management (arXiv:2206.00059). arXiv. https://arxiv.org/abs/2206.00059 \n"}, {"review_id": "nVXnmKYEvu", "reviewer": "Reviewer_nRAM", "summary": "The authors of the paper proposed a suite of off-line reinforcement learning methods utilizing Mixture-of-Expert Language Models (MoE LMs) to train dialogue management agents. Moreover, they experimented their RL methods on two open-domain dialogue datasets and showed better overall performance of their methods (MoE specific offline RL) over SOTA offline RL methods.", "questions": "See weakness section.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: In their experimental section, it is shown that their MoE specific RL methods outperformed the SOTA offline RL methods.\n\nS2: Human evaluation is also done by recruiting 80 workers.\n\nS3: It is shown that those agents which have better performance utilize all the knowledge of different experts in a balanced way.", "weaknesses": "W1: In the conclusion section of the paper, it is stated that their specialized offline RL methods have better sample efficiency, however, I did not see any experimental proof for it.\n\nW2: In the introduction section (page 2), they described their first component of their methods two times: “Our methods consist of three main components: 1) a primitive LM which, using a probabilistic encoder and decoder, is capable of generating diverse semantic intents 1) a primitive LM that uses a probabilistic encoder-decoder pair to generate sentences with diverse semantics and intents”.\n\nW3: In the section 6 (page 7) before experiment 1, the second appendix is not referenced properly: “More details and results can be found in Appendix E and ??”\n\nW4: The implementation is not provided so that one cannot reproduce their results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for the useful feedback aimed at improving our paper.  Please find individual responses to comments below.\n\n### Conclusion indicates improved sample efficiency\nThe improved efficiency arises from both modeling and algorithmic choices. First,  we adopt the MoE LM framework (Chow et al., 2023) and greatly simplify the action space of the RL dialogue management problem, as we no longer rely on RL to directly control the token-level autoregressive generation of the language model but rather to select the best utterance to output at the current conversation turn from the pool of candidate responses generated by MoE-based model. Second, we developed our offline dialogue RL planning algorithms for MoE-LMs under the IQL offline RL methodology, which has been shown in Snell et al., 2023  to have improved performance. While these factors intuitively improve sample efficiency, we acknowledge the reviewer’s comment that explicit analysis has not been conducted in our paper, and therefore we will update the conclusion section in the final paper to clarify these points and soften our claims about sample efficiency improvement.\n\n\n### Typo in the introduction section\nThanks for catching this, we will remove the duplicate texts in the final draft.\n\n### Reference in the experiment section\nWe apologize for the formatting issue, we meant to say Appendix A and E, we will fix the appendix references in the final version of the paper.\n\n### Implementation details are unclear\nUnfortunately due to IP concerns at this point, our institution has not approved our request for open-sourcing code. We indicated that restriction in our initial submission checklist and will try to release the code by the final submission timeline. In the meantime, we tried our best to provide detailed explanations about experimental setup, model architectures, and RL training procedures in Appendix B to D (and we also follow the implementation details of the original MoE paper: Chow et al., 2023), so that the reader can implement these concepts.\n\n### References\n\nSnell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2023). Offline RL for Natural Language Generation with Implicit Language Q Learning (arXiv:2206.11871). arXiv. https://arxiv.org/abs/2206.11871\n\nChow, Y., Tulepbergenov, A., Nachum, O., Ryu, M., Ghavamzadeh, M., & Boutilier, C. (2022). A Mixture-of-Expert Approach to RL-based Dialogue Management (arXiv:2206.00059). arXiv. https://arxiv.org/abs/2206.00059 \n"}, {"review_id": "YmYNyHfumv", "reviewer": "Reviewer_BhVQ", "summary": "This paper tackles issues with using RL for dialogue management related to covariate shift when using offline RL and the requirement for many online human-bot interactions when using online RL, and both suffering from a large action space. Current NLP models lack the ability to plan dialogue interactions beyond the next interaction, which is a crucial aspect of successful conversation. To address these points, the authors propose a mixture-of-expert language model (MoE-LM) and design several offline RL algorithms with different benefits. The structure of the MoE-LM is a hierarchical one, where there are a number of experts each optimised for a different intent (things like empathy, rage, etc.) and a dialogue management model that chooses one of the expert utterances conditioned on the conversation history. This latter aspect should help with non-myopic objectives that are naturally part of conversations. The authors compare their MoE-LMs trained with several different algorithms to SotA offline RL algorithms as well as behavior cloning and a bandits method that greedily optimises the next conversation turn (just reward maximisation). On two datasets, they show their method outperforms the offline RL baselines in terms of return in a simulated conversation (DialoGPT). They also show that the best performing RL algorithm for MoE-LMs which they propose has a more uniform selection of experts than the worst performing one.", "questions": "Answers to the questions written in Weaknesses would address my main concerns with this paper.\nSome small other questions/suggestions that are not related to my score:\n- I don't really follow contribution 2; how does leveraging pre-trained LMs and prior regularization result in high-level dialogue management?\n- I would rewrite line 58-62, very hard to parse with usage of \"-- --\" twice\n- Line 102 should be textual citation (citet not citep)\n- Table 1 and 2 need more comprehensive captions. Without reading the main text a reader should understand what the numbers and error bars refer to. What are your methods, what are the takeaways from these numbers, etc.", "limitations": "There's no separate limitations section or discussion of the limitations of your proposed method; that would be a welcome addition.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Very clear and detailed explanation of the algorithms, well-motivated and important problem. Strong results of proposed method compared to baselines on two different dialogue tasks.", "weaknesses": "Some of the claims seem to not be substantiated by the results of the experiments:\n- It seems likely that the MoE-LMs generate more diverse conversations, but strictly you can't assume that based on a higher return in terms of sentiment; can you supplement the results with metrics that evaluate diversity?\n- It's not clear to me from experiment 2 that the MoE method is able to do long-term planning. Again, it seems likely given the performance increase over the offline methods and especially the Bandit, but simply based on the higher return in sentiment this can't be claimed. Can you isolate / evaluate the effect of long-term planning? How do we know from these results that the method is actually better at long-term planning and the performance increase is not due to other aspects that result in a better sentiment?\n- I might have missed it but there does not seem to be an explicit treatment of the sample-efficiency of your method over others; can you quantify this?\n\nIt's pretty hard to interpret these results based solely on sentiment calculated over conversations with a simulated user. A baseline that would help interpret this to some extent is a simple prompted LLM to hold these conversations; what kind of sentiment would it achieve? Additionally, can you do a small human eval comparing your method to bandits and the best-performing offline RL method? Perhaps this can be combined with the questions above about diversity and long-term goal achieving / planning, asking humans to rate the conversations along these axes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the useful feedback. We will address the individual comments in the following shortlist of responses.\n\n### Diversity of MoE responses; Diversity metrics\nWe concur with the reviewer that high sentiment improvement in a conversation doesn't necessarily equate to utterance diversity. Though our return metric doesn't directly promote diversity, it is facilitated by the diverse semantic representation space and various candidate utterance generators within the pretrained MoE LMs. Empirically, as seen in Figure 2 and 3, the MoE-specific offline RL methods, while scoring some of the best returns, also tend to select responses from more diverse experts. This implies a greater variety in utterances. Conversely, utterance diversity doesn't guarantee high conversation return. For instance, the agent trained with the KLC offline RL method produces diverse responses but ranks among the worst performers, as shown in Table 1.\n\n### Can the MoE method perform long-term planning; How to justify the efficacy of long-term planning\nThe reviewer astutely questioned the long-term planning capabilities of the MoE-based RL method. Initially, we didn't compare with bandit-based baselines, as the advantages of RL-based methods for long-term planning are well-established in dialogue management (see Snell et al., 2023, Jaques et al., 2020). We followed similar considerations to Jaques et al., 2020, using the same datasets.\n\nTo address the question, we evaluated the MoE-VRL offline RL method against a bandit agent, using a discounting factor of 0.0 (gamma=0.0), on the Cornell dataset. The 5-step cumulative model-based return for the bandit agent was **1.53**, significantly lower than the MoE-VRL results at a discounting factor of 0.8 (**3.62**). This experiment highlights the effectiveness of RL-based MoE dialogue management for long-term planning. We will include a discussion and the results in the final paper to highlight the benefit of long-term planning.\n\n### Sample efficiency\nThe improved efficiency arises from both modeling and algorithmic choices. First,  we adopt the MoE LM framework (Chow et al., 2023) and greatly simplifying the action space of the RL dialogue management problem, as we no longer rely on RL to directly control the token-level autoregressive generation of the language model but rather to select the best utterance to output at the current conversation turn from the pool of candidate responses generated by MoE-based model. Second, we developed our offline dialogue RL planning algorithms for MoE-LMs under the IQL offline RL methodology, which has shown in Snell et al., 2023  to have improved performance. While these factors intuitively improve sample efficiency, we acknowledge the reviewer’s comment that explicit analysis have not been conducted in our paper, and therefore we will update the conclusion section in the final paper to clarify these points and soften our claims about sample efficiency improvement.\n\n### Baseline comparisons with prompting LLMs\n\nComparing the responses of our MoE-LM agents with the ones generated by prompting LLMs is an interesting direction for future research. \n\nWe would like to bring to the reviewer's attention that (i) rather than developing SOTA chatbots, the motivation of the work is to research different offline RL methods that make MoE LMs effective for multi-turn dialogue management; and (ii) the MoE LMs in our experiments are much smaller (~42M parameters) than standard LLM-based chatbots that generate diverse responses with various persona (e.g., a full GPT2 model has 1.5B parameters). With such a difference in model sizes, one may not expect the current MoE LMs can match the behaviors of any commercialized chatbots. On the other hand, it is already quite impressive to see that MoE LMs do possess different language skills, persona and have the capabilities to “smartly” switch among different language skills to improve the conversation.   \n\n### Human evaluation on diversity and RL performance\nAppendix E provides a human evaluation of different offline RL methods w.r.t to fluency and sentiment improvement in the overall conversation. Performance comparison of RL-based agents and the myopic, bandit counterpart has only been done via their corresponding cumulative returns (see above comments). We will add the human evaluation of the bandit agent in the final paper.  \n\n### Contribution 2\nThis contribution summarizes our arguments about the sample efficiency (see above comments) improvements of specialized offline RL methods for dialogue management with MoE-LMs, via leveraging the MoE structure to simplify the RL dialogue planning problem and utilizing specialized offline RL methods to better solve this problem. We will clarify the presentation of contribution 2 in the introduction by including the above explanations.\n\n### Table 1 & 2\nWe acknowledge the reviewer’s confusion caused by the condensed presentation of these tables and will include more detailed explanations in the final paper. These tables present the average return (discounted sum of per-turn rewards in the dialogue conversations) of the dialogue agent of interest, accumulated over a 5-turn conversation. The return is averaged over 100 conversations, and the standard error is also provided.  A higher value indicates the corresponding agent is able to perform better dialogue planning and results in better overall sentiment improvement. \n\n### Formatting issues\nThanks for suggesting several modifications to improve the readability of our paper, we will incorporate them in the final draft.\n\n\n### References \nSnell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2023). Offline RL for Natural Language Generation with Implicit Language Q Learning. https://arxiv.org/abs/2206.11871\n\nChow, Y., Tulepbergenov, A., Nachum, O., Ryu, M., Ghavamzadeh, M., & Boutilier, C. (2022). A Mixture-of-Expert Approach to RL-based Dialogue Management. https://arxiv.org/abs/2206.00059 \n"}]

Author rebuttal:
## Sample Utterances

We acknowledge the reviewer’s comments on the lack of sample dialogues. We focused on displaying quantitative studies in the original submission. We included a snippet of sample dialogues in the attached PDF to showcase the effectiveness of our methods in sequential conversations, and we will add more sample dialogues in Appendix E in the final version.




Meta-review: This paper proposes off-line RL methods using Mixture-of-Expert (MoE) LM (involving a general LM and several specialized intent-accounting-for LMs) to train dialogue policy and outperform SOTA offline RL, showing that those agents can utilize all the knowledge from different experts in a balanced way.

This paper provides a reasonable and more flexible solution for a dialogue manager to learn its policy. By leveraging the capability of LMs, this approach can allow a dialogue agent to be better controlled and also take advantage of powerful LLMs (e.g. ChatGPT; end-to-end conversational models). This direction may attract a lot of interest because leveraging diverse powerful LLMs may be a practical solution for building commercial products.
Decision: Accept (poster)