SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multiple techniques in value-function parameterization have been employed to tackle the overestimation bias. Fujimoto et al. (2018) proposed maintaining two  $Q$ -functions, and a dual  $Q$ -function chooses the minimum value between them to avoid overestimation. Jaques et al. (2019) applies dropout in the  $Q$ -function to maintain an ensemble of  $Q$ -values, and outputs the minimum value to avoid overestimation. By utilizing these methods within the MoE-LM framework, we can propose the following variants of offline RL algorithms: (i) SAC that uses a dual  $Q$ -function and the actor-critic updates in (4)-(6), (ii) EnsQ that uses an ensemble of  $Q$ -functions and the same updates; and (iii) KLC that uses an ensemble of  $Q$ -functions and a latent KL-regularized actor-critic update.
Apart from the actor-critic approach that iteratively improves the value functions and policy, Implicit  $Q$ -Learning (IQL) (Kostrikov et al., 2021), a value-based offline RL algorithm, has recently shown success in tackling various problems, including task-oriented dialogue management (Snell et al., 2022). Within our MoE-LM framework, we propose the IQL-DM algorithm, whose value function  $V(z)$  minimizes the following loss:  $L_{V} = \mathbb{E}_{(z,z_{a})\sim \Phi (\mathcal{D})}[L_{2}^{\tau}(Q_{\mathrm{tar}}(z_{a}) - V(z))]$ , where  $L_2^\tau$  is the
expectile regression operator (Koenker and Hallock, 2001) of estimating the top- $\tau$  expectile statistics. The  $Q$ -function of IQL-DM is updated identically to that of actor-critic in (4), which estimates  $Q(z_{a}) \approx r + \gamma V(z_{+})$  via a least-square loss (Bradtke and Barto, 1996). The  $V$ -function estimates the top- $\tau$  quantile of the state-action  $Q(z_{a})$  random variable at every latent state  $z$ . As  $\tau$  approaches one,  $\tau \rightarrow 1$ , the IQL updates converge to the optimal  $Q$ -function,  $Q^{*}(z_{a})$ , i.e.,  $\mathbb{E}_{(z_a,r,z_+) \sim \Phi (\mathcal{D})}[(r + \gamma \max_{a_+} Q^*(z_{+},a_+) - Q^*(z_a))^2] \to 0$ , where  $z_{+}, a_{+} = \Phi((\mathbf{X}, a, X_{+}, a_{+}))$  for any next-action utterance  $a_{+}$ . Intuitively, IQL leverages the generalization capacity of critic functions to estimate the value of the best action without directly querying the values of unseen actions. This makes it less conservative than most offline RL methods that constrain the policy's actions to be in-distribution via behavior regularization (e.g., SAC, EnsQ, KLC).
Auto-regressive Decoding in Actor-Critic: The actor-critic methods (SAC, EnsQ, KLC) ameliorated the two issues in offline RL to a certain extent (the inner maximization is replaced with  $V$ -function learning and covariate shift is controlled by policy entropy regularization). However, implementing these methods (Eqs. 5-6) entails sampling utterances from the current policy, i.e.,  $\hat{a} \sim \Psi \circ \mathcal{G}$ , which involves expensive auto-regressive LM decoding at every training update. To resolve this issue, one may empirically replace  $\Psi \circ \mathcal{G}$  with a teacher-forcing (Toomarian and Bahren, 1995) variant  $\Psi_{\mathrm{TF}}(a) \circ \mathcal{G}$ , which replaces auto-regressive decoding with a one-step generation from the bot utterance  $a = Y$  in  $\mathcal{D}$ . This will further restrict the policy update of  $\mathcal{G}$  to be close to the behavior policy. In contrast, since IQL does not perform explicit policy updates, it directly circumvents this expensive auto-regressive sampling operation of  $\hat{a}$ .
DM Construction in MoE-LMs: Recall that in an MoE-LM, the DM policy  $\mu$  takes the encoded conversation history  $z = \Phi(\mathbf{X})$ , the  $m + 1$  candidate action utterances generated by the experts  $\{\widehat{Y}_i\}_{i=0}^m$ , and selects one of them to execute, i.e.,  $a \sim \mu(\cdot \mid z, \{\widehat{Y}_i\}_{i=0}^m)$ . Given the  $Q$ -function  $Q(z_a)$  learned by any of the above offline RL algorithms, we extract the DM policy  $\mu$  via softmax greedification over the finite set of MoE candidate utterances, i.e.,  $\mu(a \mid z, \{\widehat{Y}_i\}_{i=0}^m) \propto \exp(\beta \cdot Q(z_a))$ , where  $\beta > 0$  is the policy temperature. This DM policy uses the  $Q$  function to score different candidate utterances and returns an utterance based on the likelihood of these scores.
5 Mixture-of-Expert Offline RL
In Section 4, we presented how state-of-the-art offline RL methods are adapted to the MoE framework, which can have limitations due to being agnostic to the model architecture. Recall that MoE dialogue management is a specialized hierarchical reinforcement learning (HRL) problem, which optimizes over a restricted class of DM policies defined by the convex hull of expert policy set (whose weights are defined by the DM policy  $\mu$ ). This problem is of great interest because it reduces the original RL DM problem, with a combinatorial action space, into one that has a much smaller finite action set. In the following, we leverage the mixture-of-policy structure and develop offline RL algorithms that specifically target this HRL problem.
Stochastic-action IQL (SAIQL): The first approach applies IQL to the discrete, stochastic set of candidate action utterances  $\{\widehat{Y}_i\}_{i=0}^m$  as generated by the MoE experts. Equipped with the latent conversation data  $\Phi(D) = \{(z, z_a, r, z_+)\}$  (see Section 4) and the latent expert policies  $\{\mathcal{G}_i\}_{i=0}^m$  in the MoE-LM, we propose a DM algorithm, whose value function  $V(z)$  minimizes the following loss:
$$
L _ {V} = \frac {1}{m + 1} \sum_ {i = 0} ^ {m} \mathbb {E} _ {z, \hat {a} _ {i} \sim \Psi \circ \mathcal {G} _ {i} (\cdot | z)} \left[ L _ {2} ^ {\tau} \left(Q _ {\operatorname {t a r}} \left(z _ {\hat {a} _ {i}}\right) - V (z)\right) \right], \tag {7}
$$
where  $z_{\hat{a}_i} = \Phi((\mathbf{X}, \hat{a}_i))$  is the latent state that corresponds to the action utterance sampled from the  $i$ -th expert,  $L_2^\tau$  is the expectile regression operator, and the  $Q$ -function is updated as in Eq. 4. To incorporate the maximization over candidate utterances in IQL, we compute the expectile regression over the joint latent state and expert policy distributions.
However, unlike the standard IQL DM algorithm, which avoids autoregressive decoding for policy execution, SAIQL requires auto-regressive sampling of all  $m + 1$  candidate utterances. Suppose the augmented latent conversation data  $\Phi (D)_{\mathrm{SA}} = \{(z,z_a,r,z_+,\{z_{\widehat{Y}_i}\}_{i = 0}^m)\}$ , which also includes the set of latent expert actions  $\{z_{\widehat{Y}_i}\}_{i = 0}^m$ , is available. One straightforward way to circumvent this issue is by replacing the expectation over experts with the realized candidate utter
ances, i.e., by approximating the value function in SAIQL with its unbiased empirical average  $\frac{1}{m + 1}\sum_{i = 0}^{m}\mathbb{E}_{(z,\{z_{\widehat{Y}_i}\}_{i = 0}^m)\sim \Phi (\mathcal{D})_{\mathrm{SA}}}[L_2^\tau (Q_{\mathrm{tar}}(z_{\widehat{Y}_i}) - V(z))]$ .
While having access to candidate utterances is not standard in IQL, it is necessary here to allow  $Q$ -learning to exploit quantile regression over realized candidate utterances (an approach shown to be sound in Boutilier et al. (2018)). Therefore, we termed this method stochastic action IQL (SAIQL) to reflect the stochastic action sets used in IQL training. Once SAIQL converges, the DM policy is also constructed as a softmax of  $Q$ -values applied to each candidate utterance.
The MoE-MDP is defined as  $\bar{\mathcal{M}} = (\bar{S},\bar{A},\bar{P},\bar{r},\bar{s}_0,\gamma)$ , where the state space is the product of the learned latent space  $\mathcal{Z}$  and the joint action space of the  $m + 1$  experts, i.e.,  $\bar{S} = \mathcal{Z}\times \mathcal{A}^{m + 1}$ ; the action space consists of the  $m + 1$  experts, i.e.,  $\bar{A} = \{0,\dots ,m\}$ ; the initial state  $\bar{s}_0$  is the encoding of the initial user's query and the utterances suggested by the experts in response to this query; the transition kernel models both the user's responses and the next experts' actions; and finally the reward is the same as in the original MDP. Since MoE-MDP has a finite number of actions, learning a policy  $\lambda$  is equivalent to solving a finite-action MDP, i.e.,  $\lambda^{*}\in \arg \max_{\lambda}J_{\lambda}\coloneqq \mathbb{E}[\sum_{k = 0}^{\infty}\gamma^{t}\bar{r}_{t}\mid \bar{P},\bar{s}_{0},\lambda ]$ .
Follow-the-Leading-Expert (FtLE): Banijamali et al. (2019) showed that the MoE-MDP problem is NP-hard but can be approximated by  $\max_{\lambda \in \Delta^{m + 1}}\sum_{i = 0}^{m}\lambda (i)V^{i}(z) + \mathcal{U}(\bar{\mathcal{M}})$ , where  $V^{i}$  is the value function of the  $i^{\mathrm{th}}$  expert and  $\mathcal{U}(\bar{\mathcal{M}}) > 0$  is a surrogate function that depends on the experts' stationary distributions. However, computing these distributions is generally intractable as the experts are LMs themselves. This motivates our heuristic FtLE algorithm, which ignores the second term in training a set of expert critic functions and picks the best action at each step. To efficiently parameterize the critic function, similarly to the architecture used in DQN (Mnih et al., 2013) for discrete-action RL, we define a  $(m + 1)$ -headed critic function, where each head represents the value of following an expert's policy. To train the multi-headed critic, we modify the standard critic losses as
$$
L _ {Q} = \sum_ {i = 0} ^ {m} \mathbb {E} _ {z, z _ {a}, r, z _ {+}} \left[ (r + \gamma V _ {\mathrm {t a r}} ^ {i} (z _ {+}) - Q ^ {i} (z _ {a})) ^ {2} \right], L _ {V} = \sum_ {i = 0} ^ {m} \mathbb {E} _ {z, \hat {a} _ {i} \sim \Psi \circ \mathcal {G} _ {i} (\cdot | z)} \left[ (Q _ {\mathrm {t a r}} ^ {i} (z _ {\hat {a} _ {i}}) - V ^ {i} (z)) ^ {2} \right], (8)
$$
where  $Q^i$  and  $V^i$  represent the critic-function head for expert  $i$ . To overcome the auto-regressive sampling issue in (8), we relabel the offline conversation data  $\mathcal{D}$  by assigning action utterances to train the critic function(s) whose corresponding expert(s) most likely generate those utterances. Specifically, consider the following V-function loss:
$$
L _ {V} = \sum_ {i = 0} ^ {m} \mathbb {E} _ {z, z _ {a}, Y} \left[ \mathbf {1} _ {i = i (z, Y)} \cdot \left(Q _ {\operatorname {t a r}} ^ {i} \left(z _ {a}\right) - V ^ {i} (z)\right) ^ {2} \right], \tag {9}
$$
where  $\mathbf{1}_{i = i(z,z_a)}$  selects the expert based on the best log-likelihood  $i(z,Y)\coloneqq \arg \max_{i}\log \Psi (Y|z^{i',})$  with  $z^{i,'}\sim \mathcal{G}_i(\cdot |z)$ . After learning the critic functions, the FtLE DM policy can be constructed as  $\mu (a\mid z,\{\widehat{Y_i}\}_{i = 0}^m)\propto \exp (\beta Q^{i(z,a)}(z_a))$
Value-based RL for MoE-MDP (MoE-VRL): Consider a  $(m + 1)$ -headed value function  $\Lambda$  of the MoE-MDP, where each head represents the optimal value by choosing the corresponding expert's action. Applying standard DQN, this function can be learned by minimizing the following loss:
$$
L _ {\Lambda} = \mathbb {E} _ {z, Y, r, z _ {+}} \left[ \left(r + \gamma \max  _ {i _ {+}} \Lambda_ {\operatorname {t a r}} \left(z _ {+}, i _ {+}\right) - \Lambda (z, i (z, Y))\right) ^ {2} \right], \tag {10}
$$
where  $\Lambda_{\mathrm{tar}}$  is the target  $\Lambda$ -network. For simpler exposition, we only use the partial MoE-MDP states of encoded conversations in the above DQN loss and omit the candidate action utterances. Extending to the full MoE-MDP state is straightforward but is omitted for brevity. The inner maximization over  $i_{+}$  can be computed explicitly because the MoE-MDP action space of expert indices is finite and small. Here,  $i(z,Y)$  is the same index function that attributes utterance  $Y$  to the expert most likely to generate it (based on likelihood). With the optimal value function  $\Lambda^{*}(z,i)$ , the MoE-MDP policy picks the best expert  $\lambda^{*}(z) \coloneqq \arg \max_{i} \Lambda^{*}(z,i)$ , and the DM policy can be constructed as  $\mu(a|z,\{\widehat{Y}_i\}_{i=0}^m) \propto \exp(\beta Q^{\lambda^{*}(z)}(z_a))$ , where  $Q^{\lambda^{*}(z)}(z_a)$  is the critic of the optimal expert.
6 MoE-based DM Experiments
We evaluate our MoE-based offline RL algorithms on two open-domain benchmarks common in the RL-based dialogue management literature (Jaques et al., 2019). The first one is the Cornell Movie
corpus (Danescu-Niculescu-Mizil and Lee, 2011), which consists of conversations between speakers in different movies. The second is the Reddit Casual (Ghandeharioun et al., 2019) conversations dataset, which is a subset of the Reddit corpus that only contains casual conversations.
Environment: We perform the experiment by having DM agents interact with a DialoGPT (Zhang et al., 2019) simulated-user environment. The task is to maximize user satisfaction, which is measured by the user's overall sentiment. To construct an immediate reward, we set  $r(X_{+}) \coloneqq \ell_{\mathrm{sent}}(X_{+})$ , where  $\ell_{\mathrm{sent}}(X)$  is a RoBerTa-based sentiment classifier (Liao et al., 2021), which assigns a score from  $[-1,1]$  that is inversely proportional to the (negative) positive sentiment prediction probabilities.
We pre-train the MoE-LM with either the Cornell or Reddit dataset and construct 10 experts (i.e.,  $m = 9$ , plus the primitive expert), each corresponding to an individual intent in open-ended dialogues, including "empathy", "optimism", "cheerfulness", "contentment", "dejection", "rage", "sorrow", "questioning", "exploration", etc. See Appendix B for details. The conversation lasts for 5 turns (with  $\gamma = 0.8$ ), where each turn entails a query/response from the user followed by an agent's utterance. During the agent's turn, each expert generates 5 candidate utterances, thus resulting in a total of 50 candidate utterances. To evaluate the methods, we measure the return of the trajectory generated by different agents via  $\mathbb{E}_{\mathbf{X}_0\sim \mathcal{D}}[\sum_{i = 0}^4\gamma^i r(X_{i + 1})|Y_i\sim \mathrm{LM}(.|\mathbf{X}_i),X_{i + 1}\sim P_{\mathrm{Dialog - GPT}}(.|\mathbf{X}_i,Y_i)]$
Evaluation: We employ two evaluation approaches, namely (i) a model-free approach that only utilizes the learned  $Q$  function to score candidate utterances, and where the DM policy selects the action utterance based on a softmax likelihood; and (ii) a model-based approach that uses the Value function  $(V)$  along with a learned next-user utterance model  $P_{\mathrm{user}}(X_{+}|z_{Y})$ , that optimizes the following loss:  $L_{P_{\mathrm{user}}} = \mathbb{E}_{(z_a,r)\sim \mathcal{D},\hat{X}_+ \sim P_{\mathrm{user}}(\cdot | z_a)}[(r - r(\hat{X}_+))^2]$ . We first approximate the  $Q$  function via  $Q(z_{a}) \approx r(\hat{X}_{+}) + \gamma V(\hat{z}_{+})$ , where  $\hat{X}_{+}$  denotes the next user utterance sampled from  $P_{\mathrm{user}}(\cdot | z_a)$ , then use that function to score candidate utterances, and, finally have the DM policy select the action utterance analogously. Human evaluation is also conducted on the DM performances of different offline RL agents. More details and results can be found in Appendix D.
Table 1: SOTA offline RL methods
Method	Reddit Casual	Cornell
Model Free	Model Based	Model Free	Model Based
IQL	0.53 ± 0.47	4.25 ± 0.12	-1.32 ± 0.19	1.47 ± 0.15
SAC	0.97 ± 0.52	4.13 ± 0.21	-1.55 ± 0.19	0.36 ± 0.26
EnsQ	0.10 ± 0.40	4.06 ± 0.25	-1.51 ± 0.20	0.21 ± 0.21
KLC	0.31 ± 0.46	3.69 ± 0.37	-1.46 ± 0.21	-0.07 ± 0.25
BC	-0.65 ± 0.41	-2.18 ± 0.36
Bandits	4.3 ± 0.16	1.3 ± 0.17
Table 2: MoE specific offline RL methods
Method	Reddit Casual	Cornell
Model Free	Model Based	Model Free	Model Based
EXP 1*	0.97 ± 0.52	4.25 ± 0.12	-1.32 ± 0.19	1.47 ± 0.15
SAIQL	0.81 ± 0.42	4.65 ± 0.06	-1.34 ± 0.25	2.61 ± 0.24
FtLE	1.14 ± 0.49	4.59 ± 0.07	-0.39 ± 0.24	3.51 ± 0.19
MoE-VRL	0.72 ± 0.47	4.46 ± 0.10	-0.58 ± 0.24	3.62 ± 0.17
Experiment 1: SOTA Offline RL with MoE-LMs: The goal of this experiment is to investigate the effectiveness of SOTA offline RL algorithms. In these experiments, we only make use of the primitive language model  $\mathrm{LM}_0 = (\Phi, \mathcal{G}_0, \Psi)$  to generate sample utterances. To simulate previous works using single policy settings, we fine-tune the latent base distribution  $\mathcal{G}_0$  for policy optimization while keeping the encoder-decoder  $(\Phi, \Psi)$  fixed. As mentioned in Sec. 4 we deploy the following offline RL algorithms to train the DM policy  $\mu$  of MoE-LMs: (i) SAC (Haarnoja et al., 2018) with a dual  $Q$  function critic (Fujimoto et al., 2018); (ii) EnsQ, which utilizes an ensemble of  $Q$  functions (Jaques et al., 2019) with actor-critic; (iii) KLC (Saleh et al., 2020), which utilizes the dual  $Q$  function and applies KL regularization between the latent policy  $\mathcal{G}$  and the primitive policy  $\mathcal{G}_0$ ,
i.e.,  $\mathbb{E}_{\mathcal{G}(\cdot |z)}[\log (\mathcal{G}(z'|z) / \mathcal{G}_0(z'|z))]$  in the actor-critic algorithm update; (iv) IQL (Kostrikov et al., 2021), which adopts the idea from Q learning to estimate an optimal  $Q$  function in the MoE-LM latent space. To our knowledge, our work is among the first that makes use of IQL for open-domain dialogue management. These methods have been implemented in ways where the original idea has been preserved, making the comparison fair to the original works. With each learned  $Q$  function, the bot picks the final action by sampling from a softmax distribution of  $Q$  scores overall candidate utterances. To demonstrate the efficacy of offline RL methods, we also include results from Behavior Cloning (BC) as well as simple reward maximization (Bandit) (i.e.,  $\gamma = 0$ ) for comparisons.
Table 1 presents the results of our experiments with these methods in the open-dialogue system, where a 5-turn conversation was generated. The table displays the mean return over 100 conversations with their respective standard errors. Our experiments demonstrate that model-based evaluation can significantly improve dialogue management over the model-free counterpart, even with a next-user LM that is much simpler than the Dialog-GPT user. Among most model-based and model-free evaluations, we found that IQL, originally designed to tackle offline RL problems, outperforms other RL methods. This performance can be attributed to IQL's ability to (i) alleviate  $Q$  overestimation errors due to co-variate shifts; (ii) estimate the optimal values without being overly conservative w.r.t. the behavior policy, and (iii) avert the auto-regressive utterance sampling issues in training.
Interestingly, we also found that KLC and EnsQ, two standard methods in RL-based DM, struggled to achieve satisfactory performance in our experiments. This may be due to the fact that applying dropout (for ensemble  $Q$ ) and KL regularization in the fixed MoE-LM latent space makes DM algorithms overly conservative. In contrast, SAC successfully learns a well-performing model-free DM policy but fails in the model-based regime, potentially demonstrating its instability in critic-function learning. BC also fails to provide any satisfactory performance on any of the domains, and surprisingly, Bandit method or plain reward maximization did as well as IQL, pointing to the fact that maybe the offline RL methods being used or not exactly helping in planning at all.
Experiment 2: MoE-specific Offline RL: In this experiment, we explore the benefits of leveraging the MoE framework for training offline RL agents in open-domain conversational systems. Building upon the insights from our previous experiment (Experiment 1), we propose several modifications to standard Offline RL algorithms to take advantage of the MoE framework. As mentioned in Sec. 5, we developed the following MoE-specific offline RL algorithms for DM: (i) SAIQL, which extends IQL to incorporate the multiple candidate utterances generated by the experts; (ii) FtLE, which learns a DM policy to follow the best expert policy at each step (estimation of the experts' long-term values is done concurrently with a multi-headed critic architecture and data relabeling) and (iii) MoE-VRL, which learns an optimal meta-value function over the space of experts. Leveraging the MoE-MDP formulation, solving which leads to an optimal DM policy that provides the optimal sequences of expert policy switching. We aim to evaluate the potential of these MoE-specialized offline RL algorithms over off-the-shelf offline RL methods in DM.
Table 2 shows the return observed similar to the ones displayed in Table 1. The first row in the table displays the best performance across all methods from Experiment 1, for comparison. Our results demonstrate the efficacy of the proposed methods that utilize the structure of the MoE framework in dialogue management. All the methods that used all experts while training (SAIQL, FtLE, and MoE-VRL) outperformed the SOTA offline RL methods, indicating that an offline RL algorithm that takes the candidate utterances into account can generally improve dialogue planning. Moreover, making the RL algorithms attuned to the multiple-expert structure (i.e., FtLE and MoE-VRL) indeed results in even better DM performance, emphasizing the benefits of reformulating the DM MDP using the HRL paradigm, where the DM policy is optimized over a restricted class of finite-action policies. Also, we note that only MoE-aware offline RL methods were actually able to outperform simple per-step greedification (i.e. Bandit) which hints to the fact that they were actually able to plan ahead and perform long-term credit assignments to optimize return. Whereas all the standard offline RL methods failed to do that (Table 1). Using multiple critic functions to separately estimate the value of different experts also allows us to better understand their long-term utility (of the corresponding intents) and how they affect the conversation quality. Overall, these findings highlight the potential of the MoE-specific offline RL methods to improve dialogue management performance.
Human Evaluation: Table 3 summarize the results of around 600 ratings provided by 80 on the bots' quality, in terms of fluency, and conversation-level sentiment improvement on the Reddit Casual
4The RL DM approach in Jaques et al. (2019) which applies KL regularization at the word-level LM policy is not applicable to our case because our DM policy is defined in the latent space.
Figure 2: Experiment on the Cornell dataset with Model-based evaluation(a) Histogram of frequency of expert selection. (b) KL divergence against uniform distribution.
Table 3: Human raters evaluation
Method	Avg. Fluency	Sentiment
BC	0.67 ± 0.26	0.24 ± 0.50
KLC	0.62 ± 0.27	0.66 ± 0.47
IQL	0.84 ± 0.24	0.72 ± 0.46
SAIQL	0.81 ± 0.19	0.57 ± 0.50
FtLE	0.88 ± 0.24	0.76 ± 0.48
MoE-VRL	0.72 ± 0.28	0.70 ± 0.45
dataset. We can similarly notice in this case that MoE specific offline RL methods seem to perform better both in terms of fluency and sentiment improvement over standard offline RL. The complete details of the experiment can be found in Appendix D.
Experiment 3: In this experiment, one aims to investigate the effectiveness of selecting different experts during dialogue management. To this end, we conduct a study where we measure the frequency with which utterances from different experts are selected throughout the conversation. Specifically, we wish to understand the diversity of intents selected by different offline RL algorithms.
Given approximately 200 conversation turns, we measure the frequency of the expert agents when their utterances are selected and present the frequency metric for the worst performing offline RL method (EnsQ), a good performing method (IQL), and an MoE-specific RL algorithm (such as MoE-VRL). To visualize our findings, we plot a histogram of the frequencies of different experts being selected and calculate the KL divergence of the histogram against a uniform distribution over the experts. While the uniform distribution may not be the optimal distribution of utterances, it provides a measure of how well the agents make use of different experts, along with their performance.