SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Language Model Description We make use of the MoE-2 model as described in Chow et al. (2022) which is based on transformer Vaswani et al. (2017). This variant of MoE had shown diversity in its utterances while retaining semantic fluency with low perplexity. The model was not too large that it would become too costly to use it while training. We are repeating the details of the model over here for ease of the user, but the details remain the same from Chow et al. (2022).
Our MoE uses the simple transformer architecture, where the model parameters are summarized in Table 6:
Latent distributions  $\{\mathcal{G}_i\}$  are implemented as FFN that model mean and variance of the normal distribution. We use a target entropy of 1.0. The parameters for FFN are captured in Table 7 (note: FFN has a final layer without an activation).
Table 6: Simple Transformer Architecture
Parameter	Value
Number of layers	2
Embedding hidden size	256
FFN inner hidden size	512
Attention heads	8
Key size	256
Value size	256
Dropout	0.1
Table 7:  $\left\{  {\mathcal{G}}_{i}\right\}$  FFN architecture
{G_i} FFN parameter	Value
Number of layers	1
Activation	tanh
FFN Hidden Size	128
B.2 Computational resources
Training and evaluation were run on 8 GPU instances with 32GB of RAM and a NVIDIA Tesla P100 graphics card. Training each experts takes around 2-3 days, and training each RL can take around 12 hours.
B.3 Dataset
Our models were developed using two conversational datasets, namely Reddit Casual and Cornell Movie. We obtained these datasets from the Neural Chat datasets of the MIT Media Lab, which is available at the following link: https://affect.media.mit.edu/neural chatting/datasets. These datasets comprise conversations between two speakers and each batch of training data consists of a subset of these conversations. The Reddit Casual dataset is approximately three times larger than the Cornell corpus.
B.4 Offline RL Training & Details
Table 8 summarizes the hyper-parameters that were used for training the  $Q, V$  functions.
Table 8: Hyper parameters for training the RL agents.
Hyper Parameter	Value
Number of layers (Q, V)	3
Activation	ReLU
Hidden Size	512
Epochs	100
Max Unroll	30
Batch Size	256
Learning Rate	2 × 10-3
Optimizer	Adam
τ (IQL)	0.9
Dropout (EnsQ, KLC)	0.5
We depict the minor implementations differences between the baseline RL methods that were implemented for comparison in Table 9. These tricks are often overlooked and we provide them here for the sake of completeness.
Table 9: Implementation details of different Offline RL methods
Method	Multiple Q	Dropout Q	Target V	Target Q	Learn Policy	Entropy Regularization	Behavior Policy Regularization
IQL	No	Yes	Yes	Yes	Yes	Yes	No
SAC	No	Yes	Yes	Yes	Yes	Yes	Yes
EnsQ	Yes	No	Yes	Yes	Yes	Yes	No
KLC	Yes	No	No	Yes	No	No	No
B.5 Expert Label Functions
We have used a gamut of expert language models, which constitute experts having a wide array of emotions and characteristics. The first set of six experts are sentiment-based, where to quantify the sentiment, we have used a state-of-art sentiment classifier, i.e. RoBERTa Liao et al. (2021). The sentiment detector outputs 2 types of prediction. The first set corresponds to positive, negative, and neutral, and the second prediction corresponds to 4 emotions i.e. {joy, optimism, sadness, anger}.
We define the 6 sentiment labeling functions as  $\ell_{\mathrm{pos - sent}}(Y)$ ,  $\ell_{\mathrm{neg - sent}}(Y)$ ,  $\ell_{\mathrm{joy}}(Y)$ ,  $\ell_{\mathrm{optimism}}(Y)$ ,  $\ell_{\mathrm{anger}}(Y)$ ,  $\ell_{\mathrm{sadness}}(Y)$ , which outputs a score that depends on sentiment prediction probability of any candidate bot utterance.
The remaining 4 experts deal more with conversational traits, including sentence coherence  $\ell_{\mathrm{sent - coh}}(\mathbf{X},Y)$ , question expert  $\ell_{\mathrm{question}}(Y)$ , to improve user engagement by asking questions. Finally, to encourage the agent to be able to change the topic, we provide a final reward signal which allows the agent to give exploratory utterances through  $\ell_{\mathrm{exp}}(\mathbf{X},Y)$
B.6 Model Scale Description
The number of parameters used by each expert LM is set to be the same, namely  $\theta = 42M$  for the MoE. The number of parameters used in the Q and V functions are also the same, namely  $\phi = 16M$ , and  $\phi' = 12M$ .
Table 10: Number of parameters for different algorithms,  $m$  is the number of experts
Algo Name	Number of Params
IQL	2φ + (m + 2)θ
SAC	2φ + (m + 2)θ
EnsQ	2φ + (m + 2)θ
KLC	2φ + (m + 2)θ
SAIQL	2φ' + (m + 2)θ
FtLE	2φ' + (m + 2)θ
MoEVRL	3φ' + (m + 2)θ
C Flow Chart
Figure 4 describes the flow of training of the MoE framework along with RL components, starting from Phase 1 up to Phase 3.
Figure 4: Flow Chart between different phases of the training procedure.
D Human Evaluation Experiments
We recruited 80 workers to provide a total of 600 ratings of the bots' quality, in terms of fluency, and conversation-level sentiment improvement on the Reddit Casual ChitChat dataset. Evaluating these language models with humans particularly tests these models' capabilities on generalization, since humans have the final say in judging whether a model response is natural or not. Annotators are asked to evaluate the fluency and sentiment improvement (over the conversation) of each individual sample on a scale of 0 to 1. For example, in the fluency rating 0 corresponds to "not fluent at all" and 1 corresponds to "very fluent". We obtain 600 annotations to evaluate different agent LMs trained for the Sentiment-improvement.
To evaluate the quality of sentiment improvement (for chit chat) in our language models, we conducted human evaluations on two metrics: (i) task success / sentiment improvement and (ii) fluency. In particular, let  $N$  be the number of conversations used for evaluating an arbitrary language model,
Figure 5: Evaluation Template for Human Rater Experiment for Fluency and Sentiment Improvement
$S_{\mathrm{task}}(N)$  be the number of conversations that the task is achieved. For Reddit Chat, the task metric measures user's overall sentiment improvement and the score is between  $[0,1]$ . Out of the total of  $N$  conversations, the final task metric is given by  $S_{\mathrm{task}}(N) / N$ . For fluency, let  $G(N)$  be the number of incomprehensible conversations out of the total of  $N$  conversations, then the fluency metric is given by  $(1 - G(N)) / N$ . To test for generalization, for each task and each language model under evaluation we randomly generated  $N = 100$  user-agent conversations that has not been seen in training, saved each on a Google form (whose format can be found in Figure 5 and employed raters
to obtain  $S_{\mathrm{task}}(N)$  and  $G(N)$  for all the language model and skill pairs. Results are summarized in Table 3.
E Limitations and Broader Impact
In this paper, we delve into the application of offline reinforcement learning (RL) algorithms specifically tailored for Mixture-of-Expert (MoE) dialogue management frameworks. However, due to the primary emphasis on exploring the concept of employing offline RL, our experiments were constrained to smaller language models with a capacity of approximately 20-30 million parameters. It is worth noting that larger language models have demonstrated a tendency to generate more coherent conversations. Consequently, a comprehensive evaluation of the MoE's potential utility in this context would benefit from investigating the impact of larger language models, which could provide further insights into the topic at hand. Yet, it is possible that when used maliciously, our proposed MoE-based dialogue management approach could be deployed to produce explicit or violent content (by exploiting ways to train experts with such dangerous behaviors), or to output fraudulent or plagiarized information. Finding principled ways to resolve these issues are key directions for future work.