SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2 illustrates the result for the above experiment for the Cornell dataset, in the model based setting. We observe that the worst performing agent, EnsQ, has a highly skewed distribution of expert selections, with a few experts being heavily favored over others. This suggests that EnsQ is less diverse and does not effectively utilize the full range of expert capabilities available. On the other hand, both IQL and MoE-VRL exhibit a more balanced distribution of expert selection, with utterances chosen from multiple experts throughout the conversation; i.e., their frequency distributions are closer to a uniform distribution, with much lower KL divergence distance.
However, note that there is a clear performance gap between MoE-VRL and IQL where former significantly outperforming the latter. This highlights the importance of incorporating the MoE framework to better utilize the intent of different experts in dialogue planning, rather than relying on generating a diverse set of candidate utterances. Overall, these results suggest that encouraging diversity in intents and better utilizing expert knowledge in planning can improve DM performance.
By leveraging the recent advances of MoE-LMs, we developed a suite of offline RL-based DM algorithms. Our methods significantly reduce the action space and improve the efficacy of DM. To understand how well our offline RL approaches generate diverse utterances and solve DM problems, we evaluated them on two open-domain dialogue tasks and compared them with SOTA offline RL baselines. Our results showed that by exploiting the MoE-LM structure, our specialized offline RL DM methods (i) improve the diversity of intents in bot utterances; (ii) have better sample efficiency; and (iii) yield better overall performance in both the model-based and model-free settings. Our work provides important insights on how to create scalable RL-based DM methods that train chatbots to achieve dialogue tasks and enhance user satisfaction. Future work includes fine-tuning the experts (i.e., low-level policies) with offline RL, learning the optimal semantic representation for hierarchical RL, preventing dialogue agents from generating harmful behaviors (e.g., by enforcing safety constraints in the RL algorithms), and evaluating our DM methods on more realistic problems, such as customer support, conversational recommendation, and persuasion.
We express our sincere gratitude to the Mudcats team at Google Research for their invaluable feedback and innovative ideas, which have significantly enriched this project. We would also like to thank the anonymous reviewers, whose insightful suggestions and constructive feedback were instrumental in enhancing the clarity and overall presentation of the paper. We acknowledge the generous funding and computational resources provided by Google Research, crucial for the completion of our work.
Asadi, K. and Williams, J. (2016). Sample-efficient deep reinforcement learning for dialog control. arXiv preprint arXiv:1612.06000.
Bahl, L., Jelinek, F., and Mercer, R. (1983). A maximum likelihood approach to continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, (2):179-190.
Banijamali, E., Abbasi-Yadkori, Y., Ghavamzadeh, M., and Vlassis, N. (2019). Optimizing over a restricted policy class in mdps. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3042-3050. PMLR.
Boutilier, C., Cohen, A., Hassidim, A., Mansour, Y., Meshi, O., Mladenov, M., and Schuurmans, D. (2018). Planning and learning with stochastic action sets. In Proc. of the 27th International Joint Conf. on Artificial Intelligence, pages 4674-4682.
Bradtke, S. J. and Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1-3):33-57.
Carta, S., Ferreira, A., Podda, A. S., Recupero, D. R., and Sanna, A. (2021). Multi-dqn: An ensemble of deep q-learning agents for stock market forecasting. Expert systems with applications, 164:113820.
Chien, J. and Kuo, C. (2019). Markov recurrent neural network language model. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 807-813. IEEE.
Chow, Y., Tulepbergenov, A., Nachum, O., Ryu, M., Ghavamzadeh, M., and Boutilier, C. (2022). A mixture-of-expert approach to r1-based dialogue management. CoRR, abs/2206.00059.
Chu, W., Li, L., Reyzin, L., and Schapire, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208-214. JMLR Workshop and Conference Proceedings.
Danescu-Niculescu-Mizil, C. and Lee, L. (2011). Chameleonons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. arXiv preprint arXiv:1106.3077.
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018.
Ghandeharioun, A., Shen, J., Jaques, N., Ferguson, C., Jones, N., Lapedriza, A., and Picard, R. (2019). Approximating interactive human evaluation with self-play for open-domain dialog systems. Advances in Neural Information Processing Systems, 32.
Greensmith, E., Bartlett, P., and Baxter, J. (2004). Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9).
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML.
Hurley, N. and Rickard, S. (2009). Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723-4741.
Jaques, N., Ghandeharioun, A., Shen, J., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv:1907.00456.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Verma, S., Fu, J., Yang, M., and Levine, S. (2022). Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08426.
Walker, M. (2000). An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416.
Watkins, C. J. C. H. and Dayan, P. (1992). Technical note q-learning. Mach. Learn.
Williams, J. and Young, S. (2007). Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393-422.
Wolf, T., Debut, L., Sanh, V., Chaumont, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Young, S., Gašić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., and Yu, K. (2010). The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language, 24(2):150-174.
Zhang, Y., Sun, S., Galley, M., Chen, Y., Brockett, C., Gao, X., Gao, J., Liu, J., and Dolan, B. (2019). Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.
Zhao, T., Xie, K., and Eskenazi, M. (2019). Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. arXiv preprint arXiv:1902.08858.
A.1 Diversity over all agents and Datasets
Figure 3: Diversity for all agents (a) Reddit with Model-based approximation, (b) Cornell with Model-based approximation, (c) and (d) depict the KL divergence of all agents w.r.t. to uniform distribution for Reddit and Cornell.
A.2 Different Metrics for MoE-LM's
To measure the quality of LMs learned in MoE-LM we measure the following three metrics, similar to Chow et al. (2022) for 25 generated utterances. Diversity : measured as 1 - Sparsity Hurley and Rickard (2009) of the singular values of the embedded utterances, Gram-  $\{1,2,3\}$  Li et al. (2015): Ratio of unique  $\{\mathrm{uni},\mathrm{bi},\mathrm{tri}\}$ -gram in generated utterances, and finally Perplexity Bahl et al. (1983).
Table 4: Diversity, Gram-  $\{ 1,2,3\}$  and Perplexity of the primitive LM on Reddit casual and Cornell
Dataset	Diversity	Gram-1	Gram-2	Gram-3	Perplexity
Reddit	0.14 ± 0.05	0.35	0.77	0.90	38.81 ± 17.34
Cornell	0.12 ± 0.04	0.31	0.60	0.79	43.87 ± 28.81
Table 5: Quality of each expert trained on Reddit casual & Cornell with respect to their trained label.
Dataset	Question	Exploration	Positive Sent.	Negative Sent.	Sent. Coherence	Joy	Optimism	Anger	Sadness
Reddit	0.95 ± 0.27	0.47 ± 0.21	3.29 ± 0.33	1.42 ± 0.38	0.51 ± 0.40	1.99 ± 0.38	1.25 ± 0.43	1.48 ± 0.39	2.01 ± 0.46
Cornell	1.58 ± 0.39	0.33 ± 0.17	3.55 ± 0.99	1.90 ± 0.5	0.69 ± 0.40	2.44 ± 0.71	2.11 ± 0.99	2.71 ± 0.69	3.45 ± 0.83
A.3 Sample Utterances for Reddit Chitchat
A.3.1 MoE-VRL
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been a day but in your case I am not so sure. agent: how are you doing?
2. <conv history> user: I have my test tomorrow morning, second try. agent: Failed first try because on a parallel park I was too far away from the curb. user: Hmm my driving test didn't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do a maneuver and the one my tester gave me was parallel park.
3. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: It's always a good distraction for things like this. user: Being able to use your phone is crucial. agent: I can't wait to dive into something more physical though it's been a while since I felt distracted by internet. user: Good Good points! agent: I'm glad to hear the science behind writing has been very active. user: The past few days I was reading about some writers in America who seemed very much to embrace the science of writing! agent: It's really refreshing to see another view from another perspective. user: There's nothing I really care about as a person. agent: I'm glad to hear the internet is bringing so much joy to people of needle, and it's really encouraging to read stories about other people doing the same thing to make the internet more loving and sustain the quality of the medium.
4. <conv history> user: Yeah I feel you. agent: 28 here never been in a relationship. user: I've asked dudes out and it always ends with "sorry, not attracted to black girls" agent: I'm considering going out with one of those user: If an encounter deserves to be raped, then your defense would be ridiculous. agent: If a lady wants to cheat on you, then so be it. user: Years of my life in general have been destroyed by women who are unable to be trusted. agent: If you want a real relationship, you go get laid. user: This is my experience.
5. <conv history> user: Paris has a lot of culture! agent: Paris is a cool ass city. user: I wish I had spent more time spending my time there. agent: What do you like about the people and cultures there? user: If you don't mind me asking. agent: I do not have a lot of life in the country. user: I have been to France in line up for almost a decade, and I loved it. agent: My main local family is super mixed, some are pagan, some only listen to Irish and British music, some pray for Jesus, some hold superstitious beliefs, but generally don't bother me. user: I think I picked out a lot of interesting stuff on my trip! agent: I like the cultural diversity of which there is, one cultural thing I noticed is that of France, America has very little diversity.
6. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good reason to get a good job so you really don't have to worry about the future. user: But to answer the question, yes. agent: I listen to music pretty much 24/7 and I'm at school at 9:45, so if I let my mind play with the music I would be at 8. user: Not exactly, but in a gradual way. agent: I am also a homebody too. user: That's why I need good music to relax and do you know of good albums/artists to listen to? agent: :)
7. <conv history> user: It truly is incredible agent: Is this an amazing gift for your lil buddy? user: Absolutely, it is an amazing gift that you will remember for years afterward agent: Is this an insta post?
8. <conv history> user: It's nice to see something positive that happened in a simple way. agent: Hey, positive comments are the best. user: It happened to me and it's nice that people still follow in the interests of ordinary people. agent: I'm glad you're into the hobby of collecting things made from scratch and you're a trustworthy and generous person. user: Thank you for being interesting. agent: The world needs more positivity and being thankful to share it is one of the best things I can offer as an example. user: Thank you for being interesting. agent: Hey, it's the best way to be! user: I'm glad we could qualify somehow. agent: I really admire you for doing time tests on things and putting in the effort to challenge them to new things.
A.3.2 FtLE
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been a day but in your case I am not so sure. agent: what do you mean?
2. <conv history> user: I have my test tomorrow morning, second try. agent: Failed first try because on a parallel park I was too far away from the curb. user: You should be able to test fast and pass it with time to ease yourself into the mindset of having to assess the difficulty of the test. agent: You have my utmost confidence rn. user: Yep, I'm test-whoring. agent: Here's a few more points for different folks: - I know people who are struggling with dumb questions, and who have dealt with a really tough life so far. user: - I have told them that I think their question was wise but not asking, so I was happy to see it is up and running. agent: - I understand where you're coming from. user: I were a musician for  $10+$  years, playing some classics and have a million interests that could be put together into something complex. agent: Sometimes I wonder if I'd be making money these days.
A.3.3 IQL
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been a day but in your case I am not so sure. agent: how are you going to be feeling when you get up? <conv history> user: I have my test tomorrow morning, second try. agent: Failed first try because on a parallel park I was too far away from the curb. user: Hmm my driving test didn't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do a maneuver and the one my tester gave me was parallel park.
2. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: I just finished a really long work order and I'm obsessed with being in a finished story. user: I'm getting into the story arcs of a lot of popular characters and the way they're composed and how the story arc follows. agent: It's my favorite story so far. user: I also recently started playing the game Horizon: Zero Dawn and I've found it to be really great. agent: In general, I really like characters from different genres. user: I'm also really liking What remains of agent: Very well said. user: I think that's one of the best ways to spend a story. agent: The game doesn't have its own story structure as a lot of games do, but the dialogue itself is lot more lively than most RPGs and it's (in my opinion) very immersive.
3. <conv history> user: Yeah I feel you. agent: 19 soon to go user: Dang, congrats! agent: That must be one hell of a journey... user: Yeah it is, hopefully after my vasectomy I can get
it fixed. agent: They were 36 in a while, but don't know how long they've been off for, just over a year at this point.
4. <conv history> user: Paris has a lot of culture! agent: Paris is filled with French people and there's a lot of French culture. user: If you're interested in learning about the city I think you're going to have a hard time in agent: I'm a US photographer currently living in Paris. user: I've been to Paris a little bit, but never tried the city. agent: I mainly get what you mean like people with a story, but I have a few friends who are passionate about the city. user: Maybe we shared something that made them feel more passionate about the city, since they share a similar artistic sense. agent: Oh definitely! user: Paris's got a lot of culture! agent: I'm a US photographer living in Paris, and I go there mainly for work.
5. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good reason to get a bike. user: But if it is just for fun and to get out of the house, then why even have a car? agent: It is a good way to make new friends and find a routine that you like to do. user: Though, a lot of people don't ride bikes where I live, so just consider it a trade-off. agent: Maybe in your area, you could get a work truck and move your weekend to an apartment or house. user: Thanks for the suggestions! agent: I might work on Monday to Thursday because I live near a really awesome university and there is a lot of stuff to do. user: Thanks for the response! agent: Maybe off the beaten path, but if you ever get bored of where you live (aka the lower 4 miles to 9), then it's a good way to save some money too.
6. <conv history> user: It truly is incredible agent: Is this an amazing gift for your dad? user: My dad's mother did it for him when he was in a great place, and he's always dreamed of sharing it with him. agent: All of my aunties and cousins talk about it, my childhood friends still cherish it from their time, and no matter how difficult life challenges and stresses me out, I always remember that I must one day present the gift to my dad. user: It was a leather cap for his birthday that he really liked, something that he had worn off to him as a kid so he knew it was actually his. agent: I'm really excited that I get to be the only one with a gift for him. user: It definitely fills me with joy and pride. agent: This is a great gift for your dad and a great way to show him how much appreciated he is! user: It's quite the thing to do, but my dad would have felt even more proud if he knew! agent: It's really cool just knowing what people have accomplished in what's his day.
7. <conv history> user: It's nice to see something positive that happened in a simple way. agent: Hey, OP! user: That's nice to read. agent: I'm glad you posted it and hope your day gets better. user: Life can be hard sometimes. agent: But it's worth seeing a positive message of the kind kind in those times. user: Thanks for sharing. agent: i tend to be pretty negative so it's nice to see a positive effect that others have on me user: Really glad you made improvements, OP. agent: Kind of funny because I'm Another positive change to see the odds of good in what we'll call "life hacks".
A.3.4 SAIQL
1. <conv history> user: 11:15 here, it's been a day for sure. agent: I am so tired. user: For you it has been a day but in your case I am not so sure. agent: how are you doing?
2. <conv history> user: I have my test tomorrow morning, second try. agent: Failed first try because on a parallel park I was too far away from the curb. user: Hmm my driving test didn't have any parking in it. agent: Source: took my test a couple of months ago user: In the UK you have to do a maneuver and the one my tester gave me was parallel park.
3. <conv history> user: I'm getting back into writing and reading to be more immersed in the concept of writers. agent: I never actually had much experience with books in school and I'm reading Lord of the Rings for the first time. user: I'm also playing Wizard of Legend (1999) again. agent: Haven't even been able to finish the second one yet. user: Hey, I'd totally be interested in that if you get around to it. agent: Is it still as good as I remember? user: I read it for the first time when I was younger but then it kind of sucked but I guess the novelty of the story kept me going since it was a different tone and style than the other. agent: Wizard of Legend is definitely a good read, then. user: I agree that it keeps you talking, writing is a great way to put it. agent: You can spend long hours on the wiki page, learning stuff on every major concept, just amazing writing.
4. <conv history> user: Yeah I feel you. agent: 28 here never been in a relationship. user: I've asked dudes out and it always ends with "sorry, not attracted to black girls" agent: I'm
definitely bordering on racist. user: I'm afraid of myself. agent: I'm anxious of as much racism as there is racism. user: I'm afraid of myself. agent: About three years ago there was a video on my Facebook from my mate posted a picture of a black young guy with his hand on her shoulder and I instantly went to the guy's house and then looked at his hand on her shoulder, I laughed and told him that's how I was kinda proud of him for that either he's okay with black girls or I'm just not attracted. user: Yet here he is going out of his way to touch my face, hold my hand, look at my face, etc. agent: I somehow don't think he attracted anything, I think he is attractive (his friends describe him as pretty attractive, him having a huge voice and looking interesting) and maybe he saw me being attractive and wanted to do something about it.
5. <conv history> user: Paris has a lot of culture! agent: You'll have to correct peoples assumptions, but personally I would strongly recommend Paris to you. user: It's got a lot of culture. agent: And open world is a must for me! user: If I didn't have to deal with French! agent: How is... Paris a lot of people don't know someone from Paris, or maybe they have friends there/ stayed there for 6 months.
6. <conv history> user: Is your daily commute to school/work long and tedious? agent: That's a good way to get to where I work. user: But I don't mind. agent: It's the weekends for me. user: Not really fine fine fine. agent: I'm hopeful that I'll be able to get to the job I want by next summer. user: Thanks! agent: Nice, the new home the place is going to be at least decent. user: Might be better to go in and ask if you can live with them till then agent: It's not a bad commute, and not bad all the time.
7. <conv history> user: It truly is incredible agent: Is this an adorable pretendr? user: It's the only thing on my block that actually feels like I'm serious about not breaking up with someone. agent: Okay, so I don't know if this is an r/aww thing or what, but I always cutey thinking someone is so sweet and thoughtful, especially when you see how genuine they are. user: It always makes me a smile to see their faces and it's quite enough if they respond to it! agent::)
8. <conv history> user: It's nice to see something positive that happened in a simple way. agent: Hey, OP! user: That's nice to read. agent: I've been like that before. user: Let's just say I read it and it really is nice that someone apparently never had to deal with whatever it was and just decided to write about it agent: Yeah, you're right! user: This probably doesn't help the situation, but i think the OP was afraid to write about it and I like the fact that we occasionally assume the other person was the positive person all the time. agent: Yeah, it always makes things feel better, especially when you can just put a smile on someone's face and think about what can be said.
This section describes more details about our experimental setup to evaluate the algorithms.
B.1 Model parameters and Description