SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\mathcal {C} (\phi) = \max  _ {\phi} \underset {(c _ {h}, c _ {o}, \tau , \alpha_ {h}) \sim \mathcal {D} ^ {\text {o f f}}} {\mathbf {E}} \log P _ {\phi} \left(\alpha_ {h} \mid \tau , c _ {o}\right). \tag {4}
$$
Just like the language modeling objective, this classification objective is averaged over each token in the dialogue sequence. Our full training objective then becomes:
$$
\max  _ {\theta , \phi} \mathcal {L} _ {C T X} (\theta) + \beta * \mathcal {C} (\phi), \tag {5}
$$
where  $\beta$  is a hyper-parameter and  $\mathcal{L}_{CTX}(\theta)$  is the standard context-conditional language modeling objective as defined in Section 3.
4.3 Task Pretraining
As observed by Liu et al. (2021), for some structured tasks, such as table question answering, pretraining on a simplified version of the given task with a synthetic context can help the model to focus learning on the "skills" that are most relevant to utilize the task context, which leads to improved downstream task performance. We instantiate this idea in our method by pre-training our model on a simplified (dialogue-free) version of the task. Instead of simultaneously modeling all the details of the raw dialogue, as is required to learn  $P(\tau | c_{o})$ , the key observation here is that in our case the task reward only depends on the tuple  $\{c_{h}, c_{o}, a_{T}\}$ . This enables us to effectively learn to execute the task by only modeling  $P(c_{h}, a_{T} | c_{o})$ , without any dialogue at all. By pre-training our model to first learn this simplified distribution, we effectively focus on learning the necessary skills for completing the task. It is expected that the skills learned during this pre-training phase should also generalize and transfer when we later perform training on the real dialogue. The particular instantiation of this principle in the case of AirDialogue is described in Section D.
4.4 Model-Based Dialogue Rollouts
While the methodology discussed so far can produce effective policies, language models also represent task dynamics, as discussed in Section 3. We can leverage this fact to further improve the performance of our fine-tuned models by performing model-based planning at test-time, using both the policy and dynamics components in concert to further maximize task reward. A full dialogue trajectory can then be formed by concatenating this sampled future trajectory  $\tau_{\geq t}$  with the current state of the dialogue  $\tau_{< t}$  i.e.,  $\tau = \{\tau_{< t}, \tau_{\geq t}\}$ . We perform the model-based planning by sampling  $k$  such future trajectories from the final fine-tuned model, and ranking them according to an estimated reward function  $\hat{R}(\tau, c_o)$  (see Appendix E.1). Then, we improve upon the policy  $\pi$  from which we took the samples by taking the action (i.e., the next utterance)  $a_t$  which receives the highest estimated reward among the sampled trajectories. This rollout sampling procedure is identical to the one used by Lewis et al. (2017).
5 CALM for AirDialogue
In this section, we instantiate our proposed method, CALM, for the AirDialogue flight booking task (Wei et al., 2018). We first give an overview of the task, and then describe how to do relabeling and context conditioning on this specific task.
5.1 AirDialogue Dataset
Dataset overview. The AirDialogue dataset (Wei et al., 2018) is a recently published large-scale airline reservation dataset based on the aforementioned task. The dataset includes 402,038 conversations. The dataset involves three distinct tasks: booking, canceling, and changing flights. We describe the booking task in detail below.
Flight booking task. The (human) customer is given a set of 12 trip requirements, and the flight agent (bot) is provided with a table of 30 flights. The goal of the flight agent is to book a flight from the table for the customer which meets all their requirements, or to correctly inform them that no such flight is available. To determine task success, the flight agent must predict an explicit action at the end of the dialogue indicating the flight that was booked or inform no flight available. See Figure 7 for an example conversation from the dataset.
5.2 Processing Tables
The AirDialogue booking tasks require efficiently querying a flight table containing flight information (e.g., departing location, ticket price)
given to the agent prior to the conversation. In order to successfully complete the booking task, the agent needs to be able to filter, select, and integrate information from the flight table based on the customer's preferences inferred from the dialogue.
Instead of treating the tables as unstructured sequences (Wei et al., 2018; Jiang et al., 2021) or as SQL databases (Chen et al., 2020), CALM models tables as an observable context consisting of a set  $c_{o} = \{f_{1}, f_{2}, f_{3}, \dots, f_{N}\}$  of table rows. These rows are then input to our model as a set of embeddings (see appendix C and G for more details).
5.3 Relabeling AirDialogue with CALM
While the AirDialogue dataset only includes one flight table for each dialogue, there are potentially many flight tables compatible with each dialogue as each flight can appear in many tables. We hence implement our relabeling procedure as described in Section 4.1 as follows. We perform rejection sampling on the observable context (i.e., the table of flights)  $c_{o} \sim q(c_{o}|\tau, c_{h})$ , sampling until we obtain a new context  $(c_{h}, c_{o}, \tau)$ , which gives maximum reward possible  $R(\tau, c_{h}, c_{o}) = \max_{c_{o}} R(\tau, c_{h}, c_{o})$ . The prior distributions  $p(c_{o})$  and  $p(c_{o}|c_{h})$ , from which the tables in the AirDialogue dataset were sampled, are provided with the dataset. By rejection sampling from  $p(c_{o}|c_{h})$ , we can effectively sample from the posterior  $q(c_{o}|\tau, c_{h})$  within a certain computational budget. In this setting,  $c_{o}$  denotes tables and there are exponentially many tables which correspond to a task success under a given dialogue. Therefore, with our relabeling approach, we increase the number of near-optimal task examples exponentially, which makes it much easier for the language model to learn to query the flight table.
Our relabeling is approximately valid according to the condition specified in Section 4.1. While the customer does not have access to the flight table and therefore is not directly affected by our relabeling, there are still some minor edge-cases in which over-optimism about the dynamics could be learned by our policy. If for example, in the dataset the customer were to occasionally reject the first flight that we suggest, our policy may learn to assign a small probability to the action of initially offering the wrong flight, relying on them subsequently rejecting it such that we can later recover and offer the correct one. However, in practice we observe that these cases are rare in AirDialogue.
success rate
CALM (greedy)	0.88 ± 4e-3
LM(GPT2-small) (greedy)	0.38 ± 1e-3
AirConcierge (greedy)	0.81 ± 7e-3
CALM (planning)	0.90 ± 2e-3
LM(GPT2-small) (planning)	0.74 ± 7e-3
Human	0.88
Table 1: Comparison of our method and baselines across all tasks. Using greedy decoding, our method matches human performance, greatly improving over baselines. Adding roll-outs (32 samples) further improves task completion.
5.4 Table Selection as Auxiliary Loss
The primary high-level action involved in AirDialogue is the decision of which flight table entry, if any, to recommend to the user. We therefore implement our auxiliary objective as a classification head on top of the language model, trained to predict the flight table entry that meets the customer's requests. Specifically, our set of high-level actions  $\mathbf{A}$  is the set of flight table rows  $\{f_1,f_2,f_3,\dots,f_N\}$  plus an additional item  $f_{0}$ , corresponding to the case in which no flights meet the customer's requirements. If  $f^{*}$  is the flight recommended in the dialogue, then our auxiliary objective is:
$$
\mathcal {C} (\phi) = \max  _ {\phi} \underset {(c _ {o}, \tau) \sim \mathcal {D} ^ {\text {o f f}}} {\mathbf {E}} \log P _ {\phi} (f ^ {*} | \tau , c _ {o}). \tag {6}
$$
In this section, we empirically evaluate the performance of CALM on AirDialogue (Wei et al., 2018). We first show that CALM outperforms the SOTA on the AirDialogue dataset by around  $7\%$  in the standard simulated evaluation protocol proposed by Chen et al. (2020), which prior work denotes as "self-play" (see Appendix E), and this matches human-level performance as reported by Wei et al. (2018). Beyond this, we also perform a comprehensive set of ablation studies to validate the necessity of each component of CALM.
Experiment Setup and Baselines. We compare CALM on AirDialogue with two baselines. The first is AirConcierge, the previous SOTA on AirDialogue, which explicitly parses and executes SQL queries from the dialogue (Chen et al., 2020). The other is a standard language model (denoted as LM(GPT2-small)) trained on a dataset filtered for successful task examples, without any of our context-aware language modeling techniques (see Appendix Section A for more details on dataset filtering). CALM uses the fine-tuned GPT2-small model (Radford et al., 2018) as the backbone of the policy and dynamics model. After learning the
Figure 3: Example dialogues generated by CALM (in green) in the simulated evaluation. Despite being end-to-end, CALM produces highly coherent and sensible outputs.
Figure 4: Task success as a function of the number of rollout samples. Note that successful task completion improves with more rollout samples.
dynamics model, both CALM and the LM(GPT2-small) can employ two different planning strategies: (1) a simple greedy decoding of the next utterance (equivalent to beam search with beam-width one) and (2) the rollout planning as described in Section 4.4. For AirConcierge, we only evaluate greedy decoding, as this method cannot be easily adapted for producing full rollouts. Rollout planning requires a method for predicting the reward of a given dialogue, and we describe our specific reward predictor for AirDialogue in Appendix Section E.1.
Results for Task Success. In terms of task success, CALM outperforms the prior SOTA (AirConcierge) by approximately  $7\%$ , achieving  $88\%$  task success when using greedy decoding from the language model (see Table 1). Compared with AirConcierge, where all reasoning about the task context is done outside of the language model, CALM does all of the filtering, selecting, and responding with relevant flight table entries within the language model, in a fully end-to-end manner. Meanwhile, CALM also improves over LM(GPT2-small) by  $50\%$  in terms of task success, indicating the necessity of our context-aware approach for goal-oriented tasks.
CALM	LM(GPT2-small)	AirConcierge
Perplexity	1.63	1.59	-
BLEU	32.88	35.75	27.75
Table 2: BLEU score and perplexity results. CALM improves on task success without sacrificing generation quality.
We further evaluate the performance of various methods, when utilizing the rollout planning technique. As shown in Figure 4, as the number of rollout samples increases, the performance improves for all methods. Remarkably, applying the rollout planning to CALM further increases total task success by  $2\%$ , raising it to  $90\%$  and matching human performance on the AirDialogue task. The baseline LM(GPT2-small) benefits much more from rollout planning than CALM, and we suspect that at around  $90\%$  task completion, the performance becomes bottlenecked by the customer bot's mistakes, therefore we only observe less gain from rollout planning with CALM.
Results for Language Quality. To quantitatively measure the generated language quality, we present perplexity and BLEU for all methods in Table 2. CALM performs similarly to LM(GPT2-small) and outperforms AirConcierge significantly.
Ablation Study. To examine the effectiveness of each single component in our method, we train and evaluate four ablations of CALM. Each of these ablations removes one of the components in our approach: task relabeling (Section 4.1), auxiliary loss (Section 4.2), and table pre-training (Section 4.3). Beyond this, we also examine CALM without both task relabeling and pre-training. As shown in Table 3, removing any one of these components drops task success by at least  $10\%$ , and in most cases much more than that. This shows that each piece of our method plays a critical role in helping CALM to effectively learn the goal-oriented task.
Success Rate
CALM	0.88 ± 4e-3
LM(GPT2-small)	0.38 ± 1e-3
CALM w/o relabel, pre-train	0.42 ± 4e-3
CALM w/o relabel	0.66 ± 1e-2
CALM w/o pre-train	0.39 ± 3e-3
CALM w/o auxiliary loss	0.78 ± 4e-3
Table 3: Task success rate for various ablations of CALM on AirDialogue (all using greedy decoding). Removing any single component from CALM drops performance by at least  $10\%$ .
We proposed an end-to-end framework, CALM, for goal-oriented dialogue systems. Formulating end-to-end dialogue generation as a Markov decision process, CALM employs task relabeling and context-aware finetuning to steer supervised learning of language models towards specific goals, improving task performance drastically while preserving language quality. We show that this improves performance on AirDialogue over the previous state of the art, and matches previously reported human performance under the standard simulated evaluation protocol.
CALM optimizes for task-specific measures of success, and while such measures might be comparatively simple for domains such as AirDialogue, in general specifying the right success measure or reward function may present challenges. Furthermore, as with all methods based on end-to-end language models, CALM is susceptible to internal biases and inconsistencies in the language model itself. There is for example no constraint that ensures that CALM produces truthful answers, or that it avoids harmful or socially unacceptable outputs. A practical deployable dialogue system would likely require additional measures to account for such issues, analogously to how learning-based methods for self-driving vehicles might require some additional safety mechanisms to ensure constraints, and indeed further research on reward specification, ensuring truthful outputs, and other constraint strategies for dialogue systems that combine language models and reward maximization is a promising and important direction.
The context-conditioned supervised learning strategy used by CALM provides for reward maximization, but is in general not optimal for arbitrary reinforcement learning problems: in general RL settings, learning a value function with dynamic programming in general can attain significantly better returns than imitating high-performing trajectories, by recombining good parts of multiple
different trajectories (which might individually be suboptimal) (Kostrikov et al., 2021; Kumar et al., 2022). The simple supervised learning strategy works well in the domain we tested, but extending CALM to use value-based reinforcement learning methods is a promising direction for future work. Indeed, the improvement obtained from planning on top of the CALM model likely indicates that the supervised learning approach we employ has room for improvement. Additionally, the auxiliary objectives and relabeling strategies we employ require some amount of domain-specific design, and more general strategies could be developed in future.
Addressing these limitations in future work and developing more advanced methods that combine end-to-end language generation via large language models with concepts from reinforcement learning and planning is a promising research direction for making dialogue systems more capable, while also making language models more task aware. We hope that CALM will serve as an indication for the potential of such methods.
This research was supported by an AmazonBAIR Commons project, as well as the Office of Naval Research. We thank Dilek Hakkani-tur, Alexandros Papangelis, Mandi Zhao, Ruiqi Zhong, and Yang Liu for advice and feedback.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5055-5065.
Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. arXiv preprint arXiv:1607.00070.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large
scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Chieh-Yang Chen, Pei-Hsin Wang, Shih-Chieh Chang, Da-Cheng Juan, Wei Wei, and Jia-Yu Pan. 2020. Airconcierge: Generating task-oriented dialogue via efficient large-scale knowledge retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 884-897.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696-3709.
Grace Chung. 2004. Developing a flexible spoken dialog system using simulation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 63-70.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45-51.
Wieland Eckert, Esther Levin, and Roberto Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 80-87. IEEE.
Mihail Eric and Christopher D Manning. 2017. A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 468-473.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. 2020. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint arXiv:2002.11089.
Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062.
Milica Gašić, Filip Jurčicek, Blaise Thomson, Kai Yu, and Steve Young. 2011. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 312-317. IEEE.
Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In Ninth International Conference on Spoken Language Processing.
Kallirroi Georgila and David Traum. 2011. Reinforcement learning of argumentation dialogue policies in negotiation. In Twelfth Annual Conference of the International Speech Communication Association.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. 2019. Learning to reach goals without reinforcement learning.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Conference on Empirical Methods in Natural Language Processing, pages 2333-2343. Association for Computational Linguistics.
Peter A Heeman. 2009. Representing the reinforcement learning state in a negotiation dialogue. In 2009 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 450-455. IEEE.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. volume 33.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456.
Haoming Jiang, Bo Dai, Mengjiao Yang, Tuo Zhao, and Wei Wei. 2021. Towards automatic evaluation of dialog systems: A model-free off-policy evaluation approach. arXiv preprint arXiv:2102.10242.
Leslie Pack Kaelbling. 1993. Learning to achieve goals. In *IJCAI*, pages 1094–1099. CiteSeer.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134.
Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul A Crook, Y-Lan Boureau, and Jason Weston. 2019. Recommendation as a communication game: Self-supervised bot-play for goal-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1951-1961.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.