SlowGuess/NAACL_2022_OCR huggingface dataset:
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. 2022. When should we prefer offline reinforcement learning over behavioral cloning? International Conference on Learning Representations.
Esther Levin, Roberto Pieraccini, and Wieland Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on speech and audio processing, 8(1):11-23.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In *Conference on Empirical Methods in Natural Language Processing*, pages 2443-2453.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Conference on Empirical Methods in Natural Language Processing.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of NAACL-HLT, pages 2060-2069.
Jingjing Liu, Stephanie Seneff, and Victor Zue. 2010. Dialogue-oriented review summary generation for spoken dialogue recommendation systems. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pages 64-72.
Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jianguang Lou. 2021. Tapex: Table pre-training via learning a neural sql executor. arXiv preprint arXiv:2107.07653.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. 2020. Learning latent plans from play. In Conference on Robot Learning, pages 1113-1132. PMLR.
Michael F McTear. 2002. Spoken dialogue technology: enabling the conversational user interface. ACM Computing Surveys (CSUR), 34(1):90-169.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2017. Coherent dialogue with attention-based language models. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3252-3258. AAAI Press.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pretrained auto-regressive model. arXiv preprint arXiv:2005.05298.
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Herve Frezza-Buet. 2011. Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):1-21.
Dean A Pomerleau. 1989. Alvinn: An autonomous land vehicle in a neural network. Technical report, Carnegie Mellon University, Artificial Intelligence and Psychology.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. 2018. Temporal difference models: Model-free deep rl for model-based control. arXiv preprint arXiv:1802.09081.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. OpenAI Blog.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. 2018. Semi-parametric topological memory for navigation. arXiv preprint arXiv:1803.00653.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152.
Satinder Singh, Michael Kearns, Diane Litman, and Marilyn Walker. 1999. Reinforcement learning for spoken dialogue systems. Advances in neural information processing systems, 12:956-962.
Satinder Singh, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the njfun system. Journal of Artificial Intelligence Research, 16:105-133.
Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach. Oxford University Press on Demand.
Pei-Hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 147-157.
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arXiv preprint arXiv:1606.02689.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. 2020a. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. arXiv preprint arXiv:2006.06814.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. 2020b. Critic regularized regression. volume 33.
Wei Wei, Quoc Le, Andrew Dai, and Jia Li. 2018. Airdialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3844-3854.
Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014. The dialog state tracking challenge series. AI Magazine, 35(4):121-124.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface's transformers: State-of-the-art natural language processing.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2019a. Alternating recurrent dialog model with large-scale pre-trained language models.
Yifan Wu, George Tucker, and Ofir Nachum. 2019b. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky. 2016. Strategy and policy learning for nontask-oriented conversational systems. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 404-412.
Tangming Yuan, David Moore, and Alec Grierson. 2008. A human-computer dialogue system for educational debate: A computational dialectics approach. International Journal of Artificial Intelligence in Education, 18(1):3-26.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1208-1218.
A AirDialogue Dataset Filtering
When training the LM(GPT2-small) and Customer Bot, we filter the dataset by only keeping the successful task examples. This is be achieved by simultaneously checking for successful task completion and whether a set of simple string matching heuristics are satisfied in the dialogue. Our heuristics aim to ensure that strings corresponding to each of the customer's flight requirements and the customer's goal are explicitly present in the dialogue. This combination of filtering steps reduces the size of the training set by  $26\%$ . Despite this, we find that this is still more than enough data for the model to successfully learn the task.
A.1 Rollout Planning
In Figure 5, we show the rollout planning procedure, which described in Section 4.4.
Figure 5: Our dialogue rollout planning procedure. To generate our response, we sample entire dialogues from the language model and then re-rank the predicted dialogues with a reward function.
B Training Our Customer Bot
Our customer bot is fine-tuned from GPT2-small (124M parameters), using the standard language modeling objective. We used the Huggingface Transformers library's implementation of GPT2 (Wolf et al., 2020). The customer's flight requirements are provided to the model as a prefix to the dialogue, which formatted as a comma separated list consisting of the customer's goal and flight requirements. We trained the customer bot for maximum 10 epochs with early stopping on the filtered dataset. For training, it takes around 1 day on 4 GPUs. Specifically, we trained using Adam with learning rate 1e-4 and batch size 8. Our customer bot achieves a perplexity of 1.47 on the development set and a BLEU score of 38.5.
C Fight Agent Bot Details
All our flight agent bots are fine-tuned from GPT2-small (124M parameters) using the standard language modeling objective. We used the
Huggingface Transformers library's implementation of GPT2 (Wolf et al., 2020). Similar as the customer bot, we trained for maximum 10 epochs with early stopping on the filtered dataset, which takes roughly 1 day on 4 GPUs. Specifically, we trained using Adam with learning rate 1e-4 and batch size 8. We implement the final action prediction as a sequence of tokens generated at the end of each dialogue. The flight table is passed to the model as a prefix of flight embeddings, where each embedding is produced by summing embeddings corresponding to each attribute of a given flight (e.g., flight arrival/departure day/location, flight price, etc.).
D AirDialogue Task Pretraining
Initialized using GPT2-small (124M parameters), we further pre-train our flight-agent bots by training on simplified task sequences. Specifically, these sequences consist of our flight table followed by a comma separated list of the customer's flight requirements and a string representing the final action taken. We also apply our auxiliary loss and task-relabeling techniques during this pre-training.
We pre-train on 4 million unique samples, using batch size 64 and Adam with learning rate 1e-4, which takes around 2 days on 4 GPUs. During pre-training, we found that it took around 2 million unique samples before the model suddenly started to learn the task of querying the flight table, and it took roughly 2 million more samples before it became proficient at querying the table. Both the unusual progression of learning during this pre-training phase and the high sample complexity needed to learn the task, indicates the difficulty in learning to query the flight table. This calls for future work about further investigate the challenges in learning complex logical functions using neural networks.
E Self-Play Evaluation
Prior works primarily evaluate bots for the flight agent through "self-play" (Chen et al., 2020; Wei et al., 2018). We follow the same evaluation protocol in our work. Basically, we train a bot to play the role of the customer during evaluation and compute task success by simulating conversations against this bot. We run all self-play evaluations on the same subset of 1,000 dialogue scenarios, randomly selected from the validation set.
All models are evaluated against the same customer bot. including models for the baselines. We find that when running against our self-play bot,
Figure 6: Our reward prediction method. We train a model to parse the customer's flight requirements from the dialogue. We execute these flight requirements against the table and compare the output to the flight that was actually booked; this determines the reward (i.e. if the correct flight was booked or not).
task completion success for prior methods is increased, sometimes by more than  $8\%$  (from what was reported by such prior works under the same evaluation setting). The only difference is the specific model used for customer's side of the conversation, and we conjecture that this difference is likely due to the architecture difference and the details of our dataset filtering. This significant change in evaluation performance compared with prior works, not only indicates the quality of our customer bot, but also suggests the importance of accounting for these factors in evaluating and comparing dialogue systems. We release the code and model weights for our customer bot at https://sea-snell.github.io/CALM_LM_site/.
E.1 AirDialogue Reward Predictor for Rollout Planning
To execute rollout planning, we need a reward predictor which can estimate whether a given dialogue is a successful example of task completion or not. In the case of AirDialogue, we found that the most robust way to estimate this reward is the following: we first fine-tune a RoBERTa-base model (123M parameters) to predict the customer's ground-truth goal and flight requirements from the set of dialogues in the training set. We used the Huggingface Transformers library's implementation of RoBERTa (Wolf et al., 2020). We do not filter the training-set when training this model. Once this model is trained, our procedure for predicting dialogue success is the following:
1. Given a dialogue, use our RoBERTa model to predict the customer's goal and flight requirements.
2. We then execute this predicted information against the agent's flight table and reservation
dep. city	ret. city	dep. month	ret. month
0.76	0.76	0.77	0.77
dep. day	ret. day	dep. time	ret. time
0.76	0.76	0.94	0.94
class	price	connections	airline
0.92	0.37	0.95	0.97
flag, to produce a set of valid final actions.
3. If the final action taken in the dialogue is within the set of predicted final actions, then predict that the current dialogue is successful, otherwise predict that it is unsuccessful.
See Figure 6 for a visual illustration of this procedure. Our model obtains  $94\%$  accuracy in predicting the reward of the dialogues in the validation set (see Table 4 for a more extensive breakdown of the model's accuracy).
F Example Conversation in AirDialogue
In Figure 7, we showcase a specific example for the conversation in AirDialogue.
Table 4: Our RoBERTa parser's accuracy in predicting each of the customer's flight requirements. The parser predicts 5 out of 12 flight requirements with  $>90\%$  accuracy and 11 out of 12 with  $>70\%$  accuracy. The price requirement has the lowest accuracy because it is often not explicitly mentioned in the dialogue; the model has to rely on priors for prediction in these cases.
Greeting	Customer: Hello.
Agent: Hello, how can I help you today?
Customer: Could you help me in booking a flight ticket from AUS to EWR?
Agent: Sure, may I know your planned travel dates?
Customer: I want to fly on Aug 31, return on Sept 02.
Agent: May I know your connection limit?
Customer: I want direct flight.
Questioning	Agent: Is there any class preference for you?
Customer: No.
Agent: Do you have any other specifications?
Customer: No.
Agent: May I know your good name?
Customer: My name is Kimberly Roberts.
Suggest a Flight	Agent: We found an airline: American Airlines, flight number: 1013 and price is 100 with one connection.
Customer: Ok, proceed to booking.
Agent: Your flight ticket has been reserved.
Customer: Thank you.
Goodbye	Agent: Thank you for reaching us.
Submit: book , Kimberly Roberts , 1013
Figure 7: An example conversation in AirDialogue. Conversations generally begin with a greeting followed by some questioning / information gathering, and then finally the agent suggests a flight before ending the conversation.
G Previous Approaches to Flight Table Processing
Prior works (Wei et al., 2018; Jiang et al., 2021) typically input the table directly into a language model, expecting that the skill of querying the table will be naturally learned via the standard language modeling objective. We found this approach to
under-perform in our experiments. These findings are also consistent with recent works which show that pre-training transformers for querying tables can significantly improve the transformer's performance on downstream tasks which use tables (Liu et al., 2021). AirConcierge (Chen et al., 2020) takes a different approach, and explicitly predicts and executes SQL queries based on the dialogue. This approach obtains the SOTA task success on AirDialogue, but it involves several complex components, requires the ability to preform semantic parsing on the dialogue, and of course requires additional domain knowledge about the format and structure of the flight table, which represents the task context. In our work, we show that applying CALM for AirDialogue can close this gap by inducing task learning from language models and achieve end-to-end learning from the flight table, without sacrificing the generated language quality.
H Error Analysis
In Table 5 we present a detailed breakdown of model errors. As expected, determining the flight to book, if any, is consistently shown to be the most challenging sub-task, as evidenced by the lower "flight success rate" and the lower F1 scores for "no flight", "book", and "change" on LM (GPT2-small). In particular, "change" has a low recall, precision, and F1 score for all models because it makes up a very small  $0.4\%$  of the training data. Lastly, the "constraint success" row shows that even when CALM books the wrong flight, the flight it does books meets  $>80\%$  of the customer's flight requirements on average.
CALM	LM (GPT2-small)	AirConcierge
full success rate	0.88±4e-3	0.38±1e-3	0.81±7e-3
status success rate	0.92±3e-3	0.84±2e-3	0.90±1e-3
flight success rate	0.88±4e-3	0.39±1e-3	0.82±5e-3
name accuracy rate	0.99±2e-3	1.0±8e-4	0.99±1e-3
book R/P/F1	0.85±8e-3	0.06±3e-3	0.81±1e-2
0.86±8e-3	0.05±2e-3	0.70±9e-3
0.85±6e-3	0.05±3e-3	0.75±1e-2
no flight R/P/F1	0.82±1e-2	0.36±9e-3	0.59±5e-3
0.80±1e-2	0.74±5e-3	0.93±6e-3
0.81±1e-3	0.49±9e-3	0.72±5e-3
cancel R/P/F1	0.98±2e-2	1.0±0.0	1.0±0.0
0.95±3e-2	1.0±0.0	0.75±1e-2
0.97±2e-2	1.0±0.0	0.86±7e-3
change R/P/F1	0.25±8e-2	0.0±0.0	0.0±0.0
0.33±1e-1	0.0±0.0	0.0±0.0
0.28±1e-1	0.0±0.0	0.0±0.0
no reservation R/P/F1	0.99±5e-3	0.99±2e-3	0.99±3e-3
0.99±2e-3	0.99±2e-3	0.99±3e-3
0.99±3e-3	0.99±2e-3	0.99±3e-3
constraint success	0.81±9e-3	0.71±3e-3	0.89±1e-3
Table 5: Detailed statistics for model errors. All models are evaluated with greedy decoding. In addition to the full task success rate, we report success rate for each sub-component of the full task (status / flight / name). We also report recall (R), precision (P), and F1 score for task success under each type of high-level action (book / no flight / cancel / change / no reservation). Lastly, we report the average fraction of the customer's flight requirements that are met when the agent books the wrong flight (constraint success).