SlowGuess/ACL_2020_OCR huggingface dataset:
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Yichao Lu, Manisha Srivastava, Jared Kramer, Heba Elfardy, Andrea Kahn, Song Wang, and Vikas Bhardwaj. 2019. Goal-oriented end-to-end conversational models with profile features in a real-world setting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 48-55, Minneapolis - Minnesota. Association for Computational Linguistics.
Liangchen Luo, Wenhao Huang, Qi Zeng, Zaiqing Nie, and Xu Sun. 2019. Learning personalized end-to-end goal-oriented dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6794-6801.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings. 2019. Meta-learning for low-resource natural language generation in task-oriented dialogue systems. AAAI.
Julien Perez, Y-Lan Boureau, and Antoine Bordes. 2017. Dialog system & technology challenge 6 overview of track 1-end-to-end goal-oriented dialog learning. Dialog System Technology Challenges, 6.
Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2639-2649, Florence, Italy. Association for Computational Linguistics.
Janarthanan Rajendran, Jatin Ganhotra, and Lazaros C Polymenakos. 2019. Learning end-to-end goal-oriented dialog with maximal user task success and minimal human agent use. Transactions of the Association for Computational Linguistics, 7:375-386.
Janarthanan Rajendran, Jatin Ganhotra, Satinder Singh, and Lazaros Polymenakos. 2018. Learning end-to-end goal-oriented dialog with multiple answers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3834-3843, Brussels, Belgium. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 1199-1208.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5265-5274.
Weikang Wang, Jiajun Zhang, Qian Li, Mei-Yuh Hwang, Chengqing Zong, and Zhifei Li. 2019. Incremental learning from scratch for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3710-3720, Florence, Italy. Association for Computational Linguistics.
Jason D. Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 665-677, Vancouver, Canada. Association for Computational Linguistics.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Mark Woodward and Chelsea Finn. 2016. Active oneshot learning. NIPS (Deep Reinforcement Learning Workshop).
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 1-10, Melbourne, Australia. Association for Computational Linguistics.
A.1 Extended-bAbI dialog dataset
We extend bAbI dataset (Bordes et al., 2017) into a larger dialog dataset consisting of multiple domains, where each domain has its own ontology and the candidate response set. The main task is a response retrieval problem, where the dialog system needs to select the right response for current dialog history from the given candidate response set. The size of candidate sets in each domain are shown in Table 3. The total number of dialogs for each task is 3000 (1500/500/1000 for train/dev/test set respectively). More statistics are given in Table 4. Detailed dialog samples of extended-bAbI can be found in Table 6.
Domain	# responses
restaurant	333
flights	71
hotels	472
movies	68
music	56
tourism	47
weather	22
Table 3: The number of candidate responses in each domain.
Item	number
# of domains	7
# of dialog tasks	35
# of total system responses	1069
# of total templates for user utterances	685
# vocabulary size	386
# of sentences per dialog	12.4
# of words per sentence	4.9
Table 4: Statistics of extended-bAbI dataset.
A.2 Reward Settings
(Woodward and Finn, 2016; Rajendran et al., 2019) defined rewards for  $S$  as follows:
-  $R_{req}$ : if human is requested
-  $R_{cor}$ : if human is not requested and the model prediction is correct
-  $R_{inc}$ : if human is not requested and the model prediction is wrong
This kind of reward setting did not penalize the case that human is requested but model prediction is correct, and the value of rewards is fixed during reinforcement learning.
Here we propose better reward definition. Let a batch of data be  $\mathcal{D} = \{d_1, d_2, \ldots, d_i, \ldots\}$ . The label of each data is set as positive if the response predictor  $P$  selects a wrong answer and negative if not. The prediction of each data is set as positive if the switch  $S$  chooses human and negative if not. Then we calculate the F1 scores for positive data and negative data separately and take the average of them to get a scalar value score  $(D)$ . Each data  $d_i$  in the batch  $\mathcal{D}$  is assigned to a reward value by computing the difference of total scores with or without  $d_i$
$$
r _ {i} = s c o r e (\mathcal {D}) - s c o r e (\mathcal {D} - d _ {i})
$$
In this way, the reward function is less sensitive to the model prediction accuracy and can self-adjust dynamically during training. Both situations that human is asked while the model predicts correct and human is not asked while the model predicts wrong are given negative rewards.
A.3 Case Studies
There are two sample dialog generated by different models adapted with 10 dialogs on extended-bAbI in Table 5. It shows that MDS can predict the response correctly and choose to request human properly.
A.4 MultiWOZ Data
Table 7 shows a bAbI-format dialog example transferred from MultiWOZ. The data and code for dialog transformation of MultiWOZ and dialog generation of extended-bAbI are attached in the supplemental material.
sample 1 target domain = music
Usr	how do you do
Sys	hello what can i do for you
Usr	i ’d like to hear taylor ’s music
Sys	what kind of music do you like
Usr input	i would like to hear some country songs
Correct answer	api_call taylor country
Predictions:
Mem	hello what can i do for you	[incorrect]
MetaMem	what kind of music do you like	[incorrect]
Mem+C	what do you think of this song:entityOrder=0	[incorrect][no-request]
MDS	api_call taylor country	[correct][request]
sample 2 target domain = weather
Usr	hello
Sys	hello how i can help you
Usr	i want to know the weather in guangzhou
Sys	which date do you want to check
Usr input	i want to know the weather today
Correct answer	api_call london today
Predictions:
Mem	which date do you want to check	[incorrect]
MetaMem	api_call guangzhou today	[correct]
Mem+C	hello how i can help you	[incorrect][no-request]
MDS	api_call guangzhou today	[correct][no request]
Table 5: Prediction of different methods on two sampled dialogs. The flag [incorrect] or [correct] indicate the result of response prediction. The flag [request] or [no-request] indicate whether human is requested or not.
Domain: music
Usr:	what's up
Sys:	how can i help you
Usr:	how can i find some music
Sys:	any singer in your mind
Usr:	i'd like some songs by my idol taylor
Sys:	what kind of music do you like
Usr:	do you have country music
Sys:	api_call taylor country
Usr:	nice
Sys:	what do you think of this song: entityOrder=0
Usr:	do you have other suggestions
Sys:	how about another song: entityOrder=1
Usr:	let's do it
Sys:	great let me play this song for you
Usr:	what is the album of the music
Sys:	here you are entityOrder=3 album
Usr:	thanks a lot
Sys:	can i help you with anything else
Usr:	no thank you
Sys:	you are welcome
Domain: weather
Usr:	i want to know the weather tomorrow
Sys:	which city would you like to check
Usr:	hangzhou please
Sys:	api_call hangzhou tomorrow
Usr:	what's the weather today
Sys:	api_call hangzhou today
Usr:	thanks
Sys:	it's my pleasure
Table 6: Two dialog samples from music and weather domains on extended-bAbI.
Domain: restaurant
Usr:	I would like a v_price restaurant which serves v_food food.
Sys:	api_call v_food v_price
Usr:	What is their address and what part of town are they in?
Sys:	Inform Addr Area
Usr:	Sorry what type of food do they serve?
Sys:	Inform Food
Usr:	Thank you. goodbye.
Sys:	general_bye
Table 7: An example of bAbI-format MultiWOZ dialog.