SlowGuess/NAACL_2022_OCR huggingface dataset:
- Random Curriculum is the user goal sam
(a) User goal 1
(b) User goal 7
(c) User goal 13
Figure 6: The difficulty curves of three user goals in the Movie domain.
pling standard for most dialogue policy models.
- Slot-Number Curriculum is proposed by ACL-DQN (Zhao et al., 2021a), which takes the number of Inform_slot and Request_slot contained in user goals as the difficulty criterion  ${}^{5}$  .
VACL Curriculum is a generic global curriculum obtained during the training of our proposed method.
For all models, we use a single layer perceptron with 80 neurons and RMSprop optimizer with a 0.001 learning rate and 16 minimum batch size.
The discount factor  $\gamma$  is set to 0.9. The buffer size is  $10k$  for all models except HER, which has a buffer size of  $100k$ . Whenever the current average success rate reaches the maximum and is greater than 0.3, the experience replay will be emptied. The maximum allowable number of conversations  $L$  defaults to 30, except for the movie domain which are 40. An  $\epsilon - greedy$  strategy is used to achieve exploration, where  $\epsilon = 0.1$ . The learning rate of both the teacher model and the student model of ACL-DQN is 0.01. The slot thresholds used in ACL-DQN for dividing user goal sets are 5, 6, 7 and 6, 7, 8 and 8, 9, 10, respectively. The stitch threshold in the HER follows the optimal thresholds of their papers, which is set to 0.2. For a fair comparison, the time cost and data cost for all models are consistent. We used the same 120 conversations to pre-filled experience replay for policy initialization and difficulty initialization and the same training epoch for policy optimization and difficulty adjustment. All results are the average
Index	Difficulty	Slot Number	Request_slot	Inform_slot
12	46.7863	6	\	‘city’: ‘seattle’, ‘numberofpeople’: ‘2’, ‘theater’: ‘regal meridian 16’, ‘starttime’: ‘8:45 pm’, ‘date’: ‘tomorrow’, ‘movename’: ‘the big short’
99	46.1902	5	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘numberofpeople’: ‘4’, ‘movename’: ‘brothers grimsby’
22	43.4684	5	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘numberofpeople’: ‘4’, ‘movename’: ‘zootopia’
90	29.8568	7	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘city’: ‘portland’, ‘state’: ‘oregon’, ‘numberofpeople’: ‘4’, ‘movename’: ‘star wars’
77	15.6063	7	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘city’: ‘birmingham’, ‘state’: ‘al’, ‘numberofpeople’: ‘2’, ‘movename’: ‘zootopia’
115	6.6631	6	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘state’: ‘california’, ‘numberofpeople’: ‘4’, ‘movename’: ‘zootopia’
92	-13.6473	7	‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘date’: ‘tomorrow’, ‘city’: ‘philadelphia’, ‘numberofpeople’: ‘4’, ‘movename’: ‘deadpool’, ‘zip’: ‘19101’
16	-14.8769	6	‘date’: ‘UNK’, ‘theater’: ‘UNK’, ‘starttime’: ‘UNK’	‘city’: ‘Petaluma’, ‘numberofpeople’: ‘4’, ‘movename’: ‘eddie the eagle’
84	-15.4901	6	\	‘city’: ‘seattle’, ‘numberofpeople’: ‘2’, ‘theater’: ‘regal meridian 16’, ‘starttime’: ‘8:45 pm’, ‘date’: ‘tomorrow’, ‘movename’: ‘hail caesar’
Table 3: Information of selected user goals in the acquired movie global curriculum.
values of 1000 dialogues from 10 turns with different random seeds, each run tested on 100 dialogues. The colored areas between the curves are 0.5 times the standard deviations of each episode.
4.4 Effectiveness Evaluation
The results of our proposed VACL and the comparison models in the three domains are reported in Table 1 and Figure 3. It is obvious that the VACL brings solid improvement, affirming the effectiveness of VACL. Although PER achieves almost the second-best performance at the final epoch, it can be seen from Table 1 that PER learns slowly in the early stage. In contrast, HER has shown competitive performance in the movie domain, while it seems to have little advantage in the other two do
It is worth noting that the difficulty of user goals does not change regardless of the case.
Figure 7: The learning curves of DQN agents trained with different curriculums in Movie, Restaurant, and Taxi domains.
(a) Movie
Figure 8: The learning curves of different agents (HER and PER) trained with our global curriculum in Movie, Restaurant, and Taxi domains.
mains. We also notice that ACL-DQN(A) performs better than ACL-DQN(B) and ACL-DQN(C). We conjecture that the difficulty metric of user goal has an extremely complex criterion, not just the slot number of user goals. Thus, ACL-DQN(A) learns more hidden difficulty rules, allowing dialogue agents to benefit more from it. It also validates our assumption about the difficulty of dialogue tasks. Although SDLP learns efficiently in the early stages, its performance is affected by the forgetting problem caused by its pacing function. Moreover, the integration of SDPL into our framework achieves better performance, which further strengthens the contribution of our approach. The experimental results are presented in Appendix .4.
4.5 Ablation Analysis
To further analyze the contribution of two components to the proposed method, we conduct the ablation test. As shown in Table 1 and Figure 4, replacing either component hugely affects the performance of dialogue policy, especially in the more difficult restaurant and taxi domains. Among them, VRCL performs worst over other models in all domains while VOCL has slight improvements over DQN and SNCL. And both of them are worse than VACL. This result implies that the validity of the training scheduler and the accuracy of our difficulty
measurer. Although SNCL produces better performance than DQN, it is still limited by the fixed user goal difficulties and fails to train a good policy. The result also demonstrates that user goal difficulties evaluated by our difficulty measurer have more accurate than simply using the slot number. In conclusion, these two components benefit the VACL to a large extent.
4.6 Versatility Analysis
We suppose that our VACL can build a generic, elastic global curriculum while training a good policy, which could guide different students learning without extra re-training. To verify our conjecture, we plot and observe the curves of each user goal's difficulty during the training process. We find that almost all user goal difficulties converged, and we randomly select three user goals with 10 random seeds and draw their average difficulty curves shown in Figure 6. The results initially verify our conjecture.
The effectiveness of the global curriculum needs to be demonstrated before verifying its generality. Hence, we integrate different global curriculums into a classic representative of the DRL-based dialogue policies, DQN policy, and see if our global curriculum performs best among all of them. Figure 7 reports the effect of different global curricu
Agent	Movie	Rest.	Taxi
Success	Rating	Success	Rating	Success	Rating
DQN	0.17	2.13	0.04	0.93	0.05	1.03
ACL-DQN(A)7	0.28	2.86	0.22	2.25	0.32	2.46
HER	0.30	2.47	0.13	1.64	0.31	1.97
PER	0.34	2.96	0.22	2.41	0.35	2.60
SDPL	0.34	2.78	0.20	2.35	0.38	2.73
VACL*	0.39	3.13	0.28	2.96	0.44	3.02
VACL-DQN	0.36	3.05	0.23	2.61	0.40	2.82
Table 4: Human evaluation of different agents in Movie, Rest. and Taxi domains. Scores with * are statistically significant (p < 0.05).
lum on the DQN model. It is obvious that training with our VACL curriculum achieves the best performance across three domains and consistently outperforms other curriculums by a large margin. Interm of slot-number curriculum, it always improves very little.
To further glean insight regarding the generality of our global curriculum, we integrate our global curriculum and SDPL curriculum with different students (HER, and PER) and see whether its performance is improved. Too many the number of training leads the performance of student model converged too quickly to clearly observe the effect of our global curriculum. Therefore, we reduced the number of training by a factor of 5. Figure 8 reports the learning curves of different students on our global curriculum. It is obvious that training with our global curriculum produces significant improvements while the SDPL curriculum does not.
Therefore, we can conclude that our global curriculum could be applied to guide diverse student learning without extra effort on re-training, which verifies the versatile of the VACL framework.
4.7 Global Curriculum Analysis
To analyze the factors affecting the difficulty of user goals, we further compare user goals with varying difficulties in the VACL curriculum. Taking the movie domain as an example, we randomly selected three user goals in the VACL curriculum located at the head (blue), middle-part (yellow), and tail (gray) positions respectively, and grouped their information into Table 3. It can be seen that the slot number does not fully reflect the user goal difficulty. For example, although user goal 115 increases its difficulty by adding one inform_slot over 22, user goal 77 with 7 slots is easier than
user goal 84 with six slots. In addition, we also observed an interesting phenomenon that, both have the same slot-value pairs except moviename, user goal 12 and user goal 84 have the exact opposite difficulty, while user goal 99 and user goal 22 have approximate difficulty. We conjecture that the user goal difficulty is more relevant to the slot value in inform_slot, which determines the amount of available information. This theoretical result is further analyzed in Appendix .2.
4.8 Human Evaluation
To evaluate the feasibility of our VACL from a human perspective, we recruited real users to interact with different agents trained for  $300^{8}$  epochs using the platform and human evaluation criteria provided by MDC. The results are shown in Table 4. Consistent with simulation results, VACL also achieves better performance from a human perspective. Furthermore, we observe that the performance of VACL-DQN (DQN with our generic global curriculum) is close to VACL and outperformance of all others. It also verifies the effectiveness of our generic global curriculum. The detailed criteria for the human evaluation are shown in Appendix .5.
This paper proposes a versatile adaptive curriculum learning (VACL) framework for task-oriented dialogue policy learning. It supports evaluating the difficulty of dialogue tasks only using the learning experiences of dialogue policy and adaptively curriculum-conditioning based on the learning needs of students. This framework presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. Moreover, we explored the versatility of VACL that it succeeds in training a good dialogue policy while building a generic curriculum to guide diverse student learning without extra effort on re-training. Additionally, we analyzed the ranking of the generic global curriculum, which gave us some inspiration to refine the difficulty criteria for dialogue tasks. An important direction for our future work is how to expand our approach from single-domain tasks with different difficulties to multi-domain tasks, e.g. MultiWoz (Budzianowski et al., 2018). This paper is an essential cornerstone for such an investigation.
From the above experiments, it is observed that all models have converged and achieved optimal performance at 300 epoch.
It is worth noting that SDPL inability to produce a stable, generic curriculum. To test the point, we choose the SDPL model trained after 500 epochs in three domains to evaluate the difficulty of each user goal and rank them to form an SDPL curriculum.
We would like to thank the reviewers for their comments and efforts towards improving our paper. And we would like to acknowledge volunteers of the South China University of Technology who help us with the human experiments. This work was supported by the Key-Area Research and Development Program of Guangdong Province, China (Grant No.2019B0101540042) and the Natural Science Foundation of Guangdong Province, China (Grant No.2019A1515011792).
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and TieYan Liu. 2018. Learning to teach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Milica Gasic, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2013. Online policy optimisation of bayesian spoken dialogue systems via human interaction. In ICASSP, pages 8367-8371. IEEE.
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1311-1320. PMLR.
Michael Cerny Green, Benjamin Sergent, Pushyami Shandilya, and Vibhor Kumar. 2019. Evolutionarily-curated curriculum learning for deep reinforcement learning agents. CoRR, abs/1901.05431.
John Hammersley. 2013. Monte carlo methods. Springer Science & Business Media.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann. 2015. Self-paced curriculum learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2694-2700. AAAI Press.
Tae-Hoon Kim and Jonghyun Choi. 2018. Screenernet: Learning curriculum for neural networks. CoRR, abs/1801.00904.
Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688.
Xiujun Li, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems. arXiv preprint arXiv:1807.11125.
Sihong Liu, Jinchao Zhang, Keqing He, Weiran Xu, and Jie Zhou. 2021. Scheduled dialog policy learning: An automatic curriculum learning framework for task-oriented dialog system. In *Findings of ACL-IJCNLP* 2021, pages 1091-1102.
Keting Lu, Shiqi Zhang, and Xiaoping Chen. 2019. Goal-oriented dialogue policy learning from failures. In AAAI, pages 2596-2603. AAAI Press.
Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-Chin Lin, Marco Moresi, Carel van Niekerk, and Milica Gasic. 2020. LAVA: latent action spaces via variational auto-encoding for dialogue policy optimization. In COLING, pages 465–479. International Committee on Computational Linguistics.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. J. Mach. Learn. Res., 21:181:1-181:50.
Sanmit Narvekar and Peter Stone. 2019. Learning curriculum policies for reinforcement learning. In AA-MAS, pages 25-33. International Foundation for Autonomous Agents and Multiagent Systems.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. 2020. Automatic curriculum learning for deep RL: A short survey. In *IJCAI*, pages 4819-4825. ijcai.org.
Sebastien Racanière, Andrew K. Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, and Timothy P. Lillicrap. 2019. Automated curricula through setter-solver interactions. CoRR, abs/1909.12892.
Zhipeng Ren, Daoyi Dong, Huaxiong Li, and Chunlin Chen. 2018. Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning. IEEE Trans. Neural Networks Learn. Syst., 29(6):2216-2226.
Jost Schatzmann and Steve J. Young. 2009. The hidden agenda user simulation model. IEEE Trans. Speech Audio Process., 17(4):733-747.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016. Prioritized experience replay. In *ICLR*.
Oliver G. Selfridge, Richard S. Sutton, and Andrew G. Barto. 1985. Training and tracking in robotics. In *IJCAI*, pages 670–672. Morgan Kaufmann.
Pei-Hao Su, Milica Gasic, and Steve J. Young. 2018. Reward estimation for dialogue policy optimisation. Comput. Speech Lang., 51:24-43.
Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. 2017. Automatic curriculum graph generation for reinforcement learning agents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 2590-2596. AAAI Press.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. 2021. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. In ICLR. OpenReview.net.
Daphna Weinshall and Dan Amir. 2018. Theory of curriculum learning, with convex loss functions. CoRR, abs/1812.03472.
Daphna Weinshall, Gad Cohen, and Dan Amir. 2018. Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5235-5243. PMLR.
Steve J. Young, Milica Gasic, Blaise Thomson, and Jason D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proc. IEEE, 101(5):1160-1179.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. In ACL, pages 3742-3751. Association for Computational Linguistics.
Yangyang Zhao, Zhenyu Wang, and Zhenhua Huang. 2021a. Automatic curriculum learning with overrepetition penalty for dialogue policy learning. In AAAI, pages 14540-14548. AAAI Press.
Yangyang Zhao, Zhenyu Wang, Kai Yin, Rui Zhang, Zhenhua Huang, and Pei Wang. 2020. Dynamic reward-based dueling deep dyna-q: Robust policy learning in noisy environments. In AAAI, pages 9676-9684. AAAI Press.
Yangyang Zhao, Zhenyu Wang, Changxi Zhu, and Shihan Wang. 2021b. Efficient dialogue complementary policy learning via deep q-network policy and episodic memory policy. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4311-4323.
Rounds	Speaker	Content	Number of matching entries
\	\	\	991
1	U	Can I get tickets for the big short tomorrow?	4
2	S	Which city would you like?	4
3	U	I want to watch at seattle.	4
4	S	What time would you like to see it?	4
5	U	I want to watch at 8:45 pm.	4
6	S	regal meridian 16 is available.	4
7	U	I want 2 tickets please!	4
8	S	Great - I was able to purchase 2 tickets for you to see the big short tomorrow at regal meridian 16 theater in seattle at 8:45 pm.	4
9	U	Thank you.	4
10	S	Thank you.	4
Success
.1 User goals
The whole dialogue behavior of the user simulator is guided by a user goal, which ensures rationality, coherence, and consistency of the dialogue. During each conversation, the user simulator holds a fixed user goal to interact with the student model. This user goal describes the needs of users and explains the purpose of this dialogue. Generally, a user goal  $G$  includes a set of constraints  $C$  (also refer to Inform_slot) and a set of requests  $R$  (also refer to Request_slot), where  $C$  denotes the information constrained by the user and  $R$  denotes the information required by the user (Schatzmann and Young, 2009).
Taking a train-ticket booking as an example, the user's goal is to inquire about the departure time and arrival time of today's trains from Guangzhou to Shenzhen, where the user goal  $G$  is in the following form:
$$
\begin{array}{r}\mathbf {G o a l} = \left(C = \left[\begin{array}{l}\text {l o c a t i o n \_ f r o m} = G u a n g z h o u\\\text {l o c a t i o n \_ t o} = S h e n z h e n\\\text {d a t e} = T o d a y\end{array}\right], \right.\\R = \left[\begin{array}{l}\text {d e p a r t u r e \_ t i m e} =?\\\text {a r r i v a l \_ t i m e} =?\end{array}\right]\left. \right)\end{array}\tag {4}
$$
The condition that dialogue is considered to be successful is if and only if all the information provided by the user is identified accurately, the information provided by the agent meets all the constraints of the user, and a train-ticket that satisfies the above conditions is successfully reserved.
.2 Explanation of the ranking of the global curriculum
Different slots in the user goal carry different amounts of information, affecting the difficulty of
Table 5: A typical example of a conversation for user goal 12.
Rounds	Speaker	Content	Number of matching entries
\	\	\	991
1	U	Can I get some tickets for hail caesar tomorrow?	10
2	S	Which city would you like?	10
3	U	I want to watch at seattle.	10
4	S	hail caesar is available.	10
5	U	I want 2 tickets please!	7
6	S	regal meridian 16 is available.	7
7	U	Could you help me to book the tickets?	7
8	S	Great, I was able to purchase 2 tickets for you to watch hail caesar tomorrow at regal meridian 16 in seattle at 10:00 pm.	7
9	U	Oh, sorry! This is the wrong ticket!	7
10	S	Great, I was able to purchase 2 ticket for you to watch hail caesar tomorrow at regal meridian 16 in seattle at 10:00 pm.	7
11	U	Oh, sorry! This is the wrong ticket!	7
...	...
39	S	Great, I was able to purchase 2 ticket for you to watch hail caesar tomorrow at regal meridian 16 in seattle at 10:00 pm.	7
40	U	Oh, sorry! This is the wrong ticket!	7
Failure
Table 6: A typical example of a conversation for user goal 84.
user goals. User goals with more information slots (e.g., user goal 12) help the dialogue agents to locate entries in the knowledge base that satisfy the user's needs faster, thus they are. In contrast, user goals with less information slots (e.g., user goal 84) require more turns to gradually exclude unmatched entries. Therefore, such user goals are harder because dialogue agents get more penalties and even fail because of reaching the maximum number of rounds.
Take a typical conversation with user goals 84 and 12 as an example, where  $U$  represents the user and  $S$  represents the dialogue agent. As shown in Table 5 and 6, in addition to the conversation content, we also record the change in the number of movie entries that matched the current constraints in the movie knowledge base. It reflects that the amount of information of each utterance, e.g., the big short (movie name) and tomorrow (date) in the first utterance of user goal 12 directly helps the dialogue agent to filter out 4 entries from 991 entries that match all the user constraint, and after subsequent checking, the conversation is successful. Although the different slot values hail caesar (movie name) and tomorrow (date) also help dialogue agents to quickly narrow down from 991 entries, it needs more information to continue to exclude unmatched entries. In most cases, the conversation will fail due to insufficient information and reach the maximum number of rounds. Of course, there are few successful conversations, but most of them are failed examples because it reached the maximum number of rounds. Therefore, we selected typical failed examples to demonstrate. It explains that the user goals (e.g., user goal 12) with
(a) Movie
Figure 10: The learning curves of S-VACL (integrating SDPL into VACL framework) agents in Movie, Restaurant, and Taxi domains.
Figure 9: The effect of different  $\alpha$  values on performance in Movie, Restaurant, and Taxi domains.
(c) Taxi
more informative slots are easier.
.3 Training with varying values of  $\alpha$