SlowGuess/NAACL_2022_OCR huggingface dataset:
Yangyang Zhao\*, Hua Qin\*, Zhenyu Wang\*, Changxi Zhu and Shihan Wang\* South China University of Technology Utrecht University
msyyz@mail.scut.edu.cn; qhscut@163.com; wangzy@scut.edu.cn; cxzhu.cn@gmail.com; s.wang2@uu.nl
Training a deep reinforcement learning-based dialogue policy with brute-force random sampling is costly. A new training paradigm was proposed to improve learning performance and efficiency by combining curriculum learning. However, attempts in the field of dialogue policy are very limited due to the lack of reliable evaluation of difficulty scores of dialogue tasks and the high sensitivity to the mode of progression through dialogue tasks. In this paper, we present a novel versatile adaptive curriculum learning (VACL) framework, which presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. It supports evaluating the difficulty of dialogue tasks only using the learning experiences of dialogue policy and skip-level selection according to their learning needs to maximize the learning efficiency. Moreover, an attractive feature of VACL is the construction of a generic, elastic global curriculum while training a good dialogue policy that could guide different dialogue policy learning without extra effort on re-training. The superiority and versatility of VACL are validated on three public dialogue datasets.
Deep reinforcement learning (DRL) as a promising method has impressive achievements for dialogue policy learning (Young et al., 2013; Gasic et al., 2013; Su et al., 2018; Zhang et al., 2019; Lubis et al., 2020; Wang et al., 2021; Zhao et al., 2021b). But current DRL-based dialogue policy approaches mostly remain brute-force random sampling training, improving their performance at the expense of high interaction costs (Jiang et al., 2015; Ren et al., 2018; Narvekar and Stone, 2019; Narvekar et al., 2020). Inspired by human education, a novel
training paradigm, curriculum learning (CL), is proposed to improve learning performance and efficiency through training a model on a designed sequence of training tasks, rather than an arbitrary random sampling (Svetlik et al., 2017; Weinshall et al., 2018; Fan et al., 2018; Racanière et al., 2019; Green et al., 2019). Although many empirical studies demonstrated beneficial effects of CL, reporting in the field of dialogue policy remains very limited (Zhao et al., 2021a; Liu et al., 2021).
One reason for the slow adoption of such approaches is the absence of a reliable evaluation of the difficulty score of a dialogue task (Weinshall and Amir, 2018). On the other hand, the high sensitivity to the mode of progression through dialogue tasks makes it difficult to adaptively select the dialogue task at the appropriate difficulty for the current dialogue policy (Graves et al., 2017).
In this paper, we propose a versatile adaptive curriculum learning (VACL) framework, which consists of two main components: One is a difficulty measurer that evaluates the difficulty of dialogue tasks only using learning experiences of the dialogue policy (also refer to student model) and ranks them by difficulty to obtain a global curriculum, exempting from the limitation of unable human-defined difficulty. A generic, elastic global curriculum is available after the end of training. The other is a training scheduler that supports skip-level selection on the global curriculum according to the learning needs of the student model to maximize the learning efficiency. The skip-level selection could either choose a harder task to avoid wasting time on too easy tasks or return to an easier task to prevent forgetting. Besides, an attractive feature of VACL is the construction of a generic, elastic global curriculum while training a good dialogue policy that could guide different dialogue policy learning without extra effort on re-training. Our model is model-agnostic, in the sense that it can be incorporated into different student models.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 711 - 723 July 10-15, 2022 ©2022 Association for Computational Linguistics
In summary, the main contributions of this paper are three-fold: 1) We propose a novel VACL framework, which presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks. 2) We explore the versatility of VACL that it succeeds in training a good dialogue policy while building a generic curriculum to guide diverse student learning without extra effort on re-training. To our knowledge, it is the first to demonstrate the extra value of curriculum learning in dialogue policy tasks. 3) We validate the superiority and versatility of VACL on three public dialogue datasets. Additionally, we analyzed the ranking of the generic global curriculum, which gave us some inspiration to refine the difficulty criteria for dialogue tasks.
Inspired by the human education process, it is not new to apply curriculum learning into dialogue policy tasks to restructure the training process of dialogue agents (Selfridge et al., 1985). There are two categories for those approaches, one at the experiences replay level for data exploitation and the other at the task level for data collection (Portelas et al., 2020). This paper focuses here on the second kind of study.
The methods at the experience replay level can be considered as a ranking of the transitions, which can be implemented by transition selection or transition modification. Prioritized experience replay (PER), a typical transition selection method, biased selects transitions with higher TD-error (Schaul et al., 2016). However, PER is highly sensitive to parameter changes, too large parameter changes make it difficult for PER to convergence. Hindsight experience replay (HER) controls the distribution of training transitions by creating successful dialogue experiences from failed ones (Lu et al., 2019). However, not all failed experiences are equally useful for improving dialogue agents. Some provide limited help in reaching the complete tasks, while some are too similar to each other and thus redundant to be learned entirely.
In contrast, there are very limited attempts on researches at the task level. Zhao et al. (2021a) presented preliminary attempts, namely Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN). The ACL-DQN achieves teacher-student co-evolution by incorporating a teacher model to leverage the over-repetition re
Figure 1: Proposed VACL framework.
wards and the feedback from the student model to optimize its curriculum by means of RL. But it remains the drawback of RL, that is, training a good teacher model from scratch may require more experience than learning a good student model. Thus, the method equips the RL-based teacher model with different hand-crafted courses to assist it in customizing good curriculums even in the early training phase. However, the equipped courses presuppose user goals<sup>1</sup> can be ordered by the number of slots in user goals, whereas in reality, they may vary along with multiple factors (Kim and Choi, 2018), an observation that has been verified in our experiment. Recently, Liu et al. (2021) has proposed a novel Scheduled Dialog Policy Learning (SDPL) approach to assess the difficulty of user goals using the dialogue state differential space and scale the capacity of the training goal set proportionally as the training time increases. However, such a difficulty measurer is highly affected by the training dynamics of the model itself, making its curriculum only applicable to the model being trained this time. This implies that user goals that were not or less learned typically be more difficult, and retraining may yield different orders. And the pacing function in SDPL ignores the learning feedback of dialogue policies, resulting in forgotten problems and performance impairment.
The overall framework of the Versatile Adaptive Curriculum learning is shown in Figure 1, which includes two sub-modules: A difficulty measurer that measures the difficulty of user goals and ranks them by difficulty to obtain a global curriculum. After the end of training, a generic, elastic global curricu
<sup>1</sup>In the field of dialogue policy, each user goal is considered a dialogue task. Readers can refer to Appendix .1 for details on the user goal.
Figure 2: Training scheduler schematic.
lum is available to guide different students learning without extra efforts on re-training; A training scheduler model that follows the global curriculum and adaptively curriculum-conditioning according to the learning needs of the student model to maximize learning efficiency. We will describe the two modules in detail and the implementation of the integrated VACL algorithm in what follows.
3.1 Difficulty Measurer
In the context of human education, the distribution of students' exam scores reflects the difficulty of the test. We build on this intuition for constructing a difficulty measurer to evaluate the difficulty of user goals by calculating the average cumulative reward of their related samples. The greater the average cumulative returns, the better students' mastery of this user goal, which means that this user goal is easier for the student. Hence, we define user goal difficulty as:
Definition 1. (User Goal Difficulty): we assume that there exist interaction trajectories for each user goal,  $\{s_0, a_0, s_1, a_1, s_2, a_2, \ldots\}$ , whose corresponding return trajectories are  $\{r_0, r_1, r_2, \ldots\}$ . We define the average cumulative return of these sample trajectories about this user goals as the difficulty of the user goal  $g_i$ , called the user goal value function  $V(g)$ .
Average cumulative returns are attractive because they are usually cheaper to obtain and provide a more measured evaluation for user goals' difficulty without knowing the overall user goals. Since user goal value functions  $V(g)$  are presumed by sampling return trajectories and average their cumulative returns, we apply the Monte Carlo method (Hammersley, 2013) to evaluate the user goal value functions without manually presuming unknown difficulty factors. Given a user goal  $g$ , its user goal value function is evaluated as follows:
$$
V (g) = \frac {1}{N} \sum_ {j = 1} ^ {N} \sum_ {k = 0} ^ {M _ {j}} \gamma^ {k} r _ {k} ^ {j} \tag {1}
$$
where  $N$  denotes the number of all interaction trajectories corresponding to the user goal  $g$ , and  $M_{j}$  denotes the length of the sampled interaction trajectories  $j$ .
The difficulty measurer ranks user goals from easy to difficult based on calculated user goal difficulties to obtain a global curriculum for the training scheduler. New interaction trajectories generated from student learning are also taken into account in the update of the user goal value function:
$$
V _ {t + 1} (g) = V _ {t} (g) + \alpha \left(R _ {t + 1} - V _ {t} (g)\right) \tag {2}
$$
where  $\alpha$  denotes the update rate, and we experimentally evaluate the impact of varying  $\alpha$  on performance in Appendix .3.  $V_{t + 1}(g)$  and  $V_{t}(g)$  denote the value function of the user goal  $g$  at the previous and current moments, respectively.  $R_{t + 1}$  denotes the cumulative return of new interaction trajectory about the user goal  $g$ .
As training times increases and more interaction samples are collected, the user goal value functions gradually converge. Through sorting the converging user goal value functions, a generic, elastic global curriculum is obtained, which could be applied to new students learning without extra efforts on re-training  $^{2}$ . The global curriculum storage form is as follows:
$$
[ \{g _ {0}: V (g _ {0}) \}, \{g _ {1}: V (g _ {1}) \}, [ \{g _ {2}: V (g _ {2}) \} \dots ]
$$
where  $g_{i}$  denotes the index of the user goal  $g_{i}$ , and  $V(g_{i})$  denotes the difficulty of the user goal  $g_{i}$ .
3.2 Training Scheduler
Ideally, students should follow a curriculum based on global difficulty, while not wasting time on tasks that are easy for their current learning ability. As a result, instead of selecting from the global curriculum one by one, the training scheduler in VACL allows skip-level selection on the global curriculum according to the learning needs of the student model to maximize learning efficiency.
To achieve this goal, we first outline the learning needs for the student model as follows: In principle, training on easy user goals gains less information than difficult ones, and direct training on
2The relative order of the global curriculum remains roughly the same even for different dialogue policies, despite the change in their difficulty score. For example, for students at any grade level, high school lessons are always harder than elementary schools.
difficult user goals may not gain positive guidance. To maximize access to information and establish the correct location of the decision surface, student models tend to select a user goal with difficulty that matches their ability. Hence, the suitable difficulty of a well-match user goal for the dialogue policy in the current stage is satisfied as follows<sup>3</sup>:
$$
\begin{array}{l} D (g ^ {\prime}) = \left\{ \begin{array}{l l} V (g _ {i}) - | \Delta R |, & \text {i f} R _ {i} > 0 o r \Delta R = 0, \\ V (g _ {i}) + | \Delta R |, & e l s e \end{array} \right. \\ \Delta R \quad = \left\{ \begin{array}{l l} R _ {i} - V (g _ {i}), & \text {i f} R _ {i} \neq V (g _ {i}), \\ V (g _ {i + 1}) - V (g _ {i}), & e l s e \end{array} \right. \\ \end{array}
$$
where  $V(g_{i})$  denotes the value function of the current user goal  $g_{i}$ ,  $R_{i}$  denotes the current cumulative returns obtained when completing the user goal  $g_{i}$ , and  $\Delta R$  measures the current gap between the ability of the student policy and the user goal difficulty.
As shown in Fig 2,  $R_{i}$  controls the direction of the next user goal selection in the global curriculum, while  $\Delta R$  controls the distance.  $R_{i} > 0$  indicates that the dialogue policy already successfully completed the user goal  $g_{i}$ , and the next task tends to be more difficult. A high  $\Delta R$  indicates low learning gains due to a large gap between the capability of dialogue policy and the difficulty of the current user goal, and the next curriculum tends to choose a user goal that is further away from the current user goal in the global curriculum. The training scheduler continues to follow the global curriculum when the capability of student policy and the user goal difficulty are comparable,  $\Delta R = 0$ .
Finally, the training scheduler selects a user goal with the difficulty closest to the suitable difficulty  $D(g^{\prime})$  for the next training, as shown in Algorithm 1.
Teaching according to aptitude accelerates the learning efficiency of the dialogue policy, and the performance-enhanced dialogue policy generates higher quality sample trajectories, which further improve the accuracy and rationality of the teacher model in difficulty assessing and curriculum arrangement. Both of them promote each other and jointly improve the learning efficiency of dialogue policies. The implementation of the VACL framework is shown in Algorithm 2.
Algorithm 1 Find the next user goal findnearest(curriculum, value)
Input: The global curriculum  $S$  and the suitable difficulty  $D(g')$
Output: Index of the well-match user goal for the next training  $g'$
1: Initialize an empty list  $A$  to store all user goal difficulties in  $S$
2: for  $i \gets \text{len}(S)$  do
3: A.append(S[i].values())
(5) end for
5:  $A\gets np.array(A)$
6:  $i\gets (np.abs(A - D(g'))).argmin()$
7:  $g' \gets S[i].keys()$
8: return  $g'$
Algorithm 2 VACL for Dialogue Policy Learning.
1: Initialize an empty experience relay buffer  $D$  and a list  $T$  to store user goals index and their difficulty pairs
2: ① Policy and Difficulty Initialized
3: Initialize  $Q(s, a; \theta_Q)$  via pre-training on human conversational data
4: The difficulty of each user goal  $V(g)$  in human conversational data is calculated via Eq. 1 and stored in  $T$  together with their corresponding user goal  $g$
5: Initialize an initial suitable difficulty  $D(g) = (V(g)_{\text{max}} + V(g)_{\text{min}}) / 2$
6: for  $t = 1:N$  do
7: ② Global Curriculum Generation
8: The difficulty measurer ranks  $T$  by difficulty to obtain a global curriculum  $S$
9: ③ Curriculum Conditioning
10: The training scheduler selects a user goal  $g$  based on the suitable difficulty  $D(g)$ ,  $g = \text{findnearest}(S, D(g))$
11: ④ Interaction with environment
12: A User and a dialogue agent interact around selected user goal  $g$  and their generated experiences are stored in  $D$
13: ⑤ Measure Learning needs
14: The training scheduler measures the learning needs according to the current cumulative rewards  $R$  and evaluates the next suitable difficulty  $D(g)$  via Eq. 3
15: ⑥ Difficulty Adjustment
16: The difficulty measurer also uses  $R$  to update the user goal difficulty via Eq. 2
17: Sample random mini-batches from D and update  $\theta_{Q}$  via M-step minibatch Q-learning
<sup>3</sup>Since the user goal value function and the learning gains belong to the same order of magnitude, it is computationally desirable.
To evaluate the effectiveness and versatility of the proposed VACL framework, experiments are carried out on three public single datasets, MovieTicket Booking, Restaurant Reservation, Taxi Ordering (Li et al., 2016, 2018) in both simulation and human evaluation.
The experiments are carried out on the platform Microsoft Dialogue Challenge (Li et al., 2016, 2018) with three datasets across different difficulties for our experiments: Movie-Ticket Booking, Restaurant Reservation, Taxi Ordering. As far as the typical difficulty classification criteria (the number of slots in the user goal) (Zhao et al., 2020; Liu et al., 2021), the difficulty of each dataset is: movie = easy, rest. = middle, taxi = hard, as shown in Figure 5. The three datasets performed experimentally are human-annotated, and their statistics are shown in Table 2.
Figure 5: The distribution of the number of slots for user goals in each dataset.
Task	Intent	Slots	Dialogues	User goals	Mainly slot numbers	Domain
Movie-Ticket Booking	11	29	2890	128	5-7	1
Restaurant Reservation	11	30	4103	3525	5-12	1
Taxi Ordering	11	29	3094	2830	8-13	1
Table 2: The number of intents, slots, dialogues, user goals, and the range of slot number in user goals in three datasets.
To verify the effectiveness and versatility of our method and its global curriculum, we conduct experiments using the following existing dialogue policies with curriculum learning and the curriculum standards it brings as benchmarks, including:
- DQN model is implemented using a standard DQN with only direct reinforcement learning (Mnih et al., 2015).
- ACL-DQN(A/B/C) model performs automatic dialogue curriculum learning by using an RL-based teacher model to change the learning order of the dialogue agents based on their learning process, which contains three schedules (Zhao et al., 2021a). Schedule A has no fixed criteria and uses only a single teacher model. Both schedule B and schedule C ensure that the teacher model develops curriculum from easy to complex, with schedule B forcing students to learn one by one and not skipping levels while Schedule C allows it.
- HER model performs dialogue data augmentation by segmenting successful sessions from failed dialogues and stitching them with similar successful dialogue to obtain artificial successful dialogues (Lu et al., 2019).
- PER model reconstructs the training process by prioritizing and replaying samples with large temporal difference (TD) errors more (Schaul et al., 2016).
- SDPL model uses the dialogue state differential space (ICM loss) to assess the difficulty of user goals and then scales the capacity of the training goal set proportionally as the training time increases (Liu et al., 2021).
Proposed VACL
- VACL is the model we proposed that would apply to the varying environments, students, and support conditioning to handle changing tasks, exempt from the limitation of unable human-defined difficulty.
- SNCL is a variant of VACL, which replaces the Difficulty Measure with the Slot-Number based Difficulty Measure from Zhao et al. (2021a)  ${}^{4}$  .
- VOCL is a variant of VACL, which replaces the Training Scheduler with the sequential training, forcing students to learn from easiness to difficulty one by one.
In implementing SNCL, we fix the  $\Delta R$  in Eq. 2 to  $-1$ , since the slot-number based difficulty value and the  $\Delta R$  do not belong to the same order of magnitude, and their difficulty directions are opposite.
Agent	domain	Epoch = 50	Epoch = 150	Epoch = 250	Epoch = 350
Success	Reward	Turns	Success	Reward	Turns	Success	Reward	Turns	Success	Reward	Turns
DQN	0.2999	-19.13	32.27	0.3128	-17.61	32.29	0.3250	-15.97	31.93	0.3202	-16.67	32.18
ACL-DQN(A)	0.3453	-13.00	30.91	0.3366	-14.43	31.63	0.3665	-10.42	30.81	0.3566	-11.83	31.23
ACL-DQN(B)	Movie	0.2463	-26.46	34.04	0.3379	-14.41	31.91	0.3223	-16.58	32.51	0.3138	-17.71	32.74
ACL-DQN(C)	0.3266	-15.90	32.20	0.3167	-17.38	32.77	0.3077	-18.63	33.10	0.3184	-17.10	32.62
HER	0.1877	-34.14	35.34	0.3705	-9.69	30.30	0.3993	-6.09	30.01	0.4030	-5.65	30.01
PER	0.1360	-41.00	36.66	0.3534	-12.07	30.95	0.3693	-10.04	30.73	0.3830	-8.24	30.41
SDPL	0.3188	-16.42	31.38	0.3947	-6.60	29.94	0.4050	-5.20	29.60	0.4050	-5.28	29.76
VACL*	0.3883	-7.36	29.94	0.4275	-2.22	29.05	0.4388	-0.63	28.57	0.4290	-2.03	29.02
SNCL	0.3373	-13.87	30.72	0.3264	-15.50	31.33	0.3343	-14.73	31.70	0.3399	-14.06	31.69
VOCL	0.3511	-12.38	31.04	0.3569	-11.73	31.13	0.3642	-10.78	30.96	0.3635	-10.93	31.09
VRCL	0.3135	-17.02	31.31	0.2479	-26.24	33.98	0.2475	-26.38	34.16	0.2810	-21.88	33.20
DQN	0.1058	-34.34	29.74	0.1103	-34.00	29.87	0.1259	-32.48	29.61	0.1315	-31.92	29.52
ACL-DQN(A)	0.1908	-25.76	27.88	0.3618	-9.01	25.14	0.3669	-8.46	24.97	0.3737	-7.80	24.85
ACL-DQN(B)	Rest.	0.1605	-28.91	28.72	0.1486	-30.15	29.04	0.1169	-33.28	29.60	0.1131	-33.72	29.80
ACL-DQN(C)	0.2031	-24.72	28.02	0.2201	-23.10	27.83	0.2035	-24.87	28.36	0.2097	-24.18	28.09
HER	0.1016	-34.64	29.58	0.1943	-25.89	28.74	0.2129	-24.08	28.48	0.2109	-24.29	28.53
PER	0.2319	-21.65	27.06	0.3558	-9.58	25.21	0.3822	-6.97	24.74	0.3903	-6.18	24.63
SDPL	0.2499	-19.87	26.74	0.3171	-13.45	25.99	0.3458	-10.61	25.46	0.3432	-10.89	25.56
VACL*	0.3002	-14.92	25.89	0.4326	-2.06	23.98	0.4378	-1.57	23.95	0.4256	-2.76	24.14
SNCL	0.1291	-31.72	28.70	0.1751	-27.48	28.47	0.1840	-26.62	28.36	0.1828	-26.74	28.38
VOCL	0.1144	-33.51	29.61	0.1571	-29.39	29.05	0.1867	-26.47	28.54	0.1997	-25.16	28.27
VRCL	0.0735	-37.61	30.47	0.0718	-37.86	30.63	0.0799	-37.08	30.54	0.0739	-37.68	30.66
DQN	0.0974	-34.96	29.47	0.1916	-25.96	28.40	0.1925	-25.85	28.35	0.1882	-26.30	28.47
ACL-DQN(A)	0.1673	-27.57	27.26	0.4375	-1.63	24.00	0.4629	0.93	23.47	0.4623	0.90	23.42
ACL-DQN(B)	Taxi	0.0182	-43.10	31.49	0.0644	-38.61	30.80	0.1223	-32.93	29.89	0.1173	-33.40	29.91
ACL-DQN(C)	0.0233	-42.33	30.87	0.1312	-31.97	29.54	0.1160	-33.46	29.80	0.1207	-33.02	29.76
HER	0.2295	-22.01	27.36	0.4144	-4.12	24.83	0.4338	-2.22	24.52	0.4503	-0.56	24.17
PER	0.3510	-9.77	24.72	0.5434	8.98	21.84	0.5389	8.40	22.21	0.5510	9.68	21.82
SDPL	0.2645	-18.36	26.34	0.4903	3.49	23.27	0.5424	8.33	22.98	0.4743	1.81	23.76
VACL*	0.3396	-10.25	23.63	0.6321	17.74	20.30	0.6600	20.66	19.49	0.6513	19.80	19.62
SNCL	0.2322	-21.59	26.99	0.2908	-16.07	26.49	0.3004	-15.10	26.28	0.3006	-15.11	26.33
VOCL	0.1366	-30.34	27.27	0.3868	-6.91	25.46	0.4144	-4.14	24.86	0.4311	-2.61	24.82
VRCL	0.0784	-37.00	30.13	0.1310	-31.99	29.56	0.1227	-32.80	29.68	0.1294	-32.14	29.58
Table 1: The results of different agents at training epoch = {50, 150, 250, 350}. Each number is averaged over 10 runs, and each run is tested on 1000 dialogues. Best scores are labeled in blue. * denotes significant level  $p < 0.05$  with other agents. Success: average success rate, Reward: average reward, Turn: average turn.
(b) Rest.
(a) Movie
Figure 4: The ablation experiment of two components of VACL in Movie, Restaurant, and Taxi domains.
Figure 3: The learning curves of different agents in Movie, Restaurant, and Taxi domains.
(c) Taxi
- VRCLis a variant of VACL, which replaces the Training Schedule with the inversed training, forcing students to learn from difficulty
to easiness one by one.
Generic Global Curriculum