Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation	Current state-of-the-art neural dialogue systems are mainly data-driven and are trained on human-generated responses. However, due to the subjectivity and open-ended nature of human conversations, the complexity of training dialogues varies greatly. The noise and uneven complexity of query-response pairs impede the learning efficiency and effects of the neural dialogue generation models. What is more, so far, there are no unified dialogue complexity measurements, and the dialogue complexity embodies multiple aspects of attributes---specificity, repetitiveness, relevance, etc. Inspired by human behaviors of learning to converse, where children learn from easy dialogues to complex ones and dynamically adjust their learning progress, in this paper, we first analyze five dialogue attributes to measure the dialogue complexity in multiple perspectives on three publicly available corpora. Then, we propose an adaptive multi-curricula learning framework to schedule a committee of the organized curricula. The framework is established upon the reinforcement learning paradigm, which automatically chooses different curricula at the evolving learning process according to the learning status of the neural dialogue generation model. Extensive experiments conducted on five state-of-the-art models demonstrate its learning efficiency and effectiveness with respect to 13 automatic evaluation metrics and human judgments.

Curriculum Plausibility ::: Dialogue Analysis ::: Attributes Independence	So far, we have analyzed five dialogue attributes. A question might be raised that how well the proposed attributes correlate with each other. To validate the correlations of these conversation attributes, we summarize the statistics of the Kendall $\tau $ correlations for each dataset in Table TABREF12. We find that these attributes, in general, show little correlations with each other. This partially validates that dialogue complexity involves multiple perspectives.

Conclusion	In this paper, we propose an adaptive multi-curricula dialogue learning framework, to enable the dialogue models to gradually proceed from easy samples to more complex ones in training. We first define and analyze five conversational attributes regarding the complexity and easiness of dialogue samples, and then present an adaptive multi-curricula learning framework, which chooses different curricula at different training stages according to the learning status of the model. Extensive experiments conducted on three large-scale datasets and five state-of-the-art conversation models show that our proposed learning framework is able to boost the performance of existing dialogue systems.

Curriculum Plausibility ::: Conversational Attributes ::: Query-relatedness	A conversation is considered to be coherent if the response correlates well with the given query. For example, given a query “I like to paint”, the response “What kind of things do you paint?” is more relevant and easier to learn than another loosely-coupled response “Do you have any pets?”. Following previous work , we measure the query-relatedness using the cosine similarities between the query and its corresponding response in the embedding space: $\textit {cos\_sim}(\textit {sent\_emb}(c), \textit {sent\_emb}(r))$, where $c$ is the query and $r$ is the response. The sentence embedding is computed by taking the average word embedding weighted by the smooth inverse frequency $\textit {sent\_emb}(e)=\frac{1}{|e|}\sum _{w\in {}e}\frac{0.001}{0.001 + p(w)}emb(w)$ of words , where $emb(w)$ and $p(w)$ are the embedding and the probability of word $w$ respectively.

Curriculum Plausibility	Intuitively, a well-organized curriculum should provide the model learning with easy dialogues first, and then gradually increase the curriculum difficulty. However, currently, there is no unified approach for dialogue complexity evaluation, where the complexity involves multiple aspects of attributes. In this paper, we prepare the syllabus for dialogue learning with respect to five dialogue attributes. To ensure the universality and general applicability of the curriculum, we perform an in-depth investigation on three publicly available conversation corpora, PersonaChat , DailyDialog  and OpenSubtitles , consisting of 140 248, 66 594 and 358 668 real-life conversation samples, respectively.

Babies learn to speak by first imitating easy and exact utterances repeatedly taught by their patient parents. As children grow up, they learn grade by grade, from simple conversations to more complex ones. Inspired by such human behaviors of learning to converse, in this paper, we introduce curriculum learning to bring the neural dialogue model with easy-to-complex learning curriculum, where the model first learns from easy conversations and then gradually manages more complicated dialogues. Nevertheless, organizing a curriculum with increasing difficulty faces insurmountable obstacles: 1) automatic evaluation of dialogue complexity is a non-trivial task.  defined the difficulty for the training examples with respect to the sentence length and word rarity in neural machine translation.  expressed the difficulty regarding the value of the objective function. So far, there is no unified approach in measuring dialogue complexity. 2) Unlike the single metric of complexity in other tasks, dialogue complexity embodies multiple aspects of attributes —the specificity and repetitiveness of the response, the relevance between the query and the response, etc. As such, in this paper, we study the dialogue distributions along five aspects of attributes to gather multiple perspectives on dialogue complexity, resulting with five curricula accordingly.

Curriculum Plausibility ::: Dialogue Analysis ::: Distributions among Attributes	The distributions of the data samples regarding the aforementioned five attributes are shown in Figure FIGREF11. Although the attribute score distributions on three corpora are similar, they also have disparities: 1) Outliers frequently appear among all the distributions, which exhibits the uneven dialogue complexity. 2) In terms of query-relatedness and continuity, to our surprise, the medians of the two distributions on PersonaChat are obviously smaller than the corresponding distributions on DailyDialog and OpenSubtitles. PersonaChat is manually created by crowd-sourcing, while DailyDialog and OpenSubtitles are collected from almost real-life conversations. 3) With respect to the model confidence (the negative loss value), the median of PersonaChat is relatively smaller, which illustrates that it is more difficult for the neural dialogue generation model to learn from PersonaChat.

Experiments ::: Model Analysis ::: Single vs Multi-curricula	To further glean the insights regarding the effects of the five conversational attributes on the proposed learning framework, we conduct the ablation test using the SEQ2SEQ model by only exploiting a single attribute during the curriculum learning. Table TABREF26 reports the ablation test results on the DailyDialog. We observe that the curriculum learning leads to consistent performance improvements, even with one single conversational attribute. When applying the multi-curricula learning method to the model, we observe the nearly best performance.

Curriculum Dialogue Learning ::: Adaptive Multi-curricula Learning	Dialogue complexity consists of multi-perspectives of attributes. We extend the naive single curriculum learning into the multi-curricula setting, where we provide the neural dialogue generation model with five different learning curricula, and each curriculum is prepared by ordering the training set in terms of the corresponding attribute metric accordingly. Scheduling multiple curricula in the same learning pace is obviously inappropriate. Enlightened by the phenomenon that children usually adjust the learning progress of multiple curricula dynamically in order to acquire a good mark, we further introduce an adaptive multi-curricula learning framework, to automatically choose different curricula at different learning stages according to the learning status of the neural dialogue generation model. The adaptive multi-curricula learning framework is established upon the reinforcement learning (RL) paradigm. Figure FIGREF18 illustrates the overall learning process. The multi-curricula learning scheme is scheduled according to the model's performance on the validation set, where the scheduling mechanism acts as the policy $\pi $ interacting with the dialogue model to acquire the learning status $s$. The reward of the multi-curricula learning mechanism $m_t$ indicates how well the current dialogue model performs.

Curriculum Plausibility ::: Conversational Attributes ::: Model Confidence	Despite the heuristic dialogue attributes, we further introduce the model confidence as an attribute, which distinguishes the easy-learnt samples from the under-learnt samples in terms of the model learning ability. A pretrained neural dialogue generation model assigns a relatively higher confidence probability for the easy-learnt samples than the under-learnt samples. Inspired by , , we employ the negative loss value of a dialogue sample under the pretrained model as the model confidence measure, indicating whether a sampled response is easy to be generated. Here we choose the attention-based sequence-to-sequence architecture with a cross-entropy objective as the underlying dialogue model.

Our proposed adaptive multi-curricula learning framework, concentrating on different curricula at evolving learning process according to the learning status of the underlying model, enables dialogue systems gradually proceed from easy to more complex samples in training and thus efficiently improves the response quality. Curriculum learning in NLP.  examined curriculum learning and demonstrated empirically that such curriculum approaches indeed help decrease training times and sometimes even improve generalization.  managed curriculum learning as an optimization problem. Curriculum learning has also been applied to many NLP tasks. To name a few,  applied self-paced learning for neural question answering.  proposed a curriculum learning based natural answer generation framework, dealing with low-quality QA-pairs first and then gradually learn more complete answers.  proposed curriculum pointer-generator networks for reading comprehension over long narratives.  applied curriculum learning for neural machine translation (NMT), aiming to reduce the need for specialized training heuristics and boost the performance of existing NMT systems. In our work, instead of organizing the curriculum only from a single aspect, we provide an adaptive multi-curricula dialogue learning framework, grounding our analysis on five conversation attributes regarding the dialogue complexity.

Conventional curriculum learning organizes the training samples into one curriculum, whereas we employ multiple curricula for dialogue learning. Enlightened by the phenomenon that children usually adjust the learning focus of multiple curricula dynamically in order to acquire a good mark, we further propose an adaptive multi-curricula learning framework, established upon the reinforcement learning paradigm, to automatically choose different curricula at different learning stages according to the learning status of the neural dialogue generation model. Detailed analysis and experiments demonstrate that the proposed framework effectively increases the learning efficiency and gains better performances on five state-of-the-art dialogue generation models regarding three publicly available conversational corpora. Code for this work is available on https://github.com/hengyicai/Adaptive_Multi-curricula_Learning_for_Dialog.

To enable the proposed framework to be aware of the learning progress $\varrho _i$ regarding each attribute $i$, we also exploit $\varrho =\lbrace \varrho _0, \varrho _1, \cdots , \varrho _{k-1}\rbrace $ for state representations, where $k$ stands for the number of curricula, here $k=5$, and $\varrho _i$ can be simply measured as the learning steps on the attribute $i$. The multi-curricula learning framework samples a scheduling action $a_t$ per step by its policy $\Phi _\theta (a|s)$ with parameters $\theta $ to be learnt, and the scheduling action $a_t \in \lbrace 0, 1, \cdots , k-1\rbrace $ chooses one of the curricula. Then, a mini-batch of dialogue instances is sampled from the top $f(\varrho _i)$ portions of the chosen curriculum. The dialogue model is validated every $\Gamma $ training steps and the curriculum policy is updated at $\Gamma $-round intervals according to a reward $m_\Gamma $.

Related Work	Neural dialogue generation. Neural generation models for dialogue, despite their ubiquity in current research, are still far from the real-world applications. Previous approaches enhancing neural dialogue generation models mainly focus on the learning systems by incorporating extra information to the dialogue models such as relevant dialogue history , topics , emotions , out-sourcing knowledge  or exemplars . Latent variables ,  also benefit the model with more diverse response generations. In contrast with the previous researches, which pay most attention to the underlying dialogue models, in this work, we concentrate on the dialogue learning process and investigate how the performance of existing dialogue models can be improved on the conversation corpora with varying levels of complexity, by simply adapting the training protocols.  attributed the generic/uninteresting responses to the high-entropy utterances in the training set and proposed to improve dataset quality through data filtering. Though straightforward, the filtering threshold need be carefully chosen to prevent the data size decreasing too much. ,  proposed to investigate instance weighting into dialogue systems. However, it is difficult to accurately define the “weight” of an example in conversation systems, since the dialogue data is of high diversity and complexity.

Three annotators, who have no knowledge about which system the response is from, are then required to evaluate among win (response$_1$ is better), loss (response$_2$ is better) and tie (they are equally good or bad) independently, considering four aspects: coherence, logical consistency, fluency and diversity. Cases with different rating results are counted as “tie”. Table TABREF25 reveals the results of the subjective evaluation. We observe that our multi-curricula learning framework outperforms the vanilla training method on all the five dialogue models and the kappa scores indicate that the annotators came to a fair agreement in the judgment. We checked the cases on which the vanilla training method loses to our multi-curricula learning method and found that the vanilla training method usually leads to irrelevant, generic and repetitive responses, while our method effectively alleviates such defects.

Curriculum Plausibility ::: Conversational Attributes ::: Repetitiveness	Repetitive responses are easy to generate in current auto-regressive response decoding, where response generation loops frequently, whereas diverse and informative responses are much more complicated for neural dialogue generation. We measure the repetitiveness of a response $r$ as: where $I(\cdot )$ is an indicator function that takes the value 1 when $w_i \in \lbrace w_0, \cdots , w_{i-1}\rbrace $ is true and 0 otherwise.

Experiments ::: Overall Performance and Human Evaluation	The automatic evaluation results of our proposed multi-curricula learning framework and the comparison models are listed in Table TABREF21. Compared with the vanilla training procedure, our curriculum learning framework 1) brings solid improvements for all the five dialogue models regarding almost all the evaluation metrics, 2) achieves competitive performance across three datasets, affirming the superiority and general applicability of our proposed framework. We also notice that the relative improvements of Distinct on OpenSubtitles are much larger (up to 122.46%) than the other two experiment datasets. We conjecture that the OpenSubtitles, with extremely uneven-complexity dialogue samples, benefits more from the multi-curricula learning paradigm. We conduct a human evaluation to validate the effectiveness of the proposed multi-curricula learning framework. We employ the DailyDialog as the evaluation corpus since it is closer to our daily conversations and easier for humans to make the judgment. We randomly sampled 100 cases from the test set and compared the generated responses of the models trained with the vanilla learning procedure and the multi-curricula learning framework.

Curriculum Dialogue Learning ::: Single Curriculum Dialogue Learning	We first illustrate how a dialogue generation model exploits the curriculum by taking single curriculum dialogue learning as an example, where the curriculum is arranged by sorting each sample in the dialogue training set $\mathcal {D}_{train}$ according to one attribute. Then, at training time step $t$, a batch of training examples is sampled from the top $f(t)$ portions of the total sorted training samples, where the progressing function $f(t)$ determines the learning rate of the curriculum. Following , we define the progressing function $f(t)$ as $f(t)\triangleq min(1, \sqrt{t\frac{1-c_0^2}{T} + c_0^2})$, where $c_0 > 0$ is set to 0.01 and $T$ is the duration of curriculum learning. At the early stage of the training process, the neural dialogue generation model learns from the samples drawing from the front part of the curriculum. As the advance of the curriculum, the difficulty gradually increases, as more complex training examples appear. After training $T$ batches, each batch of training instances is drawn from the whole training set, which is same as the conventional training procedure without a curriculum.

Curriculum Plausibility ::: Conversational Attributes ::: Specificity	A notorious problem for neural dialogue generation model is that the model is prone to generate generic responses. The most unspecific responses are easy to learn, but are short and meaningless, while the most specific responses, consisting of too many rare words, are too difficult to learn, especially at the initial learning stage. Following , we measure the specificity of the response in terms of each word $w$ using Normalized Inverse Document Frequency (NIDF, ranging from 0 to 1): where $\text{IDF}(w)=\log {\frac{N_r}{N_w}}$. $N_r$ is the number of responses in the training set and $N_w$ is the number of those responses that contain $w$. $\text{idf}_{min}$ and $\text{idf}_{max}$ are the minimum and maximum IDFs, taken over all words in the vocabulary. The specificity of a response $r$ is measured as the mean NIDF of the words in $r$.

Curriculum Dialogue Learning	We propose an adaptive multi-curricula learning framework to accelerate dialogue learning and improve the performance of the neural dialogue generation model.

Curriculum Plausibility ::: Conversational Attributes ::: Continuity	A coherent response not only responds to the given query, but also triggers the next utterance. An interactive conversation is carried out for multiple rounds and a response in the current turn also acts as the query in the next turn. As such, we introduce the continuity metric, which is similar to the query-relatedness metric, to assess the continuity of a response $r$ with respect to the subsequent utterance $u$, by measuring the cosine similarities between them.

Introduction	Teaching machines to converse with humans naturally and engagingly is a fundamentally interesting and challenging problem in AI research. Many contemporary state-of-the-art approaches , , , , , ,  for dialogue generation follow the data-driven paradigm: trained on a plethora of query-response pairs, the model attempts to mimic human conversations. As a data-driven approach, the quality of generated responses in neural dialogue generation heavily depends on the training data. As such, in order to train a robust and well-behaved model, most works obtain large-scale query-response pairs by crawling human-generated conversations from publicly available sources such as OpenSubtitles . However, due to the subjectivity and open-ended nature of human conversations, the complexity of training dialogues varies greatly . Table TABREF1 shows samples drawn from OpenSubtitles , which contains millions of human-human conversations converted from movie transcripts. The response of the third sample “Yurakutei kikuhiko.” looks quite strange in terms of the given query, while the first sample is clearly easier to learn. The noise and uneven complexity of query-response pairs impede the learning efficiency and effects of the neural dialogue generation models.

A positive reward is expected if a multi-curricula scheduling action $a_t$ brings improvements on the model's performance, and the current mini-batch of training samples is drawn consulting with the scheduling action $a_t$. The neural dialogue generation model learns from those mini-batches, resulting with a new learning status $s_{t+1}$. The adaptive multi-curricula learning framework is optimized to maximize the reward. Such learning process loops continuously until the performance of the neural dialogue generation model converges. More specifically, the learning status of the dialogue model is represented as the state. Similar to other curriculum learning framework , , the learning status consists of several features, including the passed mini-batch number, the average historical training loss, the loss value on the training data, the margin value of predicted probabilities and the last validation metric values.

To accelerate the neural dialogue learning, $m_\Gamma $ is defined as the ratio of two consecutive performance deviations on a held-out validation set: $m_\Gamma =\frac{\delta _{\Gamma }}{\delta _{\Gamma _{\text{prev}}}} - 1$. The performance deviation $\delta _{\Gamma }$ is calculated in terms of 13 automatic evaluation metrics $\lbrace \xi _1, \xi _2, \cdots , \xi _{13}\rbrace $ used in the experiments: where $\xi _i^{\Gamma }$ is the evaluation score of metric $i$ computed at the current validation turn and $\xi _i^{\Gamma _{\text{prev}}}$ is computed at the previous validation turn. Each score is normalized into $[0,1]$. The curriculum policy is trained by maximizing the expected reward: $J(\theta )=\mathbb {E}_{\Phi _\theta (a|s)}[M(s,a)]$, where $M(s,a)$ is the state-action value function. Since $M(s,a)$ is non-differentiable w.r.t.

Experiments ::: Experiment Settings	We perform experiments using the following state-of-the-art models: (1) SEQ2SEQ: a sequence-to-sequence model with attention mechanisms , (2) CVAE: a conditional variational auto-encoder model with KL-annealing and a BOW loss , (3) Transformer: an encoder-decoder architecture relying solely on attention mechanisms , (4) HRED: a generalized sequence-to-sequence model with the hierarchical RNN encoder , (5) DialogWAE: a conditional Wasserstein auto-encoder, which models the distribution of data by training a GAN within the latent variable space . We adopt several standard metrics widely used in existing works to measure the performance of dialogue generation models, including BLEU , embedding-based metrics (Average, Extrema, Greedy and Coherence) , , entropy-based metrics (Ent-{1,2})  and distinct metrics (Dist-{1,2,3} and Intra-{1,2,3}) , .

Acknowledgments	This work is supported by the National Natural Science Foundation of China-Joint Fund for Basic Research of General Technology under Grant U1836111 and U1736106. Hongshen Chen and Yonghao Song are the corresponding authors.

Experiments ::: Implementation and Reproducibility	Our experiments are performed using ParlAI . Regarding model implementations, we employ a 2-layer bidirectional LSTM as the encoder and a unidirectional one as the decoder for the SEQ2SEQ and CVAE. The hidden size is set to 512, and the latent size is set to 64 for CVAE. For the Transformer, the hidden size, attention heads and number of hidden layers are set to 512, 8 and 6, respectively. In terms of HRED and DialogWAE, the utterance encoder is a bidirectional GRU with 512 hidden units in each direction. The context encoder and decoder are both GRUs with 512 hidden units. Regarding the curriculum length $T$, we set its value in the following manner: we train the baseline model using the vanilla training procedure and compute the number of training steps it takes to reach approximately 110% of its final loss value. We then set $T$ to this value. Each model is trained using two protocols: the vanilla training procedure without using any curriculum and our proposed adaptive multi-curricula learning procedure, keeping other configurations the same.

Experiments ::: Model Analysis ::: Effects of Adaptive Multi-curricula Learning	Adaptive multi-curricula learning enables the model to choose different curricula at different learning stages according to the learning status of the underlying model. As shown in Table TABREF27, we notice the performance drops when replacing the RL-based curriculum policy with the random policy, indicating that choosing different curricula according to the learning status of the model benefits the model training. When training the model with anti-curriculum learning, i.e., feeding examples to the model in the complex-to-easy manner, we also observe consistent performance decreases, affirming the effectiveness of the easy-to-complex learning manner.

Experiments ::: Model Analysis ::: Learning Efficiency	Figure FIGREF28 shows comparative results when training the SEQ2SEQ model on DailyDialog with different training protocols. As shown in Figure FIGREF28, our training method accelerates the learning effectively and consistently outperforms the baseline by a large margin in most cases.

Experiments ::: Model Analysis ::: Multi-curricula Learning Route	To glean insights on how the proposed adaptive multi-curricula learning framework performs, we present the choosing curriculum distributions $\pi (a_t|s_t)$ during the model learning in Figure FIGREF29. We notice that the model focuses more on the curriculum of “query-relatedness” at the initial learning stage. As the learning proceeds, the model gradually turns its attention to other curricula. At the final stage, the model pays more attention to the “model confidence” curriculum. Such dynamic learning route is quite similar to the human learning behavior.

Experiments ::: Model Analysis ::: Examples with Different Learning Frequencies	As shown in Table TABREF30, the most frequently learnt examples are comprehensively far better than those seldom learnt examples, which exhibits the effectiveness of the adaptive multi-curricula learning framework.

$\theta $, in this work, we use REINFORCE , a likelihood ratio policy gradient algorithm to optimize $J(\theta )$ based on the gradient: where $v_t$ is the sampled estimation of reward $M(s_t, a_t)$ from one episode execution of the policy $\Phi _\theta (a|s)$. In our implementation, $v_t$ is computed as the terminal reward $m_\Gamma $.

Question: What five dialogue attributes were analyzed?
Answer: Model Confidence, Continuity, Query-relatedness, Repetitiveness, Specificity