SlowGuess/ACL_2020_OCR huggingface dataset:
Yiping Song $^{1}$ , Zequn Liu $^{1}$ , Wei Bi $^{2}$ , Rui Yan $^{3}$ , Ming Zhang $^{1*}$
$^{1}$ Department of Computer Science, School of EECS, Peking University, Beijing, China  $^{2}$ Tencent AI Lab, Shenzhen, China
$^{3}$ Wangxuan Institute of Computer Technology, Peking University, Beijing, China  
songyiping, zequnliu, mzhang-CS, ruiyan@pku.edu.cn  
victoriabi@tencent.com
Training the generative models with minimal corpus is one of the critical challenges for building open-domain dialogue systems. Existing methods tend to use the meta-learning framework which pre-trains the parameters on all non-target tasks then fine-tunes on the target task. However, fine-tuning distinguishes tasks from the parameter perspective but ignores the model-structure perspective, resulting in similar dialogue models for different tasks. In this paper, we propose an algorithm that can customize a unique dialogue model for each task in the few-shot setting. In our approach, each dialogue model consists of a shared module, a gating module, and a private module. The first two modules are shared among all the tasks, while the third one will differentiate into different network structures to better capture the characteristics of the corresponding task. The extensive experiments on two datasets show that our method outperforms all the baselines in terms of task consistency, response quality, and diversity.
Generative dialogue models often require a large amount of dialogues for training, and it is challenging to build models that can adapt to new domains or tasks with limited data. With recent advances in large-scale pre-training [Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018], we can first pre-train a generative model on large-scale dialogues from the non-target domains and then fine-tune on the task-specific data corpus [Wang et al., 2019a; Alt et al., 2019a; Klein, 2019]. While pre-training is beneficial, such models still require sufficient task-specific data for fine-tuning. They cannot achieve satisfying performance when very few examples
are given [Bansal et al., 2019]. Unfortunately, this is often the case in many dialogue generation scenarios. For example, in personalized dialogue generation, we need to quickly adapt to the response style of a user's persona by just a few his or her dialogues [Madotto et al., 2019; Zhang et al., 2018]; in emotional dialogue generation, we need to generate a response catering to a new emoji using very few utterances containing this emoji [Zhou et al., 18; Zhou and Wang, 2018]. Hence, this is the focus of our paper - few-shot dialogue generation, i.e. training a generative model that can be generalized to a new task (domain) within  $k$ -shots of its dialogues.
A few works have been proposed to consider few-shot dialogue generation as a meta-learning problem [Madotto et al., 2019; Qian and Yu, 2019; Mi et al., 2019]. They all rely on the popular model-agnostic meta-learning (MAML) method [Finn et al., 2017]. Take building personalized dialogue models as an example, previous work treats learning dialogues with different personas as different tasks [Madotto et al., 2019; Qian and Yu, 2019]. They employ MAML to find an initialization of model parameters by maximizing the sensitivity of the loss function when applied to new tasks. For a target task, its dialogue model is obtained by fine-tuning the initial parameters from MAML with its task-specific training samples.
Despite the apparent success in few-shot dialogue generation, MAML still has limitations [Zintgraf et al., 2019]. The goal of generative dialogue models is to build a function mapping a user query to its response, where the function is determined by both the model structure and parameters [Brock et al., 2018]. By fine-tuning with a fixed model structure, MAML only searches the optimal parameter settings in the parameter optimization perspective but ignores the search of optimal network structures in the structure optimization perspective. More
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5832-5841 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
over, language data are inherently discrete and dialogue models are less vulnerable to input changes than image-related models [Niu and Bansal, 2018], which means gradients calculated from a few sentences may not be enough to change the output word from one to another. Thus there is a need to develop an effective way to adjust MAML for large model diversity in dialogue generation tasks.
In this paper, we propose the Customized Model Agnostic Meta-Learning algorithm (CMAML) that is able to customize dialogue models in both parameter and model structure perspective under the MAML framework. The dialogue model of each task consists of three parts: a shared module to learn the general language generation ability and common characteristics among tasks, a private module to model the unique characteristic of this task, and a gate to absorb information from both shared and private modules then generate the final outputs. The network structure and parameters of the shared and gating modules are shared among all tasks, while the private module starts from the same network but differentiates into different structures to capture the task-specific characteristics.
- We propose the CMAML algorithm that can customize dialogue models with different network structures for different tasks in the few-shot setting. The algorithm is general and well unified to adapt to various few-shot generation scenarios.
- We propose a pruning algorithm that can adjust the network structure for better fitting the training data. We use this strategy to customize unique dialogue models for different tasks.
- We investigate two crucial impact factors for meta-learning based methods, i.e., the quantity of training data and task similarity. We then describe the situations where the meta-learning can outperform other fine-tuning methods.
Few-shot Dialogue Generation. The past few years have seen increasing attention on building dialogue models in few-shot settings, such as personalized chatbots that can quickly adapt to each user's profile or knowledge background [Zhang et al., 2018; Madotto et al., 2019], or that respond with a specified emotion [Zhou et al., 18; Zhou and Wang, 2018]. Early solutions are to use explicit [Tian et al., 2017; Zhang et al., 2018; Zhou et al., 18] or implicit [Li et al., 2016b; Zhou and
Wang, 2018; Zhou et al., 18] task descriptions, then introduce this information into the generative models. However, these methods require manually created task descriptions, which are not available in many practical cases.
An alternative promising solution to building few-shot dialogue models is the meta-learning methods, especially MAML [Finn et al., 2017]. Madotto et al. (2019) propose to regard learning with the dialogue corpus of each user as a task and endow the personalized dialogue models by fine-tuning the initialized parameters on the task-specific data. Qian and Yu (2019) and Mi et al. (2019) treat the learning from each domain in multidomain task-oriented dialogue generation as a task, and apply MAML in a similar way. All these methods do not change the original MAML but directly apply it to their scenarios due to the model-agnostic property of MAML. Thus, task differentiation always counts on fine-tuning, which only searches the best model for each task at the parameter level but not the model structure level.
Meta-learning. Meta-learning has achieved promising results in many NLP problems recently due to its fast adaptation ability on a new task using very few training data [Yu et al., 2019; Wang et al., 2019b; Obamuyide and Vlachos, 2019b; Alt et al., 2019b]. In general, there are three categories of meta-learning methods: metric-based methods [Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Ye and Ling, 2019] which encode the samples into an embedding space along with a learned distance metric and then apply a matching algorithm, model-based methods [Santoro et al., 2016; Obamuyide and Vlachos, 2019a] which depend on the model structure design such as an external memory storage to facilitate the learning process, and optimization-based methods [Finn et al., 2017; Andrychowicz et al., 2016; Huang et al., 2018] which learn a good network initialization from which fine-tuning can converge to the optimal point for a new task with only a few examples. Methods belonging to the first two are proposed for classification, and those in the third category are model-agnostic. Therefore, it is intuitive to apply the optimization-based methods, in which MAML is most popular, for dialogue generation tasks.
However, some researchers found that the original MAML has limited ability to model task-specific characteristics in the image or text classification scenarios [Jiang et al., 2018; Sun et al.,
5833
Figure 1: The proposed CMAML algorithm applying on the personalized dialogue systems. Each customized dialogue model Seq2SPG consists of a shared, a private, and a gating module. The shared and gating module are the same among users and are trained on all tasks. The private module is unique for each user to describe this user's persona, and is trained on the corresponding and similar tasks. The lines in color indicate the data flow directions.
2019; Liu et al., 2020]. Jiang et al. (2018) build an attention layer over the convolutional layers, where the convolutional layer is for general features and the attention layer is for task-specific features. Sun et al. (2019) propose to learn a task-specific shifting and scaling operation on the general shared feed-forward layers. However, the involved operations in these two methods such as shifting and scaling are designed for feed-forward networks, and can not be applied to the generative models which generally rely on Seq2seq [Sutskever et al., 2014] models with recurrent GRU [Cho et al., 2014] or LSTM [Hochreiter and Schmidhuber, 1997] cells. In this paper, we propose a new meta-learning algorithm based on MAML that can enhance task-specific characteristics for generation models.
3 Dialogue Model
In this section, we firstly describe the network structure of the proposed dialogue model, and then briefly introduce its pre-training.
We aim to build dialogue models for different generation tasks in the few-shot setting. Now, we first describe the dialogue model of each task to be used in our training algorithm. It involves three network modules and noted as Seq2SPG (in Figure 1):
Shared Module. It gains the basic ability to generate a sentence and thus its parameters are shared among all tasks. We employ a prevailing Seq2seq dialogue model [Bahdanau et al., 2014]. At each decoding step  $t$ , we feed the word  $x_{t}$  and last hidden state  $h_{t-1}$  to the decoding cell, and obtain an output distribution  $o_{s}$  over the vocabulary.
Private Module. It aims at modeling the unique characteristics of each task. We design a multilayer perception (MLP) in the decoder to fulfill this goal. Each task has its unique MLP network, which starts from the same initialization and then evolves into different structures during training. At each decoding step  $t$ , the MLP takes the word  $x_{t}$  and the output  $h_{t-1}$  of the shared module at step  $t - 1$  as input, then outputs a distribution  $o_{p}$  over the vocabulary. In our experiments, we also explore different inputs for the private module.
Gating Module. We use a gate to fuse information from the shared and private modules:
$$
g _ {s} = \tanh \bigl (W _ {s} [ o _ {s}, o _ {p} ] + b _ {s} \bigr)
$$
$$
g _ {p} = \tanh  \left(W _ {p} \left[ o _ {s}, o _ {p} \right] + b _ {p}\right) \tag {1}
$$
$$
o = g _ {s} \circ o _ {s} + g _ {p} \circ o _ {p}
$$
where  $W_{s}, W_{p}, b_{s}, b_{p}$  are parameters,  $\circ$  is elementwise product, and  $o$  is the word distribution.
3.2 Training Overview
For the rest of the paper,  $p(\mathcal{T})$  denotes the task distribution,  $T_{i}$  denotes the  $i$ -th task to be trained,  $D_{i}^{train}$  and  $D_{i}^{valid}$  denotes the training and validation corpus of task  $T_{i}$ , and  $\theta_{i}$  denotes all training parameters of the dialogue model for  $T_{i}$ , which include parameters  $\theta^s / \theta_i^p / \theta^g$  in the shared/private/gating module respectively. We consider a model represented by a parameterized function  $f$  with parameters  $\theta$ . The model training for all tasks consists of two steps: pre-training and customized model training.
In pre-training, CMAML employs the vanilla MAML to obtain a pre-trained dialogue model as the initial model  $\theta$  for all tasks. At the beginning of the MAML,  $\theta$  are randomly initialized. Then, two
5834
main procedures perform iteratively: meta-training and meta-testing. In meta-training, MAML first samples a set of tasks  $T_{i} \sim p(\mathcal{T})$ . Then, for each task  $i$ , MAML adapts  $\theta$  to get  $\theta_{i}'$  with the task-specific data, which is,
$$
\theta_ {i} ^ {\prime} = \theta - \alpha \nabla_ {\theta} \mathcal {L} _ {D _ {i} ^ {t r a i n}} (f (\theta)) \tag {2}
$$
In the meta-testing, MAML tests tasks  $T_{i} \sim p(\mathcal{T})$  with  $\theta_{i}^{\prime}$  to obtain the losses and then updates  $\theta$  by
$$
\theta = \theta - \beta \nabla_ {\theta} \sum_ {T _ {i} \sim p (\mathcal {T})} \mathcal {L} _ {D _ {i} ^ {v a l i d}} \left(f \left(\theta_ {i} ^ {\prime}\right)\right) \tag {3}
$$
Here,  $\alpha$  and  $\beta$  are hyper-parameters.
In standard MAML, each task obtains its parameters  $\theta_{i}$  by fine-tuning the pre-trained  $\theta$ . However, recall that fine-tuning fails to search the best model in the network structure perspective. Also, the generative models are less vulnerable to input changes, thus a few utterances may not be enough to adapt  $\theta$  into diverse  $\theta_{i}$  for different tasks. To address these issues, we do not perform direct fine-tuning on each task, but design our second training step - Customized Model Training, in which the pretrained private module can evolve into different structures to capture the characteristics of each task and encourage model diversity.
4 Customized Model Training
After obtaining the pre-trained model  $\theta$  from MAML, we employ Customized Mode Training with the following two updating steps:
- Private Network Pruning. This step is applied for the private module only, which is to differentiate the MLP structure of each task. Each task has a different MLP structure by retaining its own subset of active MLP parameters in order to characterize the uniqueness of this task.
- Joint Meta-learning. In this step, we re-train parameters of all three modules of each task using MAML again, but each private module is with its pruned MLP structure now. Also, similar tasks with similar pruned MLP structures are jointly trained in order to enrich the training data.
In the following, we will describe these two steps respectively as well as the gradient update of the whole dialogue model.
4.1 Private Network Pruning
After pre-training, dialogue models of different tasks remain the same parameters  $\theta$ , including
$\theta^s /\theta^p /\theta^g$  in the shared/private/gating module. In this step, the private module with parameters  $\theta^p$  will evolve into different structures with parameters  $\theta_i^p$  to capture the task's unique characteristics.
First, we fine-tune the whole dialogue model of each task from the MAML initialization with its own training data and add an L-1 regularization on the parameters of the private module. The goal of L-1 regularization here is to make the parameters sparse such that only parameters beneficial to generate task-specific sentences are active.
Second, we apply an up-to-bottom strategy to prune the private MLP for each task. This is equal to selecting edges in the fully connected layers in the MLP. We do not prune the layers connected to the input and output of the MLP. For the rest layers, we start the pruning from the one closest to the output first. For the  $l$ -th layer, we consider layers above it ( $>l$ ) are closer to the output, and its lower layers ( $< l$ ) are closer to the input. When we process the  $l$ -th layer, its upper layers should already be pruned. We only keep edges of the current processed layer whose weight excels a certain threshold  $\gamma$ . If all edges in the  $l$  layer connected to a node is pruned, all edges connected to this node in the  $l-1$  layer will also be pruned. In this way, the parameters in private module  $\theta^p$  differentiate into  $|T|$  parameters  $\theta_i^p$ , where each  $\theta_i^p$  is a subset of  $\theta^p$ . The pruning algorithm described above is illustrated in Algorithm 1.
4.2 Joint Meta-learning
So far, every task has a unique network structure in its private module. Now we jointly train the whole dialogue models of all tasks.
We start from the pre-trained MAML initialization again. For the shared and gating modules, all tasks share the same parameters, and they are trained with all training data. The private module, which is to capture the uniqueness of each task, is supposed to be trained on task-specific data. However, we do not have sufficient training data for each task in the few-shot setting, thus the private module may not be trained well. Fortunately, all private modules evolve from the same MLP structure, and similar tasks naturally share overlapped network structures, i.e. remaining edges after pruning are overlapped. This inspires us to train each edge in the private MLP by all training samples of tasks in which this edge is not pruned.
Concretely, we train the private MLP in this way:
5835
Algorithm 1: Private Network Pruning
Input: All parameters  $\theta^p$  in the private MLP module, the sparsity threshold  $\gamma$ , the total number of layers  $L$  in the private MLP module.
Output: The pruned parameters  $\theta_i^p$  in private module for task  $T_{i}$
Finetune  $\theta^p$  on the training data of  $T_{i}$  with L-1 regularization to obtain  $\theta_i^p$ .
for  $j\in \{1,\dots ,L\}$  do  $E_{j}\gets$  All edges (i.e. parameters w.r.t. each edge) in the  $j$  -th layer in  $\theta_i^p$ $N_{j}\leftarrow$  All nodes in the  $j$  -th layer in  $\theta_i^p$
$E_{keep}\gets E_{|L|}\cup E_1$
$k\gets |L| - 1;N_{keep}\gets N_{|L|}\cup N_1.$
while  $k > 1$  do
for each edge  $e$  in  $E_{k}$  do if  $e > \gamma$  and the node connected with  $e$  in  $N_{k + 1}$  is in  $N_{\mathrm{keep}}$  then  $\begin{array}{r}\big{\lfloor}E_{\mathrm{keep}}\leftarrow E_{\mathrm{keep}}\cup \{e\} . \end{array}$    
for each node n in  $N_{k}$  do for each edge e in  $E_{k}$  connected with n do if e in  $E_{\mathrm{keep}}$  then  $\begin{array}{r}\big{\lfloor}N_{\mathrm{keep}}\leftarrow N_{\mathrm{keep}}\cup \{n\} ; \end{array}$  break.   
 $k\gets k - 1$
return  $E_{keep}$  as  $\theta_i^p$
for each edge  $e$  in the MLP, if it is active in more than one tasks, its corresponding parameters  $\theta_e^p$  are updated on the data of all task  $j$ 's, in which the edge is active, i.e.  $\theta_e^p \in \theta_j^p$ :
$$
\theta_ {e} ^ {\prime p} = \theta_ {e} ^ {p} - \alpha \nabla_ {\theta_ {e} ^ {p}} \sum_ {T _ {j}: \theta_ {e} ^ {p} \in \theta_ {j} ^ {p}} \mathcal {L} _ {D _ {j} ^ {t r a i n}} (f (\theta_ {j} ^ {p})) \tag {4}
$$
where each  $\theta_i^p /\theta_i^{\prime p}$  only contains the  $\theta_e^p /\theta_e^{\prime p},$  s of all active edges in the  $i$  -th task.
During meta-testing, the loss is accumulated by the tasks that use the corresponding dialogue models, so  $\theta^p$  is updated as,
$$
\theta^ {p} = \theta^ {p} - \beta \sum_ {T _ {i} \sim p (\mathcal {T})} \nabla_ {\theta_ {i} ^ {p}} \mathcal {L} _ {D _ {i} ^ {v a l i d}} \left(f \left(\theta_ {i} ^ {\prime p}\right)\right) \tag {5}
$$
4.3 Gradient Updates
We summarize the gradient updates of the three modules in our proposed dialogue model during customized model training in Algorithm 2. For the shared and gating module, gradients are updated in the same way as MAML. The update of the private module is replaced by the above Eq. 4 and Eq. 5 introduced in joint meta-learning.
The loss function used to calculate the gradients in our model is the negative log-likelihood of generating the response  $r$  given the input query  $q$  as,
$$
\mathcal {L} = - \log p (r | q, \theta^ {s}, \theta^ {p}, \theta^ {g}) \tag {6}
$$
Algorithm 2: Customized Model Training
Input: The distribution over the task set  $p(\mathcal{T})$ , the step size  $\alpha$  and  $\beta$ .
Output: The customized dialogue models  $\theta^s \cup \theta_i^p \cup \theta^g$  for every task  $T_i$ .
for each  $T_{i}$  in  $T$  do
$\begin{array}{r}\big{\lfloor}\theta_i^p\gets \mathrm{Private\_Network\_Pruning}(T_i). \end{array}$
while not converge do
Sample a batch of tasks  $T_{i}\sim p(\mathcal{T})$    
for each sampled task  $T_{i}$  do Adapt  $\theta^s /\theta^g$  to  $\theta_{i}^{\prime s} / \theta_{i}^{\prime g}$  with  $D_{i}^{train}$  using Eq.2; Adapt  $\theta_i^p$  to  $\theta_{i}^{\prime p}$  with  $D_{i}^{train}$  using Eq.4. Update  $\theta^s$ $\theta^g$  with  $D_{i}^{valid}$  using Eq.3. Update  $\theta_i^p$  with  $D_{i}^{valid}$  using Eq.5.
return  $\theta^s\cup \theta_i^p\cup \theta^g$
We perform experiments in Persona-chat [Madotto et al., 2019] and MojiTalk [Zhou and Wang, 2018], which are treated as few-shot dialogue generation tasks in previous work [Zhang et al., 2018; Madotto et al., 2019; Zhou and Wang, 2018; Zhou et al., 18]. Persona-chat has 1137/99/100 users for training/validation/evaluation, and each user has 121 utterances on average. We follow the previous work [Madotto et al., 2019] and concatenate all the contextual utterances including the query as the input sequence. We regard building a dialogue model for a user as a task on this dataset. MojiTalk has 50/6/8 emojis for training/validation/evaluation. Each training/validation emoji has 1000 training samples on average, and each evaluation emoji has 155 samples on average. We regard generating responses with a designated emoji as a task. On both datasets, the data ratio for meta-training and meta-testing is 10:1.
We implement our shared module based on the Seq2seq model with pre-trained Glove embedding [Pennington et al., 2014] and LSTM unit, and use a 4-layer MLP for the private module<sup>1</sup>. The dimension of word embedding, hidden state, and MLP's output are set to 300. In CMAML, we pretrain the model for 10 epochs and re-train each model for 5 steps to prune the private network. The L-1 weight in the re-training stage is 0.001, and the threshold  $\gamma$  is 0.05. We follow other hyperparameter settings in Madotto et al. [2019].
$^{1}$ Code is available at https://github.com/zequnl/CMAML
5836
5.3 Competing Methods
- Pretrain-Only: We pre-train a unified dialogue generation model with data from all training tasks then directly test it on the testing tasks. We try three base generation models: the Seq2seq [Bahdanau et al., 2014] and the Speaker model [Li et al., 2016b] and the Seq2SPG proposed in Section3.1. Speaker incorporates the task (user/emoji) embeddings in the LSTM cell, and the task embeddings of testing tasks are random parameters in this setting.
- Finetune: We fine-tune the pre-trained models on each testing task, denoted as Seq2seq-F, Speaker-F and Seq2SPG-F.
- MAML [Madotto et al., 2019]: We apply the MAML algorithm on the base model Seq2seq and Seq2SPG, and note them as MAML-Seq2seq and MAML-Seq2SPG. MAML-Seq2SPG uses the same base model as the proposed CMAML but does not apply the pruning algorithm, which helps to verify the effectiveness of the pruning algorithm and joint meta-learning. Note that We did not apply MAML on Speaker model as it shows no improvement comparing with Seq2seq.
- CMAML: We try two variants of our proposed algorithm. CMAML-Seq2SPG is our full model (equal to CMAML in previous sections), where the dialogue Seq2SPG is the base model and pruning algorithm is applied for customizing unique model structures for tasks. CMAML-Seq2SP'G uses a different base model noted as Seq2SP'G, where the private module only takes the output of the shared module as the input. Pruning algorithm is also applied in private module for network customization.
5.4 Evaluation Metrics