SlowGuess/ACL_2020_OCR huggingface dataset:
Automatic Evaluation. We performed automatic evaluation metrics in three perspectives:
- Response quality/diversity: We use BLEU [Papineni et al., 2002] to measure the word overlap between the reference and the generated sentence; PPL, the negative logarithm of the generated sentence; Dist-1 [Li et al., 2016a; Song et al., 2017, 2018] to evaluate the response diversity, which calculates the ratio of distinct 1-gram in all test generated responses.
- Task consistency: We use C score [Madotto et al., 2019] in Persona-chat, which uses a pre-trained natural language inference model to measure the response consistency with persona description, and E-acc [Zhou and Wang, 2018] in MojiTalk, which
uses an emotion classifier to predict the correlation between a response and the designated emotion.
- Model difference: It is hard to measure the models ability of customization as we do not have the ground-truth model. Hence, we define the average model difference of pairwise tasks as the Diff Score of each method, and the model difference of a method before and after fine-tuning as  $\Delta$  Score. The model difference between  $T_{i}$  and  $T_{j}$  is the Euclidean distance of their parameters normalized by their parameter count:  $D(T_{i}, T_{j}) = \frac{||\theta_{i} - \theta_{j}||^{2}}{M}$ . Here,  $\theta_{i} / \theta_{j}$  includes all model parameters of this task,  $M$  is the total parameter number of the model. A set of models that capture the unique characteristics of each task should be different from each other and will have a higher Diff score, indicating that a large Diff score is a sufficient condition for a strong customization ability. Similarly, a model that changes a lot for task specific adaptation during fine-tuning will achieve a higher  $\Delta$  Score, indicating that  $\Delta$  Score is also a sufficient condition for a good adaptation ability.
Human Evaluation. We invited 3 well-educated graduated students to annotate the 100 generated replies for each method. For each dataset, the annotators are requested to grade each response in terms of "quality" and "task consistency" (i.e. personality consistency in Persona-Chat and emoji consistency in MojiTalk) independently in three scales: 2 (for good), 1 (for fair) and 0 (for bad). "quality" measures the appropriateness of replies, and we refer 2 for fluent, highly consistent (between query and reply), and informativeness, 1 for few grammar mistakes, moderate consistent, and universal reply, and 0 for incomprehensible or unrelated topic. "task consistency" measures whether a reply is consistent with the characteristics of a certain task, and we refer 2 for highly consistent, 1 for no conflicted and 0 for contradicted. Notice that the user description (Persona dataset) and sentences with a certain emoji (Mojitalk dataset) are provided as the references. Volunteers, instead of authors, conduct the double-blind annotations on shuffled samples to avoid subjective bias.
5.5 Overall Performance
Quality/Diversity. In the Persona-chat dataset, Pretrain-Only methods provide the borderlines of all methods. In Pretrain-Only, Seq2SPG achieves the best performance in terms of both automatic and human measurements, indicating the appropri
5837
Method	Human Evaluation	Automatic Metrics	Model Difference
Quality	Task Consistency	PPL	BLEU	Dist-1	C score/E-acc	Diff Score	Δ Score
Persona-Chat
Seq2seq	0.67	0.10	37.91	1.27	0.0019	-0.16	0.00	0.00
Speaker	0.85	0.10	40.17	1.25	0.0037	-0.14	0.00	0.00
Seq2SPG	0.67	0.03	36.46	1.41	0.0023	-0.14	0.00	0.00
Seq2seq-F	0.78	0.11	33.65	1.56	0.0046	-0.05	17.97	9.19
Speaker-F	0.87	0.25	35.61	1.52	0.0059	0.03	285.11	143.90
Seq2SPG-F	0.7	0.07	32.68	1.54	0.0045	-0.05	292.85	156.30
MAML-Seq2seq	0.97	0.37	37.43	1.54	0.0087	0.14	134.01	67.79
MAML-Seq2SPG	0.85	0.36	35.89	1.70	0.0074	0.16	401.28	198.90
CMAML-Seq2SP'G	0.98	0.58	37.32	1.43	0.0089	0.15	479.21	238.64
CMAML-Seq2SPG	1.15	0.69	36.30	1.70	0.0097	0.18	514.44	263.82
MojiTalk
Seq2seq	0.56	0.39	218.95	0.36	0.0342	0.73	0.00	0.00
Speaker	0.38	0.26	418.96	0.19	0.0530	0.70	0.00	0.00
Seq2SPG	0.77	0.46	158.74	0.64	0.0239	0.74	0.00	0.00
Seq2seq-F	0.50	0.35	217.60	0.40	0.0326	0.72	15.96	8.88
Speaker-F	0.39	0.25	403.92	0.21	0.0528	0.72	39.08	29.10
Seq2SPG-F	0.76	0.47	157.92	0.65	0.0228	0.74	72.43	40.94
MAML-Seq2seq	0.66	0.29	179.02	0.54	0.0109	0.70	183.05	117.09
MAML-Seq2SPG	0.71	0.40	181.56	0.73	0.0246	0.74	306.40	176.31
CMAML-Seq2SP'G	0.64	0.32	172.92	0.76	0.0102	0.75	142.90	81.15
CMAML-Seq2SPG	0.78	0.49	185.97	0.85	0.0210	0.77	345.42	190.64
Table 1: Overall performance in Persona-chat (top) and MojiTalk (bottom) dataset in terms of quality (Human, Perplexity, BLEU), diversity (Dist-1), task consistency (Human, C score, E-acc), structure differences among tasks (Diff Score  $(\times 10^{-10})$ ), model change after adaptation ( $\Delta$  score  $(\times 10^{-10}$ ).
Method	100-shot	110-shot	Similar Users	Dissimilar Users
PPL	BLEU	C score	PPL	BLEU	C score	PPL	BLEU	C score	PPL	BLEU	C score
Seq2seq	38.13	1.19	-0.11	37.58	1.29	-0.15	76.54	1.49	-0.03	42.87	1.10	-0.10
Speaker	40.95	1.02	-0.25	42.59	1.27	-0.06	162.44	0.65	-0.09	46.86	1.11	-0.13
Seq2SPG	39.75	1.27	-0.10	37.71	1.30	-0.15	73.58	1.32	-0.04	42.21	1.14	-0.22
Seq2seq-F	34.86	1.39	-0.03	34.14	1.52	-0.10	74.53	1.53	-0.07	42.33	1.33	-0.06
Speaker-F	37.11	1.30	-0.16	39.10	1.36	-0.06	103.81	1.04	0.04	40.47	1.40	0.01
Seq2SPG-F	37.19	1.31	0.00	37.00	1.33	-0.15	70.15	1.44	-0.04	36.22	1.35	-0.05
MAML-Seq2seq	36.94	1.47	0.03	37.20	1.53	0.07	83.17	1.52	-0.08	39.67	1.34	0.06
MAML-Seq2SPG	36.50	1.52	0.11	35.98	1.47	0.13	82.37	1.52	-0.06	39.41	1.41	0.12
CMAML-Seq2SP'G	37.18	1.46	0.11	37.08	1.44	0.09	82.56	1.50	0.00	40.50	1.40	0.13
CMAML-Seq2SPG	36.52	1.52	0.14	36.44	1.57	0.15	82.78	1.56	-0.07	39.55	1.43	0.16
Table 2: The performance on the Persona-chat dataset for impact factor analysis. The left figure is about the few-shot settings and the right is about the task similarity.
ateness of the proposed model structure. Finetune methods are better than Pretrain-Only methods in most cases. MAML methods have no better performance on BLEU scores than Finetune methods but have relatively higher Dist-1 scores. This indicates that MAML helps to boost response diversity. Enhanced with the proposed pruning algorithm, we can see great improvement for CMAML methods against all the competing methods on both quality and diversity measurements. Particularly, our full model CMAML-Seq2SPG shows clearly better performance and the reasons can be ascribed to two aspects: firstly, the proposed Seq2SPG has a better model structure for our task and secondly, the pruning algorithm makes the models more likely to generate a user-coherent response.
Most of the performance of the competing methods in the MojiTalk dataset is similar to the Persona-chat dataset, while one difference is that Speaker achieves the highest Dist-1 score among all the methods. By carefully analyzing the gener
ated cases, we find all non-meta-learning methods (Pretrain-Only and Finetune) consistently produce random word sequences, which means they completely fail in the few-shot setting on this task. However, meta-learning-based methods survive.
Task Consistency. On both datasets, Finetune methods make no significant differences on C score, E-acc and Task Consistency when compared with Pretrain-Only methods, which means that simple fine-tuning is useless for improving the task consistency. All meta-learning methods including MAML and CMAML outperforms Finetune. Compared with MAML-Seq2seq and MAML-Seq2SPG, CMAML-Seq2SPG obtain  $22.2\% / 12.5\%$  and  $11.8\% / 5.6\%$  improvement on C score and E-acc. It means that the private modules in CMAML-Seq2SPG are well pruned to better well describes the unique characteristics of each task.
We also observe that in MojiTalk, CMAML-Seq2SPG achieves good improvement compared with other baselines on the BLEU score but a lim
ited improvement on E-acc and task consistency score when compared with Persona-chat. This tells that when the training data is limited, the generative models tend to focus on the correctness of the response rather than the task consistency.
By jointly analyzing the response quality and task consistency measurement, we can easily draw the conclusion that the responses produced by our algorithm in CMAML-Seq2SPG not only is superior in response quality but also caters to the characteristics of the corresponding task.
Model Differences. Even though a high difference score among tasks does not indicate each model has captured its unique characteristics, a set of models that can capture the characteristics of themselves will have a higher different score. Hence, we present the difference scores of competing methods as a reference index. In Table 1, we can see that fine-tuning on non-meta-learning methods (Pretrain-Only and Finetune) does not boost the model differences between tasks. MAML helps to increase the model differences but is not as good as the proposed CMAML methods. CMAML-Seq2SPG achieves the highest model difference scores on two datasets as it distinguishes different tasks in both parameter and model structure level.
A higher  $\Delta$  score of a method means its produced dialogue models are more easy to finetune. All non-meta-learning methods have so much lower  $\Delta$  scores than MAML methods. CMAML-Seq2SPG has the highest scores on both datasets, indicating that the active edges in the private module are more likely to be fine-tuned to better fit the corpus of the corresponding tasks. We also observe that CMAML-Seq2SP'G has relatively low  $\Delta$  scores, which indicates its base generation model Seq2S'G is not as good as Seq2SPG.
5.6 Impact Factors
We further examine two factors that may have a great impact on the performance: the quantity of training data and the similarity among tasks.
Few-shot Settings. We only use Persona-chat dataset for analysis, because MojiTalk has too little data to further decrease. In Persona-chat, each user has 121 training samples on average, and we evaluate all the methods in a 100 and 110 samples setting (both in train and test) in Table 2 because all the methods tend to produce random sequences when each task contains less than 100 samples.
For non-meta-learning methods including
Persona	I also love vintage cars.
I am a pediatrician.
I love running and reading.
Query	Singing karaoke is a talent of mine.
Do you sing too?
Response	Not really. I am into running, books and old cars.
Seq2seq	I do not have any pets. I do not have any pets.
Speaker	No, I do not. I do not have any.
Seq2SPG	No, I do not have any pets.
Seq2seq-F	I do not have any pets. I do not have any pets.
Speaker-F	No, I do not. I do not have any.
Seq2SPG-F	No, I do not have any pets.
MAML-Seq2seq	Yes I do. I am a nurse.
MAML-Seq2SPG	I like to listen to jazz and jazz.
CMAML-Seq2SP'G	Yes, I am a doctor. I am a pediatrician.
CMAML-Seq2SPG	Yes, I am a pediatrician.
What do you do for a living?
Table 3: A case in Persona-chat dataset.
Pretrain-Only and Finetune, the quality scores improve as the quantity of training data increases, while the C scores almost remain the same as these methods are not sensitive to the differences among tasks. MAML methods have not changed too much on BLEU scores along with the data growth, but its C scores keep increasing. Both the BLEU score and C score of CMAML-Seq2SPG keep increasing with the data growth, and it always achieves the best performance among all the tasks. This proves that the customized generative models are suitable for the corresponding tasks and can always take the full potential of the training data.
Task Similarity. Again, we only use the Personachat dataset because we cannot define similarities among emojis. We construct two datasets: one contains 100 similar users and another contains 100 dissimilar users (both in train and test).
The performance of all the methods is close to each other in the similar-user setting. It means meta-learning-based methods have no advantage for similar tasks. In the dissimilar-users setting, CMAML-Seq2SPG performs best on the C score and BLEU. We draw a conclusion that user similarity influences the performance of our model. Compared to that in dissimilar-users setting, the BLEU in the similar-users setting is high, but the C score is low. The possible reason is that generative models do not distinguish similar tasks and regard all tasks as one task in training.
5.7 Case Study
We only present one case in the Persona-chat dataset due to the limited space in Table 3. Pretrain-Only and Finetune methods produce general responses with less information. MAML methods tend to generate diverse responses as their initial parameters are easier to be finetuned. Even
5839
though the user profiles are not used for training, CMAML-Seq2SPG can quickly learn the persona information "pediatrician" from its training dialogues while other baselines can not. From another perspective, the pruned private module in CMAML-Seq2SPG can be regarded as a special memory that stores the task-specific information without explicit definition of memory cells.
In this paper, we address the problem of the few-shot dialogue generation. We propose CMAML, which is able to customize unique dialogue models for different tasks. CMAML introduces a private network for each task's dialogue model, whose structure will evolve during the training to better fit the characteristics of this task. The private module will only be trained on the corpora of the corresponding task and its similar tasks. The experiment results show that CMAML achieves the best performance in terms of response quality, diversity and task consistency. We also measure the model differences among tasks, and the results prove that CMAML produces diverse dialogue models for different tasks.
This paper is partially supported by National Key Research and Development Program of China with Grant No. 2018AAA0101900/2018AAA0101902, Beijing Municipal Commission of Science and Technology under Grant No. Z181100008918005, and the National Natural Science Foundation of China (NSFC Grant No. 61772039, No. 91646202, and No. 61876196).
Christoph Alt, Marc Hübner, and Leonhard Hennig. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In ACL, pages 1388-1398, 2019.
Christoph Alt, Marc Hubner, and Leonhard Hennig. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In ACL, pages 1388-1398, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeuralPS, pages 3981-3989, 2016.
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. Learning to few-shot learn across diverse natural language classification tasks. arXiv preprint arXiv:1911.03863, 2019.
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston. Smash: One-shot model architecture search through hypernetworks. In ICLR, 2018.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In SSST-8, pages 103-111, 2014.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long shortterm memory. Neural computation, 9(8):1735-1780, 1997.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018.
Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen tau Yih, and Xiaodong He. Natural language to structured query generation via meta-learning. In NAACL-HLT, pages 732-738, 2018.
Xiang Jiang, Mohammad Havaei, Gabriel Chartrand, Hassan Chouaib, Thomas Vincent, Andrew Jesson, Nicolas Chapados, and Stan Matwin. Attentive task-agnostic meta-learning for few-shot text classification. arXiv preprint arXiv:1805.07722, 2018.
Nikita Kitaev Steven Cao Dan Klein. Multilingual constituency parsing with self-attention and pre-training. In ACL, pages 3499-3505, 2019.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In *NAACL*, pages 110-119, 2016.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. A persona-based neural conversation model. In ACL, pages 994-1003, 2016.
Luchen Liu, Zequn Liu, Haoxian Wu, Zichang Wang, Jianhao Shen, Yiping Song, and Ming Zhang. Multitask learning via adaptation to similar tasks for mortality prediction of diverse rare diseases. In arXiv preprint arXiv:2004.05318, 2020.
5840
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. Personalizing dialogue agents via meta-learning. In ACL, pages 5454-5459, 2019.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings. Meta-learning for low-resource natural language generation in task-oriented dialogue systems. In IJCAI, pages 3151-3157, 2019.
Tong Niu and Mohit Bansal. Adversarial over-sensitivity and over-stability strategies for dialogue models. In CoNLL, pages 486-496, 2018.
Abiola Obamuyide and Andreas Vlachos. Meta-learning improves lifelong relation extraction. In ACL, pages 224-229, 2019.
Abiola Obamuyide and Andreas Vlachos. Model-agnostic meta-learning for relation classification with limited supervision. In ACL, pages 5873-5879, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318, 2002.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
Kun Qian and Zhou Yu. Domain adaptive dialog generation via meta learning. In ACL, pages 2639-2649, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL: https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In ICML, pages 1842-1850, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeuralPS, pages 4077-4087, 2017.
Yiping Song, Zhiliang Tian, Dongyan Zhao, Ming Zhang, and Rui Yan. Diversifying neural conversation model with maximal marginal relevance. In IJCNLP, pages 169-174, 2017.
Yiping Song, Rui Yan, Yansong Feng, Yaoyuan Zhang, Dongyan Zhao, and Ming Zhang. Towards a neural conversation model with diversity net using determinantal point processes. In AAAI, pages 5932-5939, 2018.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, pages 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, pages 1199-1208, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeuralPS, pages 3104-3112, 2014.
Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. How to make context more useful? an empirical study on context-aware neural conversational models. In ACL, volume 2, pages 231-236, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeuralPS, pages 3630-3638, 2016.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar. Extracting multiple-relations in one-pass with pretrained transformers. In ACL, pages 1371-1377, 2019.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar. Extracting multiple-relations in one-pass with pretrained transformers. In ACL, pages 1371-1377, 2019.
Zhi-Xiu Ye and Zhen-Hua Ling. Multi-level matching and aggregation network for few-shot relation classification. In ACL, pages 2872-2881, 2019.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse few-shot text classification with multiple metrics. In NAACL-HLT, pages 1206-2015, 2019.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL, pages 2204-2213, 2018.
Xianda Zhou and William Yang Wang. Mojitalk: Generating emotional responses at scale. In ACL, pages 1128-1137, 2018.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In AAAI, pages 730-738, 18.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In ICML, pages 7693-7702, 2019.
5841