SlowGuess/ACL_2020_OCR huggingface dataset:
Harnessing those trained models could potentially allow a conversational agent to jointly exhibit all skills, with minimal additional training. Instead, one trains a top-level 'dialogue manager' which is a classifier with the dialogue context as input, that predicts which skill to use on each turn, and then outputs the utterance produced by the corresponding trained model. Specifically, we train a three-class classifier on top of BERT-base (Devlin et al., 2019) that assigns an utterance to the dataset it came from. We remove duplicate utterances present in more than one of the datasets prior to training and upsample with replacement to create equal representation in the classifier's training set. We also remove context from the utterances including topics from Wizard of Wikipedia and personas from ConvAI2 before training this classifier and when performing evaluation to prevent the classifier from relying on these (cf. the bias mitigation mentioned above).
In Section 4.1, we introduce the automated metrics and human evaluations that we use to measure and compare model performance. Section 4.2 discusses how adding personas and topic strings during multi-task training de-biases the selection of retrieval candidates from across our three skill-based tasks. Sections 4.3 and 4.4 detail the performance of our models using automated metrics on single-skill and BlendedSkillTalk benchmarks, respectively, and Section 4.5 compares the performance of the models on human evaluation: in all three cases, models trained on all three skills generally outperform those trained on individual skills.
4.1 Metrics used
We use both automated metrics and human evaluation. For automated metrics, we report hits@1 on the test set (or validation set in the case of ConvAI2 as the test set is not publicly available), out of 20 candidates for ConvAI2, and 100 candidates for ED and WoW, following the original datasets. For human evaluation, we ask workers to chat with various models and then rate the conversation along several axes:
- Knowledge: How knowledgeable was your chat partner (from 1: not at all, to 5: very)?
- Empathy: Did the responses of your chat
Utt. Selected	MT Single-Skills	MT S.-S. + BST
orig.	debiased	orig.	debiased
ConvAI2	64.4%	38.9%	61.1%	48.1%
WoW	11.3%	29.4%	10.0%	21.3%
ED	24.2%	31.6%	28.8%	30.5%
Table 4: Mitigating skill selection bias. Adding personas and topics during multi-task training (debias) results in the multi-task retrieval models selecting utterances more evenly when tested on BlendedSkillTalk compared to training on the original datasets (orig).
partner show understanding of your feelings (from 1: not at all, to 5: very much)?
- Personal: How much did your chat partner talk about themselves (from 1: not at all, to 5: a lot)?
- Overall: Overall, how much would you like to have a long conversation with this conversation partner (from 1: not at all, to 5: a lot)?
Conversations and ratings are collected at least 100 times per model, from 234 crowdsourced workers who produce a maximum of 10 of these conversations overall (across all model types). Several methods are used to filter out low quality workers that are similar to the methods used in collection of the BlendedSkillTalk dataset collection. All work by a given worker is excluded if they give the same ratings across all conversations, give utterances deemed unsafe by a safety classifier (Dinan et al., 2019a), utterances shorter than 3 words, use all-caps too frequently, or repeat themselves too much. Messages cannot be over 30 words or copy persona strings exactly.
4.2 Mitigating multi-task skill selection bias
We first examine the issue of skill selection bias in multi-task models. As we are employing multitask retrieval models that retrieve from the set of candidates across all skills, we can collect statistics on those selection choices (i.e., which datasets the chosen utterances originated from). Table 4 reports the percentage of utterances derived from the three skills for our multi-task models (MT Single-Skills and MT Single-Skills + BST) when evaluating on the BST test set. When training on the original skill datasets, we observe heavy overuse of the ConvAI2 utterances and underuse of WoW, likely because BST contains personas as input. Our bias mitigation approach described in Section 3.2 causes a substantial shift for both models,
making the use of the skills more equal. These results are then in line with the actual expected ratios in BST, as shown in Section 3.1 (Skill Annotations). In the following experiments, we thus use the debiased versions.
4.3 Results on Single-Skill Benchmarks
Automated metrics results on the original benchmarks used to gauge competency at a single skill (ConvAI2, WoW, ED) reported in the literature are shown in Table 5 (first row). Our poly-encoder models (rows 2-4) trained on single tasks match or exceed the metrics published with the corresponding benchmarks, except for ED, which is close. The single-skill models each perform the best on their respective original benchmark and not as well on other benchmarks, compared to the blended models. However, the performance of all blended models is more balanced, in the sense that none of the single-skill models does as well averaged over the three categories (except for the ED model doing a tiny bit better than the random-skill model). The model finetuned on BST shows balanced performance but fails to match the performance of the single-skill models on their original benchmarks. The performance of the Multi-Task Two-Stage model gains many points over that of simple random assignment of single-skill models (Random-Skill), and this Random-Skill model itself performs about as well as the BST-fine-tuned model on the ED and WoW benchmarks. The Multi-Task Single-Skills model performs best among the blended models, and nearly matches the performance of all single-skill models on all benchmarks (even surpassing it for the WoW benchmark).
The fact that the Multi-Task Single-Skills model does not do exactly as well as the single-skill models when evaluated using only candidates from individual benchmarks matches the observations of other work (Raffel et al., 2019). However, when evaluated with a set of mixed candidates from all single-skill tasks (where the set of candidates to choose from is tripled by included an equal number of candidates from the other two datasets), the multi-task model performs better than the individual models, suggesting that multi-task training results in increased resilience to having to deal with more varied distractor candidates. We also include metrics for "added-context", when topics and personas are added (see Section 4.2), as a san
Model	Single-skill benchmarks	Avg.
ConvAI2	WoW	ED
SOTA Reported	87.3	87.4	66.0	80.2
ConvAI2	89.4	78.4	42.6	70.1
WoW	57.3	91.8	47.7	65.6
ED	63.3	81.0	65.1	69.8
BST model	78.5	84.1	52.0	71.5
Random-Skill	71.0	83.9	52.0	69.0
MT Two-Stage	84.7	90.1	63.4	79.4
MT Single-Skills	88.8	92.8	63.2	81.6
MT Single-Skills	Added-context benchmarks	81.6
88.9	92.8	63.2
Mixed-candidates evaluation
Single-task	82.1	88.2	60.2	76.8
MT Two-Stage	77.2	86.6	59.0	74.3
MT Single-Skills	85.2	92.1	61.1	79.5
Table 5: Results on single-skill benchmarks. Top: reported values published in the papers accompanying the benchmarks, and the Poly-encoder paper. ConvAI2, WoW, ED: models trained on the corresponding benchmark. These models perform very well on the benchmark they were trained on, but not as well on other benchmarks. BST: The model fine-tuned on BST shows more balanced performance (i.e., none of the single-skill benchmarks does better at all three skills), but it is noticeably lower than each specialized model. Random-Skill: the performance of choosing a random single-skill per response is comparable to the BST model, but slightly worse on ConvAI2. MT Two-Stage: guiding the generation by an actual task classifier as opposed to random selection increases performance on all skills. MT Single-Skills: this model performs best among the blended skills architectures, and nearly matches the single-skill model performance (and surpasses it in the WoW case). Added-context benchmarks: when the benchmark contexts are augmented with a persona and topic as described in section 3.2, the evaluation results barely change. Mixed-candidates evaluation: when the set of benchmark candidates is tripled by adding candidates from the other two benchmarks in equal proportion, the performance of the best respective single-task models suffers, while the MT Single-Skills model proves more resilient. Note that Single-task averages in italics do not correspond to a single model, but an average over 3 models.
ity check, but they indeed barely change the numbers on single-skill benchmarks.
4.4 Results on BlendedSkillTalk benchmark
We show two types of results on the Blended-SkillTalk benchmark (BST). Single-skill models are tested directly on BST without any additional training in a zero-shot setting, or fine-tuned on the
Model	BST, zero-shot	+BST, FT
ConvAI2	76.8	81.7
WoW	67.5	79.4
ED	69.0	80.4
BST	-	79.2
Random-Skill	71.2	-
MT Two-Stage	71.9	-
MT Single-Skills	80.1	83.8
Table 6: Test results on BlendedSkillTalk. BST, zero-shot: the models are tested directly on the test set of BST without having been fine-tuned on the BST train set. +BST, FT: models are fine-tuned on the BST train set, then tested on the BST test set. Multi-Task Single-Skills + BlendedSkillTalk performs best. The Multi-Task Two-Stage model outperforms two of the single-skill models, but the latter work well when combined with BlendedSkillTalk fine-tuning. We hypothesize that ConvAI2 alone performs well because it has been trained to use persona contexts, that are used throughout the BST dialogues.
BST training set then tested on the BST test-set. Results for both settings are shown in Table 6. The Multi-Task Single-Skills model outperforms all single-skill model baselines, whether used in a zero-shot or fine-tuned fashion, despite being the same size. The MT Two-Stage and Random-Skill models outperform two of the three single-skill models. We hypothesize that the ConvAI2 model is doing better because it has already learned to use personas. All single-skill models show improved performance once fine-tuned on the BST train set. However, performance in the zero-shot setting is already good, which is promising in terms of generalization to unseen data.
4.5 Human Evaluation on Specific Skill Axes
Human evaluation results are shown in Table 7. Single-skill models tend to generally be rated better than the other single-skill models on the skill they were optimized for, although all single-skill models are similarly rated on the knowledge axis. Models that have been trained on multiple skills, either through multi-tasking (MT Two-Stage or MT Single-Skills) or through fine-tuning on BST, are performing well on every dimension, with the MT Two-Stage model and the MT Single-Skills fine-tuned on BST being the overall best. These two models have different advantages: the MT Single-Skills model fine-tuned on BST is more compact, being the same size as each individual single-skill model, but requires joint multi-task training, then fine-tuning. The MT Two-Stage
Model	Knowledge	Empathy	Personal	Overall quality
ConvAI2	3.2	3.1	3.4	3.0
WoW	3.3	2.9	2.7	2.6
ED	3.4	3.3	3.0	3.0
BST	3.5	3.6	3.1	3.3
Random-Skill	3.2	2.9	3.2	2.7
MT Two-Stage	3.7	3.6	3.3	3.5
MT Single-Skills	3.7	3.6	3.0	3.4
MT Single-Skills +BST fine-tuning	3.7	3.8	3.2	3.6
Table 7: Human evaluation results on individual axes of knowledge, empathy, and being personal, as well as overall quality. All results here have a  $95\%$  confidence interval of  $\pm 0.2$  or 0.3, omitted to avoid cluttering the table. Results that are within the confidence interval of the best model performance are bolded. ConvAI2, WoW, ED: models pre-trained on pushshift.io Reddit and fine-tuned on the respective datasets. For Empathy and Personal topics, the individual models tend to do better when trained on a dataset tailored for that, however they all perform similarly on the Knowledge dimension. BST: model pre-trained on pushshift.io Reddit and fine-tuned on BST. This model is showing better overall performance compared to single-skill datasets (i.e., none of the three single-skill dataset do better than BST in every dimension). MT Single-Skills with fine-tuning on BST and MT Two-Stage are performing very well on all dimensions. MT Single-Skills with fine-tuning on BST has fewer than a third of the parameters of the MT Two-Stage model, yet manages to perform as well, if not slightly better.
model only requires training a classifier to play the role of a dialogue manager by assigning utterances to one of the three single-skill benchmarks, but is overall a much bigger model, given that it uses large models for each single skill and the classifier itself. The "Random-Skill" model is bypassing the need for a classifier by simply using all three single-skill model randomly, and is rated well on the personal axis, but not as well on knowledge or empathy, which might be because talking about personal topics can always work, while knowledge and empathy have to be suited to the context.
This paper focuses on the goal of creating an open-domain conversational agent that can display many skills, and blend them in a seamless and engaging way. We have shown several ways to leverage previous work focusing on individual conversational skills, either by combining trained single-skill models in a two-stage way, by re-using the datasets for simultaneous multi-task training, and by fine-tuning on the overall blended task. We compared the performance of these schemes on BlendedSkillTalk, a new English-language dataset blending three conversation skills in balanced proportions (demonstrating knowledge, empathy, or ability to talk about oneself). We showed that multiple multi-task approaches can be effective on this task, however careful construction of the training scheme is important to mitigate biases when blending and selecting skills, while fine-tuning on
the overall blended task improves models further.
One natural extension would be to generalize these findings to other skills than the three addressed here, such as humor/wit, eloquence, image commenting, etc. This would in principle be straightforward to do as long as these additional skills have a corresponding "single-skill" dataset to train on and are sufficiently distinguishable from each other.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4536-4545, Hong Kong, China. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The
second conversational intelligence challenge (ConVAI2). In The NeurIPS '18 Competition, pages 187-208, Cham. Springer International Publishing.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Real-time inference in multi-sentence tasks with deep pretrained transformers. arXiv preprint arXiv:1905.01969.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019. MoEL: Mixture of empathetic listeners. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 121-132, Hong Kong, China. Association for Computational Linguistics.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84, Copenhagen, Denmark. Association for Computational Linguistics.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2322-2332, Brussels, Belgium. Association for Computational Linguistics.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5427-5436, Florence, Italy. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1):53-93.