SlowGuess/ACL_2020_OCR huggingface dataset:
Kurt Shuster, Da Ju, Stephen Roller Emily Dinan, Y-Lan Boureau, Jason Weston
{kshuster,daju,roller,edinan,ylan,jase}@fb.com
We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pretrained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge.
One of the goals of AI is to build a seeing, talking agent that can discuss, reason, empathize, and provide advice – in short a system that can perform natural communication displaying many of the properties expected when speaking to a human partner. Ideally, it should be able to be knowledgeable and personable, expert and engaging, serious or humorous – depending on the situation. It should be capable of answering questions, asking questions, responding to statements, having its own persona, and grounding the dialogue with external information and images.
While no single task exists that can train an agent or measure its ability on all of these axes at once, a number of distinct large-scale datasets targeting subsets of these skills have recently become available. We thus assemble these disparate tasks to
form a single challenge: dodecaDialogue, consisting of 12 subtasks. Each contains both training data to build the skills we desire for our agent, and validation and test sets to measure our agent's ability at that skill. The overall goal is a single agent that can display all these skills. As some of the subtasks have very large datasets, e.g. 2.2 billion utterances, they can possibly help the agent with other skills too.
We thus build a model capable of training and multi-tasking on all these sources. We employ a transformer-based architecture (Vaswani et al., 2017) which accepts an image, external textual information and dialogue history as input, and generates a response for a given dialogue turn. Practically, by pre-training on the largest of the subtasks and then multi-tasking on all them, we can obtain state-of-the-art results compared to existing independently reported performance on all 10 of the 12 subtasks that have previous comparable results. We hence set a strong baseline for this challenge. While many existing approaches use large-scale pre-training on general text corpora, we show that using dialogue datasets instead, which are more closely linked to the desired agent's goals, is a strong alternative.
However, many challenges remain. While multitasking performs well, and has clear benefits, as shown in other works (Liu et al., 2015; Raffel et al., 2019), when compared to fine-tuning of the same system we do obtain typically small losses. Zero-shot transfer to left-out tasks is also demanding for current approaches. We analyze these aspects, along with our model's ability to ground on external knowledge and images in conjunction with the dialogue context, the impact of decoding algorithms, analysis of the weighting of tasks during multi-tasking as well as cross-task transfer ability in order to shed light and make progress on this challenging topic.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2453-2470 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Name	Ask Questions Answer Questions Respond to Statements Knowledge Grounding Situation Grounding	Train	Valid	Test	# Turns	Resp. Length
ConvAI2	✓	✓	✓	✓	131,438	7,801	6,634	14.8	11.9
DailyDialog	✓	✓	✓	87,170	8,069	7,740	7.9	14.6
Wiz. of Wikipedia	✓	✓	✓	✓	74,092	3,939	3,865	9.0	21.6
Empathetic Dialog	✓	✓	✓	✓	40,252	5,736	5,257	4.3	15.2
Cornell Movie	✓	✓	✓	309,987	38,974	38,636	4.0	15.0
LIGHT	✓	✓	✓	✓	110,877	6,623	13,272	13.0	18.3
ELI5	✓	✓	231,410	9,828	24,560	2.0	130.6
Ubuntu	✓	✓	✓	1,000,000	19,560	18,920	2.0	18.9
Twitter	✓	✓	✓	2,580,428	10,405	10,405	2.0	15.7
pushshift.io Reddit	✓	✓	✓	~ 2200 M	10,000	10,000	2.0	35.0
Image Chat	✓	✓	✓	✓	355,862	15,000	29,991	3.0	11.4
IGC	✓	✓	✓	4,353	486	7,773	3.0	8.6
Table 1: The 12 dodecaDialogue subtasks, their sizes (number of train, valid, test utterances), and average number of turns and response length (words).
2 The dodecaDialogue Task
The dodecaDialogue task is intended to assemble important aspects of an engaging conversational agent into a single collection, where each subtask covers some of those goals. Such an agent should be able to get to know you when you first talk to it (ConvAI2), discuss everyday topics (DailyDialog, pushshift.io Reddit, Twitter, Cornell Movie), speak knowledgeably at depth (Wizard of Wikipedia, Ubuntu) and answer questions on such topics (ELI5). It must be able to handle situated conversations and demonstrate empathy (Empathetic Dialog, LIGHT). It can also discuss images, as this is a vital part of human connection (Image Chat, IGC). We note that all of the provided subtasks are in English.
The overall statistics of the subtasks are given in Table 1. We now discuss each in turn.
ConvAI2 ConvAI2 is a dataset used at the NeurIPS 2018 competition of the same name, and is based on PersonaChat (Zhang et al., 2018; Dinan et al., 2020). The training data involves paired crowdworkers having a conversation where they get to know each other, in which each is given a role to play based on sentences describing their persona, which were also separately crowdsourced (while they cannot see their partner's persona). It thus involves asking and answering questions, responding in kind, and getting to know the other speaker and engaging them in friendly conversation – useful skills for an open-domain conversational agent.
DailyDialog Li et al. (2017) built a dialogue dataset intended to reflect conversations occurring in daily life. It covers ten categories ranging from holidays to financial topics, rather than focusing on one domain. Compared to ConvAI2, these conversations seem more in keeping with partners who already know each other, and want to discuss typical life details, again useful skills for a conversational agent. The dataset is also annotated with topic, emotion and utterance acts, but here we ignore these annotations and learn only from the utterances in the dialogue turns.
Wizard of Wikipedia This task involves discussing a given topic in depth, where the goal is to both engage the partner as well as display expert knowledge (Dinan et al., 2019). The training set consists of 1247 topics and a retrieval system over Wikipedia from which the dialogues were grounded during the human-human crowdsourced conversations. The topics were also crowdsourced and range from e-books to toga parties to showers. A model can thus learn to also perform similar retrieval and grounding at test time to potentially discuss any topic if it can generalize. We use the gold knowledge version of the task. We see this skill as a core component of an agent being able to not just chitchat, but actually engage a user in discussing real information about the world, e.g. by retrieving over documents from the internet.
Empathetic Dialogues Rashkin et al. (2019) constructed a dataset of crowdworker conversations grounded in an emotional situation. In each dia
2454
logue, one speaker describes a personal situation and the other plays a " listener" role, displaying empathy during the discussion. The dataset contains descriptions of the situations being discussed with an attached emotion label, but these are not used here. Trained models are measured playing the part of the empathetic listener, an important feature of an agent to which humans wish to speak.
Cornell Movie Danescu-Niculescu-Mizil and Lee (2011) constructed a corpus containing a collection of fictional conversations from movie scripts, thus covering a large diversity of topics and emotional states.
LIGHT LIGHT (Urbanek et al., 2019) involves situated interactions between characters in a text adventure game. Similar to ConvAI2, personas for each character are given, with the training set including conversations between coworkers playing those roles. Different from ConvAI2, included are emotes and actions grounded within the game world (e.g. picking up and giving objects). As such, it measures the ability of a conversational agent to ground its discussion on a dynamic environment.
ELI5 ELI5 (Fan et al., 2019) involves long-form question answering grounded on multiple retrieved documents in order to answer common questions which people ask on the popular ELI5 subreddit. As such, the answers are in a conversational form applicable to a dialogue agent.
Ubuntu Lowe et al. (2015) built a dataset that involves in-depth discussions in solving Ubuntu problems. This studies the ability of an agent on a very focused single topic, and is also a standard benchmark in the field.
Twitter We use a variant of Twitter discussions (text-only), which have been used in many existing studies, e.g. Sordoni et al. (2015); See et al. (2019). This data naturally involves everyday discussions about topics that people care about. The public forum makes them different from the more personal discussions of some of the other tasks. This is the second largest dataset in the collection, and we thus measure in experiments its ability to help performance on other tasks.
pushshift.io Reddit We use a variant of Reddit discussions (text-only), which has also been used in several existing studies, see e.g. Yang et al. (2018); Mazaré et al. (2018); Keskar et al. (2019). Following Humeau et al. (2019), we use a previously
existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io, training to generate a comment conditioned on the full thread leading up to the comment, spanning 2200M training examples. This is the largest dataset in the collection – much larger than the others. The subreddits cover a vast range of topics, and hence this is a strong candidate for helping improve performance on other tasks via pre-training and multi-tasking. Note this dataset does not overlap with ELI5.
Image Chat Shuster et al. (2018) collected a crowdsourced dataset of human-human conversations about an image with a given personality, where the goal is to engage the other speaker. As such, it covers natural conversational responses, including displays of emotion and humor.
Image Grounded Conversations (IGC) IGC (Mostafazadeh et al., 2017) similarly involves two speakers discussing an image, here focusing on questions and responses. It only includes a validation and test set, and so we converted most of the validation set to form a small training set.
2.1 Evaluation
Metrics For all tasks, we use the following metrics: perplexity (PPL), BLEU,ROUGE-1,-2 and -L and F1, and also pick the metric most used in the literature as that subtask's 'Score' to compare to existing work.
Multi-tasking As we are interested in building a single conversational agent, we measure the ability of multi-tasked models that can perform all twelve tasks at once.
Single-Task Fine-tuning We can still compare such multi-tasked models to single-task fine-tuned baselines to assess if we have gained or lost performance. Like other works (Liu et al., 2015; Raffel et al., 2019) we also consider a multi-task followed by finetune setup in order to see if this produces better models. The latter tests if multi-tasking still proves useful in the single-task setting.
Zero-shot Transfer Finally, we consider a leave-one-out zero-shot setting whereby training is constrained to be on all the training data except for the task being evaluated. This evaluates the performance on truly new unseen tasks, an important behavior given there are always new tasks.
3.1 Existing Models and Results
Where possible, we have tried to track the best existing results for each task and provided a comparison in our final results table.
As ConvAI2 was a competition, a number of competitors built strong models on it. The best results were obtained by large pre-trained transformers (Dinan et al., 2020). In particular, Wolf et al. (2019b) pre-trained via the method of Radford et al. (2018) using the BooksCorpus dataset, resulting in the best perplexities and F1 scores. Since then, results have gotten even better with the advent of better and larger pretraining (Lewis et al., 2019), which we compare to here; the same work also reports strong results on ELI5.
He et al. (2019) recently obtained strong results on the DailyDialog and Cornell Movie tasks in terms of perplexity by pre-training on  $10\%$  of CCNEWS (Bakhtin et al., 2019), thus using 100 million sentences (2.7 billion words) and then fine-tuning a transformer based model with a multi-task strategy.
Overall, large pre-trained transformers indeed provide strong existing results on many of the tasks. Several large language modeling projects have been undertaken in order to show prowess in multi-tasking ability (Radford et al., 2019; Keskar et al., 2019), and transformer-based approaches have been adapted to language and vision tasks as well (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Shuster et al., 2018). As well as citing the relevant papers' results where possible in the experiments section, we also train a BERT-based (Devlin et al., 2019) generative model as an additional baseline.
3.2 Related Tasks and Collections
In the interests of feasibility, there are tasks we did not include in dodecaDialogue. For example, there are additional knowledge tasks (Qin et al., 2019; Moghe et al., 2018; Gopalakrishnan et al., 2019) and image-based datasets (Das et al., 2017) one could use. There are also a large number of QA tasks we did not include, e.g. Rajpurkar et al. (2016); Choi et al. (2018). In general, our choices were made based on tasks that after training might produce an engaging dialogue agent that humans naturally would want to talk to – which means either natural datasets or crowdsourced datasets where crowdworkers were encouraged to engage
one another. As computational resources and ambitions scale, it would be interesting to add more tasks as well, while retaining the twelve we have chosen here in order to continue to evaluate their success, whilst extending the scope of the entire system.
All the subtasks in the collection we use here already exist. Other research projects have also built such collection-based tasks before as well. In particular, the NLP decathlon (McCann et al., 2018), from which the name of this paper is inspired, collects together a diverse set of NLP tasks – from sentiment detection to parsing. Talmor and Berant (2019) collect a set of 10 QA datasets and build MULTIQA. Recently, (Raffel et al., 2019) also similarly multi-tasked a large set of NLP tasks, on an even bigger scale. Our work differs from these in that it is focused on dialogue tasks which naturally group together to form a conversational agent.
BERT baseline. We implement a generative baseline using BERT via adapting the model using a standard auto-regressive loss. We concatenate both the context and current generation and provide these as input to the model, using BERT's sentence embeddings to distinguish the roles in the network. Although BERT is trained to predict masked tokens, we find that fine-tuning can easily adjust its behavior to predicting the next token. Our BERT baseline is roughly equivalent to the model of Wolf et al. (2019b), but does not have a classification loss term. The implementation relies on HuggingFace Transformers (Wolf et al., 2019a). We thus fine-tune this model for each of our tasks, except Image Chat and IGC which require images as input.
Image+Seq2Seq. We use a modification of a transformer Seq2Seq architecture (Vaswani et al., 2017), additionally adding image features to the encoder. Our model is a 8 layer encoder, 8 layer decoder with 512 dimensional embeddings and 16 attention heads, and is based on the ParlAI implementation (Miller et al., 2017). We use BPE following Humeau et al. (2019) via lower-cased Wikipedia, Toronto Books, and Open Subtitles with 30k merges, giving 54,940 terms. Reported perplexities are computed with this dictionary. For image features, we use the pre-trained image features from the ResNeXt-IG-3.5B model, a ResNeXt 32 x 48d architecture (Xie et al., 2017) trained on 3.5 billion Instagram images following the procedure
BERT-based	Single Task (from scratch)	Twitter + Single Task	Reddit Only	Reddit + Single Task	MT All Tasks + FT Single Task	All Tasks MT	Leave-One-Out
Single Task (fastText init)
ConvAI2	19.4	43.3	38.9	28.7	18.3	11.4	11.2	11.3	16.4
DailyDialog	15.2	37.8	32.8	20.8	18.2	10.4	10.2	11.8	15.5
Wiz. of Wikipedia	14.1	40.7	36.0	37.3	15.3	8.7	8.5	8.7	13.2
Empathetic Dialog	23.2	47.1	40.5	23.1	14.4	11.3	11.1	11.2	13.0
Cornell Movie	29.4	46.2	44.8	34.2	27.8	20.0	19.8	22.3	25.4
LIGHT	29.7	63.6	57.5	40.0	32.9	18.7	18.7	19.0	26.9
ELI5	28.1	62.9	58.8	63.8	31.2	21.2	21.1	25.0	31.1
Ubuntu	20.7	35.8	34.5	38.5	31.1	17.3	17.2	23.3	30.8
Twitter	37.0	61.9	59.3	59.3	53.6	29.8	29.8	37.0	52.8
pushshift.io Reddit	39.0	27.8	27.8	27.8	27.8	27.8	25.8	28.0	106.3
Image Chat	N/A	40.1	37.4	31.1	32.5	18.3	18.3	21.8	29.3
IGC	N/A	86.3	79.5	23.1	14.6	10.0	10.0	10.2	12.2
dodecaScore	N/A	49.5	45.7	35.6	26.5	17.1	16.8	19.1	31.1
described by Mahajan et al. (2018). This model was previously used successfully for the Image Chat task in Shuster et al. (2018). The final encoding from the ResNeXt model is a vector of size 2048; we then use a linear layer to project into the same size as the text encoding, and add it as an extra token at the end of the transformer's encoder output, then feed them all into the decoder. During fine-tuning we train the text transformer, but leave the image encoding fixed, apart from fine-tuning the linear projection. The text transformer is fine-tuned with a standard auto-regressive negative log-likelihood (NLL) loss, following usual sequence to sequence training schemes.
Our best models are available at https:// parl.ai/projects/dodecadialogue.
Task Training We employ the ParlAI framework (Miller et al., 2017) for training on single tasks and for multi-tasking, as many of the tasks are already implemented there, along with a (multi-task) training and evaluation framework for such models.
Pre-training As pushshift.io Reddit and (to some extent) Twitter are much larger than our other tasks, we try pre-training the Seq2Seq module of our Image+Seq2Seq networks with those datasets, before multi-tasking on all of the tasks, or for evaluating single task fine-tuning.
For Reddit, the model was trained to generate
Table 2: Validation perplexity for the dodecaDialogue tasks in various settings.
Model	ConvAI2	WiZ. of Wikipedia Empathetic Dialog
Reddit	18.3	15.3	14.4
Reddit+ConvAI2	11.4	14.2	14.7
Reddit+Wiz. of Wikipedia	16.3	8.7	14.0
Reddit+Empathetic Dialog	17.9	15.3	11.3
Multi-Tasking All 4 Tasks	11.6	8.7	11.2
Table 3: Transfer performance of various multi-task models (validation perplexity).
a comment conditioned on the full thread leading up to the comment. Comments containing URLs or that were under 5 characters in length were removed from the corpus, as were all child comments. Comments were truncated to 1024 BPE tokens. The model was trained with a batch size of 3072 sequences for approximately 3M updates using a learning rate of 5e-4, and an inverse square root scheduler. This took approximately two weeks using 64 NVIDIA V100s. We note that our transformer pre-training only includes text, while our image encoder was pre-trained separately in previous work (Mahajan et al., 2018). Learning how to combine these sources occurs during fine-tuning.
It is important to note that, while compute-heavy, pre-training was conducted exactly once, and all of the subsequent fine-tuning is significantly faster to run.
Knowledge grounding	Without	With
Wiz. of Wikipedia	16.8	8.7
ELI5	21.3	21.2
Image grounding
Image Chat	19.5	18.3
IGC	10.1	10.1
Transfer Performance between Tasks We first perform a preliminary study on a subset of the tasks: Reddit, ConvAI2, Wizard of Wikipedia and Empathetic Dialogues, and report the transfer ability of training on some of them, and testing on all of them (using the validation set), reporting perplexity. The results are reported in Table 3. They show that training on pushshift.io Reddit alone, a huge dataset, is effective at transfer to other tasks, but never as effective as fine-tuning on the task itself. Moreover, fine-tuning on most of the smaller tasks actually provides improvements over pushshift.io Reddit training alone at transfer, likely because the three tasks selected are more similar to each other than to pushshift.io Reddit. Finally, training on all four tasks is the most effective strategy averaged over all tasks compared to any other single model, although this does not beat switching between different fine-tuned models on a per-task basis.
Comparison of Pre-training + Fine-tuning strategies Across all 12 tasks, we compare several pre-training strategies: using BERT, no pretraining at all, only initializing via fastText (Joulin et al., 2017), and using Twitter and pushshift.io Reddit pre-training with our Image+Seq2Seq architecture. For each variant we tune the learning rate, layers, number of heads and embedding size, with less pre-training typically requiring smaller capacity models. We then only fine-tune on a single task in these experiments, and report perplexity for that task alone, over all 12 tasks. The results are given in Table 2, reporting results on the validation set<sup>1</sup>.
The results show a clear reduction in perplexity with more pre-training, as expected. This is most easily seen by the dodecaScore (last row) that is the mean perplexity over all 12 tasks, which decreases from 49.5 (from scratch models) down to 17.1 with pushshift.io Reddit pre-training. FastText (45.7) and Twitter (35.6) initializations help, but nowhere near as much. BERT fares better, but still is clearly
Table 4: The impact of knowledge and image grounding in dodecaDialogue (validation perplexity).
Relative Task Weighting
1	2	5	10	20	50	∞
Cornell	21.9	21.5	20.6	20.1	19.9	19.8	-
Fine-tuned	20.1	20.0	20.0	19.9	19.8	19.8	20.0
ELI5	25.0	24.1	22.8	22.2	21.6	21.3	-
Fine-tuned	21.8	21.6	21.4	21.3	21.1	21.1	21.2
Ubuntu	23.1	22.2	20.6	19.6	18.6	17.4	-
Fine-tuned	18.2	18.1	17.8	17.7	17.2	17.2	17.3
Table 5: Validation perplexity on select do-decaDialogue tasks comparing relative weights of tasks during multi-tasking, followed by fine-tuning (row below). The relative task weight is the ratio of examples from that task compared to others presented during multitasking.  $\infty$  indicates single-task training.
Task	1	Beam Size	N-gram Block N = 3	Nucleus p =0.3
2	3	5
ConvAI2	20.0	21.0	21.3	21.2	21.3	18.7
WoW	35.9	37.4	37.8	37.9	37.9	31.1
Table 6: Impact of the decoding strategy on select tasks, reporting validation F1 score for the All Tasks MT model. N-gram block is for best beam size.
worse than pushshift.io Reddit pre-training. The hypothesis here is that pushshift.io Reddit yields much more effective transfer as it is a dialogue task like our others, whereas non-dialogue corpora such as Wikipedia are not. This was previously observed for retrieval models in Humeau et al. (2019). Note that we do not report results for the image dialogue tasks for BERT as that architecture does not deal with images.
Finally, as pushshift.io Reddit is so effective, we also compare to pushshift.io Reddit training only, with no fine-tuning at all across all tasks, similar to our initial study in Table 3. The performance is impressive, with some tasks yielding lower perplexity than BERT pre-training + single task fine-tuning. However, it still lags significantly behind fine-tuning applied after pushshift.io Reddit pretraining.