SlowGuess/ACL_2020_OCR huggingface dataset:
Katherine Stasaski $^{1}$ , Grace Hui Yang $^{2}$ , and Marti A. Hearst $^{1}$
2Georgetown University
$^{1}\{katie\_stasaski, \text{hearst}\} @$ berkeley.edu
huiyang@cs.georgetown.edu
Automated generation of conversational dialogue using modern neural architectures has made notable advances. However, these models are known to have a drawback of often producing uninteresting, predictable responses; this is known as the diversity problem. We introduce a new strategy to address this problem, called Diversity-Informed Data Collection.
Unlike prior approaches, which modify model architectures to solve the problem, this method uses dynamically computed corpus-level statistics to determine which conversational participants to collect data from.
Diversity-Informed Data Collection produces significantly more diverse data than baseline data collection methods, and better results on two downstream tasks: emotion classification and dialogue generation. This method is generalizable and can be used with other corpus-level metrics.
It is well-documented that neural dialogue models struggle with generating engaging, relevant responses (Li et al., 2016a) and often produce banal responses such as "Yeah." While this may be an appropriate response to a chitchat conversation, to keep a human participant engaged, diversity of responses is important. Diverse models vary the language used and the content referenced, and the generated utterances differ from the most typical conversation responses some proportion of the time. A model which only generates "Yeah," "No," and "I don't know" is not diverse and is not be engaging to converse with.
Past work has improved model diversity with innovation on model architectures and decoding strategies (Li et al., 2016a; Baheti et al., 2018; Li et al., 2017; Shao et al., 2017; Cao and Clark, 2017; Serban et al., 2017; Zhao et al., 2017). We build
upon this work to propose a novel method to collect and determine more diverse data to train these models with. Our method can be used in conjunction with existing generation-specific model innovations.
Some prior work on data collection processes has prioritized diversity. For instance, Rashkin et al. (2019) prompts crowdworkers to choose an underused emotion class to generate dialogue. This work encourages coverage of emotion classes, but does not consider the likelihood that some crowd-workers are better at producing certain types of data than others.
This paper introduces Diversity-Informed Data Collection (DIDC), a new strategy for creating a dataset of conversational utterances via selecting which participants' data to include in the collection. The strategy progressively builds up a more diverse sub-corpus from an existing larger collection. The main idea is to grow the sub-corpus by adding conversations sequentially and to assess the contribution of a new participant's utterances to the diversity of the entire sub-corpus. This strategy is also applicable to on-the-fly collection of new datasets via crowdsourcing or similar methods. We implement DIDC with three diversity metrics: Outlier, Entropy, and Mean-IDF.
Diversity-Informed Data Collection also provides a new method for finding an upper bound on a current corpus's diversity via a Corpus-Wide Oracle which has access to information about which utterances are most diverse across the corpus.
Prior work has not used corpus-level statistics to enhance the diversity of the collected data. Instead, when collecting data with crowdworkers, researchers have sought more diverse responses by altering the task (Kang et al., 2018) or by altering the stimulus (Larson et al., 2019). Prior work that trains neural dialogue models has not made use of subsets of existing datasets that exhibit properties
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4958-4968 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
of diversity.
Our experiments show this strategy yields significantly more diverse data than baseline collection processes. It also yields better, more diverse model output on two downstream tasks. Additionally, this method can be implemented for other metrics which are defined relative to the corpus.
Past work in neural dialogue generation investigates how to improve diversity in conversational responses. Additionally, past work in crowdsourcing data collection has explored optimizing crowdsourcing data collection processes.
2.1 Diverse Neural Dialogue Generation
Improving model diversity is an important goal in dialogue generation (Li et al., 2016a), with several related works proposing architecture and training improvements to increase diversity.
Decoding methods to increase model diversity include Li et al. (2016a) which proposes maximizing mutual information between the source sentence and response rather than maximizing likelihood. Other approaches have focused on beam search and incentivizing diverse beams, by adding similarity constraints at decoding (Baheti et al., 2018), penalizing items on the beam that are similar and reranking resulting items (Li et al., 2016b), or penalizing words which have already been generated in a current beam (Li et al., 2017). Shao et al. (2017) uses attention over already-generated words at decode time and beam reranking. Adding a temperature parameter to sharpen the decoder's distribution has also been studied (Cao and Clark, 2017).
Neural architecture improvements have also been explored, such as conditioning on a latent variable at decode time (Serban et al., 2017; Zhao et al., 2017) or a multi-headed attention mechanism which aims to capture different parts of the context (Tao et al., 2018). Zhang et al. (2018) explore the use of Generative Adversarial Networks to incentivize diversity. These more diverse models and decoding methods can be used in conjunction with Diversity-Informed Data Collection, since it attempts to improve the data that neural models are trained on in an earlier part of the model pipeline.
2.2 Crowdsourcing
Related work in crowdsourcing has approached the optimization problem of how to assign crowdwork-
ers to different tasks.
2.2.1 Crowdworker Task Assignment
Basu Roy et al. (2015) formulates the problem of matching crowdworkers to tasks depending on skill levels for a set of concepts, pay rates, and HIT acceptance ratio. Follow-up work extends to collaborative crowdwork, where crowdworkers need to work together (Rahman et al., 2015). Assadi et al. (2015) pursue a similar task assignment setup.
Additional work has attempted to automatically evaluate crowdworker quality of task performance and use the results to assign crowdworkers to new tasks on-the-fly (Fan et al., 2015). Further investigations have explored more adaptive assignment of tasks in real-time based on the likelihood that a participant will continually complete tasks (Kobren et al., 2015). Relatedly, Kumai et al. (2018) design a task allocation to minimize the stress of workers and maximize the resulting quality in terms of balanced skill performance.
2.2.2 Label Distribution Prediction
An additional area related to our work is crowdworker label distribution prediction. Liu et al. (2019) has a crowdsworking labeling task and trains models to predict the 50-label crowdworker distribution from 5-10 labels. Yang et al. (2018) aim to predict diversity in crowdworker answers to questions about an image to determine how many crowdworker responses are required to capture this diversity.
2.2.3 Dynamic Crowdworking Tasks
Lin et al. (2018) tackle the task of employing crowdworkers to generate or label minority class examples to feed an active-learning model. They deploy a multi-armed bandit to choose crowdworking tasks based on how cheaply a minority-class example can be generated using the technique. Our approach, by contrast, adapts a distributional constraint across the entire collection. Zhou et al. (2018) explores the related task of changing crowdworker team instruction prompts.
2.2.4 Diverse Crowdworking
Data collection approaches to incentivize diverse crowdworker output have also been studied. For instance, in EmpatheticDialogues (Rashkin et al., 2019) crowdworkers are conditioned to generate a response and an emotion (such as "afraid" or "proud") associated with it. If workers do not generate text with certain emotions, they are prompted
to select only from the underused labels. This is an example of trying to get better class coverage, but does not compare crowdworker output to the entire corpus of collected responses.
Past work has also examined how the particular crowdworking task affects the diversity of crowdworker output. Kang et al. (2018) compare two crowdsourcing tasks for use in a downstream goal-oriented dialogue system and examine resulting data diversity. While Kang et al. (2018) focus on choosing a task which produces diverse utterances, our work focuses on choosing a participant population which produces diverse data compared to data which has already been collected.
Building on Kang et al. (2018), and perhaps most similar to our work is Larson et al. (2019), which tackles the problem of detecting outlier paraphrases generated by crowdworkers. To obtain multiple ways of expressing similar intent (such as opening a bank account), crowdworkers are asked to paraphrase sentences. After a round of paraphrase collection, the most diverse (the outlier) paraphrases are identified and placed back onto the crowdsourcing platform for another round of data collection.
Our method is similarly aimed at increasing diversity of collected data. However, our method adapts the participant population for a set of tasks, which can be used in addition to an approach like Larson et al. (2019) which adapts the stimulus the population works on.
3 Diversity-Informed Data Collection
We propose a method, Diversity-Informed Data Collection, which progressively builds up a corpus, and while doing so, identifies which conversation participants produce more diverse utterances compared to the rest of the in-progress corpus. More formally, our task is to progressively build a subcorpus,  $sub_{c}$ , of a given size from a larger, precollected corpus,  $c$ , where utterances are tied to IDs of specific participants.
Our approach is aimed at building a diverse subcorpus  $\text{sub}_c$ . Our approach chooses which population of participants to collect data from for a given round. This population changes dynamically depending on calculated participant's diversity scores.
When utilizing a human-created, pre-existing corpus, we assume responses of the dataset are well-formed and of acceptable quality. With this assumption, we can maximize diversity scores without worrying that quality will be sacrificed for this
diversity. However, when using this approach to collect data on-the-fly, additional quality controls may be necessary to ensure diverse data does not come at the cost of quality.
We assess two experimental conditions: Simulated Data Collection and Corpus-Wide Oracle Upper-Bound. Simulated Data Collection is set up to mimic crowdsourcing data collection processes leveraging a large pre-collected corpus, while Corpus-Wide Oracle Upper-Bound gathers an maximally diverse sub-corpus of utterances.
For all experiments, we utilize the pre-collected EmpatheticDialogues corpus (Rashkin et al., 2019). We experiment with this corpus because it has crowdworker IDs associated with each utterance, which allows us to experiment with varying the participant population. Future work should conduct further experimentation to examine this approach's adaptability to other chitchat and goal-oriented datasets.
The corpus has a large number of utterances (100,000) over 25,000 conversations. Each conversation is centered around a situation (such as getting a promotion at work) and is associated with one of 32 emotions, such as anger, excitement, or guilt. Each conversation takes place between two crowdworkers and is an average of 4.3 turns. There are 810 unique crowdworkers in this dataset, each completing an average of 132 utterances each across an average of 61 conversations.
Our task is to select  $sub_{c}$  of size 10,000 from the larger EmpatheticDialogues corpus,  $c$ . We choose 10,000 as it is a sufficient number of utterances to train downstream models but still a small proportion (10%) of the original dataset, allowing examination of differences between sub-corpora. Implementation utilizes Cornell Convokit (Chang et al., 2019).
3.2 Simulated Data Collection
We simulate real-time crowdsourcing using a large, pre-collected corpus,  $c$ . This allows for running multiple trials, each time selecting  $sub_{c}$  and examining significance of different diversity metrics and participant selection conditions.
We simulate collecting data on-the-fly using an artificially-constructed environment (formally described in Algorithm 1), which completes multiple rounds of data collection until the progressively built sub-corpus  $size(sub_{c})$  is the desired size. The
procedure assumes a fixed number of conversation participants in each round to gather data from (set to 10 for our experiments). We collect 2 conver
sations from each participant, chosen to allow the algorithm to recover from a participant with low diversity utterances while not judging a participant on just one conversation.
Given a participant's conversation, the diversity of an utterance in that conversation is stated in Equation 1:
$$
d i v _ {u t t} = \text {C o m p u t e D i v e r s i t y} (u t t, \operatorname {s u b} _ {c}) \tag {1}
$$
where ComputeDiversity depends on the diversity metric examined. We obtain a diversity score for each participant  $p$ 's set of utterances  $(u t t s_{p})$  by averaging these diversity values:
$$
d i v _ {p} = \frac {1}{\text {s i z e} (u t t s _ {p})} \sum_ {u t t \in u t t s _ {p}} d i v _ {u t t} \tag {2}
$$
At the end of each round of data collection,  $u t t_{p}$  is added to  $sub_{c}$  for each participant. Additionally, the algorithm determines which subset of the participant population is retained for the next round based on a Participant Population Selection strategy.
Our algorithm is greedy, since the order participants are added to the simulation and the order in which conversations are sampled both affect the participant's likelihood to be retained for an additional round. However, crowdworker data collection itself is usually a greedy approach, with crowdworkers being assigned to tasks in the order they arrive and being allowed to complete many tasks until the dataset has been collected.
3.2.1 Participant Population Selection
We experiment with three conditions to determine which sub-set of current participants (participants which were involved in the most recent round of data collection) should be retained for the next round of data collection, summarized in Table 1.
Diverse Population: After collecting conversations from current participants, we choose to retain the most-diverse  $70\%$  of participants.
Above Mean Population: Any participant whose diversity average falls above the mean diversity average of  $sub_{c}$  is retained in the pool of participants.
Random Population: We compare to a special random baseline, where at each iteration we retain a random  $70\%$  of the participant population, to directly compare to the  $70\%$  of crowdworkers
Condition	Description
Diverse Population	Calculates each participant's average relative diversity for current data collection round. We retain the 70% most-diverse participants of the current round.
Above Mean Population	Calculates each participant's average relative diversity for current data collection round. Retains the participants whose diversity scores fall above the sub-corpus's mean diversity.
Random Population	Retains a random 70% of participants.
Corpus-Wide Oracle	Uses a Corpus-Wide Oracle which ranks utterances' diversities in relation to the large dataset, c. Selects the most diverse utterances from these values independent of conversations.
Table 1: Participant Population Selection conditions for Simulated Data Collection. The first three conditions are used in conjunction with Algorithm 1, while the last condition provides an upper-bound for diversity by utilizing a Corpus-Wide Oracle to determine the known most-diverse utterances.
Metric	Description
Outlier	Euclidean distance between utter- ance embedding and average em- bedding for all utterances in the sub-corpus (Larson et al., 2019)
Entropy	Entropy of utterance under a tri- gram language model trained on sub-corpus.
Mean IDF	Mean IDF value (Baeza-Yates et al., 1999) for words in utter- ance compared to the rest of the corpus.
Table 2: Diversity metrics considered for data collection.
retained in Diverse Population. We structure Random Population to collect data from roughly the same number of participants as Diverse Population, to examine differences between the resulting  $sub_{c}$  due to the selection of which participants to
retain for another round of data collection.
3.2.2 Diversity Metrics
We experiment with three diversity metrics (Outlier, Entropy, and Mean IDF), summarized in Table 2. For all metrics, a new utterance  $u_{tt}$  is compared to the sub-corpus  $sub_{c}$ .
The same utterance can have different diversity values depending on the utterances in  $sub_{c}$ . When augmenting pre-collected data, this allows for the collection of new utterances which are relatively diverse.
Outlier: The embedding-based Outlier metric was proposed by Larson et al. (2019). Each utterance is encoded using a Universal Sentence Encoder (USE), which creates a sentence embedding by averaging word embeddings and passing the representation through a feedforward neural network, originally trained in a multi-task setting with supervised and unsupervised NLP tasks (Cer et al., 2018).
An embedding of an utterance is created via:  $E_{utt} = USE(utt)$ . A mean corpus vector is computed by averaging all of  $sub_c$ 's utterance's vectors:
$$
E _ {s u b _ {c}} = \frac {1}{\operatorname {s i z e} \left(\operatorname {s u b} _ {c}\right)} \sum_ {u \in s u b _ {c}} U S E (u) \tag {3}
$$
The diversity metric is the Euclidean distance between each new utterance and the mean corpus vector, or:
$$
\sqrt {\sum_ {i} \left(E _ {u _ {i}} - E _ {s u b _ {c _ {i}}}\right) ^ {2}} \tag {4}
$$
where  $i$  is a dimension in Embedding  $E$ .
Utterances which are farther from the mean corpus vector are given a higher diversity score. For Simulated Data Collection, the mean corpus vector shifts as data is collected. Therefore, depending on which utterances are already added in the sub-corpus, outlier values will change for a given utterance.
Entropy: The Entropy score is determined by a non-neural trigram language model with smoothing for unseen words. The diversity score is given by:
$$
- \frac {1}{| x \in \operatorname {T r i g r a m} (u t t) |} \sum_ {\substack {x \in \\ \operatorname {T r i g r a m} (u t t)}} p (x) \log p (x) \tag{5}
$$
The language model is only trained on utterances in the sub-corpus.
Mean IDF: This metric calculates the mean IDF value for each word in the utterance (Baeza-Yates et al., 1999). IDF is calculated by treating each utterance in the corpus as a document. For a given utterance  $u t t_{p}$  and sub-corpus  $s u b_{c}$ , Mean IDF is calculated via:
$$
\frac {1}{| u t t _ {p} |} \sum_ {w \in u t t _ {p}} \log \left(\frac {| \{s u b _ {c} \} |}{| \{u t t | w \in u t t \} |}\right) \tag {6}
$$
where  $\{sub_c\}$  is the set of all utterances in the  $sub_{c}$ . The IDF of a word  $w$  in  $utt$  is the number of utterances in  $sub_{c}$  divided by the number of utterances containing  $w$  on a log scale.
In addition to evaluating the robustness of our approaches, multiple diversity metrics are chosen with different conceptual types of diversity in mind. Outlier uses Universal Sentence Encoder embeddings which capture content (Cer et al., 2018). Entropy considers the probability of short phrases and can capture word combination diversity. Mean IDF considers the rarity of words being used for vocabulary diversity. Depending on the downstream application for a dialogue agent, the utility of these diversity measures may vary.
3.3 Corpus-Wide Oracle Upper Bound
To provide an Upper Bound for the diversity of a sub-corpus  $sub_{c}$ , we create a Corpus-Wide Oracle which knows the value of each utterance's diversity compared to the entire corpus  $c$ . For each  $utt \in c$ , we compute diversity according to the methods in Table 2, where  $sub_{c} = c$ . For example, for Outlier, the mean corpus vector is
$$
\frac {1}{s i z e (c)} \sum_ {x \in c} U S E (x) \tag {7}
$$
which captures utterances from the entire corpus  $c$ . We calculate a Corpus-Wide Oracle diversity score,  $div_{oracle}$ , for each utterance in  $c$  for each diversity metric.
The Corpus-Wide Oracle is used to construct  $sub_{c}$  of any size consisting of the most diverse utterances. This sub-corpus can be used to compare against other collection methods, such as those in Simulated Data Collection, or as a way to enhance an existing collection by selecting out the most diverse utterances.
After the Corpus-Wide Oracle ranks each utterance by diversity, we select the utterances with the top 10,000 diversity values to form  $sub_{c}$ . This
serves as a use-case for collecting the maximally-diverse corpus for a given diversity metric.
However, the Corpus-Wide Oracle might not be the best 10,000 utterances to collect for a subcorpus. The Corpus-Wide Oracle selects the utterances with the most diversity compared to the whole corpus, but this might be too much diversity without enough context since the Simulated Data Collection methods add entire conversations (not utterances in isolation) to  $\text{sub}_c$ .
We evaluate the collected corpora both in terms of how diverse each sub-corpus is as well as performance on two downstream tasks: conversation emotion classification and dialogue generation.
4.1 Overall Diversity
The first evaluation aims to answer the question of if our methods produce more diverse sub-corpora than the Random Population baseline. We examine the hypothesis that using a collection method with knowledge of diversity will result in  $sub_{c}$  that is significantly more diverse. For each data collection method, we compare the diversity of the sub-corpus to Random Population. Because diversity values are relative to  $sub_{c}$ , diversity of  $sub_{c}$  is measured via  $div_{oracle}$  values.
Table 3 shows the resulting divoracle values for datasets collected using our methods. Each value is the average of 100 trials, in which each trial collects a 10,000 utterance sub-corpus,  $sub_{c}$ .
Significance results for all experiments use a two-sided t-test compared to the Random Population baseline. Both Diverse Population and Above Mean Population produce datasets which contain statistically significantly  $(p < 0.001)$  more diverse data compared to the Random Population baseline. The Corpus-Wide Oracle method produces the most diverse results overall, as expected as it is a collection of the top 10,000 most diverse utterances. Running Diversity-Informed Data Collection to collect datasets of size 5,000 produced similarly significant differences.
We also examine the average number of participants out of the 810 total in  $c$  that are included for each method. Note in Table 3 the difference in Average Number of Participants from Random Population and Diverse Population to Above Mean Population and Corpus-Wide Oracle. Even though Above Mean Population is more diverse than Di