SlowGuess/ACL_2020_OCR huggingface dataset:
Condition	Mean Score	Avg. #Part
Outlier	Random Population	0.974	257.4
Diverse Population	0.979*	262.1
Above Mean Population	0.978*	516.9
Corpus-Wide Oracle	1.035*	539.0
Entropy	Random Population	-5.350	257.2
Diverse Population	-5.320*	259.1
Above Mean Population	-5.294*	359.1
Corpus-Wide Oracle	-4.261*	481.0
Mean IDF	Random Population	5.455	256.2
Diverse Population	5.659*	257.7
Above Mean Population	5.613*	357.5
Corpus-Wide Oracle	7.783*	546.0
Table 3: Results for diversity scores for each method of collecting corpora, by metric (Outlier, Entropy, and Mean IDF). Higher scores are better for all metrics. Also shown are the average number of participants (Avg. #Part) included out of a possible 810. * indicates statistical significance compared to the Random Population baseline  $(\mathrm{p} < 0.001)$ .
verse Population for Entropy, it comes at the cost of more participants. Across all three diversity metrics, Above Mean Population requires about 100-200 additional participants than Diverse Population and Random Population. In an online setting where the cost to train new crowdworkers is high, the tradeoff between number of participants and diversity of content may be worth considering.
To examine the quality of the resulting  $sub_{c}$ 's, we turn to downstream task evaluation. We first examine the task of classifying a conversation's emotions from utterance text. Following Larson et al. (2019)'s justification, we would expect more diverse  $sub_{c}$  to result in higher classification accuracies, because more diverse responses should cover more variation in how people express emotions in conversation.
4.2.1 Classification Method
We follow the methodology of Larson et al. (2019) who propose evaluating the diversity of goal-oriented intent paraphrases. For their use case, classification models predict the intents from the paraphrase. For our case, each conversation in the EmpatheticDialogues corpus is associated with an emotion, such as anger or guilt. There are 32 such emotions throughout the corpus. The classification
Condition	SVM	Fast-Text
Outlier	Random Population	0.224	0.050
Diverse Population	0.234*	0.052
Above Mean Population	0.229	0.077*
Corpus-Wide Oracle	0.100*	0.057*
Entropy	Random Population	0.218	0.052
Diverse Population	0.212†	0.049
Above Mean Population	0.254*	0.065*
Corpus-Wide Oracle	0.134*	0.102*
Mean IDF	Random Population	0.220	0.052
Diverse Population	0.236*	0.052
Above Mean Population	0.257*	0.064*
Corpus-Wide Oracle	0.131*	0.065*
Table 4: Results for downstream classification accuracy averaged over 5-fold cross-validation over 10 trials: higher is better. The task is classification of emotions from a set of 32 possible given the text of dialogue responses in  $sub_{c}$ . † and * indicate p<0.05 and 0.001 respectively compared to Random Population.
task is to predict which of the 32 emotions is expressed from a given utterance. Following Larson et al. (2019), we use two classification models:
- Bag-of-Words SVM
- FastText classifier
Bag-of-Words SVM is an SVM using TF-IDF word features for prediction. The FastText classifier uses a neural classification model on top of fastText sentence embeddings (Joulin et al., 2017). The sub-corpora we collect using the different methods serve as the datasets to train these classification models.
4.2.2 Classification Results
Classification task results are summarized in Table 4. Reported scores are averaged 5-fold cross-validation and averaged over 10 runs of datasets collected from each method.
While most conditions show Diverse Population significantly outperforms Random Population, it performs worse than Random Population with Entropy SVM and Entropy FastText and performs the same in Mean IDF FastText. Above Mean Population, on the other hand, outperforms the Random Population baseline on all conditions. This could potentially be due to the larger number of participants included in Above Mean Population. Surprisingly, Corpus-Wide Oracle does not perform the best in each category. We conjecture that too many diverse responses do not allow a classifica
tion model to learn common characteristics.
4.3 Generation
Because the ultimate goal of collecting more diverse dialogue data is generating more diverse text, we evaluate diversity of neural text generation models trained on resulting corpora.
4.3.1 Generation Method
Our task is to generate the next utterance in a dialogue, where the data collection processes collect utterances for  $sub_{c}$ . To train generation models, the input is the most recent parent utterance for each  $utt$  in  $sub_{c}$ , and  $utt$  is the target sentence to generate. When  $utt$  is the starting utterance in a conversation, the input is the situation associated with the conversation (such as planning a vacation).
We train Sequence-to-Sequence models (Sutskever et al., 2014) with a 2-layer bidirectional encoder, hidden size 500, word vector size 64, Adam optimizer (Kingma and Ba, 2014), learning rate 0.001, trained for 3000 steps with batch size 32. Models are implemented using OpenNMT (Klein et al., 2017). We opt to use a standard model as it has fewer parameters to learn from smaller sub-corpora. We use the same parameter settings for all trained models.
4.3.2 Generation Results
Generation task results are summarized in Table 5. We report on both mean and median length of model responses. Distinct-1 and Distinct-2 measure the proportion of unigrams and bigrams respectively in the set of model responses which are unique (Li et al., 2016a). We also report diversity of the generated responses calculated by the metrics used in  $sub_{c}$  collection (see Table 2).
Our method results in models which produce more diverse output compared to baseline Random Population data collection. Interestingly, Diverse Population and Above Mean Population split the win on producing more diverse outputs. CorpusWide Oracle diversity results are sometimes lower and overall shorter in length than other methods; a potential reason is this condition only samples utterances, not conversations.
Responses from the model trained on each  $sub_{c}$  are evaluated with all 3 diversity metrics, to examine potential interactions. Collecting  $sub_{c}$  with Entropy results in higher Mean IDF (and vice versa) compared to Random Population. Collecting  $sub_{c}$  with Outlier results in slightly lower Mean IDF
(and vice versa) for Diverse Population and Above Mean Population compared to Random Population. There is not a consistent signal between Outlier and Entropy. Future work can further examine the relationships among these diversity metrics.
Diversity Considerations: Compared to a random baseline, Diversity-Informed Data Collection results in more diverse data than Random Population, which is shown to be more effective on downstream tasks. Future work can explore the effect of simultaneously optimizing multiple desirable measurements of diversity.
However, we acknowledge that maximum diversity might not be what is desired and does not always result in the best downstream task performance, as indicated by the low Corpus-Wide Oracle downstream task performance. While we have not examined the tradeoff between diversity and quality, this can be explored in future work.
Generalizability: Diversity-Informed Data Collection is generalizable to metrics other than diversity. Concretely, DIDC should be used when a desired metric (1) can compare one sample (or set of samples) to the in-progress dataset and (2) has variation among the participant population.
Additionally, Diversity-Informed Data Collection can be applied to areas outside of dialogue data collection. For instance, DIDC could apply to collecting data with different emotions or sentiment. Another extension is to a specialized application domain, such as collecting dialogues for educational tutoring purposes, where our method could be used to collect more data from students who generate text consistent with certain types of misconceptions.
Crowdworking Deployment: We evaluated on simulated crowdworking data by leveraging an existing corpus. This choice stems from the desire to test multiple runs of methods in a controlled environment, to reliably determine significance, and to work with data with an assumed level of quality. That said, our approach can be applied to real crowdworking tasks. Data can be gathered from several participants in parallel, where crowdworkers are added and offered new tasks or assigned qualifications based on their diversity.
If our method is deployed in paid crowdsworking tasks, Diverse Population might be more cost-effective. In this particular investigation, we find
Condition	Mean Length	Median Length	D-1	D-2	Outlier	Entropy	Mean IDF
Outlier	Random Population	7.6	7	0.114	0.296	0.981	-3.088	5.504
Diverse Population	9.7	7	0.110	0.279	0.989*	-3.354*	5.297§
Above Mean Population	8.1	7	0.063	0.169	0.960*	-3.083	5.067*
Corpus-Wide Oracle	3.8	4	0.204	0.448	1.042*	-2.968*	6.789*
Entropy	Random Population	8.8	8	0.101	0.265	0.981	-3.281	5.263
Diverse Population	7.7	7	0.122	0.317	0.978	-3.197§	5.411†
Above Mean Population	6.6	6	0.092	0.226	0.982	-3.057*	5.474*
Corpus-Wide Oracle	4.9	5	0.112	0.316	0.985§	-2.935*	5.781*
Mean IDF	Random Population	6.1	6	0.120	0.294	0.988	-3.036	5.526
Diverse Population	6.7	6	0.131	0.322	0.986	-2.955§	5.797§
Above Mean Population	7.2	7	0.071	0.187	0.976*	-2.937*	5.655
Corpus-Wide Oracle	3.4	3	0.214	0.449	1.008*	-2.421*	8.327*
Table 5: Downstream model generation results; higher numbers are better for all metrics. †, §, and * indicate p<0.05, 0.01, and 0.001 respectively. As Distinct-1 and Distinct-2 are summary statistics, we did not test significance.
Diverse Population requires 100-200 fewer participants than Above Mean Population to create a dataset. Due to the time required to train new participants, there is a tradeoff between training a new worker and collecting more data form current participants.
Caution should be taken in using this method on-the-fly without a quality check. Standard quality control methods (e.g., crowdworker qualifications, manual examination, crowdworker verification) should be deployed for from-scratch data collection.
Crowdworker Fairness: Another important consideration for a live deployment is the crowdworker's perspective of fairness. Because some crowdworkers are retained for more data collection than others, communicating this possibility to crowdworkers is essential (Brawley and Pury, 2016). Crowdworking best practices involve disclosing which quality metrics are being used to workers to set clear expectations (Bederson and Quinn, 2011). Additionally, combining our method with a method which alters the task crowdworkers complete (Kang et al., 2018) as opposed to restricting the crowdworking population could be a way to balance fairness with crowdworkers. Different task and population combinations could allow for all crowdworkers to participate in more tasks.
We propose a method, Diversity-Informed Data Collection, which leverages this to produce more
diverse datasets than the standard approach, and which performs better on downstream tasks. We define diversity of an utterance compared to the other utterances in a corpus. This allows for measurement of the impact of adding each utterance to the corpus. Working under the same assumption that a subset of participants produce diverse data compared to the corpus, our method can be extended to other diversity measures and can be modified to work with other corpus-level metrics.
This work was supported by an AWS Machine Learning Research Award, an NVIDIA Corporation GPU grant, a UC Berkeley Chancellor's Fellowship, a National Science Foundation (NSF) Graduate Research Fellowship (DGE 1752814) and an NSF CAREER Award (IIS-1453721). We thank the three anonymous reviewers for their helpful comments. We additionally thank Cathy Chen, David Gaddy, Daniel Fried, Lucy Li, and Nate Weinman for their helpful feedback.
Sepehr Assadi, Justin Hsu, and Shahin Jabbari. 2015. Online assignment of heterogeneous tasks in crowdsourcing markets. In Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015, November 8-11, 2015, San Diego, California, USA, pages 12-21. AAAI Press.
Ricardo A. Baeza-Yates, Berthier Ribeiro-Neto, et al.
1999. Modern Information Retrieval, chapter 3, Modeling. ACM press New York, USA.
Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. 2018. Generating more interesting responses in neural conversation models with distributional constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3970-3980, Brussels, Belgium. Association for Computational Linguistics.
Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. 2015. Task assignment optimization in knowledge-intensive crowdsourcing. The VLDB Journal—The International Journal on Very Large Data Bases, 24(4):467-491.
Benjamin B. Bederson and Alexander J. Quinn. 2011. Web workers unite! addressing challenges of online laborers. In CHI '11 Extended Abstracts on Human Factors in Computing Systems, CHI EA '11, page 97-106, New York, NY, USA. Association for Computing Machinery.
Alice M. Brawley and Cynthia L.S. Pury. 2016. Work experiences on mturk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54:531 - 546.
Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 182-187, Valencia, Spain. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2019. Convokit: The cornell conversational analysis toolkit.
Ju Fan, Guoliang Li, Beng Chin Ooi, Kian-lee Tan, and Jianhua Feng. 2015. Icrowd: An adaptive crowdsourcing framework. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD '15, page 1015-1030, New York, NY, USA. Association for Computing Machinery.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association
for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, and Jason Mars. 2018. Data collection for dialogue system: A startup perspective. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 33-40, New Orleans - Louisiana. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL.
Ari Kobren, Chun How Tan, Panagiotis Ipeirotis, and Evgeniy Gabrilovich. 2015. Getting more for less: Optimized crowdsourcing with dynamic tasks and goals. In Proceedings of the 24th International Conference on World Wide Web, WWW '15, pages 592-602, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
Katsumi Kumai, Masaki Matsubara, Yuhki Shiraishi, Daisuke Wakatsuki, Jianwei Zhang, Takeaki Shionome, Hiroyuki Kitagawa, and Atsuyuki Morishima. 2018. Skill-and-stress-aware assignment of crowd-worker groups to task streams. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Zurich, Switzerland, July 5-8, 2018, pages 88-97.
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 517-527, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2157-2169, Copenhagen, Denmark. Association for Computational Linguistics.
Christopher H. Lin, Mausam, and Daniel S. Weld. 2018. Active learning with unbalanced classes and example-generation queries. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Zürich, Switzerland, July 5-8, 2018, pages 98-107. AAAI Press.
Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher Homan. 2019. Learning to predict population-level label distributions. In *Companion Proceedings of The 2019 World Wide Web Conference*, WWW '19, pages 1111-1120, New York, NY, USA. ACM.
H. Rahman, S. B. Roy, S. Thirumuruganathan, S. Amer-Yahia, and G. Das. 2015. Task assignment optimization in collaborative crowdsourcing. In 2015 IEEE International Conference on Data Mining, pages 949-954.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 3295-3301. AAAI Press.
Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating long and diverse responses with neural conversation models. CoRR, abs/1701.03185.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. 2018. Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages
4418-4424. International Joint Conferences on Artificial Intelligence Organization.
Chun-Ju Yang, Kristen Grauman, and Danna Gurari. 2018. Visual question answer diversity. In Sixth AAAI Conference on Human Computation and Crowdsourcing.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 1815-1825, Red Hook, NY, USA. Curran Associates Inc.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664, Vancouver, Canada. Association for Computational Linguistics.
Sharon Zhou, Melissa Valentine, and Michael S. Bernstein. 2018. In search of the dream team: Temporally constrained multi-armed bandits for identifying effective team structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18, pages 108:1-108:13, New York, NY, USA. ACM.