Common Law Annotations: Investigating the Stability of Dialog System Output Annotations
Year: 2023

Abstract:
Metrics for Inter-Annotator Agreement (IAA), like Cohen’s Kappa, are crucial for validating annotated datasets. Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure validity or reproducibility. While researchers are encouraged to increase annotator agreement, this can lead to specific and tailored annotation guidelines. We hypothesize that this may result in diverging annotations from different groups. To study this, we first propose the Lee et al. Protocol (LEAP), a standardized and codified annotation protocol. LEAP strictly enforces transparency in the annotation process, which ensures reproducibility of annotation guidelines. Using LEAP to annotate a dialog dataset, we empirically show that while research groups may create reliable guidelines by raising agreement, this can cause divergent annotations across different research groups, thus questioning the validity of the annotations. Therefore, we caution NLP researchers against using reliability as a proxy for reproducibility and validity. https://github.com/jsedoc/ common-law-annotations

heading: Future Work
text: In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerbertVoss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. Harry Bunt, Alex Chengyu Fang, Nancy M. Ide, and Jonathan J. Webster. 2010. A methodology for designing semantic annotation languages exploiting syntactic-semantic iso-morphisms. In Proceedings of the Second International Conference on Global Interoperability for Language Resources (ICGL 2010), Hong Kong, pages 29–46. Joan Byamugisha, C. Maria Keet, and Brian DeRenzi. 2017. Evaluation of a Runyankore grammar engine for healthcare messages. In Proceedings of the 10th International Conference on Natural Language Generation, pages 105–113, Santiago de Compostela, Spain. Association for Computational Linguistics. Giuseppe Carenini and Jackie C. K. Cheung. 2008. Extractive vs. NLG-based abstractive summarization of evaluative text: The effect of corpus controversiality. In Proceedings of the Fifth International Natural Language Generation Conference, pages 33– 41, Salt Fork, Ohio, USA. Association for Computational Linguistics. Amanda Cercas Curry, Dimitra Gkatzia, and Verena Rieser. 2015. Generating and Evaluating LandmarkBased Navigation Instructions in Virtual Environments. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pages 90–94, Brighton, UK. Association for Computational Linguistics. Hyungtak Choi, Siddarth K. M. , Haehun Yang, Heesik Jeon, Inchul Hwang, and Jihie Kim. 2018. SelfLearning Architecture for Natural Language Generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 165–170, Tilburg University, The Netherlands. Association for Computational Linguistics. Philipp Cimiano, Janna Lüker, David Nagel, and Christina Unger. 2013. Exploiting ontology lexica for generating natural language texts from RDF data. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 10–19, Sofia, Bulgaria. Association for Computational Linguistics. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. Kelly Cuccolo, Jon E Grahe, Martha S Zlokovich, John Edlund, rick miller, Susana Gallor, Jordan R Wagge, Kaitlyn M Werner, Albert L Ly, Fanli Jia, and et al. 2022. NICE: CROWD. Kelly Cuccolo, Megan S. Irgens, Martha S. Zlokovich, Jon Grahe, and John E. Edlund. 2021. What Crowdsourcing Can Offer to Cross-Cultural Psychological Science. Cross-Cultural Research, 55(1):3–28. Publisher: SAGE Publications Inc. Seniz Demir, Sandra Carberry, and Kathleen F. McCoy. 2008. Generating textual summaries of bar charts. In Proceedings of the Fifth International Natural Language Generation Conference on - INLG ’08, page 7, Salt Fork, Ohio. Association for Computational Linguistics. Jan Milan Deriu and Mark Cieliebak. 2018. Syntactic Manipulation for Generating more Diverse and Interesting Texts. In Proceedings of the 11th International Conference on Natural Language Generation, pages 22–34, Tilburg University, The Netherlands. Association for Computational Linguistics. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. Abdurrisyad Fikri, Hiroya Takamura, and Manabu Okumura. 2018. Stylistically User-Specific Generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 89– 98, Tilburg University, The Netherlands. Association for Computational Linguistics. Katja Filippova and Michael Strube. 2008. Dependency tree based sentence compression. In Proceedings of the Fifth International Natural Language Generation Conference on - INLG ’08, page 25, Salt Fork, Ohio. Association for Computational Linguistics. 12324
Sarah E. Finch and Jinho D. Choi. 2020. Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236–245, 1st virtual meeting. Association for Computational Linguistics. Dimitra Gkatzia, Helen Hastie, Srinivasan Janarthanam, and Oliver Lemon. 2013. Generating student feedback from time-series data using reinforcement learning. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 115–124, Sofia, Bulgaria. Association for Computational Linguistics. Jon E. Grahe, Kelly Cuccolo, Dana C. Leighton, and Leslie D. Cramblet Alvarez. 2020. Open Science Promotes Diverse, Just, and Sustainable Research and Educational Outcomes. Psychology Learning & Teaching, 19(1):5–20. _eprint: https://doi. org/10. 1177/1475725719869164. Byron B. Hamilton, Judith A. Laughlin, Roger C. Fiedler, and Carl V. Granger. 1994. Interrater reliability of the 7-level functional independence measure (FIM). Scandinavian journal of rehabilitation medicine, 26(3):115–119. Vrindavan Harrison and Marilyn Walker. 2018. Neural Generation of Diverse Questions using Answer Focus, Contextual and Linguistic Features. In Proceedings of the 11th International Conference on Natural Language Generation, pages 296–306, Tilburg University, The Netherlands. Association for Computational Linguistics. Nicole Hober, Tülay Dixon, and Tove Larsson. 2023. Toward increased reliability and transparency in projects with manual linguistic coding. Corpora, 18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations. Eduard Hovy and Julia Lavid. 2010. Towards a ‘Science’ of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics. International journal of translation, 22(1):13–36. David M. Howcroft, Anya Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169–182, Dublin, Ireland. Association for Computational Linguistics. Stephanie Inglis. 2015. Summarising Unreliable Data. In Proceedings of the 15th European Workshop on
Natural Language Generation (ENLG), pages 95– 99, Brighton, UK. Association for Computational Linguistics. Stephanie Inglis, Ehud Reiter, and Somayajulu Sripada. 2017. Textually Summarising Incomplete Data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 228–232, Santiago de Compostela, Spain. Association for Computational Linguistics. Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel Weld. 2022. GENIE: Toward reproducible and standardized human evaluation for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11444–11458, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460–8478, Dublin, Ireland. Association for Computational Linguistics. Kittipitch Kuptavanich, Ehud Reiter, Kees Van Deemter, and Advaith Siddharthan. 2018. Generating Summaries of Sets of Consumer Products: Learning from Experiments. In Proceedings of the 11th International Conference on Natural Language Generation, pages 403–407, Tilburg University, The Netherlands. Association for Computational Linguistics. J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159–174. Publisher: [Wiley, International Biometric Society]. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 22 140:55–55. Saad Mahamood and Ehud Reiter. 2011. Generating affective natural language for parents of neonatal infants. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 12– 21, Nancy, France. Association for Computational Linguistics. Saad Mahamood and Ehud Reiter. 2012. Working with clinicians to improve a patient-information NLG system. In INLG 2012 Proceedings of the Seventh International Natural Language Generation Conference, pages 100–104, Utica, IL. Association for Computational Linguistics. 12325
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A Dialog Research Software Platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics. Priscilla Moraes, Kathy McCoy, and Sandra Carberry. 2014. Adapting Graph Summaries to the Users’ Reading Levels. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pages 64–73, Philadelphia, Pennsylvania, U. S. A. Association for Computational Linguistics. Yusuke Mori, Hiroaki Yamane, Yusuke Mukuta, and Tatsuya Harada. 2019. Toward a Better Story End: Collecting Human Evaluation with Reasons. In Proceedings of the 12th International Conference on Natural Language Generation, pages 383–390, Tokyo, Japan. Association for Computational Linguistics. Gabriel Murray, Giuseppe Carenini, and Raymond Ng. 2010. Generating and validating abstracts of meeting conversations: a user study. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics. Alice Oh and Howard Shrobe. 2008. Generating baseball summaries from multiple perspectives by reordering content. In Proceedings of the Fifth International Natural Language Generation Conference on - INLG ’08, page 173, Salt Fork, Ohio. Association for Computational Linguistics. Silviu Paun, Ron Artstein, and Massimo Poesio. 2022. Learning from Multi-Annotated Corpora. In Statistical Methods for Annotation Analysis, pages 147– 165. Springer International Publishing, Cham. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. Place: Cambridge, MA Publisher: MIT Press. James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning. O’Reilly Media, Inc. Raheel Qader, Khoder Jneid, François Portet, and Cyril Labbé. 2018. Generation of Company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 254–263, Tilburg University, The Netherlands. Association for Computational Linguistics. Ehud Reiter, Albert Gatt, François Portet, and Marian van der Meulen. 2008. The importance of narrative and other lessons from an evaluation of an NLG system that summarises clinical data. In Proceedings
of the Fifth International Natural Language Generation Conference on - INLG ’08, page 147, Salt Fork, Ohio. Association for Computational Linguistics. Ehud Reiter, Roma Robertson, and Liesl Osman. 2000. Knowledge acquisition for natural language generation. In Proceedings of the first international conference on Natural language generation - INLG ’00, volume 14, page 217, Mitzpe Ramon, Israel. Association for Computational Linguistics. Sashank Santhanam and Samira Shaikh. 2019. Towards Best Experiment Design for Evaluating Dialogue System Output. In Proceedings of the 12th International Conference on Natural Language Generation, pages 88–94, Tokyo, Japan. Association for Computational Linguistics. Björn Schlünder and Ralf Klabunde. 2013. Greetings generation in video role playing games. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 167–171, Sofia, Bulgaria. Association for Computational Linguistics. Martin Schweinsberg, Nikhil Madan, Michelangelo Vianello, S. Amy Sommer, Jennifer Jordan, Warren Tierney, Eli Awtrey, Luke Lei Zhu, Daniel Diermeier, Justin E. Heinze, Malavika Srinivasan, David Tannenbaum, Eliza Bivolaru, Jason Dana, Clintin P. Davis-Stober, Christilene du Plessis, Quentin F. Gronau, Andrew C. Hafenbrack, Eko Yi Liao, Alexander Ly, Maarten Marsman, Toshio Murase, Israr Qureshi, Michael Schaerer, Nico Thornley, Christina M. Tworek, Eric-Jan Wagenmakers, Lynn Wong, Tabitha Anderson, Christopher W. Bauman, Wendy L. Bedwell, Victoria Brescoll, Andrew Canavan, Jesse J. Chandler, Erik Cheries, Sapna Cheryan, Felix Cheung, Andrei Cimpian, Mark A. Clark, Diana Cordon, Fiery Cushman, Peter H. Ditto, Thomas Donahue, Sarah E. Frick, Monica Gamez-Djokic, Rebecca Hofstein Grady, Jesse Graham, Jun Gu, Adam Hahn, Brittany E. Hanson, Nicole J. Hartwich, Kristie Hein, Yoel Inbar, Lily Jiang, Tehlyr Kellogg, Deanna M. Kennedy, Nicole Legate, Timo P. Luoma, Heidi Maibuecher, Peter Meindl, Jennifer Miles, Alexandra Mislin, Daniel C. Molden, Matt Motyl, George Newman, Hoai Huong Ngo, Harvey Packham, Philip S. Ramsay, Jennifer L. Ray, Aaron M. Sackett, Anne-Laure Sellier, Tatiana Sokolova, Walter Sowden, Daniel Storage, Xiaomin Sun, Jay J. Van Bavel, Anthony N. Washburn, Cong Wei, Erik Wetter, Carlos T. Wilson, Sophie-Charlotte Darroux, and Eric Luis Uhlmann. 2016. The pipeline project: Pre-publication independent replications of a single laboratory’s research pipeline. Journal of Experimental Social Psychology, 66:55–67. João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. ChatEval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 60–65, Minneapolis, Minnesota. Association for Computational Linguistics. 12326
Advaith Siddharthan, Matthew Green, Kees van Deemter, Chris Mellish, and René van der Wal. 2012. Blogging birds: Generating narratives about reintroduced species to promote public engagement. In INLG 2012 Proceedings of the Seventh International Natural Language Generation Conference, pages 120–124, Utica, IL. Association for Computational Linguistics. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from Disagreement: A Survey. J. Artif. Int. Res. , 72:1385–1470. Place: El Segundo, CA, USA Publisher: AI Access Foundation. Kees van Deemter. 2000. Generating vague descriptions. In Proceedings of the first international conference on Natural language generation - INLG ’00, volume 14, page 179, Mitzpe Ramon, Israel. Association for Computational Linguistics. Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, and Luou Wen. 2021. Underreporting of errors in NLG output, and what to do about it. In Proceedings of the 14th International Conference on Natural Language Generation, pages 140–153, Aberdeen, Scotland, UK. Association for Computational Linguistics. Sebastian Varges. 2006. Overgeneration and ranking for spoken dialogue systems. In Proceedings of the Fourth International Natural Language Generation Conference on - INLG ’06, page 20, Sydney, Australia. Association for Computational Linguistics. Jason Weston and Kurt Shuster. 2021. Blender Bot 2. 0: An open source chatbot that builds long-term memory and searches the internet. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond Goldfish Memory: Long-Term Open-Domain Conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180–5197, Dublin, Ireland. Association for Computational Linguistics. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : LargeScale Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270–278, Online. Association for Computational Linguistics. Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing Precise and Robust Dialogue Response Evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 26–33, Online. Association for Computational Linguistics. 12327
A Appendix
A. 1 Generic LEAP As stated in subsection 3. 1 has task-specific properties that need to be chosen by researchers. We used LEAP with a particular set of parameters for our experiment. Figure 4 is a generic flowchart for LEAP. We acknowledge that LEAP is extremely rigid. For our experiments, this was important as we desired to minimize variance between groups. In practice, researchers may desire more flexibility (e. g