LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset
Year: 2023

Abstract:
Collaboration increasingly happens online. This is especially true for large groups working on global tasks, with collaborators all around the world. The size and distributed nature of such groups make decision-making challenging. This paper proposes a set of dialog acts for the study of decision-making mechanisms in such groups, and provides a new annotated dataset based on real-world data from the public mail-archives of one such organization – the Internet Engineering Task Force (IETF). We provide an initial data analysis showing that this dataset can be used to better understand decision-making in such organizations. Finally, we experiment with a preliminary transformerbased dialog act tagging model.

heading: 1 Introduction and Related Work
text: Motivation Online collaboration has been used for many years by large distributed organizations. The increasing availability of high-speed Internet connections and collaboration tools, along with the Covid-19 pandemic, are making it ever more prevalent. Large distributed organizations of this type often undertake important tasks. For example, the Internet Engineering Task Force (IETF) and the World Wide Web Consortium (W3C) are responsible for developing the technical standards that underpin the Internet. Consequently, understanding the decision-making processes in this type of organization is essential to increase transparency and accountability, to facilitate tracking of decisions and the reasoning behind them, and to understand alternatives that were considered (or not) and the voices that were (or were not) heard.
Goals Most studies of decision making in text (e.g. Hsueh and Moore, 2007; Fernández et al., 2008; Bui and Peters, 2010) rely on annotation and analysis of Dialogue Acts (DAs). We adopt this approach and label emails from public IETF mailing lists with DAs. Our aim is to answer the following
research questions: RQ1: What is an appropriate set of DAs to use for this annotation task?; RQ2: How do communication patterns change through the life-cycle of a decision discussion?; and RQ3: How do different types of participants differ in how they contribute to the process? The overall goal of these questions is to better understand the mechanisms underlying the decision-making process in a large, distributed, collaborative organization.
Related Datasets The most notable email-based related dataset is the Enron Corpus (Klimt and Yang, 2004), covering over 200K messages of Enron employees in various positions within the organization. However, in-house emails of a single closed company are not representative of communication in larger, more diverse collaborations.
Datasets specifically relevant for studying decision making include AMI (McCowan et al., 2005) and ICSI/MRDA (Janin et al., 2003; Shriberg et al., 2004). However, the AMI dataset is not “real”: it uses actors acting out small-group meetings on predefined topics. In contrast, the ICSI dataset is based on naturally occurring meetings at the International Computer Science Institute (ICSI). While both are annotated with general dialogue act labels, AMI also includes specific decision-oriented dialogue acts provided by Fernández et al. (2008). Despite this, they are not representative of interaction in large groups, or online collaborative settings. Consequently, we annotate a new dataset tailored to address our research questions. We denote it as Largeorganization Email-based Decision-dialogue-act Analysis dataset – LEDA.
There are important differences between LEDA and AMI/ICSI. First, while AMI/ICSI are transcribed face-to-face, real-time, in-person, and small-group meetings. LEDA contains emails from mailing-lists, asynchronous, and from a large decentralized, globally spread group. Second, AMI/ICSI discuss mostly self-contained, focused
6080
topics (design, research-group progress); LEDA discusses the more long-term, complex task of designing Internet-standards. We further provide a more detailed comparison of LEDA with AMI in Appendix A.
Contributions First, we propose a taxonomy of DA labels for large-group email decision-making. Second, we provide a novel dataset labeled with DAs. Third, we provide data analyses exploring decision-making communication patterns within the IETF. Fourth, we provide a preliminary DA prediction model on this dataset, which can serve as a reference baseline model for future work.

heading: 2 Dataset
text: Our data consists of emails from the IETF mailing list archive.1 The IETF is a typical example of decision making in a large, distributed, online collaborative community; it has rich metadata available via the IETF DataTracker;2 and the data is publicly available with appropriate consent.3
IETF background The IETF is a large, open, voluntary organization tasked with developing Internet standards (Flanagan, 2019; McQuistin et al., 2021; Khare et al., 2022). It is comprised of working groups (WGs), each focusing on a relatively narrow field: e.g., RMCAT4 WG focuses on specific Real-time Media Congestion Avoidance Techniques. Each WG has one or more participants as chairs. During its development, an Internet standard is called a draft. Drafts are discussed in the mailing lists (the archive has >2M emails, predominantly in English, between 56k participants over 20 years) and in several live meetings yearly. After sufficient revision and review, a draft becomes an Internet standard.
Data preparation The email archive consists of threads (sets of emails connected with reply-to relations, forming a tree-like structure). Given a particular draft, we extract all threads with at least one message that mentions the draft in either the subject or body. We do this for four drafts, chosen by an IETF expert to span a range of technical areas. We opted for entire threads over a smaller number of drafts (rather than more drafts but with partial
1http://mailarchive.ietf.org/arch/ 2http://datatracker.ietf.org/ 3www.ietf.org/privacy-statement/ 4http://datatracker.ietf.org/wg/rmcat/
threads) to ensure a full view of the draft discussion and agreement process over its life-cycle.
We then preprocess all messages, splitting them into Quote, Signature, or Normal segments using custom heuristics developed for this data. A Normal segment contains text written by the author of the message. A Quote segment contains text written by someone else, which is being quoted. A Signature segment contains signatures (name, company name, website). Normal segments are useful for analysis, while the rest introduce noise. We also keep track of quoting relations between segments.
Label set calibration As our starting point, we take the DA labels defined in the ISO 24617-2 standard (Bunt et al., 2012). Cross-referencing with labels in datasets from related work and manual inspection of the IETF data suggested that much of the complexity in the standard is not needed for our goals. This was confirmed in several initial rounds of annotations where we observed considerable confusion between the very fine grained ISO 24617-2 DAs on our data. After each iteration, we simplified the label set by removing irrelevant labels for email communication (e.g., rhetorical devices such as pauses) and aggregating hard to distinguish labels (e.g., accepting a request and agreeing to an opinion). Table 1 presents our twolevel taxonomy with three coarse grained labels divided into eleven fine-grained ones, which was obtained after four rounds of calibration.
Annotation Annotation of each segment with DA labels was carried out by seven student annotators, all with a background in linguistics. A segment can be assigned several DAs simultaneously (a multi-label setting). During the calibration rounds, annotators provided feedback which helped modify the taxonomy and instructions. For the final annotation, they were provided a detailed set of instructions and an annotation tool specifically developed in-house.
Table 1 reports data statistics and inter-annotator agreement (IAA). Each thread is annotated by at least two annotators. To measure IAA, we considered both Fleiss’ Kappa and Krippendorff’s Alpha, but neither supports multi-label annotation. Instead, we consider one annotator’s labels as “gold labels,” and another’s as “classifier predictions.” We calculate the F1 score for all annotator pairs and average them. This calculation is performed on a subset of 15 threads labeled by all annotators. For some
labels, the annotation is inherently difficult, as reflected in the IAA. Manual inspection reveals that many of these disagreements may be impossible to completely resolve as the task is subjective (Uma et al., 2021). For example, ClarificationElicitation is more often implicit ("I don’t see why ...") than explicit ("Can you explain why ..."), introducing disagreement. However, recent work (Pavlick and Kwiatkowski, 2019; Basile et al., 2021; Leonardelli et al., 2021) shows it is viable to design models and evaluation measures that account for this inherent ambiguity instead of trying to resolve it. Accordingly, we release all individual annotators’ labels with the text data and code. 5 While covering only four drafts, LEDA is of substantial size (8230 segments, 2196 messages, 363 authors), with the drafts hand-picked by an IETF expert to ensure they are representative. We focus on trends that are very prominent and supported by statistical significance tests. Finally, an inspection of plots for individual drafts revealed that the main trends outlined in the remaining sections were consistent across all four drafts.

heading: 3 Analysis of gold-standard labels

heading: 3.1 Draft life-cycle
text: To address RQ1, we divide the period between the submission and publication of a draft into five equal time intervals (T1 - T5), each representing 20% of the period. We visualize the distribution of DAs falling into each of the periods. in Figure 1.6
Answer and Question are more common in the early phases, likely due to more new issues being raised and unresolved issues discussed.
ContextSetting and Extension are very frequent, increasingly so towards the end phases; we conjecture this is because those phases cover more complex issues requiring more background description.
The frequency of ProposeAction is stable throughout the cycle and noticeably higher than StateDecision. This may imply that participants prefer to discuss actionable options rather than explicitly deciding on a single one.

heading: 3.2 Different groups
text: To explore RQ2, we categorize the participants as: (1) authors of the draft being discussed, or not;
5https://github.com/sodestream/ acl2023-email-da-dataset.git
6In both figures InformationProviding is omitted because it dominates the plot and obscures other trends.
(2) influential — following (Khare et al., 2022), having top-10% centrality in the email interaction graph — or not; (3) chairs of any IETF WG, or not; (4) everyone (all participants). Figure 2 gives a visualization of DA distributions for each group.
Authors vs. non-Authors Authors are more social, give more answers, and ask fewer questions (including clarification questions). Also, they use fewer NeutralResponse, Extension, and ContextSetting, indicating shorter, more focused messages. These trends imply they take a more reactive role in the discussion. Finally, they make the most decisions in the discussion, as would be expected, since they are in charge of the writing process.
Influential vs. non-Influential Influential people use Answer, Agreement, and NeutralResponse more, making them generally more responsive. They use less Extension, ContextSetting and Thanking, implying a concise, focused communication style. As expected, they make more decisions and propose slightly more actions.
Chairs vs. non-Chairs Similar to influential participants, chairs use NeutralResponse more than non-Chairs. However, they use more ContextSetting and Extension, and do more Thanking. We find this is because chairs send a lot of emails initiating and managing discussions and review assignments. Such emails are often composed of many small segments and contain a lot of these labels.
Feedback to questions We further explored how likely the different groups are to have their questions answered. From the labeled data we obtain percentages for authors (22%), chairs (51%), influential (34%), and everyone (37%). Authors have the lowest ratio, possibly because their questions are, on average, more complex. The chairs, while they tend not to ask many questions, are the most likely to to get an answer. This is expected, as it is difficult to ignore a question from someone in that position. Surprisingly, the difference between ratios of influential participants and everyone are not statistically significant.7 Another surprising finding is that, on average, around two thirds of all questions appear to remain unanswered.

heading: 3.3 Other observations
text: ClarificationElicitation is almost nonexistent, implying either very little misunderstandings or un-
7We used a z-test with significance level 0.05.
willingness to explicitly voice it. Research on misunderstandings in dialog (Aberdeen and Ferro, 2003) implies it is likely the latter.
Most participants tend to use NeutralResponse, as opposed to Agreement or Disagreement, and between the latter two they prefer Agreement. This tendency is confirmed by related research on agreement (Stromer-Galley and Muhlberger, 2009).
ContextSetting, Extension, and NeutralResponse are, expectedly, very frequent. This implies there are a lot of boilerplate explanations around segments with more relevant DAs.

heading: 4 Automated Dialogue Act Tagging
text: We provide a preliminary DA tagging model to investigate the predictability of our DA tags, and to serve as a baseline for future work. We use a hierarchical sequence model, inspired by work in DA tagging for spoken dialogue (e.g. Li et al., 2019): the input is a sequence of segments (each one a sequence of words), and the output is a sequence of predictions, one 14-dimensional vector for each input segment, representing DA probabilities.
Each input segment is encoded into a vector; we use the [CLS] token of BERT (Devlin et al., 2019). The sequence of segment vectors is then passed to a Bidirectional-LSTM (Hochreiter and Schmidhuber, 1997); each BiLSTM hidden state vector is passed through a linear layer (shared for all time steps) to produce the output prediction vector sequence. The loss function is binary cross-entropy averaged across all labels and all elements of the sequence.
The model is implemented using PyTorch (Paszke et al., 2019) and scikit-learn (Pedregosa et al., 2011). We used a learning rate of 2−5, batchsize of 32, and LSTM hidden-layer size of 256. All other hyper-parameters are left at default values. We experiment with two variants of BERT: bertbase and bert-base-ietf (fine-tuned using language modeling loss on the entire IETF mail archive).
We split the data into train (60%), validation (20%), and test threads (20%). We report results on test threads by the model best on the validation threads. The input sequences for the model are the possible root-to-leaf paths in the input threads, following (e.g. Zubiaga et al., 2016).8
Results are given in Table 2. Predicting higherlevel labels is easier, as expected. For lower-level
8This will cause segments that are part of several paths to be processed multiple times and assigned multiple label hypotheses; we take the most common label in this case.
labels, performance is worst for labels that are conceptually more subjective (as reflected by IAA) or have very few examples.
Curiously, bert-base-ietf performs comparably to or worse than bert-base. We hypothesize the reason for this may be the specific language of the IETF (technical discussions). It may cause the additional language model training step to make the bert-baseietf model forget information generally useful for DA tagging. On the other hand, this information is retained in bert-base. If this is the case, it would hurt the performance of bert-base-ietf after further fine-tuning on the DA tagging task. However, we leave investigation of this and other hypotheses for this unexpected result to future work.

heading: 5 Conclusion
text: We have presented a taxonomy of dialogue acts (DAs) and a labeled dataset of emails. Moreover, we provided a data analysis and a preliminary DA prediction model. We hope this dataset will be useful to facilitate further research on the interaction behavior of participants in online collaboration settings. Future work could include a more detailed investigation into the underlying reasons for the observed trends. Another possibility is looking into the interaction of DAs and the participant interaction graph as described by (Khare et al., 2022). Finally, to get further insights, it would be interesting to annotate segments of with a particular DA with additional labels, e.g., explicit/implicit for Agreement or different sub-types of Question.

heading: 6 Limitations
text: One of the main limitations is that we focus solely on the IETF. Consequently, we can never be completely sure how well our findings generalize to other similar organizations without further annotation.
We are also limited by not conducting a hyperparameter search on our models. We omit this step as the main goal is not maximizing performance, but rather data annotation and analysis. In a similar vein, it is likely possible to increase performance by using a more advanced model that is either trained on dialogue-like data or is specifically designed to exploit phenomena specific to dialogue (e.g., having speaker embeddings).
We also acknowledge that many emails are longer than 512 tokens which is the limit of our BERT model and thus might have been cut short. However, most of the emails do fit into this limit.

heading: 7 Ethical Considerations
text: The IETF conditions participation by agreements and policies that explicitly state mailing list discussions and Datatracker metadata will be made publicly available.9 In our analysis we use only this publicly available data. We have discussed our work with the IETF leadership and confirmed it is conforming to all their policies.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful comments. This work was supported by the UK EPSRC under grants EP/S033564/1 and EP/S036075/1 (Sodestream: Streamlining Social Decision Making for Enhanced Internet Standards). Purver was also supported by the Slovenian Research Agency via research core funding for the programme Knowledge Technologies (P2-0103).

heading: A Appendix A: Comparison with the AMI dataset
text: In this section, we compare our dataset with the AMI dataset (McCowan et al., 2005). The counts are given in Table 3, after removing those AMI DA categories that make sense in AMI’s spoken, faceto-face setting but do not exist in the email given the text modality and non-synchronous nature (e.g. Stall, Backchannel). The distributions are roughly similar. The main difference is a lot more ClarificationElicitation and Answer in AMI. The former may reflect the explicitly decision-oriented setting of AMI (actors were tasked with making design decisions on how to build a remote control, and therefore decisions and clarity were the primary focus), and/or its synchronous speech, which participants must clarify immediately (while email can be studied over more time before replying). The latter may reflect the fact that AMI is built on live face-to-face conversations, thus leaving an articulated question ignored and unanswered would be considered rude, while in email communication, this is less problematic.

heading: B Appendix B: Computing resources
text: The prediction model experiments (two of them – bert-base and bert-base-ietf) were run on a single Nvidia QUADRO RTX 6000 GPU for 100 epochs each. For both experiments, one epoch took approximately 4 minutes. In preliminary experiments, we found the models with our hyperparameters need 14GB of video memory. They can, however, run with less memory with reduced batch size. Alternatively, larger batches could be emulated using several smaller batches and gradient accumulation (this is not implemented in our code).

heading: C Appendix C: Annotation details
text: The annotators come from diverse backgrounds but were primarily chosen as skilled linguists from the population of graduate and Ph.D. level linguistics students. They all lived in the UK and were paid an hourly wage that was slightly above average for similar tasks in the UK.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
3 A2. Did you discuss any potential risks of your work? 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2,4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 2,4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 2

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 2,7

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 7

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2,4
C 3 Did you run computational experiments? 3,4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix 2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4,6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 2, Appendix 3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix 3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 7
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. The data is already public.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix 3

heading: Future Work
text: Future work could include a more detailed investigation into the underlying reasons for the observed trends. Another possibility is looking into the interaction of DAs and the participant interaction graph as described by (Khare et al. , 2022). Finally, to get further insights, it would be interesting to annotate segments of with a particular DA with additional labels, e. g. , explicit/implicit for Agreement or different sub-types of Question