MEETINGQA: Extractive Question-Answering on Meeting Transcripts
Year: 2023

Abstract:
With the ubiquitous use of online meeting platforms and robust automatic speech recognition systems, meeting transcripts have emerged as a promising domain for natural language tasks. Most recent works on meeting transcripts primarily focus on summarization and extraction of action items. However, meeting discussions also have a useful question-answering (QA) component, crucial to understanding the discourse or meeting content, and can be used to build interactive interfaces on top of long transcripts. Hence, in this work, we leverage this inherent QA component of meeting discussions and introduce MEETINGQA, an extractive QA dataset comprising of questions asked by meeting participants and corresponding responses. As a result, questions can be open-ended and actively seek discussions, while the answers can be multi-span and distributed across multiple speakers. Our comprehensive empirical study of several robust baselines including long-context language models and recent instruction-tuned models reveals that models perform poorly on this task (F1 = 57.3) and severely lag behind human performance (F1 = 84.6), thus presenting a challenging new task for the community to improve upon.1

heading: 1 Introduction
text: Millions of meetings occur every day worldwide, which results in vast amounts of meeting transcripts. Meeting transcripts are typically long documents, often domain-specific depending on the subject matter, and contain a lot of information. Basic tasks such as catching up with a missed meeting, looking up a specific discussion or response to a query can be time-consuming. These tasks can be facilitated by NLP systems, including summarization and question-answering. To this end, several publicly available small-scale corpora of meeting
1MEETINGQA data and code is available at https:// archiki.github.io/meetingqa.html
transcripts have been released (Carletta et al., 2005; Janin et al., 2003; Garofolo et al., 2004, inter alia).
Prior NLP work on meeting transcripts mainly focuses on summarization (Oya et al., 2014; Li et al., 2019; Zhu et al., 2020, inter alia). However, lack of annotated data impedes research on other important NLP tasks in this domain. To address this gap, we introduce a question-answering (QA) task based on conversations in meeting transcripts. Specifically, we consider questions asked by participants during the meeting and aim to extract corresponding answer spans from relevant discussions among meeting participants (refer to Figure 1). This task has several practical applications such as building an interactive meeting browser/interface for navigating through transcripts and informing tasks such as meeting summarization and handling action items involving QA pairs (Kathol and Tur, 2008; August et al., 2022).
15000
While standard QA datasets consist of human generated questions either based on short supplied contexts (Rajpurkar et al., 2016, 2018; Rogers et al., 2021) or are answered using a large collection of documents (Joshi et al., 2017; Kwiatkowski et al., 2019; Zhu et al., 2021b), our task setting is challenging yet interesting in several ways. First, meeting transcripts are long documents and QA systems still struggle to understand long contexts (Pang et al., 2022; Soleimani et al., 2021). Second, successfully answering questions asked within meetings requires robust understanding of the conversation and discourse that takes place both before and after a question. Third, the multi-party spoken text falls under a different domain when compared to typical text documents. While standard long documents rarely include any meaningful (nonrhetorical) questions, multi-party meetings often involve discussions asked by one participant and answered by the rest, allowing us to use these questions to create a QA dataset. Furthermore, the conversational nature of transcribed text differs from written documents and may contain disfluencies and other artifacts. Finally, instead of using annotator-generated questions (like in Wu et al. (2022)), questions asked by participants are more open-ended and discussion-seeking, with interesting answer types that can be multi-span and/or contributed by multiple speakers (e.g., Figure 1).
To this end, we first introduce our dataset MEETINGQA, created by annotating meetings transcripts from the popular AMI (Augmented Multi-party Interaction) corpus, containing over 100 hours of meetings (Carletta et al., 2005), via a robust annotation pipeline . MEETINGQA comprises of 7,735 questions asked by participants across 166 meetings. Unlike other datasets, questions in MEETINGQA are less concise (12 words on average) and reflect queries asked in a conversational setting. The answers include realistic situations such as rhetorical questions, multiple discontiguous spans and/or contributions from multiple speakers.
Next, on MEETINGQA dataset, we test diverse models designed for long input contexts such as Longformer (Beltagy et al., 2020), and BigBird (Zaheer et al., 2020) as well as RoBERTa (Liu et al., 2019), and DeBERTa-v3 (He et al., 2020) with as much meeting context surrounding the question as possible. To incorporate the multi-span nature of answers in our dataset, we design and experiment with multi-span variants of the aforementioned
models. Furthermore, we also investigate how well recent instruction-tuned large language models fare at answering questions from MEETINGQA. Lastly, we create a silver-annotation pipeline using MEDIASUM (Zhu et al., 2021a), a corpus containing 463.6K short interview transcripts, to provide additional training data. We find that the best performance is achieved by finetuned short-context models (F1 = 57.3). Overall, we show that models struggle to identify rhetorical questions and selecting which utterances constitute the answer. Thus, model performance significantly trails behind human performance on MEETINGQA (F1 = 84.6), leaving a large potential for future improvements on this challenging task.

heading: 2 Our Dataset: MEETINGQA
text: We first describe our data collection process in Section 2.1 and then provide an extensive analysis of MEETINGQA in Section 2.2.

heading: 2.1 Data Collection
text: Question Selection. We leverage the punctuated text to identify possible questions (ending with ‘?’). We also filter out questions containing ≤ 2 words as we manually find them to be either meaningless or rhetorical. While questions are marked to facilitate annotators, we encourage them to find missed potential questions due to incorrect punctuation.
Answer Annotation. For each possible question, we ask annotators to label the set of sentences (each identified by a unique number) from the meeting transcript that form the answer. Additionally, we also collect meta-data about the question. First, we ask the annotators to label if the question was meaningful, used to filter out rhetorical, unanswered or logistical questions and incorrect punctuations. Some speakers can ask consecutive or multiple questions in the same turn that are often related and answered together. In such scenarios, we allow annotators to combine questions and provide a common answer from the meeting transcript. The annotators mark these questions using the combined question attribute. Finally, since our questions are conversation segments, they may not be self-contained. Hence, we ask annotators to mention the question context sentences (if any) separately. We refer readers to Appendix A for more details and examples from MEETINGQA.
Annotation Process. All annotators were hired by a professional crowdsourcing company TELUS.2 The company obtained consent from the crowd workers and conducted ethical reviews. To train annotators, we provide comprehensive instructions for each type of annotation with several manually annotated examples from a small subset of transcripts and different possible scenarios curated by the first author. The annotations were collected in multiple batches, starting with the first batch containing a small subset of 250 questions. We iteratively provided extensive feedback to the crowdworkers on their annotations and resolved existing issues till the annotations were satisfactory. Next, we assigned three independent annotators to each question, and calculated Krippendorff’s α = 0.73 (Krippendorff, 1980) using MASIdistance (Passonneau, 2006), indicating substantial agreement. We then collected annotations for the remaining questions in two additional batches using one annotator per question followed by a quality assurance stage to validate the outcome of the annotations. Overall, we spent $10,427 in the annotation process, amounting to $61 per meeting. For additional details refer to Appendix A.

heading: 2.2 Dataset Information and Analysis
text: After filtering and quality control, we were left with a total of 7,735 questions from 166 meetings (≈ 100 hours of meeting recordings). Size and Splits. We split our dataset into train, dev, and test sets such that questions in each split come from distinct meetings. Table 1 shows dataset statistics across different answer types, namely
2https://www.telusinternational.com/
unanswerable, multi-span, and multi-speaker (described below). Due to relatively small number of meetings in the AMI corpus and diversity in meeting content, our test set contains a larger fraction of questions from the dataset as opposed to the conventional 80:10:10 split across train/dev/test sets.
Question Types. Unlike most QA datasets, questions in MEETINGQA are extracted directly from the meeting transcripts. Consequently, we find that questions may not be concise, and may not begin with ‘wh’ prefixes, making our dataset challenging yet interesting for the community. We perform a manual analysis of question types based on 200 randomly selected questions from the test set in Figure 2 (left). First, we observe that a majority of questions in MEETINGQA are framed in a ‘yes/no’ manner, followed by ‘what’ and ‘how’ questions that are typically information-seeking. We find that in a discussion-heavy setting such as ours, yes/no questions elicit a detailed response that cannot be reduced to a direct ‘yes/no’ response in over 40% of the cases (see Figure 2 (right)). Further, manual analysis shows that nearly half the questions are subjective, i.e., seeking opinions of meeting participants, and as high as 20% of answerable questions
are framed rhetorically. Appendix A contains additional tri-gram-based analysis of questions.
Length. Figure 3 shows the distribution of the length of meeting transcripts, questions, and answers in MEETINGQA. On average, each meeting transcript comprises of 5.8K words which constitute as long documents unlikely to fit entirely in the input context of typical pretrained language models (Devlin et al., 2019; Liu et al., 2019; He et al., 2020). Further, questions and their answers contain an average of 12, and 35 words respectively.
Answer Types. Due to the nature of meeting conversations and questions asked by participants, most answers are direct responses or follow-up discussions. However, some questions are rhetorical or do not elicit any discussion. These questions are unanswerable (30% of MEETINGQA). Among answerable questions, we note two scenarios of interest: multi-span and multi-speaker answers. Multispan answers contain non-consecutive and discontinuous utterances or sentences, typically in the form of relevant discussion interleaved with irrelevant chit-chat (see examples in Appendix A). Additionally, multi-speaker answers occur when multiple participants contribute to answering a question which is typical in a discussion. Note that multispeaker and multi-span answer cases are not mutually exclusive (refer to Figure 1 for an example). We find that 40% of all answers (excluding unanswerable questions) in our dataset are multi-span and 48% of answers are multi-speaker in nature. Moreover, Figure 2 (right) shows from our manual analysis that a considerable amount of disagreement exists among speakers in multi-speaker answers, with approximately 70% of cases displaying some form of disagreement. Notably, 22% of answers involve additional follow-up or action items, which are specific to the context of meetings.
Human Performance. We estimate human performance on MEETINGQA using a random subsample of 250 questions from the test split. Each question is assigned a different annotator who had not previously annotated the meeting containing that question. Scoring the provided answers relative to the reference answers in our dataset, yields an F1 of 84.6. This breaks down to F1 of 80.7 and 86.3 for unanswerable and answerable questions respectively. The F1 score for multi-span and multispeaker answers is 88.1 and 87.7 respectively.

heading: 3 Methods
text: In this section, we investigate the difficulty level of our new MEETINGQA for state-of-the-art QA systems and establish strong baseline results. We describe strategies for retrieving contexts from transcripts in Section 3.1, followed by different QA models in Section 3.2, and silver data annotation for data augmentation methods in Section 3.3.

heading: 3.1 Retrieving Contexts from Transcripts
text: Given that meeting transcripts are very long documents, it is infeasible to input the entire transcript as context to typical QA models. Thus, we first select a smaller transcript segment that fits the model’s input length limitations. We explore two strategies to retrieve contexts as described below.
Location-based Context Retrieval. We use the relative location of the question in the meeting transcript to retrieve a context by fitting as many (complete) sentences as possible under a fixed length budget (measured in words). Further, we split this budget into two components: prefix and suffix referring to the sentences that precede and succeed the question respectively. We set the prefix budget to 50 words and the suffix budget to 250 words respectively, resulting in a total budget of 300 words.3
3Ensures context fits into QA models limited 512 tokens.
Retrieval Method Answer-Span Overlap Upper Bound
Note that the suffix budget is significantly larger than the prefix budget since we expect to find answers in sentences following the question. The sentences before the question only provide additional context to the ongoing discussion.
Score-based Context Retrieval. Alternatively, we use the question as a query and compare it to other sentences from the entire transcript via two scoring methods consistent with Pang et al. (2021). First, we retrieve sentences using ROUGE-1 score relative to the question. Second, we use cosine similarity based on sentence embeddings (Reimers and Gurevych, 2019). We concatenate sentences in the order they appear in the transcript until reaching the total length budget. Similar to location-based retrieval, we set the total budget to 300 words.
Results of Context Retrieval. Table 2 compares both retrieval methods using the same total length budget on the answerable questions split. We observe that the sentence-level overlap between extracted contexts and annotated answers for score-based retrieval is significantly lower than for location-based retrieval. We use this overlap to compute the maximum achievable performance of QA systems for each type of retrieval. Correspondingly, we find similar trends in upper-bound performance metrics (discussed in Section 4) with location-based contexts (near-perfect max F1) considerably outperforming score-based contexts (max F1 < 40). Therefore, for short-context models, we henceforth use location-based contexts.

heading: 3.2 Models for Meeting-based QA
text: We primarily focus on extractive models including both short and long-context models. Given the transcript or a segment from it (context) and the question, models are tasked with extracting answer-span(s) from the context. We use two highperforming short-context models RoBERTa and DeBERTaV3, each supporting up to 512 tokens,
with extracted context from Section 3.1. Additionally, we explore Longformer and BigBird which support longer sequences of up to 4096 tokens by utilizing a combination of sliding window and global attention mechanisms. Further, the Longformer Encoder-Decoder (LED) model supports up to 16,384 input tokens. These models allow us to use most or all portions of the transcript needed for answering the questions as the context. In case of an overflow, we use as many utterances from the transcript around the question as possible and truncate the rest. Note that these models output a single answer-span by default. Therefore, for multi-span answers, we train models to predict a span starting with first utterance and ending with the last utterance of the gold answer.
Multi-Span Models. In order to better model multi-span answers, we follow Segal et al. (2020) and pose multi-span QA as a sequence tagging task, predicting if each token in the context is part of the answer. For simplicity, we restrict ourselves to their proposed IO tagging. Thus, the answer prediction is a concatenation of all token-spans contiguously tagged with I. Similar to single-span models, we train multi-span variants of RoBERTa, DeBERTa, Longformer, and BigBird models.
Instruction-Tuned Models. Furthermore, we use FLAN-T5 (Chung et al., 2022), a publiclyavailable instruction-tuned model, to study zeroshot performance on our MEETINGQA. Given the relatively large size of contexts and distinct nature of our task, we rely on succinct instructions instead of few-shot demonstrations. Furthermore, due to the model’s generative nature, we cannot directly use the predictions for our extractive QA task. Therefore, we adapt instruction-tuned models for our setting by employing instructions that ask models to list sentences instead of directly generating answers that may be less faithful to the context. Next, we filter out predicted sentences not present in the context. While this is a strict selection criterion, it removes any possible hallucinations.4

heading: 3.3 Silver Data Augmentation
text: Due to high annotation costs of gold labels, and unavailability of similar QA datasets, we investigate automatic methods to annotate answers. We match the salient features of MEETINGQA, such as meaningful questions within the transcript and
4Appendix E shows these choices improve overall scores.
multi-speaker discussions using the MEDIASUM dataset (Zhu et al., 2021a). This dataset contains 463.6K short multi-party interview transcripts, detailed speaker information, and identifies a host or interviewer who steers the discussion via questions.
We begin by identifying the host speaker and focusing on their questions. Next, we predict which speaker(s) would answer the question by identifying speaker entities mentioned in utterances or from previous dialogue turns. Finally, we search utterances from the identified speakers until a stopping criterion is met and label it as the answer. Due to the assumptions made in the above process, models trained directly on this data could overfit on spurious correlations (Jia and Liang, 2017; Wang and Bansal, 2018). Thus, we apply various perturbations to the context such as separating the question and answer utterances, converting to unanswerable questions by removing relevant sentences, creating more speaker transitions, and masking speaker names. Refer to Appendix F for additional details.

heading: 4 Experiments and Results
text: Evaluation Metrics. Following Rajpurkar et al. (2016) we report macro-averaged F1 on the entire test set as well as on specific answer types (Section 2.2).5 However, F1 treats sequences as bagof-words, and thus, there can be a non-significant
5We also report exact match (EM) scores in Appendix C.
overlap between a random span and the target span for large span lengths. To address this, Soleimani et al. (2021) propose reporting Intersection-overUnion (IoU) defined as:
IoU = |p ∩ t| / |p ∪ t|,
where p and t are the predicted and target spans, respectively. Since our answer spans are much longer than those in SQuAD (refer to Figure 3), we also report macro-averaged IoU to measure performance.
Training Settings. We measure performance of various models in both finetuned and zero-shot settings. First, we directly finetune the base pretrained model on the model on MEETINGQA. Next, to supplement training data we explore intermediatetraining (Phang et al., 2018; Pruksachatkun et al., 2020) with SQuAD v2.0 (Rajpurkar et al., 2018)6 or a combination including silver data from Section 3.3 prior to finetuning on MEETINGQA, increasing the training data by 5x and 10x respectively. Additional details on checkpoints, hyperparameters, and training are present in Appendix B.
Turn-based Baseline. We devise a straightforward algorithm called turn-based baseline that is inspired by the automatic silver data annotation
6SQuADv2.0 is used for all models except BigBird, for which we use TriviQA due to lack of reliable existing model checkpoint on HuggingFace (Wolf et al., 2019).
algorithm explained in Section 3.3. In the turnbased baseline, when a speaker asks a question, the predicted answer includes all the subsequent utterances of other speakers until the same speaker gets another turn (stopping criterion). Note that, turn-based baseline assumes all questions can be answered and always provides single-span answers, although the predictions may be multi-speaker.

heading: 4.1 Results and Discussion
text: We report performance of various fine-tuned singlespan, multi-span models in Tables 3, and 4 respectively on the test split of MEETINGQA. Further, we evaluate zero-shot performance in Table 5. We summarize our findings below and refer readers to Appendix C for additional results.
Main Baselines and Comparison with Human Performance. Results from Tables 3 and 4 show that single-span models (narrowly) outperform the multi-span models, with the best overall performance achieved by single-span variant of DeBERTa-base (overall F1 = 57.3). Other singlespan variants of Longformer and BigBird achieve higher performance on answerable questions (up to F1 = 64.4) but have lesser overall performance due to lower F1 scores on unanswerable questions.7 Comparing to the human performance (overall F1 = 84.6), we find at least a 25 point difference in overall F1 of all finetuned models. Across various answer types, the difference in F1 scores is still at least 20 points. Similar trends holds for EM and IoU metrics too.8 In the zero-shot setting (refer to Table 5), the difference in overall scores with respect to human performance is even greater (≥ 44
7Model predictions may be biased against (or towards) empty spans impacting score of unanswerable questions.
8Following the order EM ≤ IoU ≤ F1 for all models.
points across all metrics). Furthermore, all finetuned models outperform the turn-based baseline (with the exception of LED-base), whereas the corresponding zero-shot variants fail to outperform the turn-based baseline on overall metrics. This suggests that our dataset is challenging for current QA systems, leaving significant scope for improvement via interesting future work.
Impact of Long-Context Models. We observe that in a majority cases short-context models (especially RoBERTa) outperforms long-context models (Longformer and BigBird) by 1-2 points. Furthermore, the LED model that completely fits 90% of transcripts has significantly lower overall score (≈30 F1 point difference) due to poor performance on answerable questions.9 We believe that the ability to fit larger contexts is traded-off by welloptimized design of short-context models. This is consistent with the findings of Pang et al. (2022) and suggests better long-context models may be needed to outperform shorter extractive models.
Impact of Multi-Span Models. Table 5 shows that in the zero-shot setting, multi-span variants slightly outperform their single-span counterparts for long-context models and slightly underperform for DeBERTa. In Appendix C, we find that within answer types zero-shot performance drops for unanswerable questions while improving for multi-span and multi-speaker answers. For finetuned models (Tables 3 and 4), the overall performance of multispan models is comparable if not slightly less than single-span variants.10 Notably, for short-context
9Due to this, we do not experiment further with LED models in multi-span, intermediate-training, and zero-shot settings.
10Note that we do not intermediate train multi-span models on standard extractive QA tasks such SQuAD v2.0. Therefore, gold training data for multi-span models is always scarce.
models, there is significant gain in performance for all answerable questions. Further, we observe that multi-span models consistently underperform on unanswerable questions (as high as 15 F1 points). Performance of multi-span model on unanswerable questions can be negatively impacted by even one false positive I tag, changing the prediction from unanswerable to answerable. While prior work on multi-span QA (Segal et al., 2020; Li et al., 2022) have found tagging-based approaches to outperform single-span variants, they only explore factoid questions on relatively shorter contexts. Future work can focus on improving multi-span QA for more open-ended questions like in MEETINGQA.
Impact of Intermediate Training. Silver data augmentation is effective in zero-shot settings with ≥15 point improvement for single-span longcontext models (Table 5). For finetuned models, however, we do not observe significant improvements in overall scores from intermediate-training compared to directly finetuning. Interestingly, silver data augmentation improves performance on unanswerable questions for single-span models (except DeBERTA) and multi-span models.
Instruction-Tuned Models. Lastly, Table 5 shows zero-shot performance of instruction-tuned FLAN-T5 model. We find the FLAN-T5 XL model (3B parameters) outperforms most zero-shot singlespan models and narrowly underperforms zero-shot multi-span models. Despite the design of instruc-
tions and filtering (Section 3.2), the model underperforms on unanswerable questions. Thus, we add an additional step to identify answerable questions and use model responses only for predicted answerable questions. The question classification can be done zero-shot using the same FLAN-T5 model11 or by training an external supervised model.12 We observe that using the FLAN-T5 model is more effective (yields best performance) than using a supervised model (6 F1 point drop) as the predictions of the latter are biased towards the question being unanswerable. Future work can further focus on accurately identifying answerable questions to improve overall performance.
Error Analysis. Next, we analyze some intriguing patterns in the errors within model predictions. Firstly, we observe that identifying rhetorical or unanswerable questions asked in a meeting is a challenging sub-task. Training a separate binary classification model that classifies whether a question is answerable based on the context from MEETINGQA yields only an F1= 49.2 (see Appendix B). In Figure 4a, it becomes apparent that a significant portion of errors in predictions for answerable questions stem from the model incorrectly predicting that the question is rhetorical, particularly in the
11Different instruction like based on the context, did anyone answer the given question? This elicits a ‘yes’/‘no’ response.
12Using the sequence classification head of RoBERTa-base model trained on questions from MEETINGQA (Appendix B).
zero-shot setting. Additionally, in case of multispan answers, single-span models exhibit higher fraction of errors where predictions include sentences not present in the gold answer, in contrast to their multi-span counterparts (for details refer to Appendix D). This follows from the construction of single-span models, as described in Section 3.2. Lastly, for multi-speaker answers, we analyze the overlap in speakers (measured via IoU) of predicted and gold answers in Figure 4b. We find that even incorrect predictions of finetuned models contain roughly 55% speaker overlap with the gold answer, i.e., models can effectively predict which speakers answer the question. However, incorrect predictions in the zero-shot setting contain only 30% speaker overlap indicating that zero-shot models may struggle to predict which speakers answer the question. Future works can explore methods to effectively identify rhetorical questions and predict which speakers answer the question to improve overall performance. A more detailed analysis of errors can be found in Appendix D.

heading: 5 Related Work
text: Our work builds upon prior work on meeting transcripts and question answering. Rogers et al. (2021) provide a comprehensive survey of several QA datasets and formats.
Meeting Transcripts. Several other small-scale corpora of meeting recordings or transcripts are publicly available (Janin et al., 2003; Garofolo et al., 2004; Chen et al., 2005; Mostefa et al., 2007). We restrict ourselves the most popular and frequently used AMI corpus. Other works study various aspects of summarizing meeting transcripts (Mehdad et al., 2013; Wang and Cardie, 2013; Shang et al., 2018; Li et al., 2019; Zhu et al., 2020, inter alia) or extracting action-items (Morgan et al., 2006; Purver et al., 2007; Cohen et al., 2021). The work closest to ours uses Markov models to classify dialogue-acts as questions, answers or others (Kathol and Tur, 2008).
QA on Conversational Text. Prior work comprises of QA datasets based on small chit-chat from TV shows (Sun et al., 2019; Yang and Choi, 2019) or domain-specific chat-rooms (Li et al., 2020). The QACONV (Wu et al., 2022) dataset builds on these works with conversations from multiple domains (including MEDIASUM). However, these works employ human annotators for generating
questions based on their understanding of the conversation resulting in straight-forward questions testing local information. Consequently, the answer spans of these datasets are significantly shorter, single-span, restricted to one speaker and often correspond to simple noun phrases (as high as 80% for QACONV). In contrast, questions asked by meeting participants are more open-ended, discussionseeking, and correspond to longer answers (≈ 7x) with complex multi-span and multi-speaker scenarios. Note that our work is different from conversational QA datasets that consist of a sequence of questions and answers simulating a conversation grounded in a short paragraph (Choi et al., 2018; Reddy et al., 2019; Campos et al., 2020).
Long-Context QA. Recent works show that QA models struggle to understand answer questions correctly using long contexts (Pang et al., 2022; Mou et al., 2021; Soleimani et al., 2021; Dasigi et al., 2021). However, unlike our work, the source (long) documents for building these datasets are taken from written-text domains such as books, film-scripts, research papers, or news articles.

heading: 6 Conclusion
text: In this work, we present MEETINGQA, an extractive QA dataset based on meeting transcripts to identify answers to questions asked during discussion among meeting participants. Detailed analysis of the data reveals it is a challenging real-world task. Baseline experiments with a wide variety of models show the current performance lags behind human performance by at least 25 and 44 overall F1 points for finetuned and zeroshot models respectively. This demonstrates that current QA systems find our task challenging, leaving tremendous scope for improvement. We hope that future works will aim to bridge this gap and our work fosters research in NLP tasks (especially QA) on other text domains such as meeting transcripts.

heading: Acknowledgements
text: We thank the reviewers and the area chairs for their helpful comments and feedback. We thank TELUS International for their help with data collection. We also thank Shiyue Zhang and Swarnadeep Saha for their helpful comments. This work was partially supported by NSF-CAREER Award 1846185, and NSF-AI Engage Institute DRL-2112635. The views contained in this article are those of the authors and not of the funding agency.
Limitations
Due to the structure of MEETINGQA, the answers to questions asked by participants (if any) are present in the transcript itself, making it an extractive task. Therefore, we do not extensively explore the use of generative models since the predictions do not stick to the sentences in the transcript and could possibly include hallucinations. However, we aim to mitigate hallucinations by using instruction-tuned generative models with suitably designed instructions and enforce a strict exact match criteria for filtering any possible hallucinations. Future work can explore how to adapt or evaluate non-instruction-tuned generative models on this task and better identify hallucinations with a more relaxed filtering to improve performance. We also do not report zero-shot performance of InstructGPT (Ouyang et al., 2022) as these models are not freely accessible. Additionally, we use a simple multi-span QA adaptation technique from Segal et al. (2020), but predicting answer spans by classifying each token can be difficult to train leading to slightly lower performance (discussed in Section 4.1). We hope our dataset provides additional motivation for future work on multi-span QA. Finally, MEETINGQA only comprises of publicly available meeting transcripts in English, but our methodology of data collection and model training (using multilingual variants) should still be applicable for other languages in future work.
Ethical Considerations
The human participants in our work were recruited by an external crowd-sourcing company that ensured annotators provided informed consent, were given fair compensation, and no personally identifiable information (PII) was collected or released. We use existing publicly available meeting transcripts collected by the AMI project (Carletta et al., 2005) in controlled scenarios and filtered for offensive/toxic content. We also conducted manual inspection of a random sample from annotated transcripts and did not find any toxic content or PII. Furthermore, the collected data and experiments are conducted in English and we do not claim generalization of our findings across languages. Given the broad nature of meetings, the content can fall into a number of domains, of which only a few are represented in the AMI corpus. Therefore, we do not expect models trained on MEETINGQA to generalize to certain domains such as judicial, ethi-
cal review, congressional proceedings, etc. which involve specific jargon and rules of engagement.

heading: A Additional Details on MEETINGQA
text: A.1 Tri-gram Analysis of Question Types
In contrast to most QA datasets, questions in MEETINGQA are extracted directly from the meeting transcripts and thus are conversation segments. Consequently, we find that questions may not be concise, often use auxiliary verbs, and do not typically begin with ‘wh’ or ‘how’ prefixes, making our new QA task and dataset challenging yet interesting for the community. This makes conventional analysis of question types based on prefixes less relevant here, and instead, we compute the top-25 most common trigrams from all questions, shown in Figure 5. The three most common question patterns are: ‘do you/we ...’, and ‘what ...’. Additionally, the trigrams demonstrate that our questions are open-ended and seeking opinions or thoughts from other participants that tend to elicit long responses.
A.2 Dataset format and meta-information
We provide annotations for each meeting transcript at the sentence level in ‘.json’ format, and each sentence has 4 primary attributes: displayText, speakerFaceId, sentenceId, and question which contain the sentence text, integer identifier of the speaker (unique within a meeting), integer identifier of the sentence, and information about the sentence as a question respectively. The question attribute is relevant only if the sentence is identified as a question. It contains additional attributes: possible, meaningful, questionContext, combinedQuestion, and answerSpan. First, we perform “ques-
tion selection” described in Section 2.1 and set the possible tag as True to guide the annotators. The remaining attributes are set to default values meaningful = False, answerSpan = [], questionContext = [], and combinedQuestion = [] prior to annotation. Annotators modify the question attribute during the course of the annotation and can even mark additional questions outsider our question selection criteria by setting possible = True. They lable the remaining attributes according to the “answer annotation” steps mentioned in Section 2.1. The list type attributes questionContext, combinedQuestion, and answerSpan contain sentences specified using the value of the corresponding sentenceId attributes. The domain of meeting transcripts (from AMI corpus) is a combination of elicited scenario-driven data, and natural data. We refer interested readers to the AMI project page for more information about the topic or scenario of each meeting.13
We find that out of 7.7K questions in MEETINGQA, only 66 (< 1%) additional questions were identified by the annotators that were missed by our
13Refer to documentation links: https://groups. inf.ed.ac.uk/ami/corpus/scenariomeetings. shtml, and https://groups.inf.ed.ac.uk/ami/ corpus/nonscenariomeetings.shtml.
question selection criteria. Further, 751 questions (9.7%) were annotated with additional context sentences via questionContext and a total of 784 (10.1%) were combined with another question via combinedQuestion attribute. Among the latter, an average of 2.2 (maximum 4) questions were combined and these questions were an average of 1.5 sentences apart. The average length of questionContext (when annotated) was 1.7 sentences (maximum 3) which preceded the question by 1.7 sentences. Note that for the purposes of QA evaluation, we only use the possible and answerSpan attributes. The remaining attributes serve as meta-information to understand the dataset better and can facilitate error analysis and/or future work. Also to come up with overall question counts, we ignore the combinedQuestion attributes and count all the questions individually. Therefore, this attribute serves as an indicator of when and why different questions share the same answer. Empirically, we note that the combined
questions and question context typically fit within the contexts created using location-based retrieval (Section 3.1) and are present in the input fed into QA models in the vast majority of cases.
A.3 Additional Annotated Examples
Next, we show multiple examples of snippets from meeting transcripts with QA components present in MEETINGQA in Figures 6-11. Figure 7 also contains an example of an unanswerable question asked by Speaker 4 (“my data is coming?”) which is either rhetorical or corresponds to incorrect punctuation. In such cases, annotators label meaningful = False and an empty/null answer annotation (answerSpan = []). On the other hand, Figure 11 also contains two consecutive questions asked by Speaker 1 but the annotators mark both as meaningful = True, but choose to combine them via (combinedQuestion) and share a common answer. This because the first question is more generic, and the second question builds on top of it, by providing a specific example of what is loaded and what isn’t. Further, in Figure 8 we provide an example of question which needs additional context sentences annotated via questionContext. Figures 6, 9, and 11 are diverse instances of multi-speaker and multi-span answers in our dataset.
A.4 More on Data Collection
AMI Transcript Preprocessing. The AMI corpus is a collection of 171 meeting transcripts containing manually annotated and punctuated speakerspecific XML files for each meeting. We parse these XML files and combine utterances from multiple speakers by aligning the start times into a single transcript (with speaker information) corresponding to each meeting. We then use a disfluency detector model to identify and remove disfluencies from the utterances (Jamshid Lou and Johnson, 2020).
Annotator Recruitment and Training. All annotators are hired by a professional crowdsourcing company TELUS.14 The company obtained consents from the crowdworkers before the annotation process and conducted ethical reviews. The company recruited 18 annotators, all based in the United States and native English speakers, who had previously successfully participated in text-based annotation projects. In addition to the instruction document (shared in the supplementary) curated by the first author, TELUS conducted a series of (virtual) meetings to deliver instructions, conduct example walk-through of the annotation and clarify doubts. At the end of initial training, a small batch of 5 meetings was provided to each of the annotators to calibrate performance. The responses were then compared to the good quality annotations performed by 2 project leads at TELUS manually in consultation with the authors. Feedback was pro-
14https://www.telusinternational.com/
vided to the annotators to improve the quality of annotations, and based on their final responses the top-6 best performing annotators were selected to work on the project.
Quality Control. From the pool of selected annotators, project leads recruited two of the best performing and experienced annotators to help with quality control. Any annotated meeting transcript was assigned to either of these two annotators for review. Between the two, meeting annotated by one was assigned to the other for review. After the review, minor errors in annotation were fixed directly, otherwise major errors were sent back to the respective annotators for a re-annotation of the question and in some rare cases the annotation was redone by the reviewers. At the end of annotation batch (total of 4), the transcripts were sent to the authors who extensively reviewed them and provided feedback. We also looked for typos, and other issues which were fixed promptly. The TELUS project leads did not find any toxic and offensive content at their end and no such concerns were reported in the quality control stage. Further, all communication with the annotators is done by the crowdsourcing company and no personally identifiable information (PII) is released to the authors. Additionally, their execution platform contains unique identifiers
for all annotators ensuring their PII is not released along with the annotated data. Finally, based on the feedback from annotators, we removed all questions corresponding to 5 meetings as the meeting content was hard to follow.
Time Taken. On an average across meeting transcript and annotators, it took ≈ 1 hour to annotate each meeting transcript which averages to ≈ 1.3 minutes per question. However, the per meeting annotation time strongly correlated with the length of the transcript (number of sentences). Among questions, annotators spent more time on answerable questions. This is because if the question is marked unanswerable (not meaningful), they did not have find exact answer span (in sentences) and annotated other meta-information. The project leads also ensured that the amount of time taken by each annotators was consistent with internal estimated by the intial batch used during annotator recruitment. The total production time for this project (for TELUS) was 186.5 hours.
Compensation. The annotators were compensated through a fixed hourly rate defined for each participant. No additional bonus was provided to incentivize faster turnaround times. The average hourly wage for participants was roughly $20/hour in compliance with all the federal and local local laws to ensure fair payment.

heading: B Experimental Details
text: GPU Compute. For training and/or inference we used a combination of 6 NVIDIA A10 24 GB GPUs and 2 NVIDIA RTX A6000 48GB GPUs. Directly finetuning on MEETINGQA starting from a pretrained checkpoints is quite fast and takes not more than 4 GPU hours (depending on the batch size). Intermediate training on the silver-annotated data (5 epochs) takes about 12-18 GPU hours (training time is higher for long-context models).
Hyperparameters. For the short-context models (RoBERTa and DeBERTa-v3) we used max sequence length of 512, (stride of 128 but not utilized due to location-based context retrieval), and batch size of 16. For long context models, we used max sequence length of 4096, stride of 128, and batch size varied between 8 and 16 (depending on GPU availability). Note that there is no stride for multispan models. For model training on MEETINGQA, we use a learning rate of 3e-5, warmup ratio of 0.2, and train for 15 epochs with an early stopping
criteria set to a patience of 2 epochs (F1 on dev split). For intermediate training with silver data, we use the same hyperparameters except we train for 8 epochs with the same early stop criteria. The hyperparameters values used are pretty standard and were not tuned explicitly for MEETINGQA.
Model Sizes. RoBERTa base and large models comprise of 125M and 355M parameters respectively, while DeBERTa base and large models comprise of 86M and 304M parameters. On the other hand, Longformer-base comprises of 149M parameters. Since BigBird is initialized with the RoBERTa checkpoints they share the same model size. Finally, instruction-tuned models FLAN-T5 LARGE, XL and XXL consist of 770M, 3B, and 11B model parameters respectively.
Pretrained Checkpoints. For models without intermediate-training we use the standard checkpoints for all models available on HuggingFace. For the score-based context retrieval in Section 3.1, we use HuggingFace’s evaluate library for computing ROUGE1 and the multi-qa-MiniLM-L6-cos-v1 model from the sentence-transformers python package for embedding cosine similarity. During silver data annotation, we used the en_core_web_sm from spacy package for NER. For intermediate training (SS) we used the following pretrained checkpoints (base size):
• RoBERTa: deepset/roberta-base-squad2 • DeBERTa: deepset/deberta-v3-base-squad2
• Longformer: mrm8488/longformer-base-4096finetuned-squadv2 • BigBird: google/bigbird-base-trivia-itc
Licensing. We used the AMI dataset that has CC-BY-4.0 license. Our released data will have the CC-BY-NC license. We do not violate the constraints put in the MEDIASUM dataset to use interview files for reasearch purposes only.
Instructions/Prompts. For instruction-tuned FLAN models we use the following prompt template to generate sentences from the context that answer the question.
[CONTEXT] Based on the conversation above, which sentences from the converstation answer [SPEAKER]’s question: [QUESTION]
Here, [.] is a placeholder filled in separately for each instance/question. Additionally, for the ‘self ask’ setting, we first use a prompt (shown below) to get the model to output if the question is answerable. If the model outputs “no”, filter out those questions and use an empty string as
the predictions. We find answers to the remaining questions using the prompt above.
[CONTEXT] Based on the conversation above, did anyone answer [SPEAKER]’s question: [QUESTION] Respond “yes” if answered, “no” otherwise.
Binary Answerable Classification Model. As mentioned in Section 4.1, we train a separate supervised RoBERTa-base model to detect if a question is answerable. This is formulated as a binary classification task, therefore we train the sequence classification head on questions from MEETINGQA. We use the same hyperparameters as for single-span RoBERTa models mentioned above. The final performance of this model, is not as strong with an overall F1 = 49.2. This indicates that even a simple binary task formulation from MEETINGQA is challenging and requires thorough understanding of meeting discussions.

heading: C Additional Results
text: Building on Tables 3 and 4, which contain F1 and IoU scores, we present the exact match (EM) scores
in Tables 6 and 7 for finetuned single-span and multi-span respectively. While the relative trends across the models remains the same, we find that the EM scores are the lowest because it reflects predictions that perfectly match the reference. Another noteworthy observation is that the EM scores of all single-span models on the multi-span split is 0. This can be explained by the training procedure of single-span models (described in Section 3.2). The models are trained to predict a single “super-span” starting from the first sentence in the reference to the last sentence in the reference. Therefore, even in the theoretical best-case-scenario, the models would predict a single super-span containing all the reference sentences interleaved by irrelevant sentences for questions with multi-span answers. We analyze errors due to this in Appendix D.
In Table 5, we only present the overall scores for various models. All the scores on different
splits are given in Table 8. We observe that for all single-span models (except RoBERTa on unanswerable questions) adding silver data in intermediate training helps improve performance across all splits. Furthermore, the multi-speaker and multispan splits consistently pose a challenge for all models evaluated in a zero-shot setting. Also, within answer-types performance drops for unanswerable questions while improving for multi-span and multi-speaker answers. The challenge posed by unanswerable questions can be explained by the multi-span adaptation (Section 3.2). By posing question answering as a token-classification task, even one false positive (I tag instead of all Os) in token label, changes the answer prediction from unanswerable to answerable. Finally, we note that when using a pipeline-approach of isolation unanswerable questions separately, we find the errors in this step cascade and are reflected in the per-
formance on answerable questions of the overall system. These systems perform better on unanswerable questions (not identified by regular instruction and filtering), however the false-positives decrease performance on answerable questions, even more so when using an external supervised model.
In Table 8, we see clear scaling of performance as we move from FLAN-T5 large (770M) to FLANT5 XL (3B). However, for FLAN-T5 XXL (11B) the performance of unanswerable, multi-span and multi-speaker questions increases (≥ 8 F1 points) but performance on other answerable questions decreases (≈ 9 F1 points) which in turn reduces the overall performance as compared to FLAN-T5 xl. Table 9 evaluates the performance of RoBERTalarge and DeBERTa-large architectures for singlespan and multi-span models in both finetuned and zero-shot settings. The corresponding performance of the base models can be found in Tables 3, 4, and 5 respectively. We do not observe any significant increase in performance when using the larger checkpoints, thus leaving ample room for future work to bridge the gap between model and human performance on MEETINGQA.

heading: D Error Analysis
text: In this section, we analyze error patterns across models discussed in Sections 3.2 and 4 in detail. First, we note that for the unanswerable questions split, any error corresponds to the model predicting a non-empty answer span. The frequency of this for a given model can be calculated by 100 − F1 score for this split (provided in Tables 3, 4, and 8). However, for answerable questions, errors in model predictions are diverse as categorized below.
I. Prediction is an empty-span (unanswerable)
II. Predicted span contains a sentence not present in the gold or annotated reference span
III. At least one of the sentences in the reference span is not present in the predicted span
IV. Combination of errors with respect to reference span (both II and III)
Therefore, whenever the model prediction does not exactly match the annotated reference span, we can put it in one of the above 4 categories. We perform this analysis for various finetuned single-span, finetuned multi-span models as well as zeroshot single-span, multi-span and instruction tuned models discussed in Sections 3.2 and 4. For brevity, we
12.2%
6.7%
65.4%
15.7%
FLAN T5 XL
41.0%
4.7%
42.9%
11.4%
FLAN T5 XL (self ans)
85.9%
1.3%
10.0%
2.8%
FLAN T5 XL (ext ans)
Predicted unanswerable Predicted sentence not in ref. Both ref. errors Ref. sentence not in prediction
pick representative models from different possible combinations of intermediate training data. This is illustrated in Figures 12-15 with error I shown in red, error II shown in yellow, error III shown in blue, and error IV shown in green.
Table 3 shows very similar performance different intermediate training data configurations for a given model architecture. Thus, we present error distribution for single-span models directly finetuned on MEETINGQA in Figure 12. We find that most of the errors belong to categories II-IV. The DeBERTa model has a relatively high unanswerable prediction error which is primarily because its predictions skew towards unanswerable as explained by the F1 score on No Answer (unanswerable) split in Table 3. Next, in Figure 13 we show the error distributions on the corresponding multispan models finetuned directly on MEETINGQA. We observe that, for all model (except RoBERTa) the frequency of incorrectly predicting unanswerable goes down as well as the prediction span con-
taining sentences outside the reference (error II). However, the frequency of hybrid error IV increases significantly. This can partly be explained by the design of single-span and multi-span models. As mentioned in Section 3.2, training data of the single span model involves creating a single “super-span” starting from the first sentence in the reference to the last sentence in the reference. This by construction involves error II and supervision on this data directs the model to include irrelevant sentences in the answer span if it is sandwiched between two relevant sentences. Also, for multi-span models an unanswerable prediction implies all tokens are labeled with the O tag, and even one false positive (I tag) would make the prediction answerable. Due to this, one can expect these models to mispredict empty spans less frequently.
Interestingly, when we look at zero-shot performance of single-span and multi-span models in Figure 14, we find relatively high frequency of error I and very low frequency of error II. The errors III
and hybrid IV also become more common. These models which are trained only on intermediate data, do not generalize in their ability to predict when a question is unanswerable. Further, we find that in the zero-shot evaluation setting, model predictions are shorter than the reference span by at least one sentence on average (≈ 2 sentences for multispan split). This indicates in zero-shot evaluation models are more likely to predict a part of the answer than output spans that covers all sentences in the reference, containing extra sentences that lie outside the reference.
Finally, we look at the errors of instruction-tuned FLAN-T5 models in Figure 15. When using FLANT5 with appropriate instruction and filtering we find that most of the errors are hybrid, i.e. predicted sentences do not cover the reference span entirely and also contains irrelevant sentences. When we add the self-ans pipeline on top of it with additional instructions to spot unanswerable questions, the predictions contain more empty spans (relatively) which is reflected in the increase in frequency of error I and the No Answer F1 (in Table 8). Surprisingly, when we use an external supervised model to predict unanswerable questions, it contributes to the vast majority of errors in the pipeline (error I). This is consistent with the fact that the test F1 score of this model on the task of classifying questions as answerable or not was only 49.2.
So far, we have analyzed errors in predictions for all the answerable questions. Next, we focus our attention on questions with multi-span and multispeaker answers. Within the multi-span split, we calculate the fraction of incorrect predictions (as per exact match) that are multi-span, denoted by multi-span preds (%). Similarly, for multi-speaker split, we calculate the fraction of incorrect predictions (as per exact match) that are multi-speaker in nature, denoted by multi-speaker preds (%). Further, we compare the list of speakers in the reference and predicted spans using Jaccard similarity (IoU) denoted as speaker IoU. We compute and report these metrics for all the aforementioned models in Table 10.
As expected, due to the single-span training, none of the predictions of the single-span models are multi-span in nature. On the other hand, even incorrect predictions of the finetuned multispan models are multi-span in nature at least half of the times. However, a significant fraction, between 29-46%, of the errors in this split can be attributed
to single-span predictions for various models. For zero-shot models, over 90% of incorrect predictions are single span (also vast majority of all predictions are single span). On the multi-speaker split, the incorrect predictions of finetuned models are multi-speaker in nature. However, the speaker IoU (< 65) indicates that predicted spans often miss utterances from relevant speakers in the reference and also include irrelevant utterances from other speakers. Zero-shot models on the other hand, only tend to give single-speaker responses which is the primary source for errors. Note that, relatively high frequency of error I or prediction unanswerable spans also contributes in driving down the values of these metrics (empty spans have no span or speaker information).
E Instruction-tuned Model Ablations
In Section 3.2, we describe adaptation of generative FLAN-T5 model to our extractive setting by (i) designing instructions that ask models to list which sentences from the context contain the answer (mentioned in Section B); and (ii) filtering out all sentences from the model response that are not present in the context to remove any possible hallucinations. To show the importance of both these steps, we first compare with an instruction eliciting a direct response (answer) from the model, mentioned below.
[CONTEXT] Based on the conversation above, state the answer(s) given to [SPEAKER]’s question: [QUESTION]
We call this direct instruction as opposed to the list instruction mentioned used in Sections 3.2 and 4.1. Further, we examine the importance of filtering by comparing raw model responses to their filtered counterparts. The comparison on a random subset of 500 question from the dev splits is shown in Table 11. We find that our chosen type of instruction (list) significantly outperforms the direct
instruction (tends to be more abstractive). Furthermore, filtering consistently improves overall performance especially when using direct instructions possibly due to higher number of hallucinations in the corresponding model answers.

heading: F Silver Data Augmentation Details
text: Section 3.3 describes the silver data annotation process to annotate publicly available interview transcripts from CNN and NPR (Zhu et al., 2021a) for extractive QA task similar to MEETINGQA. We first identify the subset of speakers that act as the host or interviewer and focus on questions asked by these speakers to generate answer annotations.
Based on the utterance containing the question, we first automatically identify speaker(s) answer the question using a rule-based approach. This is done by (i) finding speakers mentioned in the question sentence or utterance using an off-the-shelf NER model (mentioned in Appendix B), (ii) identifying speakers from previous speaker turns if the same speaker takes the turn after the host speaker assuming this is a case of follow-up questions, or (iii) in the absence of first two conditions, all speaker that take turns after the host. Finally, we search utterances corresponding to identified speakers in the transcript until a stopping criterion (max number of utterances, or reach the next host utterance) is
met and label it as the answer. From the 463.6K transcripts, only 15% of the files have identify host who steer the interview and have sufficiently high frequency of questions. However, each of these transcripts result in roughly 20 annotated questions on average. For intermediate training, we sample a total of 150K questions from this set and split it randomly into train, dev, test splits in 80 : 10 : 10 ratio. Table 12 shows a few examples of silver answer annotations for questions asked in MEDIASUM interviews.
Perturbations. First, we add random sentences between the question and answer utterances to prevent a location bias in which model predicts sentences that immediately follow the question as the answer. Second, we create scenarios where the question is unanswerable by removing annotated answer spans from the context. Third, we replace speaker names in the context with a numeric identifier because information about speaker names are not always available in the transcript including AMI dataset. For multi-span models, we further insert random sentences from elsewhere in the transcript in between annotated answers to facilitate better span selection. Finally, the number of speaker turns in MEDIASUM are 10x smaller than those in AMI dataset (refer to Table 2 of Zhu et al. (2021a)). Therefore, we create more speaker transitions by splitting a long speaker utterance into shorter utterances by multiple speakers.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 2, 3 and 4 verify the claims made.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2
B1. Did you cite the creators of artifacts you used? No response.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix B
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 8, Appendix B
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 8, Appendix A
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2, 8, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3-4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B, Sec 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B, Sec 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sec 4, Appendix B, C
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2, Appendix A
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A and supplementary data folder
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 2, Appendix A
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 2, Appendix A
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Ethical Review was conducted by crowd-sourcing company.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix A

heading: Future Work
text: We do not observe any significant increase in performance when using the larger checkpoints, thus leaving ample room for future work to bridge the gap between model and human performance on MEETINGQA