SYLLABUSQA: A Course Logistics Question Answering Dataset (2024)

Abstract: Automated teaching assistants and chatbots have significant potential to reduce the workload of human instructors, especially for logistics-related question answering, which is important to students yet repetitive for instructors. However, due to privacy concerns, there is a lack of publicly available datasets. We introduce SYLLABUSQA 1, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5, 078 open-ended course logisticsrelated question-answer pairs that are diverse in both question types and answer formats. Since many logistics-related questions contain critical information like the date of an exam, it is important to evaluate the factuality of answers. We benchmark several strong baselines on this task, from large language model prompting to retrieval-augmented generation. We introduce Fact-QA, an LLM-based (GPT-4) evaluation metric to evaluate the factuality of predicted answers. We find that despite performing close to humans on traditional metrics of textual similarity, there remains a significant gap between automated approaches and humans in terms of fact precision.

heading: 1 Introduction
text: In educational applications, artificial intelligence (AI) approaches have shown significant promise in improving learning outcomes (Aleven et al., 2016; VanLehn, 2011), by automatically providing feedback to students or engaging in tutoring dialogues with them. The key idea is to use AI to create an ondemand virtual teaching assistant to interact with many students simultaneously; see, e.g., Khamigo from Khan Academy (Academy, 2022). These approaches can scale up the effort of expert human teachers and tutors, and relieve them from doing repetitive tasks so that they can focus on providing personalized feedback or designing new learn-
1Dataset and code: https://github.com/ umass-ml4ed/SyllabusQA
ing content (Adamson and Rosé, 2012). In higher education, one promising avenue for AI-powered teaching assistants to reduce human effort is course logistics-related question answering (QA): answering student questions on logistics whose answers can be directly found or inferred from the syllabus.
There exist many approaches for automated QA in online courses (both logistics-related and content-related), using tools from rule-based AI systems (Raamadhurai et al., 2019; Feng et al., 2006) and expert systems with knowledge bases (Goel and Joyner, 2017; Goel et al., 2021) to end-to-
10344
end text generation (Zylich et al., 2020). Recently, large language model (LLM)-based approaches have shown great promise to improve the coverage and answer quality over traditional QA approaches (Hicke et al., 2023). See Section A in the Supplementary Material for a detailed discussion on related work. Unfortunately, these approaches are mostly developed and evaluated on proprietary data due to student privacy concerns, which prevents more researchers from contributing to the development of automated QA systems for education.
For evaluation, especially in logistics-related QA that often contains critical information, the factuality of predicted answers is more important than measuring surface textual features. Moreover, text similarity metrics may not be suitable for some open-ended natural language generation tasks (Amidei et al., 2018). As an example, the answer “The final exam will be on Dec 15”, has high surface-level textual similarity with the reference answer, “The final exam is on Dec 14”, but contains a critical factual error that may lead to significant negative consequences to students. Meanwhile, human instructors and teaching assistants often answer student questions in a concise way, without giving any unnecessary information. Therefore, it is important for LLM-based approaches to generate answers that are both concise and precise.
Contributions In this paper, we introduce the SYLLABUSQA dataset for course logistics-related QA. We publicly release this dataset and hope that it can be a benchmark for future work on developing and evaluating automated QA approaches for teaching assistance. Our contributions are: First, we collect a diverse set of 63 real course syllabi covering 36 majors across 12 universities, and employ crowd annotators to write 5, 078 logisticsrelated QA pairs with the goal of simulating what students would ask in a real-world course. Second, we detail the diverse composition of syl-
labi and QA pairs in SYLLABUSQA, in terms of syllabi domain, question types, answer sources, and different language styles. We lay out metrics to evaluate different aspects of open-ended automated QA approaches, in terms of both surface textual similarity and more importantly, the factuality of predicted answers grounded in the syllabus. Third, we conduct extensive experiments to benchmark the QA performance of several strong baselines on SYLLABUSQA. Overall, LLM-based approaches perform similar to humans on surfacelevel textual similarity metrics but worse on factuality metrics. We found that fine-tuning on real QA pairs from SYLLABUSQA performs much better than LLM prompting approaches and that retrievalaugmented generation is especially important.
To the best of our knowledge, SYLLABUSQA is the first publicly available real-world course logistics-related QA dataset. SYLLABUSQA assesses automated QA models on various natural language understanding aspects including handling challenging question types, from reasoning-based ones to adversarial ones, understanding of long input documents sourced from diverse course majors, processing complex input formatting including tables and schedules, and answering open-ended questions in a similar way as human instructors and TAs. Table 2 highlights the difference between SYLLABUSQA and existing datasets; see also Section 3.1 for a detailed discussion.

heading: 2 SYLLABUSQA Data Collection
text: We now detail how we construct SYLLABUSQA to address some of the limitations in existing datasets.

heading: 2.1 Source Syllabi
text: The source texts we include in this dataset are anonymized course syllabi, based on which QA pairs are generated. We collected 63 unique syllabi from instructors across 12 unique universities
worldwide, at both undergraduate and graduate levels. We include a wide range of diverse course subjects, from science and engineering to humanities and business, as shown in Figure 1. We also collected syllabi from courses with different formats: lab-based ones, project-based ones, lecturebased ones, etc. To ensure anonymity, we manually replaced personally identifiable information on instructors and teaching assistants with placeholders, along with any private information like platform login codes. On average, there are 8 pages per syllabus, with a minimum of 2 and a maximum of 27. We store all syllabi in their raw PDF format. Apart from the considerable variation in length, the diverse information formats in the syllabi, such as lists, tables, boxes, schedules, forms, and even diagrams and images, make QA on SYLLABUSQA especially challenging: doing so requires techniques ranging from parsing, information retrieval, to longdocument QA.

heading: 2.2 Design Considerations

heading: Categorization by Question Types
text: SYLLABUSQA is designed to include diverse question types across common logistics-related questions that human instructors and teaching assistants often face. There are 7 question types defined below, with real examples of each type:
• Yes/No: questions that have answers consisting of a single word yes/no answer, e.g., “Is there a separate lab section for this class?” • Single-factual: questions that have answers containing a single explicit fact from the syllabus, e.g., “When are office hours?” • Multi-factual questions that have answers combining multiple explicit facts from the syllabus, e.g., “What software and applications are used in this class?” • Single-hop reasoning: questions that have implicit answers requiring a single reasoning step, e.g., “I have not yet taken Advanced Biology, can I still take this course?”
• Multi-hop reasoning: questions that have implicit answers requiring multiple reasoning steps, e.g., “Can I start the class six weeks in and get an A?” • Summarization: questions that have answers that require summarizing information from multiple parts of the syllabus, e.g., “Could you tell me how class participation grades are broken down?” • Adversarial: questions that have no answers due to insufficient information, e.g., “Can I contact the instructor over Zoom?”
Categorization by Answer Source Questions in SYLLABUSQA can also be categorized by the source of their answers as defined below:
• Explicit: questions that have answers directly from the syllabus, including question types Yes/No, Single-factual, and Multi-factual. • Implicit: questions that have answers not explicitly in the syllabus and require inference, including question types Single-hop reasoning, Multi-hop reasoning, and Summarization. • Insufficient Information questions that do not have answers due to insufficient supporting information in the syllabus, including questions of type Adversarial.
The combination of multiple question types and answer sources makes SYLLABUSQA challenging for automated QA approaches. The involvement of adversarial questions, in particular, requires models to not hallucinate information not present in the syllabus, which is a critical test to pass before any potential deployment to real students.

heading: 2.3 Annotation Procedure
text: Annotation Guidelines For simplicity, we use a straightforward annotation task: given a syllabus, we ask an annotator to simulate logistics-related QA pairs seen in real course classrooms. They write 2 QA pairs for each of the 7 question types for a total of 14 QA pairs, to ensure QA pairs are
evenly distributed across question types. Before the task, we guide them through a tutorial to help them get familiar with the task and write diverse QA pairs. We give them a list of 14 diverse QA examples manually written by us, covering a wide spectrum of QA examples seen in classrooms, with 2 examples for each question type, on a single example syllabus. We encourage annotators to write open-ended questions with answers written in their own words, giving them the flexibility to come up with questions that are difficult to answer. See Section K in the Supplementary Material for the instructions we provide.
For explicit questions with answers directly found in the syllabus, including the Yes/No, SingleFactual, and Multi-Factual types, an annotator additionally provides one, one, and up to five answer spans respectively, i.e., snippets directly copied from the reference syllabus, supporting their answer. For Single-hop and Multi-hop reasoning questions, an annotator additionally provides one and up to five reasoning steps respectively, written in their own words to detail their thought process before arriving at the final answer. For Summarization questions, an annotator additionally provides up to five answer spans from the syllabus used to construct the summary. This additional information provided by annotators encourages their answers to be cohesive and faithful to the reference syllabus and can be further used to aid development of automated QA approaches. For Adversarial questions, annotators do not write answers and we use “No/insufficient information” as the answer.
Crowdworker Recruitment SYLLABUSQA is designed to include questions written in different styles. Therefore, we recruited a large pool of over 200 annotators across two popular platforms, Amazon Mechanical Turk (AMT) and Prolific, all with an undergraduate bachelor’s degree or above, located in the United States or Canada. To ensure diversity in language style, we asked each annotator to write a maximum of 14 QA pairs per syllabus, and a maximum of 112 total QA pairs across various syllabi. We collected this data over 5 months.
Quality Control To ensure high data quality, we asked each annotator on AMT to write 14 QA pairs on an assigned syllabus in a screening session. We only include data from those who pass the screening test in our dataset and invite them for additional annotation rounds. For annotators on Prolific, in addition to Prolific’s screening process, we manu-
ally checked a random subset of QA pairs written by each annotator and found that their data meets our quality standards. For further quality control, we also run heuristics-based checks to filter out unsuitable QA pairs. Since our source syllabi are anonymized, we discard questions asking for personally identifiable information about instructors and/or teaching assistants. We also post-process answers to Yes/No type questions to a single word since some contain additional explanations.
Agreement among Annotators To aid development of automated QA approaches, we explicitly aimed at encouraging language diversity across QA pairs in SYLLABUSQA by employing a large pool of annotators to reflect the diversity in real questions asked by students in classrooms. Therefore, inter-annotator agreement may be low, especially if evaluated on traditional text similarity measures. To formally measure this agreement, we had an independent expert annotator with extensive teaching experience as a college instructor, write oracle answers for around 20% of randomly sampled questions from the test set. We provide this expert with the same instructions and tutorial as the annotators and ask them to first select the question type before writing the answer accordingly.
Following related work on open-form QA datasets (Xu et al., 2022), we do not use traditional inter-annotator agreement metrics since we asked annotators to write QA pairs in their own words and style, which makes these metrics unsuitable (Amidei et al., 2018). Instead, we mainly evaluate agreement on factual information overlap, which measures the overlap in key facts contained in the annotator’s answer and an oracle answer. We use a GPT-4-based evaluation of precision and recall of factual information present between the answers written by the expert and annotators, which we detail later in Section 4.2. This metric results in an inter-annotator precision of 0.707 and recall of 0.664, indicating high overlap in factual information. As a reference, agreement measured in traditional surface textual similarity metrics is 0.419 in ROUGE-L F1 (Lin, 2004) and 0.684 in BERTScore (Zhang et al., 2020). This result is not surprising since we found that although annotatorwritten answers have varied surface language styles, they contain very similar key information.

heading: Qualitative Analysis of Ground-truth Answers
text: We analyze annotator-written ground-truth answers where an independent expert annotator manually
verified a random sample of QA pairs from 10% of the test set on: 1) precision and 2) recall. For precision, we test if the ground-truth answer is relevant to the question and supported by the syllabus. For recall, we test if the ground-truth answer contains all the relevant information required to answer the question. We found that 82% (77%) of groundtruth answers have perfect precision (recall), 11% (16%) of ground-truth answers have partial precision (recall), and 7% (7%) of ground-truth answers have poor precision (recall). Through a qualitative error analysis, we find the following error types leading to less-than-perfect precision/recall in ground-truth answers: 1) imperfect annotator recall/human error, 2) ambiguous syllabus information, and 3) ambiguous annotator-written answers. We provide details and examples in Section C in the Supplementary Material.

heading: 3 SYLLABUSQA Data Analysis
text: Statistics We randomly split the dataset into trainval-test splits with a proportion of roughly 60%- 20%-20% in terms of QA pairs. We ensure no overlap in syllabi across splits to test the generalization ability of QA approaches to answer questions based on unseen new syllabi. We show detailed statistics of SYLLABUSQA in Table 3. Overall, there are 5, 078 QA pairs, almost uniformly distributed across 7 question types. There are 2, 177 Explicit QA pairs, 2, 181 Implicit pairs, and 720 Insufficient Information pairs. We also show a breakdown by question type in Section B in the Supplementary Material.
Question Diversity We investigate question diversity in SYLLABUSQA, especially across questions based on the same reference syllabus. Questions with similar semantics across different syllabi are acceptable since their answers are grounded in different surrounding contexts in different locations. We compute the average intra-syllabus question pair similarity, then average across all syllabi, resulting in a ROUGE-L F1 score (Lin, 2004) of 0.126, which indicates low intra-syllabus question similarity. This observation validates our design to encourage annotators to write diverse QA pairs with multiple question types and answer sources.

heading: 3.1 Comparison to Existing Datasets
text: We compare SYLLABUSQA to existing datasets both in the education domain and for more generalpurpose QA in Table 2. To the best of our knowl-
edge, SYLLABUSQA is the first real publicly available course logistics-related QA dataset in educational settings; the data in (Labutov et al., 2018) is synthetically generated. ChaTA (Hicke et al., 2023) uses a proprietary dataset (with a publicly available sample) based on QA pairs on Piazza from an introductory computer science course. In contrast, SYLLABUSQA is public and more diverse by design, covering 36 majors. FairytaleQA (Xu et al., 2022) is a public dataset on reading comprehension QA from short fairytale story snippets of around 150 tokens. In contrast, SYLLABUSQA covers logistics-related QA with much longer context since the syllabi average 8 pages and 5K tokens long. CliCR (Šuster and Daelemans, 2018) tests QA on medical documents with automatically generated Cloze-type questions with document spans as answers. In contrast, SyllabusQA is crowdsourced with diverse types of open-ended QA pairs written simulating teaching assistants.
Compared to general-purpose QA datasets, NewsQA (Trischler et al., 2017) contains QA pairs grounded on news articles while QuAC (Choi et al., 2018) has QA on multi-turn dialogues, both with answers as text spans. In contrast, SyllabusQA contains QA pairs grounded on long course syllabi, which require techniques from long document QA. It also contains questions with open-ended answers. NarrativeQA (Kočiský et al., 2018) is sourced from movie scripts and books in plain text format. In contrast, SyllabusQA is sourced from course syllabi, containing different source text formats including tables and diagrams. HotpotQA (Yang et al., 2018) focuses on multi-hop reasoning questions. In contrast, SyllabusQA has seven different question types including summarization, multi-factual, and adversarial. OpenBookQA (Mihaylov et al., 2018) contains multiple-choice questions grounded on a set of facts. In contrast, SyllabusQA has different types of questions with open-ended answers.

heading: 4 QA Performance on SYLLABUSQA
text: We benchmark several strong baselines using stateof-the-art LLMs on SYLLABUSQA. For opensource LLMs, we use the popular LLaMA 2 (Touvron et al., 2023) model family, including the chat variants of LLaMA 2-7B, LLaMA 2-13B, and LLaMA 2-70B. Using open-source LLMs ensures the reproducibility of results and mitigates student privacy concerns in real-world deployment. For completeness, we also benchmark against GPT-
4 (OpenAI, 2023) enabled with the retrieval assistant, a highly capable but proprietary LLM. We explore different approaches in our experiments including zero-shot prompting and supervised finetuning (SFT), with and without help from retrievalaugmented generation (RAG) techniques. We show all prompts in Section J and model training details in Section F in the Supplementary Material.

heading: 4.1 Approaches
text: In the basic Zero-shot approach, we prompt pretrained LLaMA 2 models using only the question and system instructions in the prompt. We also experiment with Zero-shot with RAG, where we additionally include the top-5 relevant syllabi chunks retrieved in a process detailed below. As a sanity check, we include a Search Baseline where the answer to a question is simply the top-1 retrieved syllabus chunk, which reflects what a student would get using a simple keyword search (see Section E in the Supplementary Material).
In SFT, we fine-tune LLaMA-2-7B and LLaMA2-13B on the SYLLABUSQA training set. In SFT with RAG, we additionally include the top-5 relevant syllabi chunks retrieved from the reference syllabus and construct the input prompt similar to zero-shot with RAG for a fair comparison.
In Retrieval-augmented Generation (RAG), we retrieve chunks in the course syllabi that are relevant to the question and include them in the input to LLMs. Although a shorter syllabus can be completely included in the prompt, we chose to retrieve only relevant chunks from the syllabus to allow for generalization to longer syllabi. Moreover, using long input contexts packed with irrelevant information can decrease model performance (Liu et al., 2023). In our RAG pipeline, we parse syllabi from raw PDFs to text files using Adobe Acrobat’s PDFto-text parser, which accurately parses varied text formats including tables, schedules, and lists. To accommodate the limited context length of LLMs, we chunk each syllabus to a maximum size of 1000
characters per chunk and an overlap of 200 characters between adjacent chunks to preserve context. We use BM25 (Robertson et al., 2009), a popular and effective ranking function, to retrieve the top-5 syllabus chunks to include in the input.
We experiment with Chain-of-Thought prompting (Wei et al., 2022), leveraging the rich meta information of SYLLABUSQA. Specifically, we use an SFT with RAG approach on LLaMA-2-13B to first predict the question type, then generate reasoning steps written by the annotator for Singlehop and Multi-hop reasoning type questions only, before finally predicting the answer.
We also benchmark two additional approaches to provide more comparisons to the approaches above. The first is GPT-4 with Retrieval Assistant in a zero-shot prompting setting: we use the gpt-4-1106-preview model with the retrieval assistant from external files enabled, where we upload the raw PDF syllabus to GPT-4 and then prompt it to answer a question, along with system instructions explaining the task. Our prompt is shown in Section J in the Supplementary Material. This approach provides us with an upper bound on performance of the zero-shot approach. The second is Human Performance, an estimated upper bound of QA performance on SYLLABUSQA for any approach. We simply calculate the agreement between answers written by the human expert and annotators on about 20% of the test set. This measure underestimates human performance, though, since our annotators are not instructors who are familiar with the syllabi, making them susceptible to imperfect recall, especially on longer syllabi.

heading: 4.2 Metrics
text: We now detail the evaluation metrics we use for the QA task on SYLLABUSQA. We report surface textual similarity using traditional metrics like ROUGE-L F1 (Lin, 2004) and recent ones like BERTScore F1 (Zhang et al., 2020). Since these metrics are more aligned with language style rather
than the factuality of an answer, we also use several metrics to measure the factuality of answers, which are more robust to surface textual features in answers than standard metrics.
Factuality Metrics We design a novel LLMbased (GPT-4) evaluation metric which we call Fact-QA, inspired by FActScore (Min et al., 2023), to evaluate the factuality of predicted answers. FActScore evaluates information precision by breaking the generated text into a series of atomic facts and then computing the percentage of atomic facts supported by a reference knowledge source, such as Wikipedia. The key modification we make in our adaptation is to use the annotatorwritten ground truth answer as the reference source. We then swap the predicted and ground truth answers to compute information recall.
To compute precision, we compute the proportion of facts in the predicted answer supported by the reference answer, denoted as Fact-QA Precision. To compute recall, we compute the proportion of facts in the reference answer supported by the predicted answer, denoted as Fact-QA Recall. Intuitively, if a predicted answer has high precision, it is less likely to contain incorrect or irrelevant facts, signifying limited LLM hallucination. If a predicted answer has high recall, it is more likely to have covered all relevant facts needed to answer the question. Both aspects are important since a desirable answer should cover all necessary facts but nothing irrelevant, which may distract students when they read the answer. We also aggregate these metrics and compute a Fact-QA F1 score. Please see Section D in the Supplementary Material for implementation details and our prompts.
We note that ideally, we should use the syllabus as the knowledge source to evaluate the factuality of an answer by providing it to GPT-4 for retrieval. However, due to the high cost associated with the OpenAI API, we use facts from the reference answer instead, which we found to be a high-quality summary of facts in the syllabus that are relevant to a question. Specifically, we examined 10% of questions from the test set, where we compared the Fact-QA Precision and Recall scores obtained by providing the entire syllabus to GPT-4 vs. providing only the annotator-written answer. Results show a high correlation, with Pearson correlation coefficients of 0.8223 and 0.7369, respectively, which suggest that when computing the Fact-QA metric, using facts from the much shorter reference answer
is a good proxy for using the full syllabus.
Qualitative Analysis of Fact-QA We (the authors) investigate the effectiveness of our Fact-QA metric by performing a qualitative analysis to judge whether GPT-4’s output is accurate on 5% of the test set. We find that GPT-4’s outputs are correct, partly flawed, and poor about 78%, 15%, and 7% of the times, respectively. By examining GPT-4’s errors in extracting facts and verifying claims, we find four major error types: 1) failures in logical reasoning, 2) overly granular atomic fact extraction, 3) arithmetic errors, and 4) incorrectly handling unanswerable questions (see Section D.2 in the Supplementary Material).
We (the authors) perform a human evaluation to examine the factual similarity between the expert annotator-written answers and the annotatorwritten ground-truth answers, on 5% of the test set. We find moderate to high Pearson correlation coefficients of 0.7785, 0.6602, and 0.7735 with FactQA Precision, Fact-QA Recall, and Fact-QA F1, respectively, indicating that Fact-QA is an effective metric (see Section D.3 in the Supplementary Material).

heading: 5 Results, Analysis, and Discussion
text: We report the performance on both surface textual similarity and factuality metrics for all approaches in Table 4, averaged over questions on the test set of SYLLABUSQA. We also provide stratified performance by question type in Table 5. We show qualitative example outputs across question types in Section H and an error analyses of failed cases in Section I in the Supplementary Material.
Models are good at capturing surface textual similarity The best-performing approach is LLaMA-2-13B with SFT combined with RAG and Chain-of-Thought (LLaMA-213B+SFT+RAG+CoT). This approach performs similarly to humans on surface-level textual similarity metrics. This observation suggests that training automated teaching assistant chatbots has some promise towards reducing human workload for course logistics-related QA. However, zero-shot approaches perform much worse than SFT on the same underlying LLM, which suggests that training on actual QA pairs is necessary to adapt to the diverse language styles among real instructors. We note that human performance is underestimated since we asked annotators to use
Model Factuality Surface Textual Similarity
Fact-QA F1 ↑ Fact-QA Precision ↑ Fact-QA Recall ↑ ROUGEL F1 ↑ BERTScore F1 ↑
diverse styles in their answers, which exaggerates disagreement (Amidei et al., 2018).
Models fall behind humans on factuality We note that the performance of the best approach, LLaMA-2-13B+SFT+RAG+CoT, falls behind human performance on the factuality metrics and is especially lower on precision (by 10%), which indicates that LLMs are prone to hallucination. As one example, the best approach predicts “Grading breakdown is exams (30%), attendance (20%), case analysis (25%), and group project (25%)” while the reference answer is “Grading breakdown is attendance (10%), discussion (10%), assignments (20%), exams (40%), and group project (20%)”. The predicted answer scores highly on surface textual similarity but is inaccurate. See additional qualitative examples of LLM hallucination in Section I in the Supplementary Material. We also found that LLMs perform worse on questions with implicit answers, especially reasoning-based ones, compared to questions with explicit answers (see Table 5). These observations suggest that there is room for improvement before these approaches can be deployed in real-world educational settings, since course logistics questions can often be highstakes especially when involving exams and grading, requiring answers to be factually accurate. Nonetheless, we believe that SYLLABUSQA, our
FACT-QA metric, and these results serve as a useful benchmark for the research community moving forward.
Retrieval boosts performance We see that RAG provides a significant boost in LLM performance. In the zero-shot setting, LLaMA-2-70B combined with RAG shows an increase of 3.5% on BERTScore F1 and 19% on Fact-QA F1 than its non-RAG counterpart. Similarly, in the SFT setting, LLaMA-2-13B combined with RAG shows an increase of 8% on BERTScore F1 and 30% on Fact-QA F1 than its non-RAG counterpart. This improvement may even be more significant when using other retrieval methods such as dense passage retrieval (Karpukhin et al., 2020), with the option of training the retriever using meta information of answer spans associated with explicit type questions, which we leave for future work.
SYLLABUSQA is challenging even for GPT-4 For the latest model from the GPT-4 family (gpt4-1106-preview as of Dec 1, 2023) enabled with retrieval assistant where we upload the syllabus as a PDF file, we see that performance on textual similarity metrics is generally lower than that of SFT approaches which have been fine-tuned to adapt to the style of answers. GPT-4 performs similarly to humans on harder questions with implicit answers, especially reasoning-type ones (see Table 5). On
Fact-QA Recall, GPT-4 surpasses human performance, which suggests that the retrieval assistant is highly effective. This result also indicates that human annotators may not retrieve relevant information perfectly, especially for long syllabi (8 pages on average).
However, recall is only one side of the story. Compared to human performance on precision, GPT-4 with retrieval assistant underperforms, with a gap of 25% in Fact-QA Precision and 12% in Fact-QA F1 (possibly overestimated due to Yes/No type questions). This result indicates that while GPT-4 can retrieve relevant facts from the syllabus, its predicted answer is not always fully supported by the reference answer. Moreover, GPT-4 has a low Fact-QA F1 score on adversarial questions with insufficient supporting information in the syllabus (see Table 5). We show examples of GPT-4 hallucination in Section I in the Supplementary Material. This factuality gap shows SYLLABUSQA is challenging even for state-of-the-art LLMs.
Chain-of-Thought improves answer factuality Chain-of-Thought with SFT and RAG leverages question type information, leading to improved performance on factuality metrics. Question type prediction (among 7 types) has an accuracy of 54%, much higher than random chance. If the question type is Single-hop or Multi-hop reasoning, the model then generates reasoning steps before predicting the answer. Investigating performance across question types (see Table 5), we see a significant increase in Fact-QA F1 performance on multi-hop reasoning questions. However, we see a small drop in performance on single-hop reasoning questions, possibly because these simpler questions
do not benefit from intermediate reasoning steps.

heading: 6 Future Work and Conclusions
text: In this work, we introduced SYLLABUSQA, a diverse dataset consisting of real-world course syllabi and corresponding course logistics-related QA pairs. We also benchmarked the performance of several strong, LLM-based automated QA baselines on this dataset. Results show that finetuning and retrieval-augmented generation are helpful but there remains a significant gap between the performance of LLM-based approaches and that of humans on answer factuality. We make the SYLLABUSQA dataset, our code, and evaluation metrics publicly available to facilitate future work on automated teaching assistant development. There are plenty of avenues for future work. First, we can leverage question meta information to improve the performance of LLM-based QA approaches, and use overgenerate-and-rank to improve their robustness. Second, we can collect human labels on answer factuality and further improve QA approaches using methods such as direct policy optimization (Tian et al., 2024).

heading: Limitations
text: We run extensive automated and manual checks to control for artifacts in SYLLABUSQA. However, given the large-scale crowd-sourced annotation process, there could be artifacts present. Although we manually check a subset of SYLLABUSQA for acceptable data quality, we leave a full evaluation for future work. Further, due to the imperfect recall of human annotators on the reference syllabus, we found that the ground truth an-
swers might not always be comprehensive or factually correct. SYLLABUSQA contains syllabi from courses taught in English. An important future direction is extending SYLLABUSQA to courses taught in other languages, making the benefits of automated teaching assistants accessible to students across different communities.

heading: Ethical Considerations
text: While we took several steps to ensure diversity in our dataset, we cannot guarantee diversity across student cultures, genders, or other demographics because we did not collect this information about annotators. As such, there is a risk of bias in our data, and we welcome future studies that investigate such risks in SYLLABUSQA. Our dataset is intended to be used for research purposes, and not to train systems deployed in classrooms without prior extensive risk mitigation. For real-world deployment of automated teaching assistants, the privacy of student data is critical, highlighting a drawback of using closed-source models like GPT-4. We hope that SYLLABUSQA serves as an important first step in introducing a dataset and evaluation benchmark for the development of open-source automated teaching assistant methods.

heading: Acknowledgements
text: The authors would like to thank all course instructors who agreed to share their syllabus and all SYLLABUSQA annotators. We thank the Learning Agency Lab for funding the data collection process, and Perpetual Baffour, Alex Franklin, Siyuan (Mei Mei) Li, Natalie Rambis, Jules King, Ulrich Boser, Hasnain Heickal, Jaewook Lee, Nischal Ashok Kumar, and Wanyong Feng for helpful discussions. We also thank the anonymous reviewers for their helpful comments. The authors are partially supported by the NSF under grant IIS-2202506.

heading: A Related Work
text: In online courses, especially discussion forums, many prior works have attempted automated QA. A well-known example is Jill Watson (Goel and Joyner, 2017) and its later variants (Goel et al., 2021). Jill Watson takes an expert systems approach using a knowledge base of human-authored QAs; when a question comes in, it searches through the knowledge base and determines whether an answer can be retrieved or generated. Other relevant works include the Curio SmartChat system (Raamadhurai et al., 2019) and the DiscussionBot system (Feng et al., 2006). These works focus on retrieving relevant information and then use it in pre-defined interaction mechanisms with students. More recently, several works have approached this problem via automated, end-to-end QA systems. Zylich et al. (2020) used TF-IDF (Salton and Buckley, 1988) to retrieve relevant documents and trained a recurrent neural network for end-to-end QA. While they also used a classifier to determine whether a question is answerable, they did not validate it on actual questions that are not answerable. In contrast, SYLLABUSQA contains such adversarial questions by design. The authors of a concurrent work (Hicke et al., 2023) employed LLMs with document retrieval to improve QA performance. While they also perform a human evaluation of the usefulness and accuracy of answers on a proprietary dataset, their focus is more on contentrelated QA rather than logistics-related QA.

heading: B Dataset
text: Dataset Collection We do not collect any personally identifiable information from crowd workers. We paid 18-20 US dollars per HIT which on average took between 60-90 minutes to complete, which is above the minimum wage in the United States. We collected explicit consent from each annotator before starting the task, and we informed them that their data would be used to train AI models for educational research. Our data collection process in this work is IRB-approved.
We take several steps to ensure that SYLLABUSQA is properly anonymized. We ensure that all names and identifying information, such as phone numbers, are removed from syllabus documents and file names. Additionally, we do not include the account IDs of crowd annotators in the dataset to ensure that answers are not linked to annotators. We additionally screen for
offensive content in the data by manually checking a random subset of QA pairs written by each annotator and running automated heuristics like the profanity-check Python library.
Stratified Statistics by Question Type We show a breakdown of QA pairs in SYLLABUSQA by question type in Table 6.
Comparison of Full Test Set vs 20% Test Set As described in Sec 2.3, 20% of questions were randomly sampled from the test set to compute agreement which was also used as an estimate of human performance. We also report GPT-4 with Retrieval Assistant performance on this same 20% test set due to budgetary constraints. To investigate whether this 20% test set is an accurate representation of the full test set, we first report statistics on the distribution of question types comparing the full test set to the 20% test set in Table 7. We observe that both sets have a sufficient number of questions from each question type.
We additionally report model performance on the 20% test set as well as the full test set in Table 13. We find similar model performance trends on both sets implying that the 20% test set allows for a fair comparison.
License We are releasing our dataset under the CC BY-NC-SA (Attribution-NonCommercialShareAlike) License, and clarify that the intended use is for research purposes.

heading: C Analysis of Ground-truth Answers

heading: C.1 Factuality Analysis of Ground-truth Answers
text: We perform a factuality analysis of annotatorwritten ground-truth answers on two aspects: 1) precision and 2) recall. For precision, we test whether the information contained in the groundtruth answer is relevant to the question and supported by the syllabus. For recall, we test whether the major relevant information required to answer the question is contained in the ground-truth answer. An independent expert annotator manually verified a random sample of 10% of the test set. For each QA pair, the expert annotator read the question, read the corresponding syllabus, and rated the ground-truth answer on precision and recall. Precision and recall were both individually rated using a three-point scale indicating poor, partial, and perfect scores.
Through this manual analysis performed by an independent expert annotator, we found that 82% (77%) of ground-truth answers have perfect precision (recall), 11% (16%) of ground-truth answers have partial precision (recall), and 7% (7%) of ground-truth answers have poor precision (recall).
This error analysis highlights that human performance is underestimated. Annotators are susceptible to imperfect recall, especially on longer syllabi, reducing human performance. Ideally, human performance should be measured using the same professors who have written the course syllabi, thereby having close to perfect recall of its contents, leading to an improvement in human performance.
We performed a qualitative error analysis and found the following error types leading to less-thanperfect precision/recall in ground-truth answers: 1) imperfect annotator recall/human error, 2) ambiguous syllabus, and 3) ambiguous answers. We provide examples for each of the error types in Table 8.

heading: C.2 Correlation of Ground-truth Answers with Syllabus Text
text: For an illustration of how ground-truth answers to implicit vs explicit question types differ, we provide an example QA pair for multi-hop reasoning (implicit) and multi-factual (explicit) type questions in Table 9. We observe that implicit question types have answers that can be very different from the syllabus text spans often requiring reasoning and summarization, while for explicit question types, the answers are more correlated to the text spans of the syllabus although written in an openended natural style.

heading: D FACT-QA Metric
text: D.1 Implementation Details and Prompts
For the Fact-QA metric, we use GPT-4, specifically gpt-4-1106-preview, as the underlying LLM for Fact-QA. In our prompts, we do not reveal to GPT4 the roles of the answers and instead refer to them as “answer 1” and “answer 2”. Given a question and two answers, we have GPT-4 extract all atomic facts from answer 1 and compute how many are supported by answer 2. We prompt GPT-4 in this way twice for each predicted answer, setting answer 1 to the predicted answer and answer 2 to the reference answer for Fact-QA Precision and vice versa for Fact-QA Recall. We provide the prompt and example output for Fact-QA in Table 17.

heading: D.2 Qualitative Analysis of Fact-QA Metric
text: We investigate the effectiveness of our Fact-QA metric by performing a qualitative analysis of GPT4’s outputs when evaluating the expert annotatorwritten answers compared against the ground-truth answers on 5% of the test set. We find that for around 78% of evaluations, GPT-4 executes the task correctly and assigns scores that are a good representation of the factual similarity between the answers. For around 15% of evaluations, GPT-4 had minor issues with Fact-QA being a satisfactory representation of the factual quality of the answer. For the remaining 7% of evaluations, either GPT-4
had major issues or the metric is not a good reflection of answer quality.
Among the strengths of using Fact-QA, GPT-4 is effective at handling answers with varied styles, is capable of extracting atomic facts from the first answer which can be suitably compared to the second answer, and exhibits excellent (implicit) reasoning to determine which facts are supported by the second answer. Additionally, we find that hallucinations are surprisingly rare, possibly due to the straightforward nature of the Fact-QA task of comparing two answers.
For a qualitative error analysis, we additionally examine the subset of partly flawed (15%) and poor (7%) cases where GPT-4 makes errors in either extracting atomic facts from the first answer or verifying claims stated in these facts by the second answer. First, we observe that GPT-4 occasionally fails with logical reasoning. For example, GPT4 determined that a fact claiming a grade would be a zero was supported by an answer claiming a grade would be hurt, when only the converse is true. Second, we observe that GPT-4 at times tends to extract multiple facts by breaking a single sentence into 3 or more facts. This granular extraction can cause scores to be overly low when some information is not supported by the second answer. Third, we observe that GPT-4 often fails to understand the meaning of “No/Insufficient Information”, inferring this to be the same as a direct “No” answer to a question. Fourth, we observe that GPT-4 at times struggles with arithmetic to determine if facts are supported. For example, GPT-4 fails to identify that 87% and 26/30 are roughly equal. We note that many of these issues could be fixed with further prompt engineering, which we leave for future work.
To conclude our analysis of the Fact-QA Metric, we present several observations that could lead to improvements in the metric for future work. First, we note that a fact unsupported by the second answer is treated equally to a fact directly contradicted by the second answer. An additional penalty for contradicting information would be helpful. Second, we note that for some answers not relevant to the question, GPT-4 still attempts to interpret them. Automatically assigning a score of 0 to irrelevant answers as an initial screening might help.

heading: D.3 Correlation of Fact-QA Metric with Human Evaluation
text: We (the authors) additionally perform a human evaluation to examine the factual similarity between the expert annotator-written answers and the annotatorwritten ground-truth answers, on 5% of the test set. We set up our human evaluation imitating the FactQA evaluation process. For the manual analysis, we use only the question, the predicted answer, and the ground-truth answer, matching Fact-QA. For precision, we assess if the factual information contained in the predicted answer is supported by the ground-truth answer. For recall, we assess if the factual information contained in the ground-truth answer is supported by the predicted answer. We assign a human evaluation precision/recall score of 0, 0.5, and 1 for poor, partial, and complete precision/recall, respectively. We find moderate to high Pearson correlation coefficients of 0.7785, 0.6602, and 0.7735 with Fact-QA Precision, Fact-QA Recall, and Fact-QA F1, respectively, indicating that Fact-QA is an effective metric.

heading: D.4 Average Number of Atomic Facts in Ground-truth Answers
text: To illustrate the role of atomic facts in the FactQA metric, we report statistics on the number of atomic facts per ground-truth answer in the test set, stratified by question type in Table 10.

heading: D.5 Simple Example Illustrating the Fact-QA Metric
text: We provide a simple example to illustrate our FactQA metric. In the example shown in Table 11, Answer 1 is the ground-truth answer, and Answer 2 is the predicted answer. We get a Fact-QA Recall score of 1/2. To find Fact-QA Precision, we swap Answer 1 and Answer 2 to get a Fact-QA Precision
score of 1.

heading: E Search Baseline as a Sanity Check
text: We implement a simple rule-based keyword search baseline as a useful sanity check for model performance on SyllabusQA. The search baseline imitates a student searching for an answer within the syllabus using keywords from the question. We chunk syllabus text documents into chunks of 200 characters with an overlap of 50 characters between adjacent chunks to preserve context. We chose a chunk size of 200 characters since this roughly corresponds to the mean number of tokens in an answer in the train set of SyllabusQA (30.3 tokens/answer), and the answer length provides a balance between recall and precision. We retrieve the top-1 chunk using BM-25, a strong and effective retrieval function simulating a manual keyword search performed by a student. We simply output this retrieved top-1 chunk without any modification as the predicted answer.
The search baseline is competitive with Zeroshot LLaMA with RAG on textual similarity and Fact-QA Precision as seen in Table 4. Since Zeroshot LLaMA is not fine-tuned on SyllabusQA, the answers generated are sometimes in a different chatbot style, which reduces textual similarity scores. In contrast, the search baseline outputs an unfiltered text snippet from the syllabus. On FactQA precision, the choice of using only the top-1 retrieved snippet ensures higher precision but reduces recall since Zero-shot LLaMA with RAG synthesizes information from top-5 retrieved snippets.
While serving as a useful sanity check, the search baseline has major drawbacks compared to our models including 1) not being able to answer questions in a conversational style, 2) not being able to reason about facts spread across retrieved snippets to answer complex questions, 3) not being able to summarize multiple facts spread across the syllabus in a concise answer, and 4) not being able to identify when an answer is unanswerable from the retrieved snippets.
We illustrate through qualitative examples in Table 12 questions, especially of type implicit, on which the Search Baseline performs poorly including multi-reasoning and summarization-type questions. We show the top-1 syllabus snippet searched/retrieved by BM-25 as the predicted answer.

heading: F Experimental Setup
text: For SFT with LLaMA models, we use the AdamW optimizer with a batch size of 8, a learning rate of 1e-4, and a cosine learning rate scheduler. We warm up for 3% of training steps and use gradient clipping for training stability. We use the Parameter Efficient Fine-Tuning (PEFT) library from HuggingFace (Wolf et al., 2020) to load LLaMA models and train via low-rank adaptation (LoRA) (Hu et al., 2022) (LoRA α = 32,LoRA r = 16,LoRA dropout = 0.05). We use full precision for LLaMA-7B and LLaMA13B models and 8-bit precision for inference on LLaMA-70B. We fine-tune for 3 epochs with early stopping on the validation set on a single A100 80GB GPU, with each epoch taking up to 4 hours. For RAG, we use the Okapi BM25 (Robertson et al., 2009) ranking function to retrieve the top5 syllabus chunks with default parameters (k1 = 1.5, b = 0.75). For generation, we use nucleus sampling with top-k (p = 0.95, k = 50). Wherever possible, we use standard hyperparameters and do not do extensive parameter tuning since we aim to benchmark the performance of various baselines on SYLLABUSQA. Our goal is not to develop the best QA model in this work. Due to high computational and Open AI API cost, we report performance on one run of model baselines. For ROUGE we use the rouge-score library with Porter stemmer enabled, and for BERTScore we use the bert-score library with microsoft/deberta-xlarge-mnli as the underlying model. We additionally note that we used GitHub Copilot minimally in the writing of our code. All software we use in the development of this work is open source. We are consistent with the terms and intended use of all software and with the OpenAI API.

heading: G Additional Results
text: G.1 Model Performance on Full Test Set vs 20% Test Set
We reported GPT-4 with Retrieval Assistant and human performance on the 20% test set in our main results in Table 4. To investigate whether this is fair, we compare the performance of all models on the full test vs 20% test set in Table 13 using BERTScore F1 as the evaluation metric. We find similar model performance trends on both sets implying that the 20% test set allows for a fair comparison.

heading: H Qualitative Analysis
text: We show example qualitative outputs from the bestperforming models for questions from each type in Table 14.

heading: I Qualitative Error Analysis
text: We show examples of factually incorrect answers indicating model hallucination, or answers with additional irrelevant information, from the bestperforming model, LLaMA-2-13B + SFT + RAG + CoT in Table 15, and from GPT-4 with Retrieval Assistant in Table 16.

heading: J Prompts
text: We list all prompts for reproducibility.

heading: J.1 Prompts for LLaMA Models
text: For models from the LLaMA-2 family, we match their pre-training prompt style and use the following format: 

 
 You are a teaching assistant. Answer questions from students on course logistics. 
 {RAG context text for RAG baselines} ### The question is: {question text} ### The answer is: 

The RAG context text for RAG baselines is constructed as follows:
Here are snippets from the course syllabus which could be helpful in answering the question: ### Snippet 1: {snippet 1 text} ### Snippet 2: {snippet 2 text} ### Snippet 3: {snippet 3 text} ### Snippet 4: {snippet 4 text} ### Snippet 5: {snippet 5 text} The relevant snippets from the course syllabus were added above.

heading: J.2 Prompts for LLaMA Models with Chain-of-Thought Prompting
text: We combine Chain-of-Thought prompting using the following prompt style:


 
 You are a teaching assistant. Answer questions from students on course logistics. First, choose the question type from one of these 7 options: 1) yes/no, 2) single factual, 3) multi factual, 4) single reasoning, 5) multi reasoning, 6) summarization, and 7) no answer. For questions of type single reasoning, provide a single reasoning step. For questions of type multi reasoning, provide up to 5 reasoning steps. For all other question types, don’t provide any reasoning steps. Then, write the answer to the question. 
 {RAG context text for RAG baselines} ### The question is: {question text} ### The question type is: 


heading: J.3 Prompts for GPT-4 with Retrieval Assistant
text: For GPT-4 with Retrieval Assistant, we use the following system instructions: You are a teaching assistant. Answer questions from students on course logistics using the attached course syllabus document in your knowledge base. If an answer is not contained in the course syllabus output ‘No/insufficient information’.
Given the system instructions above, we use the following prompt for GPT-4 with Retrieval Assistant: The question is: {question text} The answer is:

heading: K Dataset Collection: Annotator Instructions
text: We provide the instructions given to annotators in Table 18. We provide the tutorial given to annotators containing 14 diverse QA pairs based on an example syllabus from a course in Computer Science in Table 19.

heading: L Misc: Attribution
text: Icons used in Figure 1 are from www.flaticon. com.

heading: Future Work
text: We note that many of these issues could be fixed with further prompt engineering, which we leave for future work. To conclude our analysis of the Fact-QA Metric, we present several observations that could lead to improvements in the metric for future work. First, we note that a fact unsupported by the second answer is treated equally to a fact directly contradicted by the second answer. An additional penalty for contradicting information would be helpful. Second, we note that for some answers not relevant to the question, GPT-4 still attempts to interpret them. Automatically assigning a score of 0 to irrelevant answers as an initial screening might help