SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts
Year: 2023

Abstract:
Lectures are a learning experience for both students and teachers. Students learn from teachers about the subject material, while teachers learn from students about how to refine their instruction. However, online student feedback is unstructured and abundant, making it challenging for teachers to learn and improve. We take a step towards tackling this challenge. First, we contribute a dataset for studying this problem: SIGHT is a large dataset of 288 math lecture transcripts and 15,784 comments collected from the Massachusetts Institute of Technology OpenCourseWare (MIT OCW) YouTube channel. Second, we develop a rubric for categorizing feedback types using qualitative analysis. Qualitative analysis methods are powerful in uncovering domain-specific insights, however they are costly to apply to large data sources. To overcome this challenge, we propose a set of best practices for using large language models (LLMs) to cheaply classify the comments at scale. We observe a striking correlation between the model’s and humans’ annotation: Categories with consistent human annotations (>0.9 inter-rater reliability, IRR) also display higher human-model agreement (>0.7), while categories with less consistent human annotations (0.7-0.8 IRR) correspondingly demonstrate lower human-model agreement (0.3-0.5). These techniques uncover useful student feedback from thousands of comments, costing around $0.002 per comment. We conclude by discussing exciting future directions on using online student feedback and improving automated annotation techniques for qualitative research.* *Equal contributions. SIGHT is intended for research purposes only to promote better understanding of effective pedagogy and student feedback. We follow MIT’s Creative Commons License. The dataset should not be used for commercial purposes. We include an elaborate discussion about limitations of our dataset in Section 7 and about the ethical use of the data in the Ethics Statement Section. The code and data are open-sourced here: https://github.com/rosewang2008/sight.

heading: 1 Introduction
text: Lectures are a learning experience for both students and teachers. Students learn from teachers about the subject material. Teachers also learn from students about how to improve their instruction (for Teaching Project, 2011; Pianta et al., 2008; Evans and Guymon, 1978; Hativa, 1998). However, in the online education setting, student feedback is both abundant and unstructured. This makes it challenging for teachers with online content to synthesize and learn from available feedback.
To take a step towards tackling this challenge, we contribute SIGHT (Student Insights Gathered from Higher Education Transcripts), a large dataset of 288 math lecture transcripts and 15,784 comments collected from the Massachusetts Institute of Technology OpenCourseWare (MIT OCW) YouTube channel. MIT OCW is a popular YouTube channel that offers a collection of lecture content from real MIT courses. Their courses gather up to thousands of student comments (OCW, 2020, 2023; Breslow et al., 2013), in which users express a range of feedback from excitement about the pedagogy to confusion about the course content. The dataset is a rich source of data for studying the relationship between teaching content and student commentary.
Second, we develop a rubric for categorizing different kinds of student feedback in the YouTube comments using a qualitative analysis approach. Qualitative analysis involves iteratively examining the data and accounting for the context (Corbin et al., 1990; Erickson et al., 1985; Bauer and Gaskell, 2000). For example, we examine the student comments for useful feedback categories while accounting for the online context of the instruction. Our rubric includes 9 categories of student YouTube comments, spanning from general feedback useful for encouraging instructors (e.g., “Amazing lectures!”) to specific comments on the pedagogy or technical content.
315
While qualitative analysis methods are effective in uncovering domain-specific insights, applying these methods to large sources of data is challenging (Erickson et al., 1985; Corbin and Strauss, 1990; Bauer and Gaskell, 2000; O’Connor and Joffe, 2020). Scaling annotation effectively is crucial for sifting through large amounts of unstructured data (e.g., the 15,784 comments in SIGHT) and uncovering relevant student feedback. However, qualitative methodologies often require domain-expertise. This limits the pool of analysts, which makes it expensive to find this expertise or means that only a small sample of the data can be analyzed (Harrison et al., 2019; Lee et al., 2017). Additionally, the qualitative analysis process is time-consuming because it allows for the annotation rubric to be adapted and accommodate new categories. This means that data has to be re-annotated frequently and must be analyzed flexibly. Therefore, our third contribution is proposing a set of best practices for using pretrained large language models (LLMs)—specifically ChatGPT (OpenAI, 2023)—to cheaply, quickly and flexibly annotate data at scale. We explore different prompting approaches (e.g., zero-shot, k-shot, and reasoning).
We analyze the quality of the model annotation and the diversity of user feedback. Categories with consistent human annotations (>0.9 inter-rater reliability, IRR) also display higher human-model agreement (>0.7), while categories with less consistent human annotations (0.7-0.8) correspondingly demonstrate lower human-model agreement (0.3- 0.5). Albeit imperfect, annotating with ChatGPT allows researchers to explore their entire dataset in a fast, cost-effective way. For example, we are able to sift through 15,784 comments and identify those related to student confusion and lecture pedagogy in a few hours, all under $0.002 per comment.
These comments can be invaluable for instructors looking to improve their lecture content.
In summary, we make the following contributions in this paper:
1. We create SIGHT, a dataset of 288 lecture transcripts from MIT OpenCourseWare (OCW) mathematics courses and of 15,784 annotated user comments.
2. We develop an annotation rubric of feedback types found in YouTube comments using a qualitative analysis approach.
3. We release a set of best practices for using LLMs with qualitative coding rubrics for scaling annotation.
4. We analyze the quality of the annotation and the diverse types of student feedback uncovered via our automated annotation procedure.

heading: 2 Related Work

heading: 2.1 YouTube as an Educational Platform
text: YouTube is an online platform with a vast collection of educational videos, such as from MIT OCW (OCW, 2023). Due to its popularity and large volume of high-quality education content, YouTube is an important platform for providing educational resources. For example, prior work in education have studied how YouTube provides educational video content to fields like medicine (Curran et al., 2020), support multi-modal learning through video and lecture slides (Lee et al., 2022), or allows for informative discussions in the comment section (Dubovi and Tabak, 2020; Lee et al., 2017). Prior works have not yet studied how YouTube comments can serve as feedback for instructors who host online content.
316

heading: 2.2 Student Feedback
text: Course evaluations by students are the cornerstone for providing feedback to instructors at higher educational institutions (Hammonds et al., 2017; Marsh and Roche, 1997). However, internal course evaluations receive limited student responses, are administered infrequently, and suffer from recency bias (e.g., surveys are typically administered after final examinations) (Cohen, 1981; Greenwald and Gillmore, 1997; Kim and Piech, 2023). Integrating traditional evaluations with informal evaluations from online platforms, like MOOCs and YouTube, can expand the sources of feedback.
Another challenge with course evaluations is that they are unstructured: Student evaluations can encompass a wide range of topics, from technical issues to personal opinions. This is an even more prominent issue for YouTube where videos receive a lot of spam comments. While unsupervised natural language processing (NLP) methods, such as topic modeling, have been applied to survey data to extract themes (Hujala et al., 2020), they struggle to identify specific information. Alternatively, classifiers can be trained for specific domains, e.g., sentiment classifiers for measuring class mood (Hynninen et al., 2019; Baddam et al., 2019; Gottipati et al., 2018; Alhija and Fresko, 2009; Azab et al., 2016), however they are timeconsuming to train, especially if the rubrics are modified over the course of analysis. Finally, although qualitative analysis offers powerful insights, it is typically limited to small data samples and is challenging to scale (Asselin et al., 2011; Brook, 2011; Lee et al., 2017).

heading: 2.3 LLMs for Qualitative Analysis
text: Recent advances in NLP have resulted in the development of sophisticated pretrained LLMs like ChatGPT (OpenAI, 2023). These models are appealing because they are able to generalize to many domains and follow instructions easily (Brown et al., 2020). We believe these characteristics are particularly appealing for researchers who use qualitative analysis methods and want to explore their dataset fully. Recent works have explored using ChatGPT for annotation on existing datasets and benchmarks (Kuzman et al., 2023; Ziems et al., 2023; He et al., 2023; Gilardi et al., 2023). Our work explores applying LLMs to scale annotation on a novel rubric we’ve designed for our research purposes.
Number of lecture series 10 Number of lecture transcripts 288
Number of comments 15,784 Number of labels (Section 4) 9

heading: 3 SIGHT

heading: 3.1 Lecture Transcripts

heading: 3.2 Lecture Comments
text: 317

heading: 4 Feedback Rubric
text: We develop a rubric that catalogs different types of student feedback found in SIGHT. This rubric is used for annotating the comments at scale as well (rf. Section 5). This section details how we developed this rubric.

heading: 4.1 Rubric Development
text: When creating the taxonomy, we seek to jointly maximize the following objectives.
• Provide coverage of feedback expressed in our data. We uncover categories starting from the data and manually label a subset of the data; this is a part of a qualitative research methodology known as the grounded theory approach (Corbin and Strauss, 1990).
• Provide coverage of feedback types in the literature. After we developed a set of categories directly from the data, we consult prior work on course evaluations to incorporate potentially missing themes. Specifically, we used Gravestock and Gregor-Greenleaf (2008); Chen and Hoshower (2003); Zabaleta (2007); Kim and Piech (2023) as additional sources.
• Be specific about what the feedback is about. We want to make the feedback categories targeted, enabling instructors to easily understand areas for improvement.
Our process for developing the rubric follows the procedure outlined in O’Connor and Joffe (2020); Seidel et al. (2015); Corbin and Strauss (1990):
318
Two authors read a subset of randomly selected comments and developed the initial categories collaboratively. The categories were then adapted to be specific and iterated until both authors agreed that the categories sufficiently covered the comments.

heading: 4.2 Rubric Categories
text: The final feedback categories in our rubric are detailed below. Examples of each category are shown in Table 2.
General: The comment expresses a general/bigpicture opinion about the video’s content and/or about the teaching/professional characteristics of the instructor. For example, “Amazing!!!” or “Great teacher.” would be marked as general.
Confusion: The comment asks a math-related question, expresses math-related confusion, and/or points out a math-related mistake in the video.
Pedagogy: The comment mentions an instructional method. Instructional methods include the use of examples, applications, worked out problems, proofs, visualizations, elaboration, and analogies.
Teaching setup: The comment describes or mentions the lecture’s teaching setup. Teaching setup includes the chalk, chalkboard, microphone or audio-related aspects, and camera or camerarelated aspects (e.g., angle).
Personal experience: The comment mentions the user’s personal experience or context with respect to the lecture. Personal experience or context includes the user’s own math learning or teaching experiences.
Clarification: The comment clarifies someone’s math-related misunderstanding or elaborates content from the video, and the comment includes an ‘@‘ that is immediately followed by a username.
Gratitude: The comment contains the word “thanks” or “thank”.
Non-English comment: The comment is not in English.†
†Because the lectures are conducted in English and the authors feel most comfortable English, we make the distinction between English and non-English comments.
N/A: The comment expresses a joke or is a troll comment, and/or the comment says something that is hard to connect to the video content, and/or the comment does not fall into any of the categories above.

heading: 4.3 Annotation of Sample Dataset
text: We have two annotators (co-authors) annotate a sample dataset of 280 comments based on the rubric descriptions provided above. The annotators are asked to select all categories that applied. Table 2 reports the category percentage in the sample dataset. Appendix B includes an image of the annotation interface.

heading: 5 Scaling Annotation
text: This section details how we scale annotations using our rubric and LLMs. Scaling annotation is crucial for sifting through large amounts of unstructured data (e.g., the 15,784 comments in SIGHT) and uncovering relevant student feedback. By using LLMs, we can cheaply and quickly classify comments, without the need for expensive human annotation.

heading: 5.1 Model
text: For scaling annotation, we use GPT-3.5 (gpt-3.5-turbo) through the OpenAI API (OpenAI, 2023). Although alternative models can also be used, such as text-davinci-003 from the original InstructGPT model family (Ouyang et al., 2022), or open-sourced models like Flan-T5 (Chung et al., 2022), GPT-3.5 is cheap, effective, and generally accessible for researchers without GPU support.

heading: 5.2 Prompting Methods
text: This section discusses the prompting strategies used for scaling annotation. We also experiment with other approaches, but report the most effective approaches in the main text. We detail our prior attempts and best practices in Appendix D, which we believe to be highly instructive for researchers applying this methodology to other settings.
Each comment is annotated as a binary classification task per category, i.e., does this category apply to this comment? We found that comments oftentimes contained multiple types of feedback. For example, a comment like “His teaching style seems casual and intuitive. I go to a small public college and the course is much more formal
319
and proof driven. These lectures are a great addition to (as well as a nice break from) formal proofs. Thanks MIT!” includes feedback tied to the pedagogy, personal, and gratitude categories.
Zero-shot prompting. Zero-shot prompting directly asks the model to label the category. Following prior work (Child et al., 2019; Ziems et al., 2023), we first provide the context of the comment and the comment itself, then provide instructions on the labelling task. The context of the comment includes a mention to MIT OCW, the playlist name and video name. The instructions include a description of the category, and prompts the model to respond with “true” or “false” for whether the category applies to the comment. Figure 2 shows an example of a zero-shot prompt. The zero-shot setting is the most similar to the human annotation setup.
K-shot prompting. K-shot prompting provides examples of the annotation. It first includes the instructions at the top of the prompt, then k examples that include the context, the comment, and the label. Our work uses 3-shot examples. We did not find any benefits including more than 3 examples. The instructions are moved to the top to avoid repeating the instructions after every example. Due to space constraints, we include our k-shot prompts in Appendix E.
K-shot prompting with reasoning. K-shot prompting with reasoning is similar to k-shot prompting, but additionally provides a reasoning for the label. The reasoning comes after the comment, but before the label. Due to space constraints, we include our k-shot reasoning prompts in Appendix E.

heading: 5.3 Evaluation
text: We aim to measure the effectiveness of ChatGPT in scaling the annotation process and providing instructors with useful feedback. Our evaluations are centered around the following research questions.
RQ1: How does the zero-shot approach with ChatGPT compare to human annotations across categories? We investigate this question by measuring the IRR between the human annotations and the zero-shot ChatGPT annotations.
RQ2: How does the incorporation of additional information, such as k-shot examples and reasoning, affect the model’s annotation? We in-
vestigate this question by measuring the IRR between the human annotations and the k-shot and k-shot with reasoning ChatGPT annotations.
RQ3: What are some examples of useful feedback in the scaled annotated dataset? We investigate this by performing a qualitative analysis (grounded theory approach) on the comments annotated with the confusion category.

heading: 6 Results
text: RQ1. We compare ChatGPT’s zero-shot annotation to human annotations on the sample dataset described in Section 4.3. We compute Cohen’s kappa to measure IRR within categories. Table 3 reports the human IRR as human and the average humanmodel IRR on the zero-shot setting as 0-shot. The human-model agreement never surpasses the human agreement scores. The human-model agreement also varies a lot across categories. For example, the model has fair agreement with the human annotators on pedagogy and personal (∼ 0.30) and perfect agreement on gratitude (0.92). To better understand the model’s failure modes, we manually inspect the model’s mislabeled comments. Table 4 shows comments that are representative of common failure modes on pedagogy and personal. The model seems to miss subtle references to either category, such as “explaining their algebra steps” for category pedagogy. The model also tended to mislabel student questions
320
as examples of pedagogy and personal. Categories that require more interpretation seem to be more difficult for the model to annotate in agreement with humans.
To further investigate this, we plot human against 0-shot in Figure 3. Strikingly, we observe a correlation between the model’s annotations and the humans’ annotations: Categories exhibiting greater consistency among human annotators (>0.9 IRR) also display higher agreement between humans and the model (>0.7), while categories with less consistent human annotations (0.7-0.8) correspondingly demonstrate lower levels of human-model agreement (0.3-0.5). Our findings suggest that the model’s annotations reflect the variability observed in human opinions, providing a complementary perspective to recent works such as He et al. (2023), which report models outperforming humans in annotation tasks. Our results suggest that this superior performance may not always hold, as the model’s annotation accuracy appears to be influenced by the level of human agreement. Appendix C includes additional plots of the category distributions across different annotators.
RQ2. The previous section on RQ1’s zero-shot performance indicates that the model poorly annotates categories that involve more qualitative interpretation. Each category has seems to have common failure modes. This section explores potential remedies that provide more information to the model: We provide three examples with annotated labels for each category in the prompt. These examples were selected based on the errors the model made in the zero-shot setting. We also test providing the same three examples with human-written reasoning as to why those examples are annotated with such a label.
Table 3 reports the human-model agreement scores on the 3-shot (3-shot) and 3-shot with reasoning setting (3-shot-R). The effect of the auxiliary information varies across the categories: Some categories benefit from the examples and reasoning such as pedagogy which does better on 3-shot by +0.17 and on 3-shot-R by +0.22 compared to 0-shot. However, other categories exhibit consistently worse agreement with more information, such as clarification which does worse on 3-shot by -0.32 and on 3-shot-R by -0.16 compared to 0-shot. We also experiment with
321
selectively picking the examples and tuning the reasoning, but those attempts did not result in better agreement across the categories. We flag this as an important area to address by future research for performing annotation work with LLMs.
RQ3. This section explores the annotated dataset on confusion, a category that has high humanmodel IRR and would be useful for providing instructors feedback. Of the total 15,784 comments, about 16% are annotated with confusion by the model. The model allows for easy identification of areas where students are struggling to understand the material. This information can be invaluable for teachers looking to refine their instruction (e.g., minimize confusion in their teaching material) and improve the learning experience for their students. Table 5 illustrates the diversity in subcategories within confusion. We used the qualitative research approach of grounded theory to discover these categories. There is a range of comments which ask a conceptual questions (e.g., “Can anyone explain what professor meant by pivot variables ?”) or express confusion due to a potential mistake in the lecture (e.g., “i think the explanation of the first question was a little bit wrong it seems.”) Instructors may use these identified comments to appropriately adapt their lecture content in future course iterations.

heading: 7 Limitations
text: While our work provides a useful starting point for understanding student feedback, there are limitations to our work. Addressing these limitations will be an important area for future research.
Comments may not reflect real student feedback. The comments in our dataset are from users who have chosen to post publicly on YouTube. Addi-
tionally, the comments may include features specific to this online education setting. Thus, the comments may reflect real student comments from these courses.
There is a selection bias in lecture sources. SIGHT includes lectures that may be drawn from the most successful offerings of that course. The instructional quality may not be representative of typical instruction. Thus, inferences drawn about the instruction should be interpreted with caution, as they might not generalize to other lecture settings.
We analyze only English comments. We analyze only English comments because the lecture content is given in English and the authors are most comfortable with English. As a result, our rubric may not capture the types of feedback from nonEnglish students watching lectures taught in English. In the future, the rubric and analysis should be adapted to account for the multilingual feedback setting.
We annotate a small subsample of the data To assess the validity of the automatic labels, we conduct a diagnostic study on a small, randomly selected subset of the dataset, comprising approximately 2% of the comments. Our work aims to establish a preliminary evaluation of the humanmodel agreement and model annotations, and further validation of the automatic labels is necessary. Future work can focus on acquiring such gold-standard annotations to enhance the quality and reliability of the automatic labels.

heading: 8 Future Work
text: This work contributes SIGHT (a dataset of lecture transcripts and student comments), a rubric for annotating student comments, and an analysis on the
322
annotation quality of LLMs and annotated comments.
Synthesizing student feedback effectively for instructors. Given the large volume of feedback that instructors receive, it is important to develop methods for summarizing student feedback (Hu et al., 2022). Equally important is how to present the feedback to teachers such that teachers receive it well and can easily incorporate it into their instruction (Yao and Grady, 2005; Lindahl and Unger, 2010).
Revising the lecture content with respect to student feedback. SIGHT contains per-lecture user comments. This can serve as language feedback for revising the lecture content conditioned on student feedback (Scheurer et al., 2022).
Expanding the human annotations Our work relies on two annotators (co-authors) familiar with rubric categories who annotated 280 comments from the total of 15,784 comments. Future work can investigate expanding the human annotations using our rubric, which may be useful for finetuning or evaluations. Additionally, the rubric categories focus on the themes that emerge from the comments. These can act as an initial filter on relevant versus irrelevant comments for instruction feedback. Future work can consider incorporating categories that play a more specific role for their use case, such as capturing the student’s experience in the course (Welch and Mihalcea, 2016; Ganesh et al., 2022).
Improving the model’s annotation for qualitative analysis methods Our work shows that the model does not annotate categories well that require more interpretation, even with auxiliary information. Future work can explore alternative best practices needed in prompting for these types of categories.

heading: 9 Conclusion
text: Our work contributes SIGHT, a large-scale dataset of lecture transcripts and student comments. We propose a rubric and different prompting methods for performing automated annotations on SIGHT. While we find that there is still room for improvement on reliably automating the annotation process, the dataset and rubric provide a foundation for future research to address the challenges of discovering useful feedback from students at scale.
For qualitative researchers, SIGHT offers a unique opportunity to investigate and gain insights into the feedback provided by students in online learning environments. The comments cover students’ perspectives and opinions related to math lectures. Educators can also leverage SIGHT as a valuable resource to learn from student comments and refine their teaching materials. By analyzing the feedback provided by students, educators can identify strengths and weaknesses in their instruction, discover areas that students find challenging or confusing, and gather valuable insights to enhance their teaching methodologies. We hope the dataset and methods building off of this dataset can aid educators in making data-informed decisions to optimize their instructional practices, thereby promoting a more effective learning environment.
Ethics Statement
The purpose of this work is to promote better understanding of effective pedagogy and student feedback through the use of NLP techniques. The SIGHT dataset is intended for research purposes only, and is licensed under MIT’s Creative Commons License. The dataset should not be used for commercial purposes, and we ask that users of our dataset respect this restriction. As stewards of this data, we are committed to protecting the privacy and confidentiality of the individuals who contributed comments to the dataset. It is important to note that inferences drawn from the dataset may not necessarily reflect the experiences or opinions of real students, and should be interpreted with caution. The intended use case for this dataset is to further education research and improve teaching and learning outcomes. Unacceptable use cases include any attempts to identify users or use the data for commercial gain. We additionally recommend that researchers who do use our dataset take steps to mitigate any risks or harms to individuals that may arise.

heading: Acknowledgements
text: REW is supported by the National Science Foundation Graduate Research Fellowship. We thank Professor Shannon Seidel for feedback during the initial stages of the project. We thank Yunsung Kim, Allen Nie, Haley Lepp, and Gabriel Poesia for their feedback on the manuscript. We also thank Betsy Rajala, Quinn Waeiss and Professor Michael Bernstein from the Ethics and Society Review Board for
323
guidance on the ethical and safety risks with the data used in our work.

heading: A SIGHT details
text: The playlists used in SIGHT are:
• 6.041 Probabilistic Systems Analysis and Applied Probability
• 18.01 Single Variable Calculus
• 18.02 Multivariable Calculus
• 18.404J Theory of Computation
• 18.06 Linear Algebra
• 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
• 18.100A Real Analysis
• 18.102 Introduction to Functional Analysis
• 18.217 Graph Theory and Additive Combinatorics
• 18.650 Statistics for Applications
Table 6 shows each playlist’s number of comments. These comments are collected through Google’s YouTube API as detailed in Section 3.

heading: B Human annotation interface
text: Figure 6 shows an example of what the annotation interface looks like. Each comment had to be labeled with at least one category in order to proceed to the next comment. The human annotators annotated the same comments. A total of 290 comments are manually annotated and are used to perform the annotation analysis in Section 6.

heading: C Distribution of annotations
text: Figure 4 shows the distribution of comment categories from the two human annotators and model.

heading: D Scaling annotation
text: This section documents prior attempts at scaling annotation with LLMs. We believe this is highly instructive for researchers applying LLMs to other domains to facilitate their qualitative analysis.
D.1 Multi-class classification with entire rubric
Setup We first attempted to perform multi-class classification over all 9 categories on each comment, i.e., which one category best applies to this comment? Note this is different from the multilabel classification scheme that our work performs in the main text. This method similarly provides the context of the comment and the comment itself, then provide instructions on the labelling task. The context of the comment includes a mention to MIT OCW, the playlist name, and the video name. The instructions include the entire annotation rubric: a list of the all category names and descriptions. It ends by instructing the model to respond with the category that best applies to the comment. An example of such a multi-class classification prompt is shown in Figure 7.
Results First, the human-model agreement scores were generally moderate (∼ 0.50). We found that the model did not follow constraints we had set for some categories. One constraint is on gratitude: label comments as gratitude if and only if they contain “thanks” or “thank”. We found that the model would still label comments that alluded to being grateful but did not follow this constraint as gratitude. Attempts at tuning the prompt did not result in higher IRR.
Additionally, a comment may belong to multiple categories in our rubric, making it challenging to make the model to just the one best category for the comment. The human annotators did have rules for resolving category conflicts, i.e., when a comment belongs to more than one category, which of the categories to assign. These rules were also included in the prompt, however this did not improve the model’s annotations much.
D.2 Staged divide-and-conquer multi-class classification
Setup To more directly help LLMs choose the one label that best applies to a comment, we give the LLM a series of simpler classification subtasks, i.e. a staged classification scheme. In the first stage, we ask the model to annotate categories that are more easily resolvable, e.g., gratitude, which only looks for the words “thanks” or “thank” in comments. We provide the model the option to also annotate with “none” if none of the of categories in the stage apply. These “none” comments then transition to the next stage of labelling. The
326
following stages ask the LLM to classify between a set of categories that are often mistaken for each other, and continue to pass comments that have not gotten a label to the following stage(s). One example of categories that the LLM frequently mistakes for each other are general and pedagogy: Although both categories can include comments about the teacher, pedagogy should only include comments that explicitly talk about the teacher’s instructional method, while general should include comments that expresses more general opinions about the teacher. Therefore, we tried grouping general and pedagogy together in one stage in hopes of helping the model more clearly see the difference between the two categories.
Results We found that compared to the previous classification scheme (Section D.1), asking the LLM to classify between a smaller set of categories that are often mistaken for each other does reduce the number of errors that the LLM makes between those categories. For example, from frequently mistaking pedagogy as general comments when prompted with the entire rubric (i.e., all other categories), isolating the classification between just general and pedagogy to one stage helped the LLM more accurately decide between labelling a comment as general, pedagogy, or none of the two. However, the human-model agreement scores were still at most moderate (∼ 0.60).
Our attempts to perform multi-class classification (with the entire rubric and with the staged divide-and-conquer method) led us to hypothesize that in our setting, the task of choosing only one label that best applies to a comment is too difficult
for LLMs to perform reliably. This is especially the case when comments require using one of our category conflict rules: ChatGPT did not seem to handle category conflicts as we had instructed in the prompt.
D.3 Binary classification per category This is the final classification scheme we tried and use in our main work.
Motivation Given the difficulty in performing multi-class classification and given that many comments do fit into multiple categories, we decided to implement binary classification per category. This involves prompting for each category on whether each label applies to a comment. This annotation scheme also allows each comment to be labelled with more than one category.
Additionally, in previous attempts, we found that the label names in the prompt affects the LLM’s response. For example, renaming the gratitude category as “thanks” increases the human-model agreement score because the model would otherwise mark every comment that alludes to a grateful sentiment as gratitude. Our binary classification approach eliminates the need for label names in prompts and reduces the priors that LLMs may have on certain label names.
Setup As our final prompting strategy, each comment is annotated as a binary classification task per category, i.e., does this category apply to this comment? Note that we have moved from multi-class classification (giving each comment just a single label) to multi-label classification (allowing each comment to get multiple labels). Details on this final prompting strategy can
327
be found in Section 5.2. However, note that a category’s description in the final version of our prompts may be slightly different from the human description for the category, since after multiple prompt engineering attempts, we found those new descriptions to be better at helping the LLM to detect comments in that category. For example, to help the LLM label short adjective comments like “Marvelous!!!” as general, we had to add the underlined part to the general category’s description in our general prompt: “The comment expresses a general sentiment/adjective about or expresses a general/big-picture opinion about the video’s content and/or about the teaching/professional characteristics of the instructor.”
Results Results for this binary classification per category method can be found in Section 6.

heading: E Prompts
text: This section details the prompts used for the 0-shot, k-shot and k-shot reasoning prompting strategies. We use these prompts to get the model annotations used in Section 6.
E.1 0-shot prompts • Figure 8 is the 0-shot prompt for general.
• Figure 9 is the 0-shot prompt for confusion.
• Figure 10 is the 0-shot prompt for pedagogy.
• Figure 11 is the 0-shot prompt for setup.
• Figure 12 is the 0-shot prompt for personal.
• Figure 13 is the 0-shot prompt for clarification.
• Figure 14 is the 0-shot prompt for gratitude.
• Figure 15 is the 0-shot prompt for nonenglish.
• Figure 16 is the 0-shot prompt for na.
E.2 k-shot prompts • Figure 17 is the k-shot prompt for general.
• Figure 18 is the k-shot prompt for confusion.
• Figure 19 is the k-shot prompt for pedagogy.
• Figure 20 is the k-shot prompt for setup.
• Figure 21 is the k-shot prompt for personal.
• Figure 22 is the k-shot prompt for clarification.
• Figure 23 is the k-shot prompt for gratitude.
• Figure 24 is the k-shot prompt for nonenglish.
• Figure 25 is the k-shot prompt for na.
E.3 k-shot reasoning prompts • Figure 26 is the k-shot reasoning prompt for general.
• Figure 27 is the k-shot reasoning prompt for confusion.
• Figure 28 is the k-shot reasoning prompt for pedagogy.
• Figure 29 is the k-shot reasoning prompt for setup.
• Figure 30 is the k-shot reasoning prompt for personal.
• Figure 31 is the k-shot reasoning prompt for clarification.
• Figure 32 is the k-shot reasoning prompt for gratitude.
• Figure 33 is the k-shot reasoning prompt for nonenglish.
• Figure 34 is the k-shot reasoning prompt for na.
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
349
350
351

heading: Future Work
text: While we find that there is still room for improvement on reliably automating the annotation process, the dataset and rubric provide a foundation for future research to address the challenges of discovering useful feedback from students at scale. For qualitative researchers, SIGHT offers a unique opportunity to investigate and gain insights into the feedback provided by students in online learning environments. The comments cover students’ perspectives and opinions related to math lectures. Educators can also leverage SIGHT as a valuable resource to learn from student comments and refine their teaching materials. By analyzing the feedback provided by students, educators can identify strengths and weaknesses in their instruction, discover areas that students find challenging or confusing, and gather valuable insights to enhance their teaching methodologies. We hope the dataset and methods building off of this dataset can aid educators in making data-informed decisions to optimize their instructional practices, thereby promoting a more effective learning environment