Venue: NAACL 2024

# The Promises and Pitfalls of Using Language Models to Measure

Instruction Quality in Education

Paiheng Xu

University of Maryland

paiheng@cs.umd.edu &Jing Liu

University of Maryland

jliu28@umd.edu &Nathan Jones

Boston University

ndjones@bu.edu \\ \end{tabular} &Julie Cohen

University of Virginia

jjc7f@virginia.edu &Wei Ai

University of Maryland

aiwei@umd.edu

###### Abstract

Assessing instruction quality is a fundamental component of any improvement efforts in the education system. However, traditional manual assessments are expensive, subjective, and heavily dependent on observers' expertise and idiosyncratic factors, preventing teachers from getting timely and frequent feedback. Different from prior research that mostly focuses on low-inference instructional practices on a singular basis, this paper presents the first study that leverages Natural Language Processing (NLP) techniques to assess multiple high-inference instructional practices in two distinct educational settings: in-person K-12 classrooms and simulated performance tasks for pre-service teachers. This is also the first study that applies NLP to measure a teaching practice that is widely acknowledged to be particularly effective for students with special needs. We confront two challenges inherent in NLP-based instructional analysis, including noisy and long input data and highly skewed distributions of human ratings. Our results suggest that pretrained Language Models (PLMs) demonstrate performances comparable to the agreement level of human raters for variables that are more discrete and require lower inference, but their efficacy diminishes with more complex teaching practices. Interestingly, using only teachers' utterances as input yields strong results for student-centered variables, alleviating common concerns over the difficulty of collecting and transcribing high-quality student speech data in in-person teaching settings. Our findings highlight both the potential and the limitations of current NLP techniques in the education domain, opening avenues for further exploration.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4375–4389

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Evaluating instruction quality is a crucial component for most efforts aimed at improving an education system. Measuring teaching practices with validity and reliability can help advance the understanding of effective teaching strategies, enhance teacher professional learning, and thus improve students' academic performance [1, 2, 1]. Traditional methods to assess instruction quality require trained raters to observe classrooms based on established criteria. Despite its wide use in early childhood, K-12, and higher education, a long line of research has identified a range of issues with this human-based classroom observation approach, such as its lacking consistency across schools and different learning contexts due to time and resource constraints, human subjectivity, and varying levels of expertise among raters, among others [2, 1, 13].

In light of the limitations of human-based classroom observations, an emerging literature has started to use Natural Language Processing (NLP) techniques to automatically evaluate teaching practices by analyzing transcripts of classroom recordings [2, 14, 15, 16, 17]. However, other than a few exceptions, these methods and related applications mostly focus on relatively low-inference instructional practices that rely on specific and concrete teaching moves [16], such as the frequency of focusing (or open-ended) questions [1]. In contrast, research is still scarce in exploring the feasibility of building automated measures on high-inference teaching practices, which require observers to draw conclusions and make interpretations based on many elements of teaching that are more aligned with existing classroom observation tools [15]. One exception is Wang and Demszky (2023), which uses zero-shot prompting to explore ChatGPT's ability in classifying classroom transcripts for a widely-used observation tool, Mathematical Quality Instruction (MQI) [18]. However, the authors find

the performance of ChatGPT to be quite poor.

In this paper, we set out to evaluate to what extent state-of-the-art NLP methods can measure high-inference instructional practices. We conduct our study using data collected from both traditional in-person math classrooms in K-12 schools and simulated performance tasks designed for pre-service teachers. Our study is particularly novel in its inclusion of teaching practices that are widely acknowledged to be effective for students with special needs in general math classrooms (Fuchs et al., 2008; Gersten et al., 2009). While research has been progressing quickly in the field of using NLP to measure teaching practices, to our knowledge, this is the first study that incorporates measures that target students with special needs (Wang et al., 2024). For general math education, we evaluate 11 observation variables across four dimensions of math instruction quality in in-person classrooms from MQI. Comparing the performance of pretrained language models (PLMs) for multiple high-inference teaching practices allows us to evaluate the relative performance of language models based on the nature of measures.

Our analysis shows that the performances of fine-tuned PLMs on measuring high-inference teaching practices vary by the levels of required pedagogical expertise. PLMs work decently well for variables largely depending on lexical usage such as mathematical language richness, achieving agreement levels on par with human experts. In contrast, PLMs demonstrate worse performances on variables that require further inferences that are also challenging for human experts, such as the precision of mathematical language and clarity of mathematical content.

We identify and address several challenges that negatively impact the performance of PLMs. The first challenge is the highly skewed distribution of labels due to the lack of high-rating teaching samples in practice. Across both datasets, high-rating samples consist of less than \(10\%\) of the dataset for the majority of variables. We address this issue by adopting class-weighted loss to help predict some variables, but the performance gain is limited. Another challenge comes from the length of transcripts used to infer teaching practices, which are typically from 5- to 15-minute classroom recordings. Such noisy and long inputs impede PLMs from focusing on the relevant information. We adopt a two-stage strategy by first predicting relevant sentences and then making the model focus on predicted relevant sentences. We find this strategy benefits variables whose assessment relies on multiple sentences. Thus, even if the relevance classifier falsely rules out some relevant sentences, there are still cues for PLMs to make predictions. To this end, we also use ChatGPT for zero-shot aspect summarization as the relevance model in the first stage but the extracted sentences are mostly irrelevant according to the rubrics, which is consistent with Wang and Demszky (2023).

Although it is preferable to use all dialogue in a classroom for measuring teaching practices, in traditional classroom settings, the collection of student speech data is often challenging due to microphone setups and consent requirements. Surprisingly, using teachers' utterances alone achieves reasonable performances for student-oriented observation variables. This finding underscores the potential of focusing on teacher-led discourse for comprehensive classroom assessment.

We summarize our contributions as follows. (1) We present to-date the most comprehensive study of assessing high-inference instruction quality using PLMs, encompassing both in-person and online educational settings, with a wide coverage of observation variables notably including those deemed to be important for special education. (2) We mitigate two challenges inherent to this task, stemming from its real-world setup and constraints. We introduce a two-stage strategy to mitigate the impact of long and noisy input and employ a class-weighted loss to compensate for the lack of high-rating teaching samples. (3) We demonstrate several practical implications when using PLMs to measure teaching practices, including the dependence of model performances on the required pedagogical expertise and the effectiveness of using only teachers' utterances for capturing classroom observations, even for student-centered variables.

## 2 Data

To comprehensively evaluate PLMs' ability to evaluate instruction quality, we select two datasets-the Simulation for Special Education (**SimSE**) dataset and the National Center for Teacher Effectiveness (**NCTE**) Transcript dataset (Demszky and Hill, 2023), which capture both simulated teaching sessions for pre-service teachers and in-person K-12 classrooms. We introduce the details of these two datasets and their corresponding instruction quality measurement 



### SimSE - Metacognitive Modeling

The SimSE dataset is collected as part of Teaching.1 a teaching simulation platform for pre-service teachers. All participating teachers complete 5-minute simulation sessions and receive coaching on how to better elicit students' thinking. The goal is to improve teachers' use of metacognitive modeling, a teaching practice that involves the narration of actions, decisions, and thought processes while demonstrating metacognitive strategies Archer and Hughes (2010); Montague (1992). The dataset in our analysis focuses on students with special needs in general math education, collected from Fall 2022 to Spring 2023. The simulation sessions in the dataset focus on teaching metacognitive modeling while unpacking (not solving) a word problem. Wilson and Conyers (2016); McLeskey et al. (2017) show that such a strategy is one of the most critical practices to support students with special needs.

Footnote 1: [https://www.teachsim.org/](https://www.teachsim.org/)

Specifically, metacognitive modeling contains five components, among which _Unpacking_, _Self-Instruction_, and _Self-Regulation_ are the most important. Table 1 shows the definitions and high-rating examples for these components. Rating these high-inference components requires drawing conclusions from and interpreting several elements of teaching. For example, _Unpacking_ can be applied to address context, quantities, and mathematical relationships in word problems. The core idea is that teachers should make sense of the concepts strategically so that the understanding can be generalized across problems. _Self-regulation_ mainly concerns students' emotional regulation and such statements often contain emotion-related words such as "overwhelmed" and "confused", though one can also implicitly express such emotions by saying "there is a lot of information in the problem". More importantly, successful demonstration of self-regulation requires the teacher to accompany emotion expression by regulation strategies and how these strategies support sense-making in general. _Self-instruction_ is the most difficult component to identify as it requires multiple types of evidence. Explicit, general self-instruction statements should be paired with specific statements

\begin{table}
\begin{tabular}{p{48.4pt} p{113.8pt} p{113.8pt}} \hline \hline Variable & Definition & High-Rating Examples \\ \hline Unpacking & A statement making sense of the context, quantities, or mathematical relationships. & “_That vocal ‘about’ means I do not need to know exactly how much – I need to estimate. And to estimate. I don’t allow clear, friendly numbers._” (Example of strategically addressing concepts that may be confusing for students. Other examples may address coach, context, using tables, etc.) \\ \hline Self- Instruction & An explicit statement made by the teacher which references their thinking and/or timescale related to making sense of problems. & “_Tm going to ask myself, what is this problem about?_” (Explicit general self-instruction example, such examples should be paired with specific self-instruction statements or justifications to be rated as high.) \\  & Asplituation based behind the teacher which references their thinking and/or timescale related to making sense of problems. & “_I know many apples and how many humans Nareech has in her basket._” (Explicit specific self-instruction example). \\  & Asplituation based behind the teacher which references their thinking and/or timescale related to making sense of problems. & “_What is this problem asking me? By putting it in my own worlds, I can make sure I get what’s going on._” (This justification makes clear to students why putting a problem in their own works can help them make sense of the problem) \\ \hline Self- Regulation & A self-instruction statement that supports students’ emotional regulation. & “_Tm justardian through and thought, this is a lot of information. When I’m feeling overwhelthened by the amount of information in a problem, I know I can go back and re-read the problem sentence by sentence to help me break the problem down into smaller, more manageable chunks._” (Example of explicit reference to a self-regulation strategy and reference to how it can be applied across problems more broadly) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Rating rubrics of selected metacognitive modeling components in the SimSE dataset.

\begin{table}
\begin{tabular}{p{48.4pt} p{113.8pt} p{113.8pt}} \hline \hline Dimension & Variable Definition & High-Rating Examples/Criteria \\ \hline Richness of the Mathematics & _Mathematical Language (MIANG)_ captures how fluently the teacher (and students) use mathematical language and whether the teacher supports students’ use of mathematical language. & 1. Density of mathematical language is high during periods of teacher talk. \\ Working with Students and Mathematics & _Remediation of Student Errors and Difficacies (REMED)_ records instances of remediation in which student misconceptions and difficulties with the content are addressed. _Conceptual remediation_ gets at the root of student misunderstandings, rather than requiring little the procedure or fact. \\ \hline Errors and Imprecision & _Imprecision in Language or Notation (LANGIMP)_ captures problematic uses of mathematical language or notation language. & Impecision occurs in most or all of the segment, OR imprecision obscures the mathematics of the segment. \\ \hline Student Participation in Meaning-Making and Reasoning & _Students Provide Explanations (STEXPL)_ captures when students provide a mathematical explanation for an idea, procedure, or solution. \\ \hline \hline \end{tabular}
\end{table}
Table 2: Rating rubrics of selected MQI variables in the NCTE dataset.



or justifications to support the understanding of the current problem and metacognitive thinking for future problem-solving.

The other two components of metacognitive modeling require less pedagogical expertise: _Objective_ evaluates the quality of the teacher stating the two goals: paying attention to the teacher's thinking (so that students may use it in their future problem-solving) and making sense of the problem. _Ending_ requires teachers to refrain from solving the problem or presenting strategies or operations.

All these components are scored as low, mid, and high (1 to 3) by experts, where the mid rating applies to samples that partially meet the criteria. The three-way rating scale was developed by instruction experts to balance granularity and reliability while providing meaningful differentiation of teaching quality. While a more granular scale might provide more nuanced insights, it would also exacerbate the difficulty of achieving a high inter-rater reliability, which has been demonstrated as a difficult task in classroom observations Ho and Kane (2013). We note that collecting such educational data poses significant challenges, notably the high cost of annotation and the necessity for well-defined rubrics that delineate each category unambiguously.

Dataset Statistics.The SimSE dataset contains 1,135 five-minute transcribed teaching sessions (teachers' talk only), each annotated with the five metacognitive modeling components. On average, each transcript has 416.1 words (std: 187.9). A subset of transcripts are annotated by two or three raters. The Spearman correlations between raters are shown in Table 3 (detailed computation process in Appendix B). We round the average score for each session as the prediction label. The label distributions are highly skewed towards the low ratings, which may account for up to \(80\%\) of the samples for some components, as shown in Figure 1. There might be multiple reasons why high-rating teaching samples are rare. For example, pre-service teachers have not yet worked in real classrooms and are still developing their teaching skills.

Lexical Analysis.We examine the linguistic properties of teaching language by analyzing the lexical differences between low and high rating samples. A subset of 199 transcripts have sentence-level annotations of their relevance to each of the five components. The raters were asked to select the evidence sentences that helped them score each component following the rubrics. We follow Monroe et al. (2008) and use normalized log-odds-ratio \(z\) to identify the n-grams more associated with each rating level. Given the lack of high ratings, we combine mid and high ratings and compare them with low-rating transcripts.

From the visualization for the Self-Instruction component in Figure 2, phrases like "help me" can be signals for justification statements, which are one of the required evidence for mid/high ratings. However, we observe conflicting signals for the other type of required evidence, specific self-instruction statements. Typical self-instruction phrases such as "think about", "to draw", and "figure out" are associated with mid/high ratings while similar phrases such as "lets think", "to find", and "to estimate" are associated with low ratings. Therefore, the identification of mid/high rating components requires high-inference ability and pedagogical expertise beyond lexical cues, highlighting the difficulty of automatic measurement.

The SimSE dataset focuses on students with spe

Figure 1: Label distribution for metacognitive modeling components in the SimSE dataset.

Figure 2: Top \(n\)-grams (\(n\in\{1,2,3\}\)) from sentences that are annotated relevant to the mid-high and low ratings for Self-Instruction. Only \(n\)-grams with \(95\%\) confidence level are colored (\(z>1.96\)).

 

cial needs in simulated teaching sessions. We posit that it represents a simpler context for analysis compared to the in-person classroom setting, regardless of the complexity of the variables being examined. This is because the setting of the SimSE dataset is similar to modular online teaching sessions which are relatively short and comprise teacher utterances only. We complement the SimSE dataset by using the NCTE dataset, which is collected from in-person, math classrooms in K-12 education.

### Ncte - Mqi

The NCTE dataset is the largest publicly available dataset of U.S. classroom transcripts linked with classroom observation scores. The raters evaluate math teaching quality based on MQI Hill et al. (2008) instruments along four dimensions: _Richness of the Mathematics, Working with Students and Mathematics, Errors and Imprecision,_ and _Student Participation in Meaning-Making and Reasoning_. Each dimension has multiple observation variables. We experiment on a total of eleven variables and present one variable per dimension for presentation purposes, as shown in Table 2. The description of the remaining seven variables can be found in Appendix A. Similar to the SimeSE dataset, these variables are scored on a scale of 1-3, corresponding to low, mid, and high ratings.

These variables also vary in the amount of information and inferences required for human evaluation. For example, _Mathematical Language_ (MLANG) is relatively straightforward as it mainly depends on the density of mathematical language usage, which is readily identifiable once a dictionary of such words is available. In contrast, _Remediation of Student Errors and Difficulties_ (REMED) and _Impprecision in Language or Notation_ (LANGIMP) require further inferences and information beyond the mathematical language used in classrooms. For example, high ratings of REMED should remediate student misunderstanding at a conceptual level. Such remediation behavior and relevant discussion should also occupy a large proportion of the segment used for annotation to be considered as "high". LANGIMP covers math content, including notation and languages used to convey technical terms. Its evaluation requires an understanding of these contents and the ability to identify when they are imprecisely expressed. _Students Provide Explanations_ (STEXPL) is a student-oriented variable.

Dataset Statistics.The NCTE dataset consists of 1,660 4th and 5th grade elementary mathematics observations collected by NCTE between 2010-2013. The transcripts are anonymized and represent data from 317 teachers across 4 school districts that serve largely historically marginalized students. The original dataset has transcripts for the whole class, which are typically 45-60 minutes long, while the MQI instrument observation variables apply to 7.5-minute segments. Therefore, we align the transcripts with MQI observations, resulting in 9,886 segments, each rated on 11 MQI variables.

Similar to the SimSE dataset, the label distribution in the NCTE dataset is also highly imbalanced. High-rating segments are extremely rare and low-rating segments constitute large proportions for most variables, which reverberates the practical challenge of lacking highly rated teaching activities in real-world education data.

Additionally, since the collection of student talk is challenging in practice, student utterances are often inaudible. We experimented with using teacher utterances only and including student utterances as input text. For the first input setting, we concatenate all teacher utterances as input. To include student utterances, we format the input in the transcript style, where each utterance is <speaker>:<utterance>. <speaker> can be either the teacher, a student, or multiple students. All speakers are anonymized, and the students have identifiers (e.g., student A) generated by professional transcribers when possible. We discuss the results of these two formats in Section 5. On average, each segment contains 647.6 words (std: 230.2) when using teacher utterances only, and 750.6 words (std: 259.9) when including student utterances.

Figure 3: Label distribution for MQI variables in the NCTE dataset.

 

Despite the differences in variable definitions and educational settings between metacognitive modeling in the SimSE dataset and MQI variables in the NCTE dataset, assessing the quality of these variables in both datasets requires raters to have strong domain knowledge and the ability to draw conclusions and interpret several elements of teaching. The SimSE dataset is tailored towards metacognitive modeling for students with special needs, and the NCTE dataset concentrates on the mathematical instruction quality. Therefore, we hope our study provides insights into how effectively PLMs can measure high-inference and highly domain-specific instructional practices in both simulated teaching sessions and in-person classrooms, two common educational settings currently.

## 3 Methods

Measuring instructional practices is naturally formulated as text classification tasks. To comprehensively understand language models' ability in these tasks, we experiment with a wide range of PLMs. Specifically, we fine-tuned the following models by adding a classification head on top of the hidden layers, using the implementation from Transformer Wolf et al. (2020), i.e., BERT Devlin et al. (2018), DistilBERT Sanh et al. (2019), XLNet Yang et al. (2019), and RoBERTa Liu et al. (2019). We also fine-tuned Llama2-7B Touvron et al. (2023) with QLoRA Dettmers et al. (2024) in the instruction-following style by converting the dataset to a next-word prediction setup. More discussions on model choices are in Appendix C.

We identify two primary challenges in automatically measuring teaching quality: (1) the text input is long and noisy, making it hard for PLMs to focus on the relevant sentences for each variable; and (2) the label distributions are highly imbalanced, as there are few high-rating teaching transcripts following the standards in the rubric for each observation variable. We discuss how we mitigate these two challenges with available resources next.

### Challenge of Long Input

The average lengths for simulation sessions and classroom segments in the SimSE and NCTE datasets are \(416.1\) and \(647.6\) words, respectively. Depending on the definitions of the observation variables, the amount of irrelevant text in the model input varies. As shown in Table 6 for the SimSE dataset, components like Objective, Self-Regulation, and Ending have fewer than or close to two relevant sentences per simulation session, while Self-Instruction has 8 and Unpacking has 21.

To reduce the impact of irrelevant information, we develop a two-stage prediction strategy that first predicts the relevance of each sentence and then scores on the concatenation of sentences predicted as relevant. Formally, for each component, we first build a relevance model \({\cal Y}_{R}(s_{i})\) that predicts the relevance of each sentence \(s_{i}\) in transcript \(T\). We then train a classification model \({\cal Y}_{S}(S_{R})\) whose input concatenates the sentences predicted as relevant by \({\cal Y}_{R}\), i.e., \(S_{R}=\{{\cal Y}_{R}(s_{i})=1|s_{i}\in T\}\).

Additionally, given the recent success of Chat GPT in summarization tasks Zhang et al. (2024), especially aspect summarization for conversations Yang et al. (2023), we also experiment with using ChatGPT as the relevance model in the first stage. The two-stage model can then be defined as \({\cal Y}_{S}({\cal S}(T))\), where \({\cal S}(\cdot)\) is the summarization model, and its output is directly used as the input for the classification model in the second stage.

### Challenge of Label Imbalance

High-rating transcripts are rare in our datasets and in real-world classrooms/simulation sessions as shown in the largest study on teaching quality to date by Kane and Staiger (2012). Therefore, differentiating mid- and high-rating transcripts from low-rating ones is helpful in practice. We group the mid- and high-rating transcripts and convert the three-way classification task to a binary one. We report the performances in both settings.

Moreover, the low-rating transcripts are the major class for almost all observation variables, taking up from \(53\%\) to \(89\%\) of the samples. For both binary and three-way settings, we use cross-entropy loss weighted by the inverse class frequency, defined as \({\cal L}(y,\hat{y})=-\sum_{j=1}^{c}w_{j}y_{j}\log(\hat{y_{j}})\), where \(y_{j}\) = 1 if training sample \(i\) belongs to class \(j\) and 0 otherwise. \(y\) and \(\hat{y}\) are the ground-truth label and predicted label, respectively. \(c\) is the number of classes, and \(w_{j}=\frac{N}{c\times N_{j}}\) is the weight for class \(j\), where \(N\) and \(N_{j}\) are the total number of training examples and the number of class \(j\) respectively.

## 4 Experiments

To evaluate the models and techniques introduced above, we randomly split the simulation sessions and classroom segments from the SimSE and NCTE datasets into train/dev/test sets with a ratio 

of \(60\%/20\%/20\%\). For each observation variable, we fine-tune the models mentioned in Section 3 on the training set, including variants that apply the two-stage prediction strategy introduced in Section 3.1 (only on the SimSE data due to the availability of sentence-level annotations) and the ones that use the weighted loss defined in Section 3.2.

We select the best model combination based on dev set performances and report their performances on the test set in Table 3 and Table 4 for the SimSE and NCTE datasets respectively. We follow this procedure for three-way and binary settings and report macro-F1 as the evaluation metric for both. We use Majority F1 as the baseline of predicting the majority rating to show the performance gain for each variable. For the three-way setting, we also report the Spearman correlation, a non-parametric measure of ranking correlation, to account for the ranking of the labels (i.e., low, mid, and high). We repeat experiments for each observation variable five times and report the mean and standard deviation of the performance scores.

The test set results for selected models are shown in Table 3 and Table 4 for the SimSE and NCTE datasets, respectively. Results for the other seven MQI variables are shown in Appendix A, and hyperparameter searching and other training details are described in Appendix C.

## 5 Results and Discussion

In general, models perform much better on metacognitive modeling components than MQI variables, partly due to the shorter input length and the teacher-only setup. However, we observe considerable variations across the variables. We discuss the insights in detail as follows.

LLMs work better for variables that require less pedagogical expertise.In Table 3, Objective and Ending have significant Spearman correlations over \(0.8\) in the three-way setting and F1 scores over \(0.9\) in the binary setting. However, components that require higher levels of pedagogical expertise, i.e., Unpacking, Self-Instruction, and Self-Regulation, have notably worse performance, although still obtaining comparable correlation levels as achieved by human raters. Similarly, in Table 4, LLMs work better for MLANG which can be assessed by lexical usage, while perform worse on REMED and LANGIMP, which require further inferences of mathematical knowledge. Similar trends can be observed for other MQI variables in Table 8.

Using only teacher utterances as input captures most observations, even for student-oriented variables.For NCTE, we experimented with using only teacher utterances and including student utterances (Table 4 and Table 5, additional results in Appendix A). We find that using only teacher

\begin{table}
\begin{tabular}{l r r r r} \hline \hline Component & Avg. \# Sent. & Majority F1 & F1 & Positive F1 \\ \hline Objective & 1.9 & 49±0 & 88±1 & 78±3 \\ Unpacking & 21.9 & 38±0 & 79±1 & 84±1 \\ Self-Instruction & 8.4 & 43±0 & 75±1 & 62±2 \\ Self-Regulation & 0.7 & 50±0 & 78±2 & 57±5 \\ Ending & 2.3 & 48±0 & 65±2 & 35±4 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Sentence prediction results for SimSE dataset. Avg. \(\#\) Sent. is the average number of relevant sentences in each simulation session. Positive F1 is the F1 score of the positive class (relevant sentences). F1 scores are in percentage.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{3}{c}{Three-way} & Binary \\ \cline{2-5}  & Majority F1 & F1 & Spearman & Majority F1 & F1 \\ \hline MLANG & 23±0 & 53±1 & 0.42±0.02 & 38±0 & 69±1 \\ REMED & 23±0 & 43±1 & 0.28±0.01 & 35±0 & 65±1 \\ LANGIMP & 29±0 & 44±4 & 0.20±0.04 & 43±0 & 61±1 \\ STEXPL & 28±0 & 47±2 & 0.35±0.01 & 42±0 & 69±1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Test set performance of selected models for the four selected MQI variables in the NCTE dataset. Inputs use teacher text only. F1 scores are in percentage. All Spearman correlations on the test sets are significant with 95% confidence.

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline \multirow{2}{*}{Variable} & \multicolumn{3}{c}{Three-way} & Binary \\ \cline{2-5}  & Majority F1 & F1 & Spearman & Majority F1 & F1 \\ \hline OLANG & 0.79 & 27±1 & 75±3 & 0.84±0.02 & 41±1 & 91±2 \\ Upacking & 0.57 & 22±1 & 64±1 & 0.61±0.03 & 35±2 & 77±2 \\ Self-Instruction & 0.34 & 29±0 & 60±2 & 0.56±0.04 & 43±0 & 76±2 \\ Self-Regulation & 0.62 & 31±0 & 65±4 & 0.68±0.05 & 46±0 & 85±4 \\ Finding & 0.78 & 28±1 & 81±3 & 0.90±0.01 & 43±0 & 95±1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Test set performance of selected models for metacognitive modeling components in the SimSE dataset. \(\rho_{\textit{Pater}}\) shows the weighted average Spearman correlation among human raters in the three-way setting. F1 scores are in percentage. All Spearman correlations on the test sets are significant with 95% confidence.

 

utterances achieves comparable and even better results for all MQI variables, despite ignoring student utterances. This is particularly interesting for the student-oriented variables (e.g., STEXPL) in the Student Participation in Meaning-Making and Reasoning dimension of MQI. Student talk is hard to collect in practice since it requires setting up multiple mics across the classroom and obtaining privacy consent. The fact that we can measure student activities with teacher utterances alone underscores the potential of focusing on teacher-led discourse for comprehensive classroom assessment.

The results above are from models selected from various model types and combined with different techniques described in Section 3. We next present the results and discuss their implications.

Class-weighted loss marginally improves the performancesVariables that have more balanced class distributions tend to have better performance, but the intrinsic difficulty of the variables seems to be the driving factor. For example, Self-regulation has a more skewed distribution but worse performance compared to Self-Regulation. Similar observations can be made when comparing Unpacking with Objective or Ending. Moreover, the adoption of class-weighted loss achieved best performance on 10 out of the 16 variables in the three-way setting (Tables 7 and 12) and 9 out of 16 in the binary setting (Tables 11 and 13), although the differences are small. Therefore, classified loss helps but with a marginal effect.

### Discussion

Dealing with long inputWe employ a two-stage prediction strategy, as described in Section 3.1. Due to data availability, we only have sentence-level annotations on a subset of the SimSE dataset. We apply the same experiment setup for the classification task on this sentence-level relevance prediction task. The results are shown in Table 6. From Tables 7 and 11, we find that the two-stage strategy helps Self-Instruction and Self-Regulation in the three-way setting, despite their moderate F1 scores on the relevant (positive) class in the first stage. We suspect that components with multiple relevant sentences (i.e., Self-Instruction) per session are less prone to the error of the relevance classifier. That is, missing one or two relevant sentences would result in empty inputs for Objective and Ending, which usually have only a few relevant sentences per session. For Self-Regulation, as this teaching practice depends on emotion-related expressions, we conjecture that the predicted relevant sentences are likely from mid- and high-rating samples.

We also use ChatGPT as a zero-shot summarization model to avoid reliance on costly sentence-level annotations. However, our preliminary analysis shows that the relevant utterances retrieved by ChatGPT are not helpful. For example, as shown in Figure 4, when prompted to summarize or extract utterances related to REMED, ChatGPT overestimated the mathematical instruction quality of the utterances and falsely identified relevant sentences that were leading questions regardless of whether they were addressing student errors and difficulty or whether they were at conceptual level. It failed to make inferences based on the level of pedagogical expertise required for assessing these subject matter variables. The finding is consistent with Wang and Demszky (2023), who found that \(30\%\) of the highlights retrieved by ChatGPT are unfaithful, \(55\%\) are irrelevant, and \(71\%\) are not insightful, determined by education experts. Such poor preliminary results, along with the high cost, warrant no further large-scale experiments.

Model choicesAmong all the PLMs employed in the experiments, DistilBERT, RoBERTa, and XLNet have comparable performance, outperforming BERT in many cases. Notably, fine-tuning Llama2 with parameter-efficient methods results in unsatisfactory results in these subject-matter tasks. As shown in Tables 7 and 11, it only marginally improves the majority baseline in most cases. We also experimented with using other parameter-efficient methods like p-tuning Liu et al. (2023) and train

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c c c c c} \hline \multirow{2}{*}{Component} & \multicolumn{8}{c}{Normal} & \multicolumn{8}{c}{Weighted} & \multicolumn{8}{c}{Two-stage} & \multicolumn{8}{c}{Two-stage + Weighted} \\ \cline{2-13}  & ditherment & bert & Accounts & Limita & ditherment & bert & bert & Accounts & ditherment & bert & bret & ditherment & bert & xlabel & refocus \\ \hline Objective & **74.3** & 674.83 & 691.65 & 649.69 & 44.17 & 734.25 & 562.15 & 554.26 & 654.23 & 71.44 & 72.44 & 504.23 & 621.26 & 614.21 & 656.12 & 654.12 \\ Upacking & 644.5 & 4924.2 & 572.0 & **674.4** & 449.6 & 674.5 & 596.12 & 412.27 & **674.6** & 63.66 & 564.20 & 65.99 & 65.64 & 646.34 & 382.0 & 572.20 \\ Self-Interaction & 634.0 & 574.1 & 552.6 & 468.14 & 366.17 & **634.9** & 565.18 & 4026.4 & 441.8 & 52.66 & 55.16 & 644.19 & 482.51 & 660.10 & 472.12 & 354.92 & 421.33 \\ Self-Regulation & 504.5 & 451.4 & 471.41 & 421.66 & 31.00 & 534.9 & 242.15 & 39.99 & 454.19 & 65.45 & 61.88 & 494.41 & 424.11 & 61.05 & 664.9 & 564.68 & 60.48 \\ Ending & 698.4 & 604.8 & **764.11** & 664.7 & 446.0 & 674.4 & 514.22 & 726.24 & 514.17 & 57.49 & 564.10 & 504.11 & 554.14 & 61.46 & 61.47 & 402.11 & 61.10 \\ \hline \end{tabular}
\end{table}
Table 7: The F1 scores (%) of predicting metacognitive modeling components on the Dev set of SimSE dataset in the three-way setting. The best-performing models are bolded.

 

ing Llama2 by adding a classification head on top of the hidden layers. The results are similar. Therefore, we report the results of the commonly adopted setup of fine-tuning Llama2 as a next-word prediction task (Touvron et al., 2023), by converting the dataset to instruction-following (Appendix C).

## 6 Related Work

Prior works on automatic teaching evaluations primarily focused on low-level instruction statistics such as the frequency of certain utterance-level teaching strategies employed in the classroom. Considering teachers' questions in the classroom play a crucial role in students' engagement and academic achievement (Kelly, 2007), many researchers studied question-related behaviors such as detecting utterances that contain questions (Blanchard et al., 2016; Donnelly et al., 2017), identifying authentic questions without prescripted answers Cook (2018); Kelly et al. (2018); Reilly and Schneider (2019), identifying focusing and funneling questions (Alic et al., 2022), and so on (Huang et al., 2020). Other discourse features of teachers have also been shown to influence learning achievement. For instance, Hunkins et al. (2022) detects socio-emotional supports. Demszky et al. (2021) measures teachers' uptake to provide feedback for students' contribution. Schlotterbeck et al. (2021); Xu et al. (2020) recognizes different types of teaching activities such as presenting, guiding, and management (Tan and Demszky, 2023).

Despite previous efforts on low-level evaluation, Wang and Demszky (2023) provided some preliminary results on evaluating teaching quality and providing high-level feedback, which aligns more with the current forms of observation feedback in education. However, their study focused on testing ChatGPT's zero-shot ability and only evaluated a small set of MQI variables in the NCTE dataset.

## 7 Future Work

Model Choices.Although our experiments cover a wide range of PLMs including the recently released Llama2-7B model, many advanced Large Language Models (LLMs) were beyond our scope due to resource demands. With the negative results in the zero-shot setting (Wang and Demszky, 2023), future research could focus on adapting these models for similar tasks, i.e., make LLMs better follow the instructions in the rubrics and align them with experts' preferences.

Model Explainability.Our study indicates that fine-tuned PLMs can match expert-level performance on some observation variables. However, they lack transparent explanations for their predictions. Explainable models would not only enhance trust among education practitioners (Lipton, 2018) but could also enrich the feedback that teachers receive. Our preliminary analysis using attribution score to highlight the important tokens achieves unsatisfactory results, suggesting more efforts on this venue. Additionally, PLMs often rely on spurious correlations (Zhou et al., 2023; Tu et al., 2020), underscoring the value of explainable models in uncovering prediction rationales and enhancing model robustness. Another way of providing insights into models' behaviors is to generate free-text explanations, which seems promising with the recent advance of generative LLMs.

Multimodality.Education recordings inherently encompass multiple modalities, offering rich data through visuals like facial expressions and acoustics such as vocal tones, which are pivotal for education specialists in assessing teaching quality. Yet, these features are underexplored in automated methods due to the lack of high-quality video or audio data. Advances in recording technology and the rise of online educational platforms can facilitate the collection of high-quality audio-visual content, paving the road for future research. Nonetheless, the sensitive nature of this data necessitates careful handling to prioritize privacy and ethical issues.

## 8 Conclusion

In this study, we present the most comprehensive study to date on assessing high-level instruction quality, spanning both in-person and simulated educational settings, with a wide coverage of observation variables for math education, including metacognitive components tailored for students with special needs. We find that the performance of PLMs largely depends on the level of pedagogical expertise required for assessing the observation variables, despite employing various techniques aimed at addressing the issues of lengthy inputs and the scarcity of high-rating samples. Our study enhances the understanding of the extent to which current NLP techniques work in education and brings insights into automatically evaluating high-inference instruction quality, which could reduce disparities in teachers' professional learning and, consequently, students' academic outcomes 



### Limitations

Generalizability.Although our study includes two distinct educational settings, the NCTE datset is the largest available dataset of U.S. classroom, and the SimSE dataset focuses on students with special needs, they still only capture a small proportion of U.S. classrooms and teaching activities. Moreover, the NCTE dataset was collected over a decade ago and thus may not represent the current education field, even though teaching practices have remained relatively constant over time [15; 2]. Therefore, there are limitations to the generalizability of the findings. It is important to carefully validate the measures built on these datasets before employing them in a new domain or a target population.

Lack of High-Rating Teaching Samples.One issue we tried to address in this study is the lack of high-rating teaching samples. However, this is limited by the lack of high-quality teaching activities in real-world classrooms/simulation sessions [30]. In fact, the teaching simulation platform that curated the SimSE dataset aims to improve teachers' professional development, along with a lot of ongoing efforts to provide teachers with timely feedback [46]. Moreover, with the recent advance of generative LLMs [9], researchers have started exploring the potential of using them to generate educational content [8, 7]. Despite their relatively poor performances in solving math problems [11], coaching on mathematical education [26], and remediating students' mathematical errors [26], the generated content can still be helpful to some extent.

## Acknowledgements

We thank Lindsey McLean and Marissa Pilger Suhr for curating and sharing the SimSE dataset. We thank Dorottya Demszky and Dan McGinn for their help with the NCTE dataset. We thank the financial support from the National Science Foundation through award numbers 2009939 and 2010298, as well as the Grand Challenge program at the University of Maryland.

## References

* (1)
* Adelman et al. (2003) Clement Adelman, Clem Adelman, and Roy Walker. 2003. _A guide to classroom observation_. Routledge.
* Alic et al. (2022) Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally identifying funneling and focusing questions in classroom discourse. In _Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)_, pages 224-233.
* Archer and Hughes (2010) Anita L Archer and Charles A Hughes. 2010. _Explicit instruction: Effective and efficient teaching_. Guilford Publications.
* Blanchard et al. (2016) Nathaniel Blanchard, Patrick J Donnelly, Andrew M Olney, Borhan Samei, Brooke Ward, Xiaoyi Sun, Sean Kelly, Martin Nystrand, and Sidney K D'Mello. 2016. Semi-automatic detection of teacher questions from human-transcripts of audio in live classrooms. _International Educational Data Mining Society_.
* Cohen and Mehta (2017) David K Cohen and Jal D Mehta. 2017. Why reform sometimes succeeds: Understanding the conditions that produce reforms that last. _American educational research journal_, 54(4):644-690.
* Cook (2018) Connor Cook. 2018. An open vocabulary approach for detecting authentic questions in classroom discourse. In _. Proceedings of the 11th International Conference on Educational Data Mining (EDM 2018)_.
* Cuban (1993) Larry Cuban. 1993. _How teachers taught: Constancy and change in American classrooms, 1890-1990_. Teachers College Press.
* Demszky and Hill (2023) Dorottya Demszky and Heather Hill. 2023. The NCTE transcripts: A dataset of elementary math classroom transcripts. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_, pages 528-538, Toronto, Canada. Association for Computational Linguistics.
* Demszky and Liu (2023) Dorottya Demszky and Jing Liu. 2023. M-powering teachers: Natural language processing powered feedback improves 1: 1 instruction and student outcomes. In _Proceedings of the Tenth ACM Conference on Learning@ Scale_, pages 59-69.
* Demszky et al. (2023) Dorottya Demszky, Jing Liu, Heather C Hill, Dan Jurafsky, and Chris Piech. 2023. Can automated feedback improve teachers' uptake of student ideas? evidence from a randomized controlled trial in a large-scale online course. _Educational Evaluation and Policy Analysis_, page 01623737231169270.
* Demszky et al. (2021) Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsumori B Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1638-1653.
* Desimone and Pak (2017) Laura M Desimone and Katie Pak. 2017. Instrumental coaching as high-quality professional development. _Theory into practice_, 56(1):3-12

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. _Advances in Neural Information Processing Systems_, 36.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Donnelly et al. (2017) Patrick J Donnelly, Nathaniel Blanchard, Andrew M Olney, Sean Kelly, Martin Nystrand, and Sidney K D'Mello. 2017. Words matter: automatic detection of teacher questions in live classroom discourse using linguistics, acoustics, and context. In _Proceedings of the Seventh International Learning Analytics & Knowledge Conference_, pages 218-227.
* Frieder et al. (2024) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. 2024. Mathematical capabilities of chatgpt. _Advances in Neural Information Processing Systems_, 36.
* Fuchs et al. (2008) Lynn S Fuchs, Douglas Fuchs, Sarah R Powell, Pamela M Seethaler, Paul T Cirino, and Jack M Fletcher. 2008. Intensive intervention for students with mathematics disabilities: Seven principles of effective practice. _Learning Disability Quarterly_, 31(2):79-92.
* Gersten et al. (2009) Russell Gersten, David J Chard, Madhavi Jayanthi, Scott K Baker, Paul Morphy, and Jonathan Flojo. 2009. Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. _Review of educational research_, 79(3):1202-1242.
* Grossman et al. (2013) Pam Grossman, Susanna Loeb, Julie Cohen, and James Wyckoff. 2013. Measure for measure: The relationship between measures of instructional practice in middle school english language arts and teachers' value-added scores. _American Journal of Education_, 119(3):445-470.
* Hill et al. (2008) Heather C Hill, Merrie L Blunk, Charalambos Y Charamabous, Jennifer M Lewis, Geoffrey Chelps, Laurie Sleep, and Deborah Loewenberg Ball. 2008. Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. _Cognition and instruction_, 26(4):430-511.
* Ho and Kane (2013) Andrew D Ho and Thomas J Kane. 2013. The reliability of classroom observations by school personnel. research paper. met project. _Bill & Melinda Gates Foundation_.
* Huang et al. (2020) Gale Yan Huang, Jiahao Chen, Haochen Liu, Weiping Fu, Wenbiao Ding, Jiliang Tang, Songfan Yang, Guoliang Li, and Zitao Liu. 2020. Neural multi-task learning for teacher question detection in online classrooms. In _Artificial Intelligence in Education: 21st International Conference, AIED 2020, Iframe, Morocco, July 6-10, 2020, Proceedings, Part 121_, pages 269-281. Springer.
* Hunkins et al. (2022) Nicholas Hunkins, Sean Kelly, and Sidney D'Mello. 2022. "beautiful work, you're rock stars!": Teacher analytics to uncover discourse that supports or undermines student motivation, identity, and belonging in classrooms. In _LAK22: 12th International Learning Analytics and Knowledge Conference_, pages 230-238.
* Kane and Staiger (2012) Thomas J Kane and Douglas O Staiger. 2012. Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. research paper. met project. _Bill & Melinda Gates Foundation_.
* Kelly (2007) Sean Kelly. 2007. Classroom discourse and the distribution of student engagement. _Social Psychology of Education_, 10(3):331-352.
* Kelly et al. (2020) Sean Kelly, Robert Bringe, Esteban Aucejo, and Jane Cooley Fruenwirth. 2020. Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations. _Education Policy Analysis Archives_, 28:62-62.
* Kelly et al. (2018) Sean Kelly, Andrew M Olney, Patrick Donnelly, Martin Nystrand, and Sidney K D'Mello. 2018. Automatically measuring question authenticity in real-world classrooms. _Educational Researcher_, 47(7):451-464.
* Kraft et al. (2018) Matthew A Kraft, David Blazar, and Dylan Hogan. 2018. The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. _Review of educational research_, 88(4):547-588.
* Li et al. (2023) Yu Li, Shang Qu, Jili Shen, Shangchao Min, and Zhou Yu. 2023. Curriculum-driven edubot: A framework for developing language learning chatdots through synthesizing conversational data. _arXiv preprint arXiv:2309.16804_.
* Lipton (2018) Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. _Queue_, 16(3):31-57.
* Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengaxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023. GPT understands, too. _AI Open_.
* Liu et al. (2024) Xiaoyu Liu, Paiheng Xu, Junda Wu, Jianxin Yuan, Yifan Yang, Yuhang Zhou, Fukxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, et al. 2024. Large language models and causal inference in collaboration: A comprehensive survey. _arXiv preprint arXiv:2403.09606_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Martinez et al. (2016) Felipe Martinez, Sandy Taut, and Kevin Schaaf. 2016. Classroom observation for evaluating and improving teaching: An international perspective. _Studies in Educational Evaluation_, 49:15-29 

James McCeskey, Council for Exceptional Children, Accountability Collaboration for Effective Educator Development, and Reform. 2017. _High-leverage practices in special education_. Council for Exceptional Children Arlington, VA.
* Monroe et al. (2008) Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. _Political Analysis_, 16(4):372-403. Montague (1992) Marjorie Montague. 1992. The effects of cognitive and metacognitive strategy instruction on the mathematical problem solving of middle school students with learning disabilities. _Journal of learning disabilities_, 25(4):230-248. Nazaretsky et al. (2023) Tanya Nazaretsky, Jamie N Mikkeska, and Beata Beigman Klebanov. 2023. Empowering teacher learning with ai: Automated evaluation of teacher attention to student ideas during argumentation-focused discussion. In _LAK23: 13th International Learning Analytics and Knowledge Conference_, pages 122-132. Raschka (2023) Sebastian Raschka. 2023. Finetuning llms withlora and olora: Insights from hundreds of experiments. [https://lightning.ai/pages/community/lora-insights/](https://lightning.ai/pages/community/lora-insights/). Accessed: 2024-03-05. Reilly and Schneider (2019) Joseph M Reilly and Bertrand Schneider. 2019. Predicting the quality of collaborative problem solving through linguistic analysis of discourse. _International Educational Data Mining Society_. Rosenshine and Furst (1971) Barak Rosenshine and Norma Furst. 1971. Research on teacher performance criteria. _Research in teacher education_, pages 37-72. Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. _arXiv preprint arXiv:1910.01108_. Schlotterbeck et al. (2021) Danner Schlotterbeck, Pablo Uribe, Abelino Jimenez, Roberto Araya, Johan Van der Molen Moris, and Daniela Caballero. 2021. Tarta: teacher activity recognizer from transcriptions and audio. In _Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Ultrecht, The Netherlands, June 14-18, 2021, Proceedings, Part I 22_, pages 369-380. Springer. Suresh et al. (2021) Abhijit Suresh, Jennifer Jacobs, Vivian Lai, Chenhao Tan, Wayne Ward, James H Martin, and Tamara Sumner. 2021. Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application. _arXiv preprint arXiv:2105.07949_. Tan and Demszky (2023) Mei Tan and Dortotya Demszky. 2023. Sit down now: How teachers' language reveals the dynamics of classroom management practices. Tuvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhoasale, et al. 2023. Liama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_. Tu et al. (2020) Liftu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. _Transactions of the Association for Computational Linguistics_, 8:621-633. Wang et al. (2024) Deliang Wang, Yang Tao, and Gaowei Chen. 2024. Artificial intelligence in classroom discourse: A systematic review of the past decade. _International Journal of Educational Research_, 123:102275. Wang and Demszky (2023) Rose Wang and Dortotya Demszky. 2023. Is ChatGPT a good teacher coach? measuring zero-shot performance for scoring and providing actionable insights on classroom instruction. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_, pages 626-667, Toronto, Canada. Association for Computational Linguistics. Wang et al. (2023) Rose E Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dortotya Demszky. 2023. Step-by-step remediation of students' mathematical mistakes. _arXiv preprint arXiv:2310.10648_. Wilson and Conyers (2016) Donna Wilson and Marcus Conyers. 2016. _Teaching students to drive their brains: Metacognitive strategies, activities, and lesson ideas_. Ascd. Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_, pages 38-45. Wragg (2011) Ted Wragg. 2011. _An introduction to classroom observation (Classic edition)_. Routledge. Xu et al. (2020) Shiting Xu, Wenbiao Ding, and Zitao Liu. 2020. Automatic dialogic instruction detection for k-12 online one-on-one classes. In _Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6-10, 2020, Proceedings, Part II 21_, pages 340-345. Springer. Yang et al. (2023) Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. _arXiv preprint arXiv:2302.08081_. Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. _Advances in neural information processing systems_, 32 

Eric Zelikman, Wanjing Ma, Jasmine Tran, Diyi Yang, Jason Yeatman, and Nick Haber. 2023. Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiency. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2190-2205, Singapore. Association for Computational Linguistics.
* Zhang et al. (2024) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024. Benchmarking large language models for news summarization. _Transactions of the Association for Computational Linguistics_, 12:39-57.
* Zhou et al. (2023) Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. 2023. Explore spurious correlations at the concept level in language models for text classification. _arXiv preprint arXiv:2311.08648_.

## Appendix A Additional MQI Variables

For Richness of the Mathematics, we have _LINK_ that measures students' and teachers' explicit linking and connections between different representations of a mathematical idea or procedure; _EXPL_ that evaluates the quality of the teacher's mathematical explanations; _MMETH_ that evaluates the quality when multiple procedures or solution methods are discussed. For Errors and Imprecision, we have _MAJERR_ that captures major mathematical errors and _LCP_ that captures when a teacher's utterances cannot be understood. For Student Participation in Meaning-Making and Reasoning, we have _SMQR_ that measures how students engage in mathematical thinking that has features of important mathematical practices; and _ETCA_ that captures student engagement in tasks in which they think deeply and reason about mathematics, regardless of the initial demand of the curriculum/textbook task or how the teacher sets up the task for students.

The statistics and results for these additional MQI variables are shown in Table 8 and Table 9.

## Appendix C Training Details

For BERT, DistilBERT, XLNet, and RoBERTa, the model checkpoints are bert-large-uncased, distilbert-base-uncased, xlnet-large-cased, and roberta-large. we used a fixed set of hyperparameters, where the learning rate is \(2\times 10^{-5}\) and batch

\begin{table}
\begin{tabular}{l r c c c c c} \hline \hline \multirow{2}{*}{Variable} & \multicolumn{3}{c}{Three-way} & Binary \\ \cline{2-7}  & Majority F1 & F1 & Spearman & Majority F1 & F1 \\ \hline LINK & 25x0 & 53±1 & 0.46±0.02 & 37±0 & 71±1 \\ EXPL & 25x0 & 46±2 & 0.32±0.02 & 38±0 & 66±2 \\ MMETH & 29±0 & 51±3 & 0.39±0.03 & 44±0 & 70±1 \\ MAJERR & 28±0 & 37±1 & 0.12±0.03 & 48±0 & 55±1 \\ LCP & 30±0 & 37±1 & 0.15±0.04 & 46±0 & 58±1 \\ SMQR & 27±0 & 45±2 & 0.26±0.02 & 41±0 & 62±1 \\ ETCA & 24±0 & 48±2 & 0.32±0.01 & 36±0 & 64±0 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Spearman correlation between each pair of raters in the SimSE dataset. \(N\) denotes sample size and \(-\) indicates there is only one type of rating in the samples that two raters annotated for the component, resulting in invalid correlation values.

\begin{table}
\begin{tabular}{l r r l l l l} \hline \hline \multirow{2}{*}{Variable} & \multicolumn{3}{c}{Three-way} & \multicolumn{3}{c}{Binary} \\ \cline{2-7}  & \multicolumn{1}{c}{Majority F1} & F1 & Spearman & Majority F1 & F1 \\ \hline LINK & 25x0 & 53±1 & 0.46±0.02 & 37±0 & 71±1 \\ EXPL & 25x0 & 46±2 & 0.32±0.02 & 38±0 & 66±2 \\ MMETH & 29±0 & 51±3 & 0.39±0.03 & 44±0 & 70±1 \\ MAJERR & 28±0 & 37±1 & 0.12±0.03 & 48±0 & 55±1 \\ LCP & 30±0 & 37±1 & 0.15±0.04 & 46±0 & 58±1 \\ SMQR & 27±0 & 45±2 & 0.26±0.02 & 41±0 & 62±1 \\ ETCA & 24±0 & 48±2 & 0.32±0.01 & 36±0 & 64±0 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Test set performance of selected models for the additional seven MQI variables in the NCTE dataset. F1 scores are in percentage. Inputs use teacher text only.

\begin{table}
\begin{tabular}{l r r l l l l} \hline \hline \multirow{2}{*}{Variable} & \multicolumn{3}{c}{Three-way} & \multicolumn{3}{c}{Binary} \\ \cline{2-7}  & \multicolumn{1}{c}{F1 (\%)} & Spearman & F1 (\%) \\ \hline LINK & 50±1 & 0.41±0.02 & 70±1 \\ EXPL & 46±1 & 0.31±0.03 & 65±2 \\ MMETH & 50±3 & 0.36±0.02 & 68±1 \\ MAJERR & 37±0 & 0.13±0.01 & 56±1 \\ LCP & 37±1 & 0.13±0.02 & 57±1 \\ SMQR & 45±0 & 0.26±0.03 & 62±1 \\ ETCA & 46±1 & 0.30±0.02 & 65±0 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Test set performance of selected models for the additional seven MQI variables in the NCTE dataset. Inputs are in the transcript format.

 

size is \(4\). Each model is trained for \(6\) epochs and the checkpoint with the best F1 performance on the Dev set is selected for comparisons with other models.

For Llama2, we experimented with \(7\)B and \(13\)B model versions. However, we did not observe satisfying performances with these models after several trials with different training paradigms (i.e., QLoRA and P-tuning) and hyperparameter settings. For QLora, We took inspiration from Raschka, [2023] and experimented {\(r=8,\alpha=16\)}, {\(r=16,\alpha=32\)}, {\(r=128,\alpha=256\)}, {\(r=256,\alpha=512\)}, and {\(r=512,\alpha=1024\)}. Here we report the hyperparameters for the results with QLoRA as shown in Table 12 and Table 13. The hyperparameters for QLoRA are \(r=256,\alpha=512\) with a dropout rate of \(0.1\). The learning rate for training is \(2\times 10^{-4}\), batch size is \(2\) with gradient accumulation step \(4\), trained for \(5\) epochs. Each training sample is converted to the following instruction-following prompt: Transcript: {transcript} Rating: {rating} and the prompt for test samples is Transcript: {transcript} Rating: All models are trained on one RTXA6000 GPU.

## Appendix D Additional Dev Set Performance

This section shows the Dev set performances for the metacognitive modeling components in the SimSE dataset in the binary setting (Table 11), and for the MQI variables from the NCTE dataset, in the binary (Table 12) and three-way (Table 13) settings.

## Appendix E Qualitative Analysis of Summarization with ChatGPT

Figure 4 shows a qualitative example of using ChatGPT with a summarization prompt in the first stage of the proposed two-stage prediction strategy.