“Mistakes Help Us Grow”: Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms
Authors: Kunal Handa
Margarett Clapper
Jessica Boyle
Rose E Wang
Diyi Yang
David Yeager
Dorottya Demszky
Venue: EMNLP 2023 Main (2023)
Keywords: Growth mindset
Language models
Education
Applications

Abstract: Teachers’ growth mindset supportive language (GMSL)—rhetoric emphasizing that one's skills can be improved over time—has been shown to significantly reduce disparities in academic achievement and enhance students' learning outcomes. Although teachers espouse growth mindset principles, most find it difficult to adopt GMSL in their practice due the lack of effective coaching in this area. We explore whether large language models (LLMs) can provide automated, personalized coaching to support teachers' use of GMSL. We establish an effective coaching tool to reframe unsupportive utterances to GMSL by developing (i) a parallel dataset containing GMSL-trained teacher reframings of unsupportive statements with an accompanying annotation guide, (ii) a GMSL prompt framework to revise teachers’ unsupportive language, and (iii) an evaluation framework grounded in psychological theory for evaluating GMSL with the help of students and teachers. We conduct a large-scale evaluation involving 174 teachers and 1,006 students, finding that both teachers and students perceive GMSL-trained teacher and model reframings as more effective in fostering a growth mindset and promoting challenge-seeking behavior, among other benefits. We also find that model-generated reframings outperform those from the GMSL-trained teachers. These results show promise for harnessing LLMs to provide automated GMSL feedback for teachers and, more broadly, LLMs’ potentiality for supporting students’ learning in the classroom. Our findings also demonstrate the benefit of large-scale human evaluations when applying LLMs in educational domains.

"Mistakes Help Us Grow":
Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms
Kunal Handa'
Margaret Clapper Jessica Boyle"
Rose Wang*
Diyi Yang'
David S Yeager Dorottya Demszky"
Brown University
UT Austin
Vanderbilt University
*Stanford University
kunal_handaaellumi. brown. edu ddemszky@stanfford edu
Abstract
time. Research on GMSL has shown positive ef- fects on crucial aspects of student learning, such resilience, motivation, and performance (Dweck, 2006). This language also helps reduce educa- tional gaps linked to race, ethnicity, and social class (Hecht et al., 2023; Canning et al., 2019).
Teachers' growth mindset supportive language (GMtt)-rhetoric emphasizing that one's skills can be improved over time-har been shown to significantly reduce disparities in academic achievement and enhance students' learning outcomes. Although teachers espouse growth mindset principles, most find it diffi- cult to adopt GMSL in their practice due the lack effective coaching in this area. We ex- plore whether large language models (LLMs) can provide automated, personalized coaching support teachers' use of GMSL. We estab- lish an effective coaching tool to reframe un- supportive utterances to GMSL by developing parallel dataset containing GMSLL-traine teacher reframings of unsupportive statements with an accompanying annotation guide, (ii) GMSL prompt framework revise teachers' unsupportive language, and (iii) an evaluation framework grounded psychological theory for evaluating GMSL with the help of students and teachers." We conduct a large-scale eval- uation involving 174 teachers and 1,006 dents, finding that both teachers and students perceive GMSL-trained teacher and model re- framings effective in fostering growth mindset and promoting bhallenge-seeking be- havior, among other benefits. We also find that model-generated reframings outperform those from the GMSL--rained teachers. These results show promise for harnessing LLMs to provide automated GMSL feedhack for teachers and, more broadly, LLMs' potentiality for support- ing students' learning the classroom. Our findings also demonstrate the benefit of large- scale human evaluations when applying LLMs in educational domains.
Despite many teachers embracing the impor- tance GMSL, most find difficult to imple- ment (Dweek and Yeager, 2019) due to miscon- ceptions as to what constitutes a growth mindset and the lack of effective coaching in this domain. Prior work has shown that many well-intentioned teachers often believe they are using GSML (for example, through simply adding more positive ex- pressions-- believe you"), without comprehen- sively understanding the tenets of GMSL needed to apply it in practice (Dweek and Yeager, 2019; Dweck, 2016). Helping teachers adopt growth mindset principles requires coaching and individ- ualized feedback that is currently not available at scale (Hecht et al., 2023).
Recent advancements in Large Language Mod- els (LLMs) have led to unparalleled abilities for models to follow instructions, which suggests po- tential for prompting them with growth mindset principles generate suggestions for teachers. Al- though unexplored with GMSL, L.MSs have shown success as automated educational tools in other set- tings. For example, KhanMigo (Khan Academy, 2023), an LLM-assisted tutoring bot was success- fully prompt-engineered to follow principles of So- cratic questioning and not provide the answer to students directly. At the same time, recent work has also shown that ChatGPT a model from the same family KhanMigo - was not successful at providing teachers with high-level, insightful pedagogical feedback (Wang and Demszky, 2023). Thus, the question whether LLMs are capable of generating GMSL for teachers remains open question.
1 Introduction
Growth mindset supportive language (GMSL) refers to rhetoric highlighting the potential for in- dividuals to develop their skills and abilities over
htttps: htt.h///..cmmkuaaann///ggrtttttt wwiiddt
In this paper, we seek to assess the ability of

---

GPT-4, a state-of-the-art LLM, to generate GMSL. In doing so, we make several contributions. First, working with teachers trained in growth mindset principles (henceforth referred to as "expert teach- ers"), we develop an annotation framework to guide teachers' understanding of GMSL. Leverag- ing a dataset of elementary math classroom tran- scripts (Demszky and Hill, 2023), we further collab- orate with these teachers to create open-source parallel dataset that pairs 100 unsupportive utter- ances from the transcripts with their expert-revised GMSL reframings.
Our findings not only highlight the need for en- couraging GMSL in classrooms but also offer valu- able insight into the role LLMs can provide in this endeavor. Furthermore, our results highlight the im- portance of extensive student--cntered evaluations when examining the usefulness of LLMs in edu- cational contexts, offering framework for future studies in this area. We demonstrate that through closely working with students and teachers when developing and evaluating LLM-based technolo- gies, we can help ensure their positive influence within the classroom.
Having outlined the tenets of GMSL, wc prompt engineer GPT-4 (OpenAI, 2023) to generate GMSL reframings of the unsupportive utterances. We utilize recent advancements in prompting and self-critique (Huang ct al., 2022) techniques de- sign detailed, GMSL-specciic prompt. Via our prompt design, we ensure that the reframings are personalized for each specific classroom instance and adhere to our GMSL tenets.
2 Related Works
2.1 Growth Mindset Supportive Language
Growth mindset refers the belief that intelligence and abilities are fixed and can be developed over time (Dweck, 2006; Yeager et al., 2016). This per- spective emphasizes the potential for individuals to improve their skills and intelligence through effort, learning strategies, and overcoming challenges.
Critically, also propose novel framework for evaluating GMSL, assessing teacher and model responses by recruiting 174 teachers and 1,006 students (13-15 years old) to evaluate the perceived impact that these responses would have on students. Our participants are given excerpts from a conversation between teacher and stu- dent and asked to rate the teachers on their (1) per- ceived growth mindset, (2) promotion of challenge- taking behavior, (3) alleviation of students' feelings of shame, and (4) increase of respect felt by stu- dents. Some excerpts contain the original teacher responses whereas others contain reframings (ci- ther by the expert teachers or model). We adapt these questions from instruments developed and tested by growth mindset experts (Hecht et al., 2022).
Previous studies have revealed that incorporat- ing GMSL positively and lastingly influences stu- dents' academic performance; for example, Yea- ger et al. (2019) found that exposing students to a growth mindset in a short one-hour conversation improved the grades among low-achieving students and increased enrollment to advanced mathematics courses (see also Schmidt et al. (2016) and Muenks et al. (2020) for similar findings). Results were consistent across achievement levels and countries (OECD, 2019). Additional research has also shown that growth mindset can positively impact stu- dents' engagement, persistence, and resilience the face of academic challenges (Good et al., 2003; Yeager et al., 2014; Dweck and Yeager, 2019). In contrast, when an educator does not foster and sup- port student's mindset with their own growth mindset language and behaviors, students are less likely on their growth mindset beliefs (Yea- ger et al., 2021). Furthermore, recent work has also validated the use of survey-based evaluation se- tups when judging the long-term impact of GMSL intervention: Hecht et al. (2022) found that stu- dents' hypothetical judgments of teachers' mindset supportive language track with teachers' actual be- havior and students' learning outcomes.
In our evaluation, we find that both expert teach- ers' and model reframings better support growth mindset, outperforming the original utterances across all metrics. We discover that model re- framings slightly surpass even the expert teachers' reframings and that the model with self-critique performs best. We also conduct a lexical analysis, identifying that all reframings include more explicit growth mindset supportive language than the origi- nal (e.g. empathetic validation, positioning oneself as collaborative resource, autonomy supportive language), with models including even more spe- cific diction compared to teacher utterances.
These prior works highlight the importance of using GMSL in classrooms. Yet, providing feed- back that encourages educators to incorporate this growth-mindset-rrient feedback can be incredi-

---

Original Utterance
Teacher (Reframe): Erase what you have please. Everything. And want you look again, oksy? Everything. Everything. Everything.
Conversation Context
Expert Teacher Reframing
2
Teacher (Reframe): Hmmm, okey! Three great answer. But let's look again. We're going to with a clean slate and see helps. erase everything.
Teacher: Mm-hmm, so erase your fraction and Okay. You're still wrong, So many pieces is that?
Model Reframing (0-critique)
Teacher (Reframe): Thank you for Irying, let's work through this logether. Erase your fraction, and let's take closer look number line. We can learn from this and better understand how to locate fractions.
Student: Three.
Model Reframing (1-critique)
Teacher (Reframe): Thank you for sharing your thoughts, It's great that you're actively participating! Let's erase the fraction and work together to understand how to locale fractions the number line. We'll figure this a team and conlinue improving.
Perceived Teacher Mindset
Challenge-seeking Behavior
Perceived Shame
Perceived Respect
1.0
186
Figure 1: Examples from dataset illustrating different responses student's 's mistake by teachers (Original and Expert) and GPT-4 (with and without self-critique). The figure also shows students' ratings of these responses along four dimensions from our evaluation framework: Perceived Teacher Mindset, Seallenge-Seeking Behavior (choice of easy vs hard assignments), Perceived Shame, and Perceived Respect.
bly difficult do scale. Hunkins et al. (2022) fo- cus on detecting GMSL but do not explore refram- ing unsupportive language at scale. Recent work like Ziems et al. (2022) explores the use of posi- tive reframing to generate wwwwtthmindset-orieee language. However, their study relies on social media data, which requires different approach to reframing compared to educational domain.
Kelly et al., 2018; Jensen al., 2020; Wang and Demszky, 2023), For example. Suresh al. (2021) provides feedback to teachers on their teaching moves, such as how frequently the teacher revoices student's 's idea or how frequently the teacher asks students to reason aloud. Jacobs et al. (2022) pro- vides evidence that K-12 math teachers receive this kind of feedback positively. A similar tool, M- Powering Teachers, provides feedback to teachers on their uptake of student ideas and demonstrates effectiveness in the 1 1-on-1 lcarning setting (Dem- szky and Liu, 2023) and online group instruction (Demszky et al., 2023). Altogether, these findings show a positive impact of cost-effective automated tools in educational contexts.
Our research builds upon the demonstrated ben- eficial impacts GMSL on students' long-term learning outcomes. We aim to explore the utility of LLMs as an effective and individualized method for coaching teachers to adopt growth mindset princi- ples through large-scale survey- based evaluation framework.
2.2 Automated Feedback to Educators
3 Modeling GMSL
Prior works on automated feedback tools provide analytics on student engagement and progress (Su et al., 2014; Schwarz et al., 2018; Aslan et al., 2019; Bonneton-Boor Botté et al., 2020; Alrajhi et al., 2021, among others). These tools enable teach- ers monitor student learning and intervene as needed. Recent NLP advances are able to provide teachers feedback on their classroom discourse, promoting relf-reflection and instructional devel- opment (Samei et al., 2014; Donnelly et al., 2017;
We utilize dataset of elementary math classroom transcripts (Demszky and Hill, 2023) and GPT-4 prompting techniques to create four different types of responses:
ORIGINAL: Teacher utterance from the class- room transcript labeled as unsupportive by teachers trained in GMSL.
2. EXPERT: Reframings of unsupportive exam- ples to GMSL by teachers trained in GMSL.

---

System Text
Model Reframing (0-critique)
You helpful assistant specializes correcting teachers' language be supportive students. Your lask direclly revise the teacher's utterance use mindsat-supportive language. Below are some tenets mindsel-supportiv language...
Teacher (Reframe): Thank you trying that solution. Let's erase your current answer, and together can take another look the problem and consider other ways approach believe can figure this out team.
1. Empathic validation...
2. Reappraisal of affect...
3. Seek to Understand...
4. Position yourself as collaborative resource...
5. Provide hope for change the future...
6. supporiv language throughout...
Prompt:
Make the previous response more mindsel supportive.
Prompt Text
This elementary grade
working division. They manipulatives build arrys separate manipulative into equal groups.
The following example of an in-class conversation between teacher and student contains unsupportive languago. Directly revise the teacher's last utterance, marked with "Reframe)". to be more mhindsel-supportee
Model Reframing (1 (1-critique)
Teacher (Reframe): appreciate your effort attempting that solution. Let's gently erase what you've written and revisit the problem together. We can explore different approaches better understand it. Remember, we're in this together and can learn from each attempl.
Teacher: Mm-hmm, your fraction and one. Oksy. You're still wrong. So zero and one. How many pieces is that?
Student: Three
Teacher (Reframe): Erase what you have please. Everything, And want you to look at again, okay? Everything. Everything. Everything.
Figure Our prompt design employed to elicit GMSL, reframings using GPT-4 as described in Section 3.2.
3. MODEL (O-CRITIQUES): Reframings of un- supportive examples to GMSL by GPT-4.
have 8+ years of experience and have experience teaching elementary grade levels. Both teachers women; one of them identifies Hispanic and the other white. In addition to the training these teachers had already received in growth mindset, our team provided the teachers additional guidance for GMSL principles. We work closely with the teachers and compensate them at $50/hr.
4. MODEL (1-CRITIQUE): Reframings of un- supportive examples to GMSL by GPT-4 prompted self-improve upon responses.
The subsequent sections detail the development of these four types of responses.
Identifying unsupportive utterances. To com- pile dataset of unsupportive teacher utterances (ORIGINAL), ask expert teachers to annotate all teacher utterances in random sample of tran- scripts for whether they (I) respond to student mistake, introduce a task or debrief task and (IT) so, whether they contain unsupportive language as defined by the amnotation guide described below. We follow this process until we reach 100 examples of unsupportive teacher utterances. These exam- ples are derived from eight transcripts and 100 total teacher utterances meeting criterion (I), indicating high rate (22.8%) of unsupportive language in the transcripts.
3.1 Data
Source. We source our data from the National Center for Teacher Effectiveness (NCTE) 4-5th grade math classroom transcripts (Demszky and Hill, 2023), representing 317 teachers across four school districts in New England, predominately serving low-income students of color. Transcripts are anonymized, where student and teacher names are replaced with terms like "Student A" and "Teacher B".
Expert teachers. We recruit two teachers who are alumni of the Hollyhock Fellowship Program?; a competitive two-year teacher professional learn- ing program that works with highly motivated teachers serving in Title schools. Both teachers
We worked closely with the expert teachers to create GMSL Guide that documents their process for reframing teacher utterances to
Reframing.
"https://cset.se sta.foor..tttt.oodrr.ooo adu/pd/hollyhock

---

be mindset supportive (EXPERT). The Guide in- corporates suggestions for teachers on fostering a growth mindset in classrooms. These suggestions include providing empathetic validation by express- ing gratitude to students for asking questions and demonstrating effort, positioning oneself as a col- laborative resource by utilizing inclusive terms like "we" and "us" when engaging in mooblem-solving activities with students, and seeking to understand students' thinking by asking thoughtful questions. After multiple rounds of pilot annotations and feed- back sessions, the two expert math teachers utilized the Guide to independently identify examples of unsupportive language and subsequently reframe them to foster more supportive mindset while preserving the task-related meaning of the message (a more detailed description of the process can be found in the Appendix, Section A).
Recent advances in prompting techniques have demonstrated the value of self-critique in LLMs- prompting the model to improve upon its own response- rexponse-enabling better reasoning about tasks (Huang et al., 2022). We utilize this process of self-critique when reframing, evaluating the model generations with one self-critique and zero self- critiques. For generations with one self-critique. prompt the model to improve upon its initial response (Make the previous response more mind- set supportive), evaluating only the final generation (MODEL (1 (I-CRITIQUE)).
Evaluation
We develop framework, grounded in psycholog- ical theory, for evaluating GMSL, generated re- sponses with the help of teachers and students (Sec- tion 4.1). In addition, we also conduct a lexical analysis to understand linguistic differences among the different types of responses (Section 4.2).
3.2 Designing Prompt to Elicit GMSL
We design a GMSL-oriented prompt to generate re- framings of teachers' unsupportive utterances. We first engineer GMSL-specific information given to the model via its "system text" hexh--ighhlevell instruc- tions that can contain extra content to guide the model's 's responses (Microsoft, 2023).
4.1 Teacher and Student Surveys In NLP, human evaluations of text generation tend to focus on rating the quality of generated text rather than the impacl the text may have on the tar- get recipient. In social psychology, impact evalua- tions are paramount, as researchers seek to measure whether their intervention had the desired effect on their participants. Furthermore, many psychologi- cal constructs such growth mindset are complex and subjective and hence they cannot be evaluated directly e.g. asking raters "How much growth mindset does this text convey?" is likely to led to confusion and unreliable annotations.
Here, the model receives both instructions speci- fying the model's 's role in correcting teachers' lan- guage to be more supportive of students as well as the GMSL Guide, outlining the tenets of GMSL. When prompting the model, we provide the model with (1) the classroom context in which the conver- sation to be reframed took place (c.g. This is an elementary grade math class working on geometry. They start the lesson with reviewing multiplication facts. They then begin to learn about classifying triangles focusing on types of angles.), (2) explicit instructions of the task: The following example of an in-class conversation between a teacher and student contains unsupportive language. Directly revise the teacher's last utterance, marked with "( "Reframe)", to be more mindset supportive, and (3) the utterance requiring reframing. demarcated by "(Reframe)", preceded by the two utterances directly prior. We use the model's generated re- sponse in our evaluation (MODEL (O-CRITIQUES)). This setup closely follows that of the expert teacher annotators; but, due condext-window limits, we were unable to provide the entire conversation lead- ing up to the utterance, as was seen by the expert teachers.
Measures of GMSL. To obtain measurements of GMSL, we adapt survey items validated as part of large-scale growth mindset interventions (Hecht et al., 2022). We administer surveys both teach- ers' and students' to probe their perceptions of the classroom and the teacher. The surveys include items estimating four constructs given an excerpt of a student-teaches interaction: (1) teacher's per- ceived growth mindset, (2) teacher's 's promotion of challenge-taking behavior, (3) shame felt by students in the teacher's 's class, and (4) respect felt by students in the teacher's class. These constructs operationalize growth mindset culture and were designed and tested by experts (Hecht et al., 2022). Figure 3 shows the key items on the student survey corresponding to the four aforementioned constructs. The teacher survey included largely

---

BASELINE MOTIVATIONAL FRAMEWORK (control variable)
b1 - To be honest, you can't really change how intelligent you are.
b2- You can always substantially change how intelligent are.
b3 When you try really hard subject in school, it means you can't really be good at that subject.
PERCEIVED TEACHER MINDSET (outcome variable)
ptm1 (r) This math teacher seems to bel sele that only some students will understand the hardest problems.
ptm2 (r) This math teacher seems to believe that student's can't really change how good they math.
SHAME (outcome variable)
shame (r)- If I
in this teacher's class, would feel embarrassed if got problem wrong the board in front of my peers. RESPECT (outcome variable)
respect In this class, the teacher would treat with respect. CHALLENGE-SEEKING BEHAVIOR (outcome variable)
csb Imagine that, later today or tomorrow, this math teacher handed out two extra credit assignments. You got choose which either one. One choice is an easy review-it has math problems you already know how to solve, and you will probably get most of the answers right without having to think very much. It takes minutes. The other choice solve, and you will probably get most of the problems wrong, but you might learn something new. It also takes 30 minutes. you had to pick right now, which would you pick?
do. You get same number of points
trying
a hard challenge-it has math problems you don't know how to
Figure 3: Items used in the survey evaluations for adolescents. (r) indicates reverse coded item, where higher scores are "worse".
the same items as the student survey (except csb), with minor wording adaptations. The shame and respect items were adapted so that there were from the student's point of view. Instead, teachers answered the following items: "Students in this teacher's class would probably feel embarrassed if they got problem wrong on the board in front of their peers." and "In this class, the teacher would treat students with respect." Only students were asked respond to hypothetical thallenge-taking item (csb) in which they imagine an opportunity receive extra-credit in the teacher's class. When students perceive the fictional teacher as having more of a fixed mindset, they are more likely to ac- cept the easy extra-credit assignment, regardless of the student's own bascline growth mindset beliefs.
work and consist of items b1, b2, and b3 in Figure 3. We responses these items control vari- ables in our analyses.
Then, they were shown four transcript segments one by one. Each segment included different response type (ORIGINAL, EXPERT, MODEL (0- CRITIQUES), MODEL (1-CRITIQUE). The seg- ments the participants saw and the order they viewed them in were randomized. Participants were not told to directly compare the segments and generally did not see reframed utterances from the same segment. Following each segment, partic- ipants were asked to respond to the GMSL items described above (ptm1, ptm2, shame, respect and for students, csb). These measures are used in later analyses as outcome variables. Finally, consistent with standard psychology practices, we ask all par- ticipants about their demographics. Participants responded to items about their self-identified gen- der, race and ethnicity, and mother's 's highest level of education (a socio-cconomic measure). At the end of the survey, after responding to all measures, participants were told that some of the transcripts they viewed were edited by artificial intelligence.
All items in Figure 3 except csb were rated on a 1-6 Likert scale with the choices being Strongly Disagree, Disagree, Mostly Disagree, Mostly Agree, Agree, and Strongly Agree. The csb item was rated binary scale, indicating if a student would choose easy hard assignments.
Survey design & flow. The full teacher and stu- dent surveys are included among the Supplemen- tary Materials, After the consent form, the survey first asked participants about their baseline growth mindset beliefs. These items are generally referred to as participants' baseline motivational frame-
We recruit 174 teachers and 1,006 students as evaluators. As for teachers, we recruit them from two pools: 16 teachers are alumni the Hollyhock Fellowship Program, the same pro-
Participants.

---

fessional learning program that we recruited our two expert teachers from (the two experts are not included in this pool). The other 158 teachers are recruited via Prolific (Palan and Schitter, 2017). We recruit students through Dynata', a service that helps recruit participants for surveys meeting speci- fied demographic criteria. The youngest population they are able to recruit is 13-15, and we work with this group. Our study is approved under IRB ap- proval number: HRP-UT902 STUDY0000138)
environment. For example, GMSL keywords such as "challenging" and "together." or first-person pro- nouns like "we" and "us," might be more prevalent in responses perceived to be said by a teacher with a growth mindset.
We compute the log odds ratio, latent Dirich- let prior, measure defined in (Monroe al., 2008) to estimate the distinctiveness of an n-gram ap- pearing in a particular response type (ORIGINAL, EXPERT, MODEL (O-CRITIQUES), or MODEL (1- CRITIQUE) as opposed to all other response types. The log odds estimate is normalized to standard deviation units (c.g. a score of 2 can be interpreted as the n-gram being 2 standard deviations "more associated" with a particular response type than others). We preprocess the data using Python's NLTK package (Bird al., 2009) for tokenization, lowercasing. and lemmatization, while also diseard- ing stop words and non-alphanumeric tokens. We use the Gensim Phrases Python package (Rehurek and Sojka, 2011) to retrieve unigrams and frequent bigrams in the dataset.
Analysis of survey results. We assess the ex- tent to which the four types of model responses contain GMSL by analyzing teacher and student perceptions collected through the survey. Follow- ing standard procedures in psychology, we control for raters' demographic features and their baseline motivational framework when estimating average ratings for each model response. We use the fol- lowing model to estimate these conditional means:
y = 30 + D31 + B32 +
(1)
where y is a vector representing the outcome, D is a matrix of rater demographic characteristics, B is a matrix of responses to baseline motivational framework items, So. 31 and 32 are vectors of un- known parameters to be estimated and e is a vector of residuals. We report the conditional mean com- puted when all variables D and B are at their mean.
Results
5.1 Teacher and Student Surveys
Table shows the results of the student and teacher evaluations. One high-level take-away is that rat- ings for the different items corroborate one another, surfacing a systematic ranking among the differ- ent response types in terms of GMSL: ORIGINAL EXPERT MODEL (O-CRITIQUES) < MODEL (I-CRITIQUE). We find the model responses out- perform the ORIGINAL teacher response by a large margin along all dimensions, with MODEL (I-CRITIQUE) doing 24-85% better and MODEL (O-CRITIQUES) doing 23-75% better. Notably, MODEL (1-CRITIQUE) almost doubles students' likelihood of choosing hard assignments (hypothet- ical challenge-seekingg behavior), which is the most important GMSL student outcome, in contrast ORIGINAL. As expected, EXPERT also scores bet- ter (by 17-50%) in terms of GMSL than the ORIG- INAL. However, unexpectedly, the models also consistently receive better ratings than the EXPERT teacher response, showing 8- 10% increased per- ccived growth mindset, 9-15% decreased shame, 9-10% increased respect, and importantly, 23% increased challenge-sceking behavior. We also find that self-critique facilitates GMSL, increas- ing ratings for MODEL (1 (1-CRITIQUE) compared to MODEL (0-CRITIQUES) by 1-6%. These re-
The demographic features D include binary in- dicators for the rater identifying as female, Asian, Black/Afrrcan American, White/Caucasian, His- panic/Latinx, Pacific Islander/Hawaiiann dummy coded variables for mother's education level and for teachers, whether they were part of the Pro- lific pool. For the outcomes and motivational framework B, we use the numerical Likert scale responses. To obtain a single outcome for per- ceived teacher mindset, we first reverse the scales for ptml and ptm² (so that greater values reflect more growth mindset) and average ratings for the two items. We keep the other responses as is.
4.2 Lexical Analysis
In addition to surveys, we also conduct lexical analysis understand the linguistic differences among different response types. These observa- tions can contribute a qualitative explanation of student and teacher evaluations regarding the lan- guage that nurtures growth mindset classroom
Phttps: //wwww hhttps://www.dynatt.comm com/

---

Teacher Evaluation
Student Evaluation
Perceived Teacher Mindset
Perceived Teacher Mindset
Challenge- Seeking
Response Type
Shame
Respect
Shame
Respect
ORIGINAL
3,42
4.15
3.77
3.3
4.4
3,6
0.2
EXPERT
4.38
3.11
4.73
3,94
3,67
4.47
0.3
MODEL (0-CRITIQUES)
4.69
2.7
5.03
4.14
3.39
4.78
0.35
MODEL (I-CRITIQUE)
4.8
2.63
5.19
4.25
3,34
4.89
0.37
Table I: Conditional means computed based teacher (n=174) and student (n=1,006) survey responses. The values show the average ratings of the four response types along our key outcome dimensions, while controlling for participant's demographics and baseline motivational framework. Results indicate that the model responses are consistently rated as more growth mindset supportive compared to the original and even the expert teacher responses, with MODEL (1 (I-CRITIQUE) performing slightly better than MODEL (O-CRITIQUES).
ORIGINAL
EXPERT
MODEL (0-CRITIQUES)
MODEL (I-CRITIQUE)
N-grams
Log Odds
N-grams
Log Odds
N-grams
Log Odds
N-grams
Log Odds
guy
6.40
tool
6.23
let work
7.08
teacher appreciate
6.72
let_see
5,95
let_look
6,23
keep_mind
5,06
let_collaborate
6.38
sit
5.80
teacher_oo
6.05
giving_try
5.06
lct consider
5.62
everybody
5,80
let_o
6.05
appreciate input 5,06
great_see
5.33
head
5.80
tell_u
5.82
find find_right
4.77
explore_differee difffeent
5.33
gon na
5.80
complete
5.82
think another
4.77
engaging problem
5.15
teacher_ee _ease
5.61
let_talk
5,54
teacher_alrigh
4.77
let_focus
5.15
supposed
5.39
hmmm
5.54
understand better
4.77
learn grow
5.15
right_think
5.39
great_ance Lasser
5,54
toreene
4.77
continue_exploring
4.93
go_back 5.39
toolbox
5.54
way_ apprpacpcccchhccc
4.77
_eftr_
4.93
Table 2: The n-grams with the highest log odds for each type of response. EXPERT utilizes more GMSL (as defined in the GMSL Guide) compared to ORIGINAL. MODEL (O-CRITIQUES) and MODEL (0-EL(1-4m (1-CRITIQUE) include more GMSL. than the EXPERT.
sults suggest that GPT-4 is able to successfully incorporate and self-improve along growth mindset principles given effective prompts.
especially adept at utilizing GMSL elements. One noticeable pattern is that the models, despite having been provided with context for grade level, use more formal language (e.g. appreciate, engage, collaborate) than the teacher responses, which may be harder for younger students to understand. Thus, this lexical analysis highlights the need for addi- tionally evaluating the grade-appropriatenes of teacher language beyond GMSL and importance LLMs directly with students without teacher in the loop who can adapt language to the learners' contexts, nceds, and background.
5.2 Lexical Variation Among Responses
Our linguistic analysis highlights differences in word usage across the four response types. Table 2 presents the top 10 n-grams ordered by their log odds ratio. In comparison ORIGINAL., EXPERT responses contain more pronounced examples of GMSL, such as empathic validation with phrases like great answer and inclusive expressions like let's ' look and let's talk. Both types of model re- framings exhibit even more explicit GMSL, as ev- idenced by autonomy supportive language (e.g., let's consider), positioning source (c.g., working together, let's collaborate), hope for change (e.g., continue exploring), and pathic validation (e.g., great effort and appreciate input). These findings suggest that all reframings are more effective in using GMSL compared to original utterances, while model reframings are
Discussion & Conclusion
In this study, we take steps towards helping teach- ers adopt growth mindset supportive language (GMSL) in their classroom. We establish annota- tion guide, create a parallel dataset, develop an tomated coaching tool utilizing GPT-4, and propose a framework grounded psychological theory for evaluating GMSL's effectiveness for both teach- ers and students. Our findings demonstrate that
collaborative re-

---

expert-teacher and model-generated reframings sig- nificantly improved upon the original unsupportive utterances in promoting a growth mindset, encour- aging challenge-seeking, behavior, alleviating feel- ings of shame, and enhancing the perceived respect by students. Intcrestingly, model reframings sur- passed expert-reacher reframings, highlighting the potential of LLMs in fostering GMSL adoption within educational contexts.
by expanding the data sources to include varied sub- jects, age groups, cultural backgrounds, and class- room situations, as well as by studying the effects of GMSL across a broader set of contexts. Further research on LLMs' ability to produce GMSL, across different teacher instruction styles, vernaculars, and personalities also remains necessary. Moreover, further exploration of the cultural and linguistic variability in GMSL implementation, alongside more extensive empirical studies on the long-term impact of GPT-4-based GMSL coaching, could contribute to a more comprehensive understanding of LLMs' utility in education settings.
Our work serves as foundation for eliciting and evaluating the use of GMSL by teachers and LLMs through collaboration with students and teachers. The insights derived from our analysis open av- enues for further inquiry into the role LLMs can play in the future to support teachers' communica- tion strategies and facilitate better student-teacher interactions in the classroom. Future research could build upon this foundation to explore new dimen- sions of GMSL, examine the long-term impact of GMSL coaching, and extend the application of LLMs alongside our psychologically-groon evaluation framework to address other outstanding challenges across various educational domains.
Ethics Statement
The two teachers with whom we collaborated develop the annotation guide and expert reframings were compensated at a rate of $50/hr. We met virtually with these teachers on a weekly basis and maintained available communica- tion channels via email throughout the annotation process.
Annotation.
Evaluation. All participants were compensated above the federal minimum wage. Teachers re- cruited via Prolific were paid $13-20/hr, in line with the platform's 's guidelines. Teachers recruited for evaluation via the Hollyhock Fellowship Pro- gram were paid with $15 Amazon E-gift Cards (rate of $60/hr). During the evaluation process, we provided prompt responses to workers' inquiries (within 24 hours). Within the survey, we also used ethics practices standard for human-subjects re- search. Researchers who were able to access the data and recruit participants were trained on human- subjects research and certified via their institution. This project was approved by an institutional re- view board. Adults were always given informed consent (an example of the consent in the Appendix, Section E) and were given the op- tion to not consent participate. For adolescents, parents were given an informed consent document and adolescents were also briefed on the poten- tial harms (here, we estimated no harms greater than everyday life and discomfort due to boredom or fatigue) before continuing in the survey. Both parents and adolescents were given the option to not participate based on the consent form. Partic- ipants were not required to complete any of the questions complete the survey, and were given the option to skip questions or to leave the survey entirely. For adolescents we collected aggregate
Limitations
While this study offers valuable insights into the adoption of GMSL by teachers and the potential of LLMS in supporting educational communica- tion, it is essential to acknowledge certain limita- tions. First, our analysis focuses on elementary math classroom transcripts, which may limit the gemeralizability of our findings for GMSL adoption across different subjects, age groups, educational levels. Also, our study does not extensively address the potential cultural, linguistic, or regional differ- ences in the understanding and interpretation of GMSL; we do not explicitly investigate the applica- bility of LLMs in this domain across all subject mat- ters or classroom environments. Doing remains necessary to build broadly applicable framework for GMSL across diverse cultural and linguistic backgrounds. Additionally, our evaluations may not capture the complete breadth of students' char- acteristics, interactions, or the diverse spectrum of classroom situations needing different GMSL strategies. We also relied on teachers and students to evaluate the reframings based on hypothetical scenarios. This approach may not fully reflect the long-term impact of GMSL on student outcomes in real-life settings.
be found
Future research could address these limitations

---

semi-identifiable data in the form of demographic data (such as gender, race/ethnicity, and mother's education level). Demographic data is important for evaluation and further analyses on this dataset, and cannot directly identify participant. For adult participants recruited via the Hollyhock Fellowship Program, we collected demographic information and emails to process payment. Emails were re- moved from the dataset soon as payments were completed. For the data collected via Prolific, we only collected demographic data.
Foundation to D.S.Y.
References
I. Alrajhi, A. Alamri, F. D. Pereira, and A. I. Cristea. 2021. Urgeney analysis of learners' comments: An automated intervention priority model for mooc. In International Conference on Intelligent Tutoring Sys- tems, pages 148-160.
S. Aslan, N. Alyuz, C. Tanriover, E. Mete, E. Okur, S. K. D' D'Mello, and A. Arslan Esme. 2019. Investigat- ing the impact of a real- engagement analyties technology in authentic class- rooms, pages 1-12. of the 2019 CHI conference on human factors in computing systems.
In multimodal student
Deployment. Our data, prompt design, experi- ments, and evaluations are intended solely for bet- tering student-teacher interactions and positively influencing students' learning outcomes. Although we took great care in designing our project with each of these aspects with this in mind, we rec- ognize that there may be unintended perverse use cases of our research, such as altering prompts that oppose our outlined goals. We urge parties intending on using any part of our work to align their intentions with those of this research: sup- porting educational experiences for students. Any cases that attempt to deploy this research for commercial gain are unacceptable. We implore in- dividuals who use our data, prompts, experiments, or evaluations to consider and mitigate other soci- etal, ethical, or otherwise deleterious ramifications that may arise.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.".
Nathalie Bonneton- BBttéé, Sylvain Fleury, Nathalie Gi- rard, Maälys Le Magadou, Anthony Cherbonnier, Mickaël Renault, Eric Anquetil, and Eric Jamet. 2020, Can tablet apps support the learning of handwriting? an investigation learning outcomes in kindergarten classroom. Computers & Educcatio, 151:103831.
Elizabeth Canning. Katherine Muenks, Dorainne Green, and Mary C. Murphy. 2019. Stem faculty who believe ability is fixed have larger racial achieve- ment gaps and inspire less student motivation their classes. Science Advances, 5.
Dorottya Demszky and Heather Hill. 2023. The NCTE Transcripts: A dataset of elementary math classroom transcripts.
A Acknowledgements
We would like to thank Alex Tamkin, Cameron Hecht, and Carol Dweck for useful conversations and feedback, as well as all the participants who took our study. We would also like to thank the Hollyhock Fellowship Program alumni with whom we worked to create the GMSL Guide. R.E.W is supported by the National Science Foundation Graduate Research Fellowship. This work was funded partially from an NSF IIS-2247357 to D.Y. Additionally, this work was supported by research grants to the Texas Behavioral Science and Policy Institute, including the National Science Founda- tion under award numbers 1 1761 179 and 2201928 (PI: D.S. Y.), by the National Institutes of Health under award numbers RO1HD084772 (PI: D.S.Y.) and P2CHD042849 (Population Research Center), and by the William and Melinda Gates Founda- tion under awards INV-047751 and INV-004519 (PI: D.S.Y.). This work was also supported by an Advanced Research Fellowship from the Jacobs
Dorottya Demszky and Jing Liu. 2023. M-Powering Teachers: Natural language processing powered feed- back improves 1:1 instruction and student outcomes.
Dorottya Demszky, Jing Liu, Heather C Hill, Dan Juraf- sky, and Chris Piech. 2023. Can automated feedback improve teachers' uptake of student ideas? evidence from randomized controlled trial in a large-scale online course. Educational Evaluation and Policy Analysis.
P. J. Donnelly, N. Blanchard, A. M. Olney, S. Kelly, M. Nystrand, and S. K. D'Mello. 2017. Words mat- ter: Automatic detection of teacher questions live classroom discourse using linguisties, acoustics and context. 218-227. Proceedings of the Seventh Inter- national Learning Analytics & Knowledge Confer- ence on LAK $17.
Carol S Dweck. 2006. Mindset: The new psychology of success. Random house.
Carol Dweck. 2016. Mindset: The new psychology of success.

---

Carol S Dweck and David S Yeager. 2019. Mindsets: view from two eras. Perspectives on Psychological science, 14(3):4814 (13):4814990 1-49..
Burt Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin words: Lexical feature selection and evaluation for identifying the content of political con- flict. Political Analysis, 16(4):372-403.
Catherine Good, Joshua Aronson, and Michael Inzlicht. 2003. Improving adolescents' standardized test per- formance: An intervention to reduce the effects of stereotype threat. Journal of Applied Developmental Psychology, 2466)6645-662.
Katherine Muenks, Elizabeth A. Canning, Jennifer La- Cosse, Dorainne J. Green, Sabrina Zirkel, Julie A. Garcia, and Mary C. Murphy. 2020. Does my pro- fessor think my ahility can change? students' percep- tions their stem professors' mindset beliefs predict their psychological vulnerabilitty, engagement, and performance class. Journal of experimental psy- chology. General.
Cameron A. Hecht, Christopher J. Bryan, and David Scott Yeager. 2023. A values-aligned inter- teaching
vention fosters growth and reduces inequality in educational outcomes. Pro- ceedings of the National Academy of Sciences of the United States of America, 120 25:12210704420.
OECD. 2019. PISA 2018 results (Volume III:: What school life means for students' lives. PISA, OECD Publishing.
Cameron A. Hecht, Carol S. Dweck, Mary C. Murphy, Kathryn M. Kroeper, and David Scott Yeager. 2022. Efficiently exploring the causal role of contextual moderators in behavioral science. Proceedings of the National Academy of Sciences of the United States of America, 120 1:e22 1121100....
OpenAl. 2023. Gpt-4 technical report.
Stefan Palan and Christian Schitter. 2017. Prolific.ac- subject pool for online experiments. Journal of Be- havioral and Experimental Finance, 17:22-27.
Jiaxin Huang, Shixiang Shane Gu, Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve.
Radim Rehurek and Petr Sojka. 2011. Gensim- -eaiimn-ython framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2).
Nicholas C. Hunkins. Sean Kelly, and Sidney K. D D'Mello. 2022. "beautiful work, you'le rock stars!": Teacher analytics to uncover discourse that supports undermines student motivation, identity, and be- longing in classrooms. LAK22: 12th International Learning Analytics and Knowledge Conference.
B. Samei, A. M. Olney, Kelly, M. Nystrand, S. D'Mello, N. Blanchard, X. Sun, M. Glaus, and A. Graesser. 2014. Domain independent assessment of dialogic properties of classroom discourse.
Jennifer A. Schmidt, Lee Shumow, and Hayal Z. Kackar- Cam. 2016. Does mindset intervention predict stu- dents' daily experience in classrooms? comparison of seventh and ninth graders' trajectories. Journal of Youth and Adolescence, 46:582 -022
Jennifer Jacobs, Karla Scornavacoo, Charis Harty, Abhi- jit Suresh, Vivian Lai, and Tamara Sumner. 2022. Promoting rich discussions in mathematies class- rooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education, 112: 103631.
Baruch B Schwarz, Naomi Prusak, Osama Swidan, Adva Livny. Kobi Gal, and Avi Segal. 2018. Or- chestrating the emergence of conceptual learning: A case study in geometry class. International Jour- nal of Computer-Supported Collaborative Learning. 13:189-211.
E. Jensen, M. Dale, P. J. Donnelly, C. Stone, S. Kelly, A. Godley, and S. K. Mello. 2020. Toward au- tomated feedback on teacher discourse to enhance teacher learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
Yen-Ning Su, Chia-Cheng Hsu, Hsin-Chin Chen, Kuo- Kuang Huang. and Yuch-Min Huang. 2014. Develop- ing sensor-based learning concentration detection system. Engineering Computations, 31 (162):216-230.
S. Kelly, A. M. Olney, P. Donnelly. M. Nystrand, and S. K. D'Mello. 2018. Automatically measuring ques- tion authenticity in real-world classrooms. Educa- tional Researcher, 47:7.
A. Suresh, J. Jacobs, V. Lai. C. Tan, W. Ward, J. H. Martin, and T. Sumner. 2021. Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application. arxiv. Preprint.
Khan Academy. 2023. Harnessing GPT-4 so that
all students benefit. A nonprofit approach for equal access. https://blog. kaw.hacaggaaaaaadmmm org/ harnessing-ai- so- setirraa-----ttaat------stttetee
-benefit-a- wwwaa-oooooooo-toeeeeoeeeo wwwp:oofttappprcccc-hoooa cor-rroacc--o--uuuuuu for-equal access. [Online; accessed 2024))
Rose E Wang and Dorottya Demszky. 2023. Is chat- gpt a good teacher coach? measuring zero-shot performance for scoring and providing actionable insights on classroom instruction. arXiv preprint arXiv:2306.03090.
Microsoft, 2023. Learn how to work with the chatgpt and gpt-4 models.

---

David Yeager, Jamie M. Carroll, Jenny Buontempo. Andrei Cimpian, Spencer Woody, Robert Crosnoe, Chandra Muller, Jared S. Murray, Pratik C. Mha- tre, Nicole B. Kersting, Chris Hulleman, Molly Kudym, Mary C. Murphy, Angela Lee Duckworth, Gregory M. Walton, and Carol S. Dweck. 2021. Teacher mindsets help explain where growth- mindset intervention does and doesn't t work. Psy- chological Science, 33:18 32.
A Developing the GSML Guide
The GMSL Guide described in Section 3.1 is devel- oped in collaboration with growth mindset experts and elementary school teachers trained in GMSL. The process was follows:
GMSL experts created initial draft of GMSL guide. This initial draft was informed by prior research highlighting the importance of educators' usage of GMSL (Yeager et al., 2021). The GMSL experts then met with the teachers to review the guide, review additional resources on general growth mindset informa- tion, and to discuss the internal thought pro- cess of identifying unsupportive language and how to reframe those instances into GMSL.
David S Yeager, Carissa Romero, Dave Paunesku, Christopher Hulleman, Barbara Schneider, Cintia Hinojosa, Hac Yeon Lee, Joseph O'Brien, Kate Flint, Alice Roberts, et al. 2016. Using design thinking to improve psychological interventions: The of the growth mindset during the transition to high school. Journal of educational psychology, 108(3):374.
David Scott Yeager, Paul Hanselman, Gregory M. Walton, Jared S. Murray, Robert Crosnoe, Chan- dra Muller, Elizabeth Tipton, Barbara Schneider, Chris Hulleman, Cintia P. Hinojosa, David Paunesku, Carissa Romero, Kate Flint, Alice Roberts, Jill Trott, Ronaldo Iachan, Jenny Buontempo, Sophia Yang, Carlos M. Carvalho, P. Richard Hahn, Maithreyi Gopalan, Pratik C. Mhatre, Ronald Ferguson, An- gela Lee Duckworth, and Carol S. Dweck. 2019. national experiment reveals where growth mindset improves achievement. Nature, 573:364 369.
2. The teachers were then provided three practice NCTE transcripts and were asked to indepen- dently identify utterances needing reframing and to reframe those into GMSL. Anecdotally, many of the utterances selected for reframing were the same between teachers. These ini- tial three transcripts were not used in our test set. During this phase, the teachers were also asked to document their annotation process.
David Scott Yeager, Rebecca Johnson, Brian James Spitzer, Kali Trzesniewski, Joseph Powers, and Carol S Dweck. 2014. The far-reaching effects of believing people change: implicit theories of personality shape stress, health, and achievement dur- ing adolescence. Journal of personality and social psychology. 106(6):867.
Then, the GMSL experts met with the teachers to discuss their annotation process and GMSL reframing. these meetings, we synthesized their experience of the annotation process to create an updated version of the GMSL Guide. During these conversations, we did not notice significant disagreement between the teachers.
Caleb Ziems, Minzhi Li, Anthony Zhang, and Diyi Yang. 2022. Inducing positive perspectives with text reframing. arXiv preprint arXiv:2204.02952.
4. After developing the more finalized version of the GMSL guide, the teachers then inde- pendently annotated the eight transcripts that were used in the test set. These final refram- ings were the ones used to prompt-tune and test GPT-4.
Overview of Appendix
We provide additional information on the devel- opment of the GMSL Guide, prompt designing. prompt text, and survey creation. We also provide a diverse set of examples of model and teacher re- framings in five different conversations well as evaluators' ratings of the original utterances and reframings for each conversation.
B Iterating on Prompt Design
In designing our prompts to best elicit GMSL- specific language, we tested multiple different prompting strategies and the robustness of model responses.
Our code is available at https: htt::///tttbbb com/ kunhanda/
mi mindset. Due to our IRB agreement, are not permitted to release the vey data without having individuals sign data- sharing agreement. you would like to access cither the student or teacher survey data, please contact kunal_ aunal_handa@allmm brown. edu.
B.1 What other prompts did we test?
Our final prompts excelled evoking GMSL from GPT-4 but carlier versions of the prompts did not perform as well. We tried a variety of different

---

prompts and prompting techniques before settling on the current prompt design described in the main text. We first attempted to categorize instances of unsupportive language in our dataset and provide the explicit categorization of the current example needing reframing when prompting the model. We found that this caused very predictable and robotic behavior from GPT-4. The model often outputted the same keywords used in the category (e.g. "De- briefing Task") in its generation.
model's outputs. We also tried these same per- turbations on the instructions defining the model's task to reframe utterances in the system text and came to the same conclusion. We did not, however, try making any changes to the annotation guide (which was developed in a coordinated effort with expert teachers trained in GMSL language).
B.3
What are the implications of these findings?
We also tested prompting the model via in- context few-shot learning but found that the quality of generations significantly decreased. We hypoth- esize that this is due to the challenge of extracting the broad, salient aspects of GMSL in reframings. The model often perceived too context-specific as- pects of the few-shot examples (c.g. generating a reframing mirroring that of one of its examples even if the conversational context was distinctly unrelated), ignoring the broad tenets of GMSL pro- vided in the system text. In these cases, the model also often referenced the task teachers were dis- cussing from the few-shot examples rather than applying the GMSL utilized in those examples to the task at hand.
Our efforts in designing GMSL-speciiie prompt indicate that GPT-4 is actually quite robust in its generations in GMSL contexts. But also too much context-specific information is a hindrance to accu- rate and realistic GMSL generations.
The model's strong performance in our evalua- tions came as a surprise to us-we expected the model to struggle in reframing utterances and did not anticipate students teachers judging the mod- els' utterances as better across all four of our met- rics. Although the model reframings did perform well our evaluations, we do acknowledge that ensuring the robustness and applicability of these generations across a wider variety of contexts still remains an open question that should be explored in future work. Also, understanding the reasoning behind LLMs irregular behavior in these instances (some of which were described above) should also be the subject of future research.
Once found that self-critique improved model performance in some cases, we attempted multiple different kinds of self-critique. We tried asking the model to improve upon its response be better aligned with a specific GMSL tenet (e.g. Im- prove upon the previous response express more em- pathic validation) but found that this self-critique often changed generations too significantly; the model would tailor the generation to only address this tenet, ignoring the other tenets and much of the positive change it made in its first generation.
C Example of Complete Prompt Text
Below is the explicit, step-by-step process used to prompt the GPT-4 model. This is the text used in one example. The System Text and Self- critique Prompt remain constant for all examples. However, the Example Prompt Text, Example Model Generation-MoDELL (0-CRITIQUES), and Example Model -oneration-MMoDi (1- CRITIQUE) change across examples.
We further explored the optimal number of self- critiques. We found that after one self-critique, model generations either: 1) often sounded too arcane be plausibly acceptable by students or teachers. Their rhetoric transitioned from realistic very formulaic, following the tenets sometimes verbatim. Or 2) did not significantly change. Given these two outcomes, we decided that 1 l-critique was the optimal number of self-critiques for our setup.
C.1 System Text
You are a helpful assistant who specializes in correcting teachers' language to be more supportive of the students. Your task is to directly revise the teacher's 's utterance to use mindset-supportive language. An Below are some tenets of mindset supportiiveee language to follow when encouraging students Each tenet is followed by methods by which to accomplish that tenet. Adhere to these tenets and use anddet-suppootive language when speaking with students.
B.2
What about perturbations to the prompt?
We made small perturbations to the prompt: al- tering sentence structure, using synonyms, and reordering sentences. We did not find that any of these adjustments significantly changed the
Empathic validation: Thank students for asking question, Try to summarize the concern in new words, Compliment student on

---

bravery reaching out challenging and impressive. When possible, provide legitimate, external cause for their concern (e.g. how society talks about college these days), rather than attributing it to them. not possible or relevant to attribute their concerns a legitimate external source, teachers should still attempt to validate emotions wi thout blaming the student for having them. Never directly tell them they're re wrong or mistaken. An
C.3 Example Model Generation-MooEEL (0-CRITIQUES)
Teacher (Revised): Thank you for sharing your thoughts. I appreciate your effort in remenbering our past classes. A fraction is indeed part of whole. Let's think together, what do usually use in class to represent whole? It's not candy bars pizza, but something else we' 'e discussed before.
2. Reappraisal of affect: Encourage them to view their negative emotions (fear, worry, stress, etc.) as a sign that they care deeply about something or someone, and reappraise the as resources that can empower them to take the challenge.
C.4 Self-critique Prompt
Make the previous response more mindset supportive.
3. Seek to Understand: Asking questions that will help you determine what the student has already done to approach the problem and what they are currently thinking or feeling. \n
C.5 Example Model Generation-Monn (1-CRITIQUE)
4. Position yourself as collaborative resource: Explain that you are trying to help them, Use terms like "we" or "us" to signal to the student that you are approaching challenge with them. An
Teacher (Revised): appreciate your answer, and it's great to see you engaging with the concept of fractions. A fraction does represent a part of the whole. Now, let's work together to remember the examples we' 've used in class to showcase a whole. Remember it wasn't food items like candy bars or pizza but something else we' 've talked about before. We'll figure it out together!
5. Provide hope for change in the future (i.e. event-focused mindset, e.g. growth mindset):
Explain how other people like them have been in similar situation and have gone to improve and be successful.
6. sutonomy-supportive language throughout: Try to simply tell students what do.
Instead, try offer students autonomy in their choices. You have you considered" or "You can think in this way". The goal not to tell students what think or rather giving them the space to safely explore their thoughts feelings about a problem.
D Examples of Original Utterances, Reframings, and Each Response Type's Evaluator Score
phrases like
feel, but
Below are five examples of conversations seen by student and teacher evaluators. The conver- sation excerpts include the Classroom Context, Conversational Context, and each of the four re- sponse types: ORIGINAL,, EXPERT, MODEL. (0- CRITIQUES), and MODEL (I-CRITIQUE). The scores given by both student and teacher evaluators to each of the responses are provided in Table 3, Scores were averaged across all student and teacher evaluations for a given response.*
C.2 Example Prompt Text
This is elementary grade math class working fractions. They focus with comparing fractions, with making equivalent fractions, simplifying fractions and putting fractions in order.
The following example of in-class conversation between teacher and student contains unsupportive language, Directly revise the teacher's last utterance, marked with "(Reframe)", to be more mindset- supportive: \n
We selected diversely interesting set of exam- ples to demonstrate that although model reframings often do perform the best, they are still imperfect; the specific cases in which models falter and the
Teacher: A pie. don't use any that class obviously because we would all way too much we were going to eat all that would spoil what we talk about in our past class? A whole as being what?\n Student: a fraction.
"The scores in this table are averaged, raw scores. Here. Perceived Teacher Mindset Shame are reverse coded: lower scores indicate better performance. A lower Perceived Teacher Mindset indicates the participant perceived the teacher have more of growth mindset whereas a higher score indi- cates the participant perceived the teacher to have more of a fixed mindset. A lower Shame score indicates the participant perceived the student as having less shame in the student- teacher interaction whereas higher score indicates that the participant perceived the student as having more shame in the interaction.
Teacher (Reframe): No, a fraction is part whole. what do we work with? I don't bring you candy bars. You guys don't 't bring in pizza. We talked about it. what do we use?

---

divergence of student and teacher ratings should continue be explored in the future.
D.1 Conversation 1
Classroom Context
Conversation 1. We see that EXPERT refram- ings outperform both model reframings in Per- ceived Teacher Mindset and Shame student ings and for every metric in teacher ratings.
The teacher is talking about different kinds of trian- gles. Student H misclassified isosceles triangle
in class.
Conversational Context
Conversation 2. There is an inconsistency be- tween student and teacher evaluations: in student evaluations, MODEL. (0-CRITIQUES) outperforms MODEL. (I-CRITIQUE), but this is reversed in teacher ratings as MODEL (I (I-CRITIQUE) at the top for all metrics.
Teacher: think this side here is longer than these two. These two would be equal. So if two are equal, Student H, what's it called? Isosceles, scalene, or equilateral?
Student: Scalene.
ORIGINAL
Conversation 3. We again see that EXPERT reframings do very well, obtaining the best scores for 3/4 of the student metrics. However, teachers rate MODEL (1-CRITIQUE) better across all three metrics.
Teacher: No.
EXPERT
Teacher: Thanks for sharing your answer! Let's think back about root word in scalene. What does it mean? That everyone is different the scale, right? So it can't be that one. Someone help Student H out.
Conversation 4. There is large amount of in- consistency in the student ratings; Each reframing performs the best for at least one metric. In the teacher ratings, MODEI. (O-CRITIQUES) obtains the highest ratings (tied in Respect with EXPERT).
MODEL (0-CRITIQUES)
Teacher: Thank you for your input, Student H. It's great that you're participating in the discussion. It can be bit tricky to remember the different types of triangles. In this case, when two sides are equal, it's actually called an isosceles triangle. Keep engaging with the material, and we'll all get better at classifying triangles together.
Conversation 5. In this conversation, both the teacher and student raters marked MODEL (1- CRITIQUE) as the highest in Shame. Although MODEL (1-CRITIQUE) performed the best across all other metries for both students and teachers, both teachers and students perceived it as inducing more shame than EXPERT, with students perceiv- ing as inducing the highest amount of shame amongst the four response types.
MODEL (1-CRITIQUE)
Teacher: Thank you for sharing your thoughts, Stu- dent H. appreciate your effort to answer the ques- tion. Classifying triangles can be bit challenging at first. When two sides are equal, the triangle is called an isosceles triangle, not scalene. As we continue learning and practicing together, we'll be- come more confident in identifying these types triangles. Remember, every mistake is an opportu- nity to learn and grow!
The variations in these results display the vari- ability in quality for model outputs. Although, on average, and in our regression analysis, mod- els perform better, they are not outperforming ex- pert teachers' reframings in every scenario. Fur- thermore, we see that self-critique does not al- ways improve upon generations often the MODEL (0-CRITIQULS) reframings surpass MODEL (1- CRITIQUE) reframings. The pattern of situations which self-critique hinders model performance should be explored in future work. These results also emphasize the need for future research to probe differences in preference between students and teachers in GMSL-specific contexts. As we see, teacher and student evaluations sometimes differ. The settings which students" and teachers' judgments diverge remain an open question that should be probed to truly understand the way in which GMSL can impact the classroom environ- ment and students' learning outcomes.
D.2
Conversation 2
Classroom Context
This is an elementary grade math class working on fractions. They focus on locating fractions using ruler as a number line.
Conversational Context
Teacher: How many lines [on a ruler] between zero and a half?
Student: Four.
ORIGINAL
Teacher: Point them. Between zero and
half? How many lines is that in between?
EXPERT

---

Teacher: Thanks for sharing! How did you get four? Let's point them on our ruler.
when dealing with fractions. In this case, we want to focus on distributing the 15 evenly across the groups. Let's collaborate and explore some strate- gies to achieve this. What are your thoughts on how we can share those 15 manipulatives among the existing groups?
MODEL (0-CRITIQUES)
Teacher: Thank you for participating, and it's great that you are trying to figure this out. Let's work together to figure out how many lines between zero and half the number line. Can you show me where you see the lines between zero and one half?
D.4 Conversation 4
Classroom Context
MODEL (1-CRITIQUE)
This is an elementary grade class having discus- sion about the relationship between multiplication and division. During the lesson they practice divi- sion math problems.
Teacher: I appreciate your effort in answering the question. Let's collaborate and take closer look the number line figure out how many lines are between zero and one half. Can you help me identify those lines between zero and one half?
Conversational Context
Original Teacher: Where do I write that 2 for 12? Student: You don't ' put the 2 yet.
D.3 Conversation
ORIGINAL
Classroom Context
Teacher: Yeah we do.
This an elementary grade math class working on long division.
EXPERT
Teacher: Interesting discussion about the 12 here! Let's go back and think about the 12. Once we multiple to get 12, how do split the numbers?
Conversational Context
Teacher: Interesting. You made 6 groups of 6, and then you had 15 leftover. Is that what you said? I wonder, if these 15 could go in some of the other groups. Student: They can be reduced.
MODEL (0-CRITIQUES)
Teacher: Thank you for sharing your thoughts. It can be bit confusing at times. Let's work together to figure out where we should place the 2 for 12.
ORIGINAL
Teacher: Well don't know if it would be reducing since it's not fraction that we're re making smaller, but that's big number, 15. That number is larger than the number of groups that have. So what that tells me that I can take these 15 and still spread them out some more. You guys work on that.
MODEL (1-CRITIQUE)
Teacher: Thank you for sharing your perspective. understand that might seem confusing at first. Let's collaborate and explore the correct placement of the 2 for 12, so we can learn together.
D.5 Conversation 5
EXPERT
Teacher: Oh, yes! That's great way to put it, I like that you used that word. We usually only use it with fractions but guess you can think about like we are reducing our left-over pile! That's great thinking.
Classroom Context
This is elementary grade math class working gcometry. They start the lesson with reviewing multiplication facts. They then begin learn about classifying triangles focusing on types of angles. Conversational Context
MODEL (0-CRITIQUES)
Teacher: Thank you for sharing that idea. It seems like you't re thinking about reducing num- bers, which is great skill have when working with fractions. In this situation, we're trying to spread the 15 equally among the groups we have. Let's think about how we can make that work. Can you come with a way to distribute those 15 ma- nipulatives among the existing groups?
Original Teacher: Very good. Student A, on num- ber six we have what in the corner? A little square. So is it right triangle, different type? I can't
hear you.
Student: Acute.
ORIGINAL
Teacher: No, sweetie. It's got little square corner, so it makes a square. It's right triangle. Now you know what I'm talking about, and do we think any sides are equal, Student E, in number six?
MODEL (I-CRITIQUE)
Teacher: appreciate your effort in trying to find a solution. seems like you'le re applying the con- cept of reducing, which an important technique
EXPERT
Teacher: Thanks for sharing your answer! Let's

---

remember what our square corner tells about triangles.
MODEL (0-CRITIQUES)
Teacher: Thank you for your response, Student A. When we see little square in the corner, actually represents right angle. So, this would be right triangle. It can be confusing sometimes, but now we know what to look for. As for the sides, Student E, what do you think about their lengths in number six?
MODEL (I-CRITIQUE)
Teacher: I appreciate your input, Student A. When there's a little square in the corner, signifies a right angle. That means this is right triangle. It's completely understandable to mix them up, but now we understand the difference. Moving on to the sides, Student E, do you have any thoughts about their lengths in number six?
E Additional Information on Surveys
Below, we provide an example of survey frames seen by participants. These figures provide an overview of the survey experience for participants. The frames include are: the consent form (Fig- ure 4), the survey overview (Figure 5), an example conversation transcript (Figure 6), example ques- tions (Figure 7 and Figure 8). We also include an example of the complete survey in the Supplemen- tary Materials.

---

Student Ratings
Teacher Ratings
Conversation
Response Type
Perceived Teacher Mindset
Challenge- Seeking
Perceived Teacher Mindset
Shame
Respect
Shame
Respect
ORIGNNAL
4.27
1.27
5.09
3.46
3
4.5
4
EXPERT
1.95
1.5
5.1
2.75
3
5.5
MODEL (0-CRITIQUES)
2.82
1.55
3.27
5,18
4+
4+
3*
MODEL (1-CRITIQUE)
2.61
1.22
3.44
4.78
2
4+
5*
ORIGINAL
3.46
4.17
4.33
3
EXPERT
3,2
4.7
2.5
3+
MODEL (0-CRITIQUES)
2.4
5.4
2.5
3.5
4.5
MODEL (1 (I-CRITIQLE)
2.95
1.3
4.1
5.1
2.17
2,33
5,33
ORIGINAL
3.08
1.08
4.67
4.42
2.83
3.67
4.67
EXPERT
2.45
4.9
2
3
MODEL (0-CRITIQUES)
2.95
4.7
3.5
MODEL (1-CRITIQUE)
3.82
4.11
1.75
2.5
ORIGINAl.
3.55
2.55
3
4+
4+
EXPERT
3.23
1.4
3.18
4.3
1.75
4,5
5
MODEL (O-CRITIQUES)
2,82
1.45
3.27
4.64
1.25
2
MODEL (1-CRITIQUE)
2885
1.5
3.6
5.4
2
3+
ORIGINAL
2.85
4.4
4.17
EXPERT
2.59
1.27
3.45
4.45
1.75
2.5
MODEL (0-CRITIQUES)
2.55
1.3
2,9
4.3
2.5
3
4.5
MODEL (1-CRITIQUE)
3.72
1.09
3.9
4
4
5
3
Table 3: Conversations' evaluator scores for each response type.
denotes that only one evaluator rated the response.

---

Busic Information
purpocee of the ellaty is gaie ectusstionn for practicce wege to monervv educational nat matern ais Wi are comd criing thes souey with eouts more generally by (l.. noc
teacherst
broud feodback
el telliieeees melteille
you
lowlingg
Preewvr the qu aest proviced in the online surve!
you
How long will this study take and how many people will be study?
enline will take
10x min Approc
will pato
Whet rinks and dicoomfortt might ropeeience from being this etudy?
to theee of our snnuudidgge taaks you be compeiannng have no more risk of Thamm to would repperienann innryghky life These niee of boewlter, belgaar on sscorffont complesingg ennorie
completey voluntory
may bur surm minatte st anry tine..
you
study?
You will like recclve ciroct bonent from ridipat ng in this etacy: However this rescarch sunu laule country:
mulosellonee proteesses light solloue Lolllng
How will protect Information?
tan tesorillate paymentt mentett finy present rettons papers pub acout the find iigs from this itud) therefo re include your name or amy other informan that cou idionffy you, We mav anone the dota we wittis with other your
researdhers future into on that can iderrity you.
any
compansatte you being Your compersstiinn on for curmpfeting the stucy will prowided trruuhhhenn enerl Pleasee note portial comp pensas on not offerce; you must pleptitt the ondre coudty recerve the componastiinn
atudy?
be
given yorouu
smll- silli roed Amauvon righl cellvneed your worel
this Volumnrry
any ressor, your sou prefie to perticipate this in the may stop engiimme closing the trowoer window withorow
unipwiio ion slidyy resientve il way
without poncry..
Contact Information on for the Study Team Questions about the Recearch questions thourth research,
Anchersiood
Contact information for questions about Your Rights as Reseerch Participant informet ask cuestion,, dispus concomo about study someone than the rosaarcerpp, slease can roen the lowinng
Inoct Lntonaa Aoview racard
Anonyvazd
Plecse refeeerre the protoaral framber found al the of chesumment
Your Consent
fly cliikingg thw lan befor consinee tivereel page are indicalling thei nocdle United ahear you ogree to particparee nittls end ne research soudy. or any timee you wicch to stop potlciuaing ling pul diess Latnoss minetro:
years
understane comtent and
consent laving tri survey
101 trinng SuRvey
Figure Consent required for all participants as described in the Ethics Statement.

---

Overview of This Activity
As student, are in position to provide teachers with helpful feedback about what they are (and aren't) doing well to help you learn. Unfortunately, teachers rarely receive this important feedback from their students.
Today, we'll ask you to read example of what four math teachers said during their classes, and we'll ask you to tell us what you think of these teachers. Your responses will be helpful for training future teachers so that they know how to support students like you.
This survey includes opinion questions about what teachers say to their classes. Please
do your best to give it your complete attention and answer all questions honestly.
Figure 5: Survey overview shown to participants in Section 4.1.
In this transcript, the teacher is telling the students about different multiplication properties. The associate property multiplication states that when multiplying three numbers together, regardless how they're grouped, answer always the same. For example, (3 x and x 5) 60.
Transcript:
Teacher: And today's word means when have math problem and it has parentheses, and choose to change the problem a little by moving the parenthesis, and it's still equal I've got problem. Everybody put on your thinking caps. give you clue. It's in the first column. Student A?
Student A: It's the distributive property.
Teacher: Why you think that the answer? Tell me what you are thinking.
Figure 6: Example conversation transcript shown participants in Section 4.1.

---

How much do you agree or disagree with this statement?
This math teacher makes me feel like I would be able to learn the material in their class, even struggle sometimes.
Strongly
Mostly disagree
Mostly agree
Strongly Agree
Disagree
Agree
disagree
How much do you agree or disagree with this statement?
This math teacher seems like they would reward students for trying hard and improving.
Strongly
Mostly disagree
Mostly agree
Strongly
Disagree
Agree
disagree
Agree
How much do you agree or disagree with this statement?
This math teacher seems like they would only give good grades to students natural "talent" for math.
Strongly disagree
Mostly disagree
Mostly agree
Strongly Agree
Disagree
Agree
Figure 7: Example question shown participants in Section 4.1 to understand participants' perception of hypotheti- cal teacher's growth mindset.
Imagine that, later today or tomorrow, math teacher handed out two extra credit assignments. You got choose which do, You get the same number of points for trying either one.
One choice an easy review-it has math problems you already know how to solve, and you will probably get most of the answers right without having to think very much. takes minutes.
The other choice is a hard challenge- it has math problems you don't know how to solve, and you will probably get most the problems wrong, but you might learn something new. also takes minutes.
If you had to pick right now, which would you pick?
The easy math assignment where would get most problems right
hard math assignment where would possibly learn something new
Figure 8: Example question shown to participants in Section 1 to understand participants' willingness to engage challenge-seeking behavior if they were this hypothetical teacher's class.


Reviews:
[{"review_id": "jLbzvJLKFw", "reviewer": "Reviewer_jpn8", "summary": "", "questions": "1. Could the authors describe the process of developing the annotation guide in more details to increase the validity of the developed framework? e.g., how was the initial GMSL Guide created and based on any source transcripts? Did the expert teachers identified the important suggestions separately and then reconvened to discuss and resolve disagreement (if any)?  ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! We are glad they find our empirical results insightful, value the usefulness of our annotation framework, and appreciate the novelty of our use of LLMs to provide GMSL. **We will revise the paper to add further detail about the process of developing the GMSL Guide.**\n\n> Question 1: Could the authors describe the process of developing the annotation guide in more details to increase the validity of the developed framework? e.g., how was the initial GMSL Guide created and based on any source transcripts? Did the expert teachers identified the important suggestions separately and then reconvened to discuss and resolve disagreement (if any)?\n\nThe GMSL Guide is developed in collaboration with growth mindset experts and elementary school teachers trained in GMSL. The process was as follows: \n1. GMSL experts created an initial draft of a GMSL guide. This initial draft was informed by prior research highlighting the importance of educators' usage of GMSL (Yeager et al., 2021). The GMSL experts then met with the teachers to review the guide, review additional resources on general growth mindset information, and to discuss the internal thought process of identifying unsupportive language and how to reframe those instances into GMSL. \n2. The teachers were then provided three practice NCTE transcripts and were asked to independently identify utterances needing reframing and to reframe those into GMSL. Anecdotally, many of the utterances selected for reframing were the same between teachers. These initial three transcripts were not used in our test set. During this phase, the teachers were also asked to document their annotation process.\n3. Then, the GMSL experts met with the teachers to discuss their annotation process and GMSL reframing. In these meetings, we synthesized their experience of the annotation process to create an updated version of the GMSL Guide. During these conversations, we did not notice significant disagreement between the teachers. \n4. After developing the more finalized version of the GMSL guide, the teachers then independently annotated the eight transcripts that were used in the test set. These final reframings were the ones used to prompt-tune and test GPT-4.  \n\nYeager, D. S., Carroll, J. M., Buontempo, J., Cimpian, A., Woody, S., Crosnoe, R., Muller, C., Murray, J., Mhatre, P., Kersting, N., Hulleman, C., Kudym, M., Murphy, M., Duckworth, A. L., Walton, G. M., & Dweck, C. S. (2021). Teacher mindsets help explain where a growth-mindset intervention does and doesn’t work. Psychological Science, 33(1), 18–32. https://doi.org/10.1177/09567976211028984", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is the first study attempting to use large language models to provide growth mindset supportive language in education.\n2. The annotation framework of growth mindset supportive language can be of use to other relevant researchers in the field.\n3. The empirical results regarding the perceived impact of growth mindset supportive language are insightful.", "reasons_to_reject": "None", "questions_to_authors": "1. Could the authors describe the process of developing the annotation guide in more details to increase the validity of the developed framework? e.g., how was the initial GMSL Guide created and based on any source transcripts? Did the expert teachers identified the important suggestions separately and then reconvened to discuss and resolve disagreement (if any)?  ", "paper_topic_and_main_contributions": "This paper presented an interesting study of using large language models to provide automated coaching to support teachers' use of growth mindset supportive language. Specifically, the paper made three contributions, including (1) an annotation framework to guide teacher's understanding of growth mindset supportive language; (2) an open-source dataset containing both unsupportive and supportive utterances guided by the defined annotation framework; and (3) an evaluation framework to assess the perceived impact of growth mindset supportive language in education.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "3c9Et13oan", "reviewer": "Reviewer_QY3p", "summary": "", "questions": "1. How does contextual definition of GMSL change, by subject, by teacher -- student pair, level of expertise of student as well instructor?\n\n\n2. How do the specificities of the dataset affect the automated model of GMSL generation step, approximately in how many cases was teacher trained in GMSL intervention necessary?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! We are happy they appreciate our work’s “long-term societal impacts with the use of LLMs in positive directions” and value its “exemplary role of human-ai collaboration in a difficult use that could impact learning outcomes.” **We will revise the paper to include more context of prior work that establishes the connection between student-teacher dialogue and learning outcomes. We will also add details on the need for future work to explore the cultural variables that might impact GMSL.**\n\n> Reasons 1: Some learning outcomes are difficult to segregate as learning units. For example, when student attempts to learn concept in math, there are some intermediate steps involved towards the learning goals. How can this be modeled with growth mindset better, so that student - teacher doesn't deviate from the broader learning outcome?\n\n**Because our reframings are performed on a sentence-level in context of the existing conversation, we only alter the way the learning goals are communicated and not the learning goals themselves.** Prior work has demonstrated that these sentence-level changes of GMSL improve student learning outcomes while remaining within the bounds of the class’ overall goals (Yeager et al., 2019). When initially prompt-tuning GPT-4 for the reframing task, we did not include significant conversational context. In these initial experiments, we did notice that the models sometimes struggled to maintain the central learning goal within the conversation. Our final prompting-tuning setup originated from this initial problem—we found that by providing both the classroom context, which often represents the long term learning goals (*“This is an elementary grade math class working on geometry. They start the lesson with reviewing multiplication facts. They then begin to learn about classifying triangles focusing on types of angles.”*) and the two utterances directly prior, GPT-4 was able to maintain the central topic of discussion and focus only on reframing. GPT-4 reframings maintained the same scope as the teacher reframings (thereby not straying from the learning goals) while improving upon the actual language within the reframing.\n\t\nIn future work, we hope to investigate the long-term outcomes associated with GPT-4 reframings—an unexplored but potentially impactful domain—demonstrating LLMs’ to enhance educators’ teaching experience and students’ learning outcomes in the long run. \n\nYeager, D.S., Hanselman, P., Walton, G.M. et al. A national experiment reveals where a growth mindset improves achievement. Nature 573, 364–369 (2019). https://doi.org/10.1038/s41586-019-1466-y\n\n> Weaknesses 1: Difficult to quantify learning outcomes, when quantifying student - teacher dialogue. A Teacher may provide harsh feedback at times so incite student to think in the correct direction. This could probably work in cases of some students without adverse impact on the student pysche. Providing feedback is correlated with the learning outcomes\n\nAlthough we recognize that quantifying learning outcomes through student-teacher dialogue is difficult, recent work has shown a causal relationship between students’ perceptive judgments of teachers’ mindset supportive language and teachers’ in-class behavior as well as students’ learning outcomes (Hecht et al. 2022). Previous research of GMSL has also demonstrated the long-term impact of GMSL-intervention within classroom contexts; for example, Yeager et al. (2019) found that exposing students to a growth mindset in a short one-hour conversation improved the grades of lower-achieving students and increased overall enrollment to advanced mathematics courses (see also Schmidt et al. (2017) and Muenks et al. (2022) for similar findings). \n\nThus, GMSL builds upon student’s learning goals rather than altering or overshadowing them. **We build upon this understanding to explore automating GMSL feedback so that any teacher can integrate GMSL into their instruction; currently the lack of effective GMSL coaching restricts teachers’ implementation of GMSL.** GMSL **does not** advocate for the lack of providing feedback but rather restructures the way in which the feedback is provided. GMSL feedback uses language that focuses on teaching that intellectual abilities can be developed. It does not necessitate that the feedback is not harsh or constructive, but rather that it communicates that growth is possible for the student. For instance, the following is an example from the NCTE transcript: \n\n*Teacher: Mm-hmm, so erase your fraction and put one. Okay. You’re still wrong. So zero and one. How many pieces is that?*\n\n*Student: Three.*\n\n*Teacher: Erase what you have please. Everything. And I want you to look at it again, okay? Everything. Everything. Everything.*\n\nHere, the teacher is correcting a student’s mistake. They are offering criticism, telling the student that they are wrong, and asking them to try again. However, the language used is unsupportive. Our expert teachers reframed this utterance to:\n\n*Teacher: That’s a great answer. But let’s look again. We’re going to start with a clean slate and see if that helps. Let’s erase everything.*\n\nThe reframing does not remove criticism/feedback from the teacher’s response but rather restructures the response to be more growth-oriented, improving students’ overall learning process. It is not that we want to avoid providing feedback, but rather that the feedback should be provided in such a way that students’ can best improve their learning process by way of that feedback (more examples of conversations from the NCTE dataset and their reframings are in Section C of the Appendix). The primary focus of our current paper is to explore, assuming that the growth mindset approach is generally accepted as beneficial for teachers and students, what contributions we can make from a technical standpoint.\n\nYeager, D.S., Hanselman, P., Walton, G.M. et al. A national experiment reveals where a growth mindset improves achievement. Nature 573, 364–369 (2019). https://doi.org/10.1038/s41586-019-1466-y\n\nSchmidt, J. A., Shumow, L., & Kackar-Cam, H. Z. (2017). Does Mindset Intervention Predict Students' Daily Experience in Classrooms? A Comparison of Seventh and Ninth Graders' Trajectories. Journal of youth and adolescence, 46(3), 582–602. https://doi.org/10.1007/s10964-016-0489-z\n\nMuenks, K., Canning, E. A., LaCosse, J., Green, D. J., Zirkel, S., Garcia, J. A., & Murphy, M. C. (2020). Does my professor think my ability can change? Students’ perceptions of their STEM professors’ mindset beliefs predict their psychological vulnerability, engagement, and performance in class. Journal of Experimental Psychology: General, 149(11), 2119–2144. https://doi.org/10.1037/xge0000763\n\n> Question 1: How does contextual definition of GMSL change, by subject, by teacher -- student pair, level of expertise of student as well instructor?\n\nWhile the tenets presented in the GMSL guide are applicable to a variety of contexts (e.g. empathetic validation, expressing hope for change), the **specific language** for instantiating these tenets may indeed vary by context. For example, for both a high school mathematics context and a fourth-grade context, the educator should express to students that they believe in the student’s ability to succeed despite potential difficulties. However, the language used between the highschool course and the fourth-grade classroom will be different. To use GMSL in a highschool context, the educator might say, *“I believe you are all capable of doing well in this class, even when you face difficulties.”* Alternatively, when talking to a fourth-grade classroom, the educator might say, *“I know that you are all going to be great math learners by the end of the year, even when you might think it’s hard.”* These express the same idea, and both use GMSL, but are necessarily contextually different. As our work is the first to explicate the high-level tenets of GMSL and translate it to teachers’ discourse, we decided to work with a concrete demographic—4th-6th grade math classrooms—to create a well-defined scope for annotation and evaluation, and to reduce sparsity in our resulting dataset. Working with teachers to adapt our GMSL guide to other subjects, grade-levels and student’s learning needs is a promising direction for future work.\n\n> Question 2: How do the specificities of the dataset affect the automated model of GMSL generation step, approximately in how many cases was teacher trained in GMSL intervention necessary?\n\nModel generations were very sensitive to the classroom context (grade level, subject) provided in the prompt (Section B of the Appendix). For example, in prior prompt-tuning setups that did not include the class year, the model generations contained fewer examples, and the language was more rigid. However, the overall content/information communicated did not change significantly. But, as detailed in our earlier response, if the classroom context was missing altogether, the model often strayed from the classroom’s overall learning goal, often adding in tangentially related information or widening the scope of the issue the conversation was attempting to address. We did not notice significant differences in reframing strategies employed by the teachers trained in GMSL or GPT-4 across different student-teacher pairs. That being said, we encourage future work to explore the implications of these variables in LLMs’ generation of GMSL. \n\nAcross the transcripts used for prompt-tuning and testing the GPT-4 model, 22.8% of all teacher utterances (100 out of 438 teacher utterances) required reframing (in the paper we mistakenly reported the rate to be 17%, which we will update in the revision). \n\n> Ethics Concerns: Difficult to judge with just one paper if there are no cultural aspects to be considered here -- Marshmallow Effect\n\nAs our paper is the first, to our knowledge, to both make explicit the linguistic properties of GMSL and demonstrate the use of LLMs to provide GMSL in classroom contexts, we hope to start the discussion regarding various cultural aspects that impact the utility of GMSL for students and teachers. The evaluation framework we propose in this paper can hopefully lead to future research in this area, tackling other variables, such as cultural aspects, that might impact students’ and teachers’ treatment of GMSL. We will revise our paper to elaborate on this constraint in the ethics and limitations sections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n## Reasons\n\n1. Strong factors that influence student's learning outcomes are considered. Promoting this work would effectively promote dialogue  and interest in the directions of student mental learning and its impact on learning outcomes.\n\n2. This work has long-term societal impacts with the use of LLMs in positive directions, which could help drive away some negative notions developed around generative AI as being harmful because of its capabilities. This is a timely work.\n\n## Strengths\n\n1. Difficult to measure qualitative aspects of teacher -- student dialogue, one may consider growth mindset, others may consider it as harsh feedback. This work attempts at nailing this aspect down in the form of a dataset with 2 levels of critique (0-moderated 1 - moderated) by teachers trained in GMSL.\n\n2. The paper plays an exemplary role of human-ai collaboration in a difficult use that could impact learning outcomes.", "reasons_to_reject": "\n## Reasons\n\n1. Some learning outcomes are difficult to segregate as learning units. For example, when student attempts to learn concept in math, there are some intermediate steps involved towards the learning goals\n\nRefer -- \nKoedinger, Kenneth R., and Mitchell J. Nathan. \"The real story behind story problems: Effects of representations on quantitative reasoning.\" The journal of the learning sciences 13.2 (2004): 129-164.\n\nLovett (1998). Cognitive task analysis in service of intelligent tutoring system design. \nIn Proceedings of the Intelligent Tutoring Systems Conference.\n\nKoedinger, Kenneth R., Albert T. Corbett, and Charles Perfetti. \"The Knowledge‐Learning‐Instruction framework: Bridging the science‐practice chasm to enhance robust student learning.\" Cognitive science 36.5 (2012): 757-798.\n\nHow can this be modeled with growth mindset better, so that student - teacher doesn't deviate from the broader learning outcomeS?\n\n## Weaknesses\n\n1. Difficult to quantify learning outcomes, when quantifying student - teacher dialogue. A Teacher may provide harsh feedback at times so incite student to think in the correct direction. This could probably work in cases of some students without adverse impact on the student pysche. Providing feedback is correlated with the learning outcomes\n", "questions_to_authors": "1. How does contextual definition of GMSL change, by subject, by teacher -- student pair, level of expertise of student as well instructor?\n\n\n2. How do the specificities of the dataset affect the automated model of GMSL generation step, approximately in how many cases was teacher trained in GMSL intervention necessary?", "paper_topic_and_main_contributions": "\n## Summary\n\nThe work consider utterances of teacher T -- student S dialogues in math problem solving context. Keeping a Growth Mindset Supportive Language (GMSL) approach, the paper could be used to promote T-S dialogue in positive directions. In several instances, in the heat of the moment, T may provide adverse feedback to S which could lead to S's lack of motivation, lack of engagement in the task and higher chance of dropouts. In summary, GMSL is a technique to promote T-S collaboration towards common goals. The work boils down to the partial automation of this process with the use of LLMs, in collaboration with teachers trained in GMSL. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Difficult to judge with just one paper if there are no cultural aspects to be considered here -- Marshmallow Effect "}, "extra_text": {}}, {"review_id": "qHq2ttkR4h", "reviewer": "Reviewer_swnb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! We are happy they appreciate the potential of our work to use LLMs to “encourage positive outcomes in educational settings, with some very surprising results.” **We will revise the paper to better motivate our evaluation setup and goals for future work.**\n\n> Reasons to reject: A key concern I have with the paper is in its evaluation: the authors use hypothetical survey questions to ask and measure how students and teachers would choose to act. Using this method of evaluation is quick and easy, but the key challenge remains to see if these results would hold in an actual classroom environment and impact actual student behavior and learning outcomes. This seems especially pertinent in the education domain, particularly when the paper's key novelty is in this unique setting compared to existing prior work in positive reframing (i.e. Ziems 2022, using social media data)—which, without, makes the key evaluation in this paper that of one on perception than actual educational impacts. Establishing this link between hypothetical student scenarios and actual learning outcomes is key to making this a stronger paper.\n\nWe agree with the reviewer that it is critical to evaluate the effects of GMSL on actual student learning. However, before this could be done, it was essential to examine LLM’s abilities to generate language that is perceived as GMSL. **As a result, the goal of our paper is to establish the validity of applying LLMs in instances of unsupportive language. Our positive results then can motivate future longitudinal studies deploying LLM-based tools to provide GMSL feedback in classrooms.**\nPrior work, through long-term longitudinal studies, has demonstrated the correlation between GMSL intervention and student learning outcomes across metrics such as grades, higher course passing rates, and selection of advanced mathematics courses (Yeager et al., 2019; Rege et al., 2021). These results were consistent across achievement levels and countries (OECD, 2019). Prior research has also shown the necessity of a teachers’ growth-mindset, communicated through their supportive language, to allow students to profit from their own growth-mindsets (Yeager et al., 2021). Furthermore, **follow-up studies have also demonstrated that students’ hypothetical judgments of teachers’ mindset supportive language did track with teachers’ actual behavior and students’ learning outcomes (Hecht et al., 2022).** Our research builds off of this understanding.\n\n**Our work is an important contribution because it is the first study, to our knowledge, that explores LLMs’ ability to utilize GMSL.** Classroom evaluations are long-term, logistically challenging, and expensive; we aim to first establish the feasibility of LLM’s deployment for this task. Through our teacher evaluations, we identify that teachers are receptive to LLM reframed responses. This is key to our overarching goal of supporting teachers’ instruction in classrooms and a precondition to an eventual longer, longitudinal study. Additionally, prior work demonstrates a disconnect between student perception of teachers’ utterances and teachers’ perception of their own language, leading to learning outcomes which differ from student and teacher learning goals (Dweck, 2015; Barger, Xiong, & Ferster 2022). Thus, our extensive student evaluation is necessary to establish that the GMSL reframings by LLMs are received positively by the students who are on the receiving end of the language.\n\nHaving identified LLM’s potential in this domain, the reviewer’s proposition is our ultimate goal—identifying whether these reframings impact students' learning outcomes within the actual classroom environment. **We follow a precedent for an interactive evaluation setup in education and computer science literature.** For example, Demszky et al. (2021) explored a computational framework for measuring conversational uptake (in ACL 2021) prior to conducting a large-scale experiment—Demszky et al. (2023)—in which they deployed their previously established framework in an actual classroom environment (journal article in Educational Evaluation and Policy Analysis). Their initial paper served as an early validation of their overall framework and approach prior to conducting the in-practice experiment. Our research mirrors this approach. \n\nYeager, D.S., Hanselman, P., Walton, G.M. et al. A national experiment reveals where a growth mindset improves achievement. Nature 573, 364–369 (2019). https://doi.org/10.1038/s41586-019-1466-y\n\nRege, M., Hanselman, P., Solli, I. F., Dweck, C. S., Ludvigsen, S., Bettinger, E., Crosnoe, R., Muller, C., Walton, G., Duckworth, A., & Yeager, D. S. (2021). How can we inspire nations of learners? An investigation of growth mindset and challenge-seeking in two countries. American Psychologist, 76(5), 755–767. https://doi.org/10.1037/amp0000647\n\nYeager, D. S., Carroll, J. M., Buontempo, J., Cimpian, A., Woody, S., Crosnoe, R., Muller, C., Murray, J., Mhatre, P., Kersting, N., Hulleman, C., Kudym, M., Murphy, M., Duckworth, A. L., Walton, G. M., & Dweck, C. S. (2022). Teacher Mindsets Help Explain Where a Growth-Mindset Intervention Does and Doesn’t Work. Psychological Science, 33(1), 18–32. https://doi.org/10.1177/09567976211028984\n\nDemszky, D., Liu, J., Mancenido, Z., Cohen, J., Hill, H., Jurafsky, D., & Hashimoto, T. (2021). Measuring conversational uptake: A case study on student-teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1638–1653). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.130\n\nDemszky, D., Liu, J., Hill, H. C., Jurafsky, D., & Piech, C. (2023). Can Automated Feedback Improve Teachers’ Uptake of Student Ideas? Evidence From a Randomized Controlled Trial in a Large-Scale Online Course. Educational Evaluation and Policy Analysis, 0(0). https://doi.org/10.3102/01623737231169270\n\nHecht, C. A., Dweck, C. S., Murphy, M. C., Kroeper, K. M., &amp; Yeager, D. S. (2022). Efficiently exploring the causal role of contextual moderators in behavioral science. Proceedings of the National Academy of Sciences, 120(1). https://doi.org/10.1073/pnas.2216315120 \n\nOECD. (2019). PISA 2018 results (Volume III): What school life means for students’ lives. PISA, OECD Publishing. https://doi.org/10.1787/acd78851-en\n\nBarger, M. M., Xiong, Y., Ferster, A. E. (2022). Identifying false growth mindsets in adults and implications for mathematics motivation. Contemporary Educational Psychology, 70, 102079. https://doi.org/10.1016/j.cedpsych.2022.102079 \n\nDweck, C. (2015). Carol Dweck Revisits the “Growth Mindset.” Student Well-Being. August 27, 2023, https://www.edweek.org/leadership/opinion-carol-dweck-revisits-the-growth-mindset/2015/09", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, a very interesting study on how large language models can be used to encourage positive outcomes in educational settings, with some very surprising results—LLM reframings even outperformed expert reframings! The key reason to accept here is the novel domain paired with actual human evaluation, which makes this paper stand out among its peers.", "reasons_to_reject": "A key concern I have with the paper is in its evaluation: the authors use hypothetical survey questions to ask and measure how students and teachers would choose to act. Using this method of evaluation is quick and easy, but the key challenge remains to see if these results would hold in an actual classroom environment and impact *actual student behavior and learning outcomes.* This seems especially pertinent in the education domain, particularly when the paper's key novelty is in this unique setting compared to existing prior work in positive reframing (i.e. Ziems 2022, using social media data)—which, without, makes the key evaluation in this paper that of one on *perception* than actual educational impacts. Establishing this link between hypothetical student scenarios and actual learning outcomes is key to making this a stronger paper.\n\n**After author response:** Thanks to the authors for their response; raising soundness from 2 to 3.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates whether GPT4 is able to generate growth mindset supportive language (GMSL) reframings on utterances made, originally, by teachers in a contextual classroom environment. The paper designs a new taxonomy to annotate GMSL language, uses it to prompt engineer GPT4, and evaluates the perceived impact of these reframings (and other actual expert-made reframings) through teacher and student feedback and lexical analysis.\n\nModel responses are shown to be greatly preferred against original teacher responses, and elicit challenge-seeking behavior—a desired GMSL outcome—in students, measured on a likert scale using survey responses. The lexical analysis of n-grams also shows some features of high GMSL considered language, which matches intuition.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores how LLMs can be used to support teachers in using growth mindset supportive language (GMSL). Teachers annotate classroom transcripts and provide expert reframings. GPT-4 is used to reframe student-directed language to exhibit a growth mindset. Then, student and teachers are surveyed on (1) teacher’s perceived growth mindset, (2) teacher’s promotion of challenge-taking behavior, (3) shame felt by students in the teacher’s class, and (4) respect. It is found that GPT-4 responses are often scored higher than expert responses.

This paper displays a nice application of LLMs with positive social impact. The issues mentioned by reviewer jpn8 regarding the GMSL guide seem to have largely been addressed. Reviewer swnb's point regarding testing this approach in a classroom is an important one, but the paper can stand alone prior to such testing (with the expectation of future publication of time-consuming follow-up work). This also is related to reviewer QY3p's point about actual learning outcomes.
Decision: Accept-Main