SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models
Authors: Jiayu Liu
Zhenya Huang
Tong Xiao
Jing Sha
Jinze Wu
Qi Liu
Shijin Wang
Enhong Chen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Socratic Teaching
Large Language Models
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Large language models (LLMs) are considered a crucial technology for advancing intelligent education since they exhibit the potential for an in-depth understanding of teaching scenarios and providing students with personalized guidance. Nonetheless, current LLM-based application in personalized teaching predominantly follows a "Question-Answering" paradigm, where students are passively provided with answers and explanations. In this paper, we propose SocraticLM, which achieves a Socratic "Thought-Provoking" teaching paradigm that fulfills the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery. To build SocraticLM, we first propose a novel "Dean-Teacher-Student" multi-agent pipeline to construct a new dataset, SocraTeach, which contains $35$K meticulously crafted Socratic-style multi-round (equivalent to $208$K single-round) teaching dialogues grounded in fundamental mathematical problems. Our dataset simulates authentic teaching scenarios, interacting with six representative types of simulated students with different cognitive states, and strengthening four crucial teaching abilities. SocraticLM is then fine-tuned on SocraTeach with three strategies balancing its teaching and reasoning abilities. Moreover, we contribute a comprehensive evaluation system encompassing five pedagogical dimensions for assessing the teaching quality of LLMs. Extensive experiments verify that SocraticLM achieves significant improvements in the teaching performance, outperforming GPT4 by more than 12\%. Our dataset and code is available at https://github.com/Ljyustc/SocraticLM.

SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models
Jiay Liu 1.2
Zhenya Huang 12- Tong Xiao Jing Sha?
Jinze Wu?
Qi Liu Shijin Wang Enhong Chen 1.2x
1: University of Science and Technology China
2: State Key Laboratory of Cognitive Intelligence (jy251198, tongxiao2002) (huangzhy, qiliuql, cheneh) @ustc. edu. cn; {j lingsha, jzwu4, saju4,sjwangg jaang33 Singgeeeiieeek.. com
ustc. edu cn;
Abstract
Large language models (LLMs) are considered crucial technology for advane- ing intelligent education since they exhibit the potential for an in-depth under- standing of teaching scenarios and providing students with personalized guidance. Nonetheless, current LL.M-based application in personalized teaching predomi- nantly follows "Question-Answeringg Answering" paradigm, where students are passively provided with answers and explanations. In this paper, we propose SocraticLMM, which achieves Socratic "Thookit-Provokinng" teaching paradigm that fulfills the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery. To build SocraticL.M we first propose a novel "Dean-Teri Teacher-Student multi-agent pipeline to construct a new dataset, SocraTeach, which contains 35K meticulously crafted Socratic-style multi-round (cquivalent to 208K singlc-round) teaching dialogues grounded in fundamental mathematical problems. Our dataset simulates authentic teaching sce- narios, interacting with six representative types of simulated students with different cognitive states, and strengthening four crucial teaching abilities. SocraticLM is then fine-tuned on SocraTeach with three strategies balancing its teaching and reasoning abilities. Moreover, we contribute comprehensive evaluation system encompassing five pedagogical dimensions for assessing the teaching quality of L.MSs. Extensive experiments verify that SocraticLM achieves significant improve- ments in the teaching performance, outperforming GPT4 by more than 12%. Our dataset and code is available at lhttts:/ h/pp::///tt thub. com
Introduction
Large language models (LLMs) have achieved impressive results across variety of tasks including natural language processing, translation, and presstinn-nsswring [54, 56 59] This draws widespread attention the potential of using L.MMs revolutionize intelligent education, especially personalized teaching 2 23 39), mainly due to their two advantages. On one hand, L.Mss have displayed human- like mastery and proficiency of knowledge in fundamental subjects like math and physics ① 48 Therefore, they can effectively deliver subject-specifie instructions to students like real teacher. On the other hand, LLMs exhibit exceptional understanding and adaptability to users' (i.e., students') inputs. They are capable of comprehending diverse students' demands, providing instant feedback, and engaging in interactive pedagogy. Based on these two aspects, there has been a lot research
Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Problem: Julie is reading 120-page book, Yesterday, she was able read 12 pages and today, she read twice many pages yesterday. If she wants to read half of the remaining pages lomorrow, how many pages should she read?
don't know how solve this problem.
don't know bow to solve this problem.
Julie read 12 2 - 24 pages today, So she was able read total of 12 + 24 - pages since yesterday. l'here are 120- 36 84 pages left read. Since she wants to read half of the remmining pages lomerrow, then she should read 84/2 = 42 pages.
That's okay. In this problem, need first calculate many pages are rensining. To do this, we nced to first count the numberr of pages Julie read today. you know calculate
Was 12 pages?
Nice but Julii read twiee as many pages today as she did yesterday. If she read 12 pages yesterday, how many pages would she have read today?
OK. 1 think it's 24,
Great! Now, we can calculate the total number of pages Julie has read since yesterday? Can figure out?
know, half of 84 42, so Julie should read 42 pages tomorow.
(a) "Question-Answeringg teaching paradigm
(b) TThought-Provokiigg teaching paradigm
Figure 1: Teaching paradigms: "Question-Answering" vs "TThought-Provookie
investigating to apply general LLM (e.g., CharGPT) to personalized teaching 5 18 building specific teaching LLMs, such as MathGPT E and EduChat [10
or
However, current bLss- personalized teaching methods predominantly adhere to a "Question- Answering" paradigm. As shown in Figure a), they passively offer functionalities such as providing answers to questions and explaining knowledge concepts to students' queries. In this process, they oversimplify the teaching into series of Q&As, directly delivering complete answers based on CoT (56), ToT [57], etc., which falls short of truly identifying the issues students may have and offering targeted assistance. Consequently, students may struggle comprehend the problem-solving process, lack a genuine improvement in their ability, and fail to resolve similar issues in the future.
In this paper, draw inspiration from the Socratic method of teaching [3 45 and propose Socrati- cLM, which achieves a novel "Thought-PProvoking Provoking" teaching paradigm as depicted in Figure [[b). The key of this paradigm is to engage students in dialogue to active participation in the learning process, which continually poses open-ended questions (marked red, e.g., " how to calculate it?") to encourage them to articulate their thoughts, challenge assumptions, and think independently. This process enables students to learn to solve a problem by themselves, thereby fostering a deeper mastery and ability. Compared with LLM-basee applications using prompt engineering directly (e.g.. GPT4), we aim to systematically study 1) The pedagogical demands of empower SocraticLM to fulfill these demands. 2) The teaching abilities of teachers and reinforce these abilities in SocraticLM. 3) The cognitive states of students and enable Socratic.M to accurately identify them during the teaching process. Consequently, our SocraticLM can provide higher quality guidance that is more tailored and appropriate for each student's 's needs, transitioning from a "guardian of knowledge" "choreographer of learning".
Prrnghiht- teaching and
To build SocraticLMM, we first construct a new dataset, SocraTeach, which consists of 35K high-quality, fine-grained Socratic-style multi-round teaching dialogues grounded in mathematical problems. In constructing the dataset, we propose a novel
Teacher-Studern " pipeline, implementing three LLM agents simulate the key roles in authentic teaching scenarios: Dean, Teacher, and Student. The Dean is a director that oversees and refines the Teacher's instructions before they are presented to the Student, ensuring that the whole teaching process adheres to the Socratic style. The Teacher actively and gradually guides the Student to solve a problem by generating Socratic instructions, inspired by classic pedagogical theories [13 45). The Student responds to the Teacher's instructions, where we establish a student cognitive state system that simulates six kinds of students in classroom to cover real and diverse teaching scenarios. Through multiple rounds of "Teacher-Stude Student" " interaction under the supervision of Dean, comprehensive Socratic teaching dialogue is formed. One step further, to enhance the diversity and robustness of our dataset, we summarize four types of student responses from real teaching scenarios and perform data augmentation to generate extra 22K single-round teaching dialogues, specifically tailored to enhance four corresponding crucial teaching abilities.
thtps://www.mathgt..com hhtps///eduppt.coo//
2

---

We fine-tune ChatGLM3-6b [12 on our SocraTeach dataset to obtain SocraticLM. During this process, we elaborate three training strategies to improve the pedagogical abilities while ensuring the sooblem-solvingeoo capacity of SocraticI.M simultaneously. In addition, we contribute novel evaluation system encompassing five pedagogical dimensions for assessing the teaching quality of L.MMs, which to the best of our knowledge, is the first exploration in this field. Experimental results show that our dataset can enhance the pedagogical performances of LLMs and the teaching quality of our SocraticLM surpasses GPT4 by more than 12%.
The contributions of this paper are:
We present SocraticLM, language model that achieves Socratic "Thought-Provoking teach- ing paradigm. Experimental results show that its Socratic teaching quality exceeds GPT4 by 12%, while maintaining the good problem-solving abilitt of the original ChatGLM3-66
We construct new dataset SocraTeach that contains massive, fine- aine-grinned Socratic teaching dialogues. To construct SocraTeach, we propose a novel "Dean- Soann-Teache--Stueeee agent pipeline, in which we design innovative supervisory role Dean, a cognitive state system to direct the Student's 's behavior, and an enhancement in four teaching abilities for the Teacher. This pipeline is general and can be transferred to the teaching in other subjects.
multi-
We develop five-dimensional comprehensive evaluation system to assess the teaching quality of LLMs, which to the best of our knowledge, is the first attempt in the field.
2 Related Work
LLMs-enhanced intelligent education. Large language models (LLMs) revolutionize three typical applications of intelligent education, namely automatic generation of educational resources, instant assessment of student learning outcomes, and personalized teaching assistance [26] 29, 42 For
educational resources, there is a tendency to use L.MMs to generate texthooks, exercises, etc., based on teaching goals and needs, providing teachers with richer inspiration [6]. For students' outcomes, LLMs can analyze students' homework and exams to provide assessments and feedbacks on their learning progress [9] As for the most concerned personalized teaching in this paper, one line of research uses general LLMS like ChatGPT to provide students with multi-level assistance 44] [1 58
in multiple disciplines, such writing [18), programming [5), and medical education 225 By analyzing students' learning data and behavioral patterns, these LLMs also have the potential to design unique learning paths to help students learn more effectively (20). Another line of research is to collect a large amount of teaching instructions to fine-tune large models (e.g., EduChat (T0). giving them targeted teaching capabilities such as problem solving and emotional support.
Personalized teaching dialogue dataset. Constructing teaching dialogues is the basis for building LLM-based personalized teaching systems. In the literature, early attempts relied on crowd-sourcing (e.g., CIMA 44) or rules (e.g., AutoTTtor (17) to create authentic dialogues. Subsequently, re- searchers adopted human-computer collaborative approaches. For instance, QuizBot (46 leveraged semantic similarity algorithms to analyze real students' responses and provided adaptive question by predefined teaching workflow. However, these methods required substantial manual effort or were constrained by predefined teaching procedures, resulting in limited scalability and difficulty in covering diverse real-world teaching scenarios. Recently, with LLMs demonstrating advantages in synthetic data generation 28), utilizing them to assist in teaching dialogue generation has attracted much attention. However, existing research (51) suggests that GPTSs make for bad teacher, thus cur- rent efforts mainly use LLMs to simulate students with different backgrounds [41], personalities [4]] and error types 44)], followed by human teachers providing explanations. Nevertheless, this process still requires human involvement, resulting in the latest dataset MATHDIAL 40) containing only 3K samples. Besides, these datasets also lack systematic investigation into Socratic teaching.
3 The SecraTeach Dataset
Pedagogical theories point out that there are two basic demands for Socratic teaching 13 45
1)
it is fundamentally dialogic, relying on conversations between teachers and students to facilitate learning: 2) uses probing questions to actively engage students, promoting independent thinking and encouraging them to find answers themselves. In building our SocraTeach dataset to meet these
3

---

Prehlem: Julie reading 1204page book, Yestenaay, read pages and today, many pages yesterday, half remaining pages fomoroow, how muny pages should she read?
Step-by-step Guiding Questions
many pages Julie read Inday?? many pages
Q2: How many pages did Julic since yestenduy?
read!
04: How many pages should she read tomerrow?
"Dean- Sracher-Student' Pipeline
Teaching Ability Enhancement
Multi-round Teaching Dialogue
Single Teaching Dialogue
Yy How poges did Jule read today?
S.
Was it 12 pages?
Ng Was 12 pages?)
Augmentation
Dean Judge & Revise
Tx: Nice try, but Julle reod txtce is many pages today is she did yestrday, If ste red 12 pages yesteday, con you calculate the result of 12 times 2?
Irreferant
weather tuday?
Questioning
Why coust how many'
pages Julle read today firet!
Judge
The [TeacherJss iesponse doess't it meet the tenching criseria because diseclly asked the [Stadeni] to colottsie the reult of times ineiadd poiring the crorr the [Staieni made their
Incorrect-repfy
papas?
Connectrappl
Was 11:24 piges?
Teucher
Student
Revise
This guoaium
Fyt Nioc ury but roli twies as many pagee uslay the did yesterbyy If the read 12 pages yestenday.
because
Student Cognitive State System
baw mane pages would she have read today?
Teucher
road
Coereel New
Figure 2:
Workflow of our SocraTeach dataset construction.
requirements, face the following challenges. First, for the teacher, there is considerable variability in pedagogical methodologies and presentation styles among teachers. It may be difficult for a model to learn all of them once, which may result in confusion and errors within the teaching logic. Second, for the student, in real teaching scenarios, students' cognitive states are intricate and heterogeneous (2 35). While some students have strong understanding abilities and sufficient knowledge, there are also a considerable number of students who cannot understand the problem or even lack the essential knowledge. We expect that our dataset should cover all these situations, so that the model can learn to provide different levels of guidance for students with different states.
To solve the above challenges, construct SocraTeach as follows. First, for each problem that needs to be taught, we decompose a list of step-wise guiding questions (Section B.1). On this basis, we can control the simulation of teachers by aligning the instructional approach and explanatory style with these questions. Second, we devise an innovative "DeannTeachherSSuuee three L.LM agents including "Dean", "Teacher", and "Student" collect fine-grained multi-round teaching dialogues (Section 3.2). Especially, to align with student profiles in authentic scenarios, we build a cognitive state system to simulate six kinds of students in "Student" from the aspects of comprehension, calculation, knowledge mastery, etc. (Section 3.3). Finally, to further enhance the diversity and robustness of SocraTeach, we design data augmentation methods construct additional single-round teaching dialogues for improving four crucial teaching abilities (Section 3.4.
pipeline, implementing
3.1 Problem Collection & Step-by-Step Guiding Questions
In this paper, we take the teaching of mathematical problems at the primary school level as example for exploration, because mathematics is a fundamental and critical subject and such problems involve the examination of students' basic understanding and reasoning abilities [3] 34). Our problems are sourced from two representative datasets: MAWPS (27) and GSM8K S which contain 2.3K and 8.8K problems, respectively.
To ensure that the expression style and teaching approach are consistent in simulating the teacher role. we decompose each problem into series of step-by- by-ett guiding questions, such as Q1-Q4 in Figure [ (please refer to Appendix A for details). It should be noted that to ensure the efficiency and conciseness of teaching, numerical calculation and a summary of the solution do not count as step.
3.2 The Dean-Teacher-Studett Pipeline
To create our SocraTeach dataset that achieves the Socratic TThought-Provooiin teaching paradigm, we propose novel Seai--Teahh----auee teacher-student dialogues, which consists of three LLM agents:
(DTS) pipeline to collect one-to-one, multi-round,
Dean D: Research has indicated that GPT's have inadequacies in understanding students and language expression required to serve a teacher [51]. To address this issue, propose

---

Dean agent to serve as an oversight role, which judges whether the Teacher's s instructions meet the requirements of Socratic teaching. If it thinks the instructions do not meet the requirements, it has the authority to revise them before they are presented to the Student.
The Teacher agent actively provokes the Student agent to solve problems in Socratic style, serving two primary purposes according to Socrates' educational theory [3 45). First, it should prompt the Student to think at the appropriate time with Socratic questions, such as guiding the Student to consider the next step after completing a reasoning step. Second, it needs to provide the Student with explanations of the steps and the involved knowledge points. To maintain a consistent teaching style for a given problem, the Teacher is asked to deliver teaching following the step-by-step questions constructed in Section 3.1
Teacher T:
Student S: Representing the learner within the dataset, the Student agent generates replies to the Teacher's ss instructions (i.e., questions and explanations). To ensure the authenticity and diversity of Student, we build cognitive state system that deseribes six kinds of real students in Section 3.3 and set Student to simulate one of them each time it replies.
In DTS pipeline, each teaching dialogue ((Ti, -.)... is formed by a cycle of interaction between Teacher and Student under the supervision of Dean, and each agent is simulated with GPT4, Taking Figure 2 as an example, in the first round (t = 1), the Teacher directly gives the question of the first step (i.e., Ti = Q1) constructed in Section 3.1 (this process does not need the use of LLMs). Then, the Student selects a state profile (e.g., weak knowledge mastery) from six types of cognitive portraits in Section 3.3 and generates a corresponding response S based on it (see Appendix B for prompt). After that, at t = 2, the Teacher provides instruction To in a Socratic style, in which we design pedagogical demands such as not providing answers but rather following the flow of the step-by-stee step questions. It should be emphasized that here we set up different response style for each student profile through examples (Appendix(B) 2). After the Teacher generates Tz. the Dean judges (e.g.. "... doesn't 't meet the teaching criteria") and revises it (e.g., change "can you 12 times 2?" to "how many-.. ... read today?"), focusing on 1) Whether it conforms the Socratie style. 2) Whether clearly points out the mistakes made by the Student. 3) Whether its language style resembles that of a real teacher (Appendix 3), i.e., To f D(T2). The revised response is then sent to the Student, and the next round of dialogue begins. Ultimately, if the Teacher thinks that the teaching process has been completed, will output an "[END]" token as the ending of its output, indicating to terminate the cycle. It is worth noting that although we focus teaching mathematical problems in this paper, our DTS pipeline is general and can be extended to problems in other subjects (c.g.. physics).
3.3 Student Cognitive State System
To ensure that our dataset covers the real and diverse student status throughout the teaching process, it is necessary to simulate different student cognitive states within the Studen/ agent. However, a systematic and unified definition of these states has not been established in existing research (15) Some previous studies have concentrated on states that are specific to particular subjects such as math and English 3 19 50) while others abstractly define general states based on human cognitive science, such as concentration, working memory, and logical reasoning [14] 47 53 33 Unfortunately, these definitions are either unadaptable to the teaching process or difficult to implement with LLMs. To address this issue, we review the Socratic teaching process from the perspective of students as follows. Initially, a student needs to grasp the meaning of the problem at hand. Then, he/she comprehends the instructions provided by the teachers and utilizes the computational ability and acquired knowledge to execute the instructions. Ultimately, this process fosters an interest in learning and helps to cultivate effective study results. Based on this idea, we summarize five dimensions of cognitive state:
(1) Problem Understanding: refers to the degree to which students understand the given problem.
(2) Instruction Understanding: refers to the degree to which students understand and carry out the teacher's s instructions. A student in a good state should easily accomplish these instructions.
(3) Calculation: refers to the ability to derive mathematical expressions and numbers correctly.
(4) Knowledge Mastery: refers the extent to which students have mastered knowledge.
(5) Thirst for Learning: refers to the students' desire or inclination to seek and acquire new information, ask questions, and explore possibilities.
5

---

Basically, we can define five types of students who perform poorly on one of the above dimensions. Moreover, we add sixth type of student who excels at all dimensions.
3.4 Teaching Ability Enhancement
The multi-round dialogues constructed by DTS pipeline (denoted as Diam) ensure a model to grasp the fundamental Socratic teaching paradigm. However, in Dia the Student responds only once to each instruction given by the Teacher and tends to choose simpler student portraits, leading to a lack of simulation for long-tail student responses (as discussed Appendix (D). In this section, to further enhance the diversity and robustness of our dataset, we construct more Student- Teachhr single-round dialogues Diag through data augmentation on DiaM, improving four important teaching abilities.
Specifically, in real teaching processes, students' responses can be classified as follows, First, from the macro perspective, the responses can be divided into "Irrelevant" and "Relevant". "Relevant" refers to responses directly related to the problem or instruction, while "Irvelevant" means that the responses are unrelated to the instructional content, such as asking "How's the weather today?" in Figure 2 Second, within the "Relevant" category, it can be further divided into "Questioning" and "Replying" which refers students asking questions to the teacher and answering the teacher's questions. respectively. Third, "Replying" can be further classified as "Incorrect-reply" and "C "Correct-reply" based on whether the students' responses to the teacher's 's question are correct or not. Along this line, students' responses include four categories: "Irrelevant". : "Questioning", "Incorrect-reply" and "Correct-reply", ", On this basis, there are four key teaching abilities that need targeted enhancement.
First, for "Irrelevant" responses, we expect a teacher to recognize them and redirect the conversation towards teaching, such responding "This question is unrelated ... Let's focus on the problem first.." To achieve this, we collect 200 genuine student inquiries from MOOCs that are unrelated to teaching and then construct 2, 000 single-round Stucent-Teacher Teacher dialogues by randomly inserting them into Dia and asking Teacher to refuse answering (please refer to Appendix foor details).
Second, "Questioning" " corresponds to the most crucial teaching ability, that is, a teacher should pro- vide students with accurate explanations. Regarding it, we randomly sample 2, 000 Seacher-Studen conversation ST) from Diam and use Student agent to ask three more questions St S S3 for Ti (see Appendix C for prompt). Then, we ask the Teacher agent to provide T T T ultimately forming 6, 000 single-round Student- {(S), Di )ii = 1,2,3) dialogues.
Third, a teacher should accurately identify students' " "Incorrect-reply" and point out the idea of correction. To achieve this, we similarly sample 2, 000 Teacher-Student conversation from Dias and employ rules and generation techniques to rewrite the Student's ' responses into five wrong answers. Then, we explicitly prompt the Teacher to identify the errors and provide response, obtaining another 10K instances of Student-Teacher dialogues (please refer to Appendix C for details).
Finally, to enable teachers to identify different expressions of the same "Correct-reply" for enhancing robustness, we take the same 2, 000 single-round "Teucher-Stulen"" dialogues used for "Incorrect- reply" and create two correct responses with Student (see Appendix [ for prompt). Subsequently, dialogues.
we collect Teacher's s replies and obtain another 4, 000 single-round
3.5 Dataset Overview
In summary, our SocraTeach consists of 35K multi-round dialogues Dias constructed by "Dean- Muucher-Student pipeline in Sections 3.2 and 22K single-round dialogues Dias through data aug- mentation in Section 3.4 The average number of rounds in Dia is 5.28, resulting in a total of 208K single-round dialogue examples. More statistics of SocraTeach are summarized in Appendix D
In comparison to existing teaching dialogue datasets 17 40 41 46 49), our SocraTeach first addresses the deficiency of LLMs inadequately simulating teachers [T by introducing the role of "Dean" for supervision and correction. Secondly, to the best of our knowledge, SocraTeach is the first publicly available dataset designed for Socratic teaching, which specifically enhances four key teaching abilities of Teacher. Thirdly, while existing datasets simulate different students by setting their demographic backgrounds (e.g., grade) or specific error types, SocraTeach models six cognitive states of Student during the teaching process based on pedagogical experience, which covers wider range of authentic teaching scenarios and enables L.MS to possess better teaching capabilities. Lastly, SocraTeach is fully automatically generated large-scale dataset containing 35K

---

multi-round dialogues and 22K single-round dialogues, significantly surpassing the existing datasets that rely on real human students/teachers (e.g., the latest MATHDIAL (40] contains 3K dialogues).
4 Fine-tune SocraticIM
Based on SocraTeach, we can fine-tune a SOTA LLM (e.g., ChatGLM3-6b [2] by splitting each dialogue ( (2)..)..... into multiple rounds, using the preceding context of each round {(T) Si..... S.)} as input and the Teacher's response T1+1 as output. However, it may lead to catastrophic forgetting and reduce the problem-solving ability that the model already has because these dialogues may differ from the data used for pre-training [21 30 38 Specifically, we observe decrease of 31.2% in the accuracy of SocraticLM on the GSMBK/MAWPS dataset in Section 6.2 Therefore, to enhance SocraticL_M's teaching ability without compromising its fundamental problem-solving capability, we explore the following three training strategies:
Separate Training. To maintain problem-solving ability, one direct way is to mix the dialogue and problem-solving data for training. However, we discover that it does not yield satisfactory results as shown in Section 6.2 Therefore, we adopt a separate training approach wherein we first fine-tur tune SocraticLM using dialogue data and then fine-tune it on a small amount of problem-sol -ooviing data randomly sampled from GSMSK and MAWPS. Our experiments revealed that optimal performance is achieved when the ratio a of problem-solving data to the dialogue data is approximately TO
Instruction Tuning. Inspired by (37), we employ different instructions for the dialogue data and problem-solving data, with the templates presented in Appendix F It is worth noting that, unlike the prompt for Teacher in Section 3.2 here our instruction for dialogues does not require the model to follow the step-by-step guiding questions in Section 3.1 This is because providing such information in training may lead the model to take shorteuts, that is, to simplify the teaching process into information extraction from the prompt, without truly mastering the pedagogical ability.
Mixed Prompt Setting. Training with mixed prompt settings for the same task is an important method for improving LLMs' reasoning abilities [55). To this end, in addition to the original zero-shot problem-solving data of GSM8K and MAWPS, we also construct their one-shot version for training, which consists of approximately TO of the amount of zero-shot data.
Our Socratic Teaching Evaluation System
Since there is no standard answer for the teaching process, previous metrics that calculate the similarity between model-generatee responses and annotated responses (e.g.. BLEU (43), Rouge (31) may not fully assess the teaching quality of LLMs. To address this issue, in this paper, we contribute an evaluation system encompassing five pedagogical dimensions for Socratic style and teaching abilities, which to the best of our knowledge, is the first comprehensive exploration in this field.
(I) Overall Quality (Overall): This metric is a holistic and subjective evaluation of teaching quality, requiring that the instruction satisfics Socratic style and enhances students' experience.
For Overall Quality, we randomly select 1, 000 single-round "Student-Teacher" conversations from Diass and recruit 10 well-edueated annotators to blindly rank the Teacher response pairs provided by each model and GPT4 in the same context (please refer to Appendix [Fo details). The Overall
[ppendixFFfoor W =
Quality is estimated by normalized win rate difference 1 (1 + Win+Loatt+ie! E (0, 1) (GPT4's own result 0.5). To ensure agreement of quality judge among humans, we also randomly construct 100 Teacher response pairs of SocraticLM and GPT4 and ask all annotators to judge which one is better. The Kappa score is 0.70, indicating good agreement among human annotators.
For the four Socratic teaching abilities we elaborated in Section 3.4 we propose metrics (2)-(5). For each of them, randomly select 100 corresponding single-roum round dialogues from Diag for testing.
(2) Incorrect Answer Recognition Accuracy (IARA): This dimension focuses on whether the teacher can accurately identify students" "Incorrect-reply" For example, in Figure 2 if student provides an incorrect answer (e.g., "14"), a competent teacher should be able to recognize and point it out. This process is objective and can be considered binary classification task.
7

---

Overall
IARA
CARA
SER
SRR
BLEU-4
Rouge-1
Rouge-2
Rouge-1
ChatGPT
0.29
0.42
0.93
0.62
0.19
22.8
34.3
14.4
21.3
GPT4
0.50
0.76
0.91
0.65
0.55
36.2
42.9
19.4
32.4
Vicuna-7b
0.15
0.16
0.77
0.16
0.39
22.8
34,9
14.4
22.8
Llama2-7h
0.27
0.15
0.86
0.32
0.13
28.3
35.7
24.1
Llama2-13b
0.25
0.23
0.87
0.30
0.08
27.9
36.4
14.3
23.6
Llama3-8b
0.33
0.75
0.77
0.39
0.52
27.4
33.0
10.9
22.0
ChatGLM3-66
0.11
0.18
0.87
0.46
0.07
17.1
26.2
9.1
15.7
EduChat-32b
0.37
0.48
0.77
0.40
0.03
27.8
38.4
17.9
29.2
SocraticLM (ours)
0.62
0.83
0.98
0.74
0.78
48.6
56.2
33.7
47.5
w/o Dias
0,54
0.27
0.89
0.67
0.34
42.6
52.2
32.3
44.4
w/o Irrelevant
0.57
0.79
0.87
0.69
0.43
45.7
52.3
29.8
44.2
w/o Questioning
0.58
0.74
0.92
0.53
0.83
47.8
55.0
31.9
45.9
w/o Incorrect
0.51
0.33
0.93
0.68
0.65
41.8
30.4
38.9
w/o Correct
0.60
0.70
0.58
0.70
0.76
47.4
55.1
32.8
46.6
Table I: Teaching performances. For all metrics, the higher value denotes the better performance. The best methods are highlighted in bold. The runner-up baselines are represented by underline.
(3) Correct Answer Recognition Accuracy (CARA): In contrast erro recognition, this dimension focuses on whether the model can accurately identify students' "Correct-reply" Neglecting this metric may mislead the LLMs consider any answer provided by the student as incorrect.
(4) Successful Explanation Rate (SER): This dimension focuses on whether the model can provide students with satisfactory explanations for their "Questioning" ". This metrie is subjective, but can be converted to binary classification based on students' real experience.
(5) Successful Rejection Rate (SRR): This metric is designed for the case where teacher should refuse to answer students' "Irrelevant" questions and redirect them back to the instructional content. Based whether the model refuses to answer the question, is also calculated as binary classification.
Compared with existing works in evaluating LLMs for education, our evaluation system offers three main advantages. First, it provides more comprehensive and adequate assessment. While previous works either rely on similarity metrics (e.g., BLEU 443 or limited-scalle manual evaluations (e.g.. issuing questionnaires [18), our system assesses overall teaching quality along with four key teaching abilities, which provides a more systematic organization of evaluation. Second, it enables better comparability across LLMs. Limited by the fact that student can only interact with one LLM at time, traditional human evaluations [5] 24 are difficult to compare the effectiveness of multiple models. In contrast, our system uses the same teaching dialogues shared across different models as test samples, allowing for a fair comparison of different LLMs simultancously. Third, our system is more extensive and reliable benefiting from larger SocraTeach dataset, while recent studies rely on smaller datasets, such as the latest one [40] with only around 600 testing dialogues.
Experiments
In this section, verify the effectiveness of our SocraticLM by taking ChatGPT, GPT4, Vicuna-
7b [6), Llama2-7b, Llama2-131 -1bb, Llama3-Sb [52]. ChatGLM3-61 [2], and EduChat-326 [0 as baselines. The implementation details described in Appendix G Especially, for fair comparison, for the problems taught in the testing dialogues, we omit all of their dialogues in training. In addition proposed evaluation system in Section also invite human annotators give real teacher response for each testing dialogue, taking it as standard to calculate the BLEU and Rouge.
6.1 Main Results
Table summarizes the results for all models. First, our SocraticLM, which contains 6 billion parameters, demonstrates significant improvement outperforms GPT4 by 12% on Overall, 6% on IARA, 7% on CARA, 9% on SER, and 23% on SRR, as well as over 12% in mirroring the responses of human teachers as measured by BLEU and Rouge. In Appendix H we present and analyze examples of their outputs. Second, our SocraticI. demonstrates significant improvement in SER. This indicates that the judgement and correction of our proposed Dean agent can critically boost the explanatory capabilities of large models when they function
all Socratic teaching abilities. Notably,

---

SocrateLM
M
Socniie
SocniicI
1.00
GPT4
1.0
JARA
CARA
1.0
0.8
SKK
0.75
0,5-
0.6
0.50
0.5
1-2
3-4
5-6
5-6
1-2
Number Chniding Questions
Number of Guiding Questions
Number of Guiding Questions
Number of Guading Questions
Figure 3: Performances on problems with different number of step-by-step guiding questions.
as teachers. Third, in Figure we evaluate the four teaching abilities on problems with different numbers of step-by-step guiding questions, which can reflect the difficulty of the problems. It is evident that our SocraticLM consistently outperforms GPT4 across all difficulty levels.
6.2 Ablation Study
Importance of Teaching Ability Enhancement. We explore the importance of single-round teaching dialogues Diag built for the four key teaching abilities in Section 3.4 From Table first observe significant decline (e.g., Overall Quality by 8%) when these dialogues are removed (i.e., "w/o Dias"). This illustrates the necessity of our teaching ability enhancement and confirms that our proposed four teaching abilities are effective meeting the real demands of Socratic teaching. Second, the IARA and SRR metrics decrease the most, indicating the greatest disparity between the current LLM-based teaching and human teaching may lie in the response to students' incorrect answers and irrelevant questions. Third, each time type of single-round dialogue data is eliminated, all teaching abilities will show a decline, which indicates that there is a coupling effect among different teaching abilities. Especially, the CARA metric in the absence of dialogues for students' "Correct-reply" (i.e., "w/o Correct") is even lower than when all single-round data removed ("wlo Dias"). We hold the reason may be that in this case, SocraticLM was still fine-tuned on dialogues corresponding to students' "Incorrect-relly.. This causes the model to develop a stronger tendency to perceive a student's reply as incorrect. This phenomenon further indicates that it is necessary balance different types of single-round dialogues to avoid overfitting on specific instructional patterns.
Importance of Ability-balancing Strategies. Here we discard the problem-solving data and the three ability-balancing strategies in Section 4 in training to investigate their in- fluence. From Table 2 fine-tuning without problem-solving data ("wlo Problem") will result in 31. 31.2% lower accuracy on GSM8K/MAWPS compared with Chat 6b. This could be attributed the notable dif- ferences between teaching dialogues and data used for LLM pre-training, causing a dramatic disturbance in the parameters. Besides, all three training strategies are effective. Among them, Separate T respectively. Mixed Prompt Setting might have already been employed in the LLM pre-training, hence exhibiting less noticeable improvements. Moreover, it is worth noting that SocraticLM achieves higher accuracy on MAWPS than Gha3GG. We speculate the reason is that, through fine- tuning on our SocraTeach dataset, SocruticLM indeed learns to address multiple student questions about various aspects of a single problem (e-g., asking about each reasoning step and the knowledge involved). This process allows SocraticLM to develop a deeper understanding of the problem mooblem-solving process, which in turn can improve its groblem-solving accuracy.
Overall
ACCc
ACCM
ChatGLM3-6b
0.11
0.624
0.798
SocraticLM
0.62
0.606
0.814
who Problem
0.58
0.312
0.701
w/o Separate
0.54
0.159
0.646
w/o Instruction
0.02
0.320
0.625
w/o Mixed Mixed-Prompt Prompt
0.56
0.605
0.804
Table 2:
Performance without problem-solving
data and three ability-balancing training strategies Section 4 ACCG, ACC,,s represent the accu- racy on GSM8K and MAWPS, respectively.
Tuning has the greatest influence
wroblem-solving/See
teaching
6.3 Influence of Data Scale
Data scale crucial for both the efficiency and effectiveness of training large language models. In order to investigate this issue, in this section, we vary the amount of multi-round dialogues in SocraTeach dataset and the ratio a between multi-round dialogues and problem-solving data.

---

Scale of Multi-round Dialogues. To study the impact of different data scale, we randomly select 25%, 50%, 75% multi-
0.90
Owaal
IARA
round dialogues from SocraTeach and expand it to 125% di- alogues by running DT' pipeline more times to train Socrat- icL.M. The results in Figure 4 indicate that (i) Our data is not only effective but also impactful at different scales, which can significantly enhance the teaching capability of LLMs, (ii) As the volume of data increases, we observe a corresponding increase in the teaching ability. This correlation highlights the importance of data quantity in model performance. Specifi-
0.30
FARA
0.70-
SER
0.60
0.50
25%
50%
100%
125%
Scale of Multi-roundd Dialogues
Figure 4: Effects of dialogue scale.
cally, it is noteworthy that minimum of 75% (+< 26K) dia- logues is required for surpassing the Overall Quality of GPT4, (iii) As the volume of data surpasses the 35K threshold, it tends approach saturation point where further increases in data volume yield smaller incremental benefits to the model's capability. Specifically, at the 125% data scale, the IARA metric shows decline, indicating that the root cause of this saturation is decrease in the model's ability to identify incorrect answers (the decline in Overall Quality is a subsequent result), This may be because, with the increase in multi-round dialogue data, the proportion of single-round dialogue data for "Incorrect reply" decreases. When multi-round data scale exceeds 125%, this proportion may fall below a certain threshold, which results in diminishing effectiveness.
Scale of Problem-solving Data. Figure S mance changes of SocraticLM as we adjust the ratio a between cobllem-solving data and dialogue data. The trend indicates that having too few too much problem-solving data does not lead to satisfactory problem-solving ability. Instead, a balance needs to be struck with the teaching dialogue data. In fact, an excessive introduction of wooblem-solving data may even result in 1.9% decrease in accuracy GSM8K. This could be attributed to that the parameters corresponding to SocraticI.M problem-soll vini ability might undergo disturbances after ini- tial fine-tuning using the teaching dialogues. When retraining with problem-solving data, it requires re-strike delicate balance between underfitting and overfitting of this ability.
shows the perfor-
0.85
0.80
0.75
Overal
0.70
ACOs
0.65
0.60
0.55
1/40
1/20
1/10
1/5
of Prohlem solving Data
Figure 5: Effects of problem-solving data scale.
7 Conclusion
In this paper, we introduced SocraticLM, LLM designed to facilitate Socratic "Thought-Provoking" personalized teaching. To build SocraticLM,, we proposed a "Dean-Teacher-e Teacher-Student" pipeline to construct SocraTeach dataset, which simulated six student cognitive states and strengthened four crucial teaching abilities. Besides, we developed a comprehensive teaching ability evaluation system for LLMs. Experiments demonstrated that SocraticLM significantly outperforms current LLMs such as GPT4 and validated the necessity of each component within the SocraTeach dataset. We discuss more cases, the broader impacts, limitations, and future work in Appendix H and []
Acknowledgments and Disclosure of Funding
This research was partially supported by grants from the National Key Research and Development Pro- gram of China (No.202 XFF0901005) National Natural Science Foundation of China (No. 62477444 62337001), and the Key Technologies R&D Program of Anhui Province (No.202443k099220039)
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXi::2303.08774,, 2023.
[2] Tufan Adigüzel, Mehmet Haldun Kaya, and Fatih Kürşat Cansu. Revolutionizing education with ai: Exploring the transformative potential of chatgpt. Contemporary Educational Technology, 2023.
10

---

[3] Geoffrey D Borman, Gina M Hewes, Laura T Overman. and Shelly Brown. Comprehensive school reform and achievement: A antaysss Review of educational research, 1352):225- 230, 2003.
[4] Andrew Caines, Luca Benedetto, Shiva Taslimipoor, Christopher Davis, Yuan Gao, Oeistein Andersen, Zheng Yuan, Mark Elliott, Russell Moore, Christopher Bryant, et al. On the applica- tion of large language models for language teaching and assessment technology. arXiv preprint arXiv:2307.08333, 2023.
[5] Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang- Yi Li. Gptutor: chatgpt-powered programming tool for code explanation. In International Conference on Artificial Intelligence in Education, pages 321-327. Springer, 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa, Imsys. org (accessed 14 April 2023). :6, 2023.
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:22100 10. 1416, 2022.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14166. 2021.
[9] Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gaševié, and Guanliang Chen. Can large language models provide feedback to students? case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323-325, IEEE, 2023.
[10] Yuhao Dan. Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, et al. Educhat: A lcrge-scale language model- based- chatbot system for intelligent education. arXiv preprint arXiv:2308.02773, 2023.
[11] Ian J Deary, Steve Strand, Pauline Smith, and Cres Fernandes. Intelligence and educational achievement. Intelligence, 35(1):13-22 2007.
[12] Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335, 2022.
[13] Linda Elder and Richard Paul. The role of socratic questioning in thinking, teaching, and learning. The Clearing House, 71(5):297-30, 1998.
[14] Amy S Finn, Matthew A Kraft, Martin R West, Julia A Leonard, Crystal E Bish, Rebecca E Martin, Margaret A Sheridan, Christopher FO Gabrieli, and John DE Gabrieli. Cognitive skills, student achievement tests, and schools. Psychological science, 25(3):736-744, 2014.
[15] Maren Formazin, UIrich Schroeders, Olaf Köller, Oliver Wilhelm, and Hans Westmeyer. Studierendenauswaall im fach psychologie. Psychologische Rundschau, 2011.
[16] Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. Large language models in education: Vision and opportunities. In 2023 JEEE International Conference on Big Data (BigData), pages 4776-4785. IEEE, 2023.
[17] Arthur C Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M Louwerse. Autotutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36: 180-192, 2004.
[18] Jieun Han, Haneul Yoo, Yoonsu Kim, Junho Myung, Minsun Kim, Hyunseung Lim, Juho Kim, Tak Yeon Lee, Hwajung Hong, So- Yeon Ahn, et al. Recipe: How to integrate chatgpt into efl writing education. Proceedings of the Tenth ACM Conference on Learning @ Scale, pages 416-420, 2023.
11

---

[19] Eric A Hanushek and Steven G Rivkin. Generalizations about using value-added measures of teacher quality. American economic review, 100(2):27--71,, 2010.
[20] Christothea Herodotou, Bart Rienties, Avinash Boroowa, Zdenek Zdrahal, and Martin Hlosta, large-scale implementation of predictive learning analyties in higher education: The teachers' role and perspective. Edlucational Technology Research and Development, 17:1223-1306 2019.
[21] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models, In International Conference on Learning Representations, 2021.
[22] Liya Hu, Zhiang Dong, Jingyuan Chen, Guifeng Wang. Zhihua Wang, Zhou Zhao, and Fei Wu. Ptadisc: a cross-course dataset supporting personalized learning cold-start scenarios. Advances in Neural Information Processing Systems, (6:444776-44996 2023.
[23] Zhenya Huang, Qi Liu, Yuying Chen, Le Wu, Keli Xiao. Enhong Chen, Haiping Ma, and Guoping Hu. Learning or forgetting? dynamic approach for tracking the knowledge proficiency of students. ACM Transactions on Information Systems (TOIS), 38(2):1-333 2020.
[24] Jennifer Jin and Mira Kim. Gpt-empowered personalized elearning system for programming languages. Applied Sciences, 13(23):12773, 2023.
[25] Tanisha Jowsey, Jessica Parkes,,arish, Rachelle Singleton, and Michael Todorovic. Medical education empowered by generative artificial intelligence large language models. Trends in Molecular Medicine, 2023.
[26] Enkelejda Kasneci, Kathrin Seßler, Stefan Kücheeann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
[27] Rik Koncel-Kedziorski Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157, 2016.
[28] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society, Advances in Neural Information Processing Systems, 36, 2024.
[29] Qingyao Li, Lingyue Fu, Weiming Zhang, Xianyu Chen, Jingwei Yu, Wei Xia, Weinan Zhang, Ruiming Tang, and Yong Yu. Adapting large language models for education: Foundational capabilities, potentials, and challenges. arXiv preprint arXiv:2401 08666, 2023.
[30] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
[31] Chin-Yew Lin, Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pages 74-81, 2004.
[32] Xin Lin, Zhenya Huang. Hongke Zhao, Enhong Chen, Qi Liu, Defu Lian, Xin Li, and Hao Wang. Learning relation-enhanced hierarchical solver for math word problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[33] Jia-Yo Yu Liu, Fei Wang, Hai-Ping Ma, Zhen Ya Huang, Qi Liu, En-Hong Chen, and Yu Su. A probabilistic framework for temporal cognitive diagnosis online learning systems. Journal of Computer Science and Technology, 38(6): 1203-1222, 2023.
[34] Jiayu Liu, Zhenya Huang, Zhiyuan Ma, Qi Liu, Enhong Chen, Tianhuang Su, and Haifeng Liu. Guiding mathematical reasoning via mastering commonsense formula knowledge. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1477-1488, 2023.
12

---

[35] Qi Liu, Zhenya Huang. Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Exereise-awar aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1): 1011100111, 2019.
[36] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. P- tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2022.
[37] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631 -22648. PMLR, 2023.
[38] Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.0874 2023.
[39] Haiping Ma, Changqian Wang, Hengshu Zhu, Shangshang Yang, Xiaoming Zhang, and Xingyi Zhang. Enhancing cognitive diagnosis using un- an-interacted exercises: A collaboration-as aware mixed sampling approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pages 8877-8885, 2024.
[40] Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Mathdial: A dialogue tutoring dataset with rich peda- gogical properties grounded in math reasoning problems. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5602-5621. 2023.
[41] Julia M Markel, Steven G Opferman, James A Landay, and Chris Piech. Gpteach: Interactive ta training with gpt-based students. In Proceedings of the tenth acm conference on learning @ scale, pages 226-236, 2023.
[42] Steven Moore, Richard Tong, Anjali Singh, Zitao Liu, Xiangen Hu, Yu Lu, Joleen Liang, Chen Cao, Hassan Khosravi, Paul Denny, et al. Empowering education with lmms-the next-gen interface and content generation. In International Conference on Artificial Intelligence in Education, pages 32-37. Springer, 2023.
[43] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[44] Minju Park, Sojung Kim, Seunghyun Lee, Soonwoo Kwon, and Kyuseok Kim. Empowering personalized learning through a conversation tutoring system with student modeling. arXiv preprint urXiv:2403. 14071, 2024.
[45] Richard Paul and Linda Elder. Critical thinking: The art of socratic questioning. Journal of developmental education, (1):3:3, 2007.
[46] Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe-Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L Murnane, Emma Brunskill, and James A Landay. Quizbot: A dialogue-based adaptive learning system for factual knowledge. Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1-13, 2019.
[47] Yueqi Shi and Shaowei Qu. Analysis of the effect of cognitive ability on academic achievement: Moderating role self-monitoring. Frontiers in Psychology, 13:996504, 2022.
[48] Aarohi Srivastava, Denis Kleyjo, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolatingthe capabilities of language models. Transactions on Machine Learning Research, (5). 2023.
[49] Katherine Stasaski, Kimberly Kao, and Marti A Hearst. Cima: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-64, 2020.
13

---

[50] Jianwen Sun, Rui Zou, Ruxia Liang, Lu Gao, Sannyuya Liu, Qing Li, Kai Zhang, and Lulu Jiang. Ensemble knowledge tracing: Modeling interactions in learning process. Expert Systems with Applications, 207:1 17680, 2022.
[51] Anaïs Tack and Chris Piech. The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues. International Educational Data Mining Society, 2022.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922: 09288, 2023.
[53] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang. Yu Yin, Shijin Wang, and Yu Su. Neuraled: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 35(8):8312-832) 8327, 2022.
[54] Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. Document-level machine translation with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[55] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2021.
[56) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824-24837, 2022.
[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[58] Liang Zhang, Jionghao Lin, Ziyi Kuang, Sheng Xu, Mohammed Yeasin, and Xiangen Hu. Spl: A socratic playground for learning powered by large language mode. arXiv preprint arXiv:2406.1 13919, 2024.
[59] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
14

---

A More details about problem decomposition
decomposition of each problem was provided in the GSM8K raw data, which we found largely met our needs. Thus, we use GPT4 to decompose the problems in MAWPS with the prompt in A 1.
A.1: Problem Decomposition Prompt
You are a math teacher, and want you to answer the math problems give you in steps. In each step, you need first give a guided question and then explain the process of solving meeting the following criteria:
After all the steps, the final result is the same as the Answer
The steps need to as few as possible
Not calculation process as step
Not a result summary as step
Different steps are divided by "m'
Please ensure that your steps are well-organized, precise, and easy follow, with each step building upon the previous one.
#Problem: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
#Answer: 72
#Steps: How many clips did Natalia sell in May? ** Natalia sold 48/2 24 clips in May.nHoo many clips Natalia sell altogether April and May? ** Natalia sold 48+24 72 clips altogether in April and May.
#Problem: (Here is the target problem.) #Answer: (Here the target answer.)
B Prompts of "Dean-Teacher-Studeer pipeline
B.1: Prompt of Student Agent
You are primary school student, please refer to the dialogue and generate a possible response for me. In generating a response, you first need choose one of the following SITUATIONS and then respond as student who fits that situation, meeting the following criteria:
You are not a teacher, do not give any teacher reply!
Don't ask questions about the people in the question.
Your response should cohere with the given Dialogue.
SITUATIONs: (1) student does not understand the meaning of the problem; (2) student does not understand the content explained by the teacher; (3) a student makes error in calculation; (4) student has poor knowledge mastery; (5) student has weak thirst for
knowledge; (6) student has strong abilities in aspects.
Here are some examples:
#Problem: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips May. How many clips did Natalia sell altogether in April and May?
#Dialogue: [Teacher]]How many clips did Natalia sell in May? May?lm//tudennte [[Suudent]]'m not good at math. Can you explain it again?'n] n| [eacher]]Noo Noo problem all. Let's break it down. If Natalia sold half as many clips in May as she did in April, and she sold 48 clips in April, then how many clips will she have sold in May? Let's take the number from April and half it. What do you get? #Response:Choos SITUATION (ttudueett]] get it, the half of 48 is 240.
Here is the target problem:
#Problem: (Here is the target problem.)
#Dialogue: (Here is the target dialogue.)
15

---

B.2: Prompt of Teacher Agent
am primary school student. You are teacher that always responds in the Socratic style, You could not give me the answer but always try to ask just the right question help me learn to think for myself. You need to break the problem down into simpler parts until it's at the right level for me. Please explain the above steps to me in the form of questions, starting with [Step IJ. If you think you have finished teaching, please put V"ENDJJ" at the end of your response.
(Choose one of the following examples according to Student's SITUATION): (SITUATION 1)
#Problem: Natalia sold clips to 48 of friends in April, then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
#Steps: [step 1| How many clips did Natalia sell in May?'n[step 2] How many clips did Natalia sell altogether in April and May?
#Dialogue: Teacher]: #Dialogue:Teacher]::: How many clips did Natalia sell in May?'n May?nn[Student]: Student]]: I'm confused. Does \"half as many'" mean the same number as April?
#RResponse[ Teacher]: fact, \"half manyl" means half the number. So if Natalia sold clips in April, how many clips did she sell in May?
(SITUATION 2)
#Problem: Randy has 60 mango trees on his farm. Hc also has 5 less than half many
coconut trees as mango trees. How many trees does Randy have in all on his farm?
#Steps: [step 1] How many mango trees does Randy have?'n[step 2] How many coconut trees does Randy have?'n|Step 3] How many trees does Randy have in all on his farm?
#Dialogue: Teacher]: How many mango trees does Randy have?'n [Student]: You mean how many are there totally just mango?
#Response: Teacher]: We're talking about just the mango trees now. So, considering just the mango trees, can you tell me how many Randy has according to the problem?
(SITUATION 3-6)
Here is the target problem:
#Problem: (Here is the target problem.)
#Steps: {Here is the target steps.)
#Dialogue: (Here is the target dialogue.)
B.3: Prompt of Dean Agent
You are a teaching director. The following is the teaching process of a Socratic teacher. Please evaluate the response of [Teacher] in the last round and judge whether it meets the following teaching criteria:
If the [Student] is asking commonsense struggles to understand a concept, the [Teacher] is allowed to give direct explanation. In other cases, direct answer solution the problem can be given in declarative rhetorical manner.
[[[[tudent] makes mistake, it is necessary imply by way of question what error he/she made.
[Taache]] needs to speak like teacher and does not use phrases like V"let ask you!". - The response must be smooth and fluent.
If you think the response meets the criteria, please reply [TTuue], otherwise reply [False], explain your judgement and modify it according the criteria.
16

---

Here are some examples:
#Problem: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips May. How many clips did Natalia sell altogether in April and May?
Bacllouuu:[TTccertr How many clips did Natalia sell in May?'n[[tudent I'm confused. Does \"hlf many'" mean the same number as AprccP:n[Tcahher] Not exactly, V"half many means half the amount. If Natalia sold 48 clips in April, what would be half of 48? That would tell how many she sold in May. Can you solve that? #Response:[[rurs [True].
#Problem: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
#Dialogue:[ [Teacher:::: How many clips did Natalia sell divided by 2? So, [eechrre:: You're the right track, dividing by 2 to figure out the number of clips sold in May. But let me ask you this: does the 48 add to 48 divided by 2, or do you divide 48 by #Response:| [rapooss::FFsssl False] The [TTachhe]'s response is incorrect and does not point out that the [Student] made calculation error. [Modified Teacher]: You're the right track! But you may make a mistake in calculations. Can you calculate again what is the result of 48 plus 48 divided by 2?
Is 48 plus 48
its first?
#Problem: Betty is 60 years old, and she is the oldest person in the family. Her daughter is 40 percent younger than she is, and her granddaughter is one-third her mother's age. How old the granddaughter?
#Dialoguc: [aralogu:[[Tacchr] How old is Betty's daughter?'m| (Stuutett 60 pereent of 60 is... 36, so 36? 36?in[Teacher] You're the right track thinking about pereentages, but remember when say somcone is 1*40 pereent younger!", we need to subtract that percentage from 100%. Can you revisit your calculation with this in mind?
#Response:[ [False] The [Student] correctly gets the answer of "How old is Betty's daughter" but the [Teacher] still asks the same question. [Modified Teacher]: Yes! You have got the correct result of Betty's daughter's age, Now you can calculate how old is Betty's granddaughter. Can you have a try?
Here is the target problem:
#Problem: {Here is the target problem.) #Dialogue: (Here is the target dialogue.)
C More Details about Teaching Ability Enhancement
To build single-round dialogues for "Irrelevant" student response, first randomly select 2, 000 dialogues from Di constructed by DTS pipeline. From each dialogue, we randomly select one round of Student's ss response and replace it with randomly selected question from the 200 questions collected from MOOCs. We then use Teacher agent to refuse answering the question under the supervision of Dean, finally forming 000 single-round Student-Teache dialogues.
C.1: Prompt of Questioning
#Problem: [Here the target problem.)
#Answer: (ller is the target answer.)
#Analysis: (Tlere is the target analysis.)
#Dialogue history: {Here is the target dialogue history.)
You are a primary school student, please for the last round of [Dialogue history], put forward three questions that primary school students may ask in concise language. Different questions should be separated by
To build five wrong answers for "Incorrect-rely". on one hand, we identify all the numbers (e.g.. 22") and operators (c.g., "+") in Student's 's reply and randomly introduce perturbation within the
17

---

C.2: Prompt of (In)correct-reply
You are primary school student, please rewrite the last round of [Studenti'ss reply in (Dialogue history] into (5 wrong) 2 correct replies, meeting the following criteria:
Only answer the question raised by the [Teacher] in the last round, do not ask anything clse. Reply in the tone of primary school student.
Different replies should he separated by **.
#Problem: (Here is the target problem.3
#Answer: (Here is the target answer.]
#Analysis: {llere the target analysis.
#Dialogue history: (Here the target dialogue history.)
Number Problems
11,147
Number of Multi-round Dialogues in Diam
35,151
Number Single-round Dialogues in Diag
22,000
Total Number of Single-round Dialogues Maximum / Minimum Number of Rounds
207,581
12/3
Average Number of Step-by-step Guiding Questions
3.29
Average Number of Rounds in Dias Average Length of Student's s response Average Length of Teacher's response
5.28
16.4
30.3
Table Statisties of our SocraTeach dataset.
range of 10 to one number or randomly replace one operator with another. We apply these rules to obtain two new Student's responses. On the other hand, we use GPT4 to rewrite the original Student's s response and generate three incorrect answers using prompt in 2 With these five responses, we prompt the Teacher response and obtain 10K Soudent-Teache dialogues.
D Statistics of SocraTeach Dataset
SocraTeach consists of 35K multi-round dialogues Dia and 22K single-round dialogues Dias- The average number of rounds in Dia is 5.28. On average, the Stu- dent/feacher's responses contain 16.4/30.3 words respectively. The overall statisties is summarized in Table Moreover, from Figure ( a). most teaching dialogues consist of 5-6 rounds. In 6)b) visualize the probability mass function of Student's cognitive states in SocraTeach, where (1)-(5 correspond to student portrait that performs poor in one of the dimensions in our proposed student cognitive system (Section 3.3). while (6) corresponds
As stated in Section 3.5
student with strong states in all dimensions. can be seen that the Student agent tends to simulate a student who is excellent in all aspects or alternatively, has problems in calculation ability (i.e.. "(3)") or knowledge mastery (i.e., "(4)").
0.30
Number of Dialogues
0.20
Probabiitt
0.10
0.05
8
10
12
(3)
(
(6)
Number of Rounds
Type of Student Cognitive State
(a)
(b)
Figure Distributions of Number Rounds (a) and Student Cognitive State (b).
18

---

E
Instruction Tuning Template
Figure shows the instruction template for teaching dialogue data and problem-solving solving data.
Problem
Template for teaching dialogue data
Natalia sold clips to 48 of friends April, and then she sold half many clips in May. How many clips did Natalia sell altogether April and May?
You are a Socratic teacher, please guide me to solve the problem with heuristic questions based the following information. [Problem] <Problem> [Answer] <Answer> [Analysis] <Analysis>
Answer
Analysis
Template for problem-solving data
Natalia sold 48/2 24 clips in
72
May. Natalia sold 48+24 72 clips altogether in April and May.
Please analyze and solve the following problem step step: <Problem>>
Figure 7: The templates for instruction tuning.
F
Human Evaluation for Teaching Quality
To evaluate the Overall Quality of different LLMs, we recruit 10 well-educated annotators to rank the instructions provided by each model compared with GPT4. Figure shows the template.
Dialogue ID:
Teaching Quality Evaluation
Read the following dialogue in which teacher explains mathematical problem student. students' round response, rate which teacher's instruction better from aspects: Understanding: Assess whether the teacher correctly understands the student's intention.
Explanation: Whether the teacher effoctively solves the students' problem and provides appropriate and actionable guidance.
Language: Evaluate whether the teacher's instruction conforms to the demands of Socratic teaching, including that it presented question and does give direct
Readability: Assess whether the teacher's instruction easy read not too blunt.
#Problem: store, Uno Giant Family Card costs $12. When Ivan bought pieces, he given discount of $2 each. How much did Ivan pay in all? #Answer: 100.
#Analysis: Instead of $12 each, an Uno Giant Family Card costs $12-$2 10 each. Ivan paid $10
10 100 for the ten pieces Uno Giant F Family Card.
#Steps: (1) How much does an Uno Giant Family Card cost?
(2) How much did Ivan pay in all?
#Dialogue history:
[Teache]]: How much does an Uno Giant Family Card cost?
[Student]: Why do we have to calculate discounts?
[Teacher]: Great question! When you buy something at discount, you pay less than the original cost. So, the original cost of Uno Giant Family Card helps understand how much moncy Ivan saved from the discount. Now, knowing this, you try find out how much Ivan paid in all after the discount?
[Student]: don't understand. How does the discount change the cost of each card?
(Teacher A]:
The discount is like reduction in the card's original price. In this instance, if the original price of cach card is $12 and there's discount of S2, you'd subtract the discount from the original price. So, new cost of each card would
$2, What would that be?
[Teacher B]:
That's great question! When we say '"discount", we mean reduction in the original price. So, if Ivan was given a discount of $2 for each of his ten Uno Giant family cards, how much did he pay for each one?
Teacher
Same
Teacher
Prefer
Figure 8: The template of human evaluation for Overall Quality.
19

---

G Implementation Details
Our SocraticLM obtained by P-Tuning 36 ChatGLM3-6b (not ChatGLMM-6b- for epochs with learning rate of 0.02 and batch size of 64. For dataset partition, described in Section[5] the testing is composed of 400 single-round dialogues for evaluating objective teaching ability (i.e., IARA, CARA, SER, SRR) and 1, 000 multiple-round dialogues for evaluating subjective Overall Quality. Of the remaining data SocraTeach, 10%/90% used for validation/training. All experiments are conducted server with six NVIDIA RTX 3090 GPUs. Our dataset and code is available at htttts://gttiiiie wwmm/iyysstc/ocrraiic.oo
H Additional Model Samples
In the following, present four samples of our SocraticLM and GPT4 for students' "Incorrect- reply". "Correct-reply". "Questioning" and "Irrelevans" response respectively. In the first example, both SocraticI.M and GPT4 identify the student's mistake, but GPT4 gives the answer "7" directly, which contradicts the demands of Socratic teaching. In the second example, GPT4 also gives an almost complete solution process, which does not meet the requirements of step-by-step Socratic guidance of thinking. In the third example, GPT4's expression is more stilted and lacks the sense of real human teacher. In the fourth example, SocraticLM redirects the student solve the math problem first, while GPT4 is shifted to answer the irrelevant question.
Sample 1: Incorrect-reply
#Problem: Brenda's mother made cookics for 5. she prepared 35 cookies and each of them had the same number of cookies, how many did cach of them have
#Dialogue history:
[Teacher]: How can we calculate the number of cookies each person gets?
[Student]: Can't you just divide for me?
[Teacher]: Well, you could just do that, but it's important to understand why we divide.
equally distribute something amongst certain number of people or things. In this case, if Brenda's mother made 35 cookies for 5 people, how would you divide those cookies that each person gets the amount?
Dividing allows
[Student]: Oh, do we just divide the total number of cookies by the total number of people?
[Teacher]: That's right! Now using this method, how many cookies does each person get if
Brenda's mother made 35 cookies for people?
[Student]: I'm not What is 35 divided by 5?
[Teacher]: Good question! Let's think about it. If you have 5 groups, and you need to distribute 35 items equally, how many would go each group? You start seeing how many times can fit into 35 without going over. Can you try to do that?
[Student]: So does that mean each person gets cookies?
SocraticLM:
That's a good start! But let's double-chcck. If you divide 35 by 5, what do you get?
GPT4:
Not quite. If you visualize group of 35 objects, and you divide it into 5 equal groups, you would see that each group actually contains objects. In this case, each person would get 7 cookies. Does that make sense?
Sample 2: Correct-reply
#Problem: Jeff makes 10 donuts each day for 12 days. Jeff eats one of the donuts each day. Chris then comes over and eats donuts. If donuts fit each box, how many boxes can Jeff fill with his donuts?
#Dialogue history:
[Teacher]: How many donuts does Jeff make?
20

---

[Student]: Does !"makes"" include the ones cats or not?
[Teacher]: When we say \"Jeff makes!", we're referring to the total number of donuts Jcff' creates, whether he them or So, how many donuts total did Jeff make? [Student]: Jeff' makes 10 donuts each day and he made donuts for 12 days, so he made 120 donuts total.
[Teacher]: That's correct! Now, how many donuts does Jeff' eat?
[Student]: Jeff eats donuts because one each day and made donuts for 12 days.
SocraticL.M:
Excellent! Now, if Jeff eats 12 donuts and he made 120 donuts in total, how many donuts are left after he eats?
GPT4:
Exactly! Now, there is another character introduced in the problem named Chris. How many donuts does Chris cat? Given that Chris cats donuts, how we calculate the total number remaining donuts? Once we have the total number of remaining donuts, what do we do to find out the number boxes required? Now, can you calculate the number boxes that can Jeff needs store all of his remaining donuts?
Sample 3: Questioning
#Problem: Matias is salesman in bookstore. He sold 7 books Tuesday and three times as many on Wednesday. If the sales from Wednesday were tripled on Thursday, how many books did Matias sell during these three days combined?
#Dialogue history:
[Teacher]: How many books did Matias sell on Wednesday? [Student]: What are you asking?
SocraticL.M:
Let's break it down. If Matias sold books Tuesday, and he sold three times many Wednesday, how many books did he sell on Wednesday?
GPT4:
How many books did Matias sell Tucsday? And if he sold three times more Wodnesday, can you calculate how many books that would be?
Sample 4: Irrelevant
#Problem: Solomon collected three times many cans Juwan. Levi collected half of what Juwan collccted. Solomon collected 66 cans. How many cans did the boys collect in all? #Dialogue history:
[Teacher]: How many cans did Juwan collect?
[Student]: How study grammar?
SocraticLM:
That's a great question! But let's first figure out how many cans Juwan collected. If Solomon collected three times many Juwan, and Solomon collected 66 cans, can you calculate how many cans Juwan collected?
GPT4:
That's entirely different topic, but can help. Study of grammar means understanding the structure of a language. We have verbs, nouns, adjectives, adverbs, ctc. a sentence. Understanding how these work together is studying grammar. Have you tried reading about grammar before?
21

---

I Broader Impacts
In this paper, we propose a large language model SocraticLM achieve Socratic teaching, which is crucial pedagogy in intelligent education and has significant impact our daily lives. On the one hand, as explained in Section 3.3 and 3.4 our SocratieLM can provide instructions to students with different cognitive states and various types of responses. Therefore, it can support multiple personalized applications in classrooms and online platforms, delivering high-quality instructions more efficiently and conveniently to students from various backgrounds, thus promoting educational equity and rapid development. On the other hand, our work, we use GPT4 to simulate conversations between teachers and students generate dialogue data, entirely without the involvement of real human teachers or students. This approach provides a way to collect data without interacting with or testing real humans, reducing the burden of human interaction while also avoiding privacy, security, and other ethical concerns. However, since our model needs to trained with multi-round teaching dialogue data, in order to expand more subjects (e.g., physics), it may require more educational resources (e.g.. problems textbooks) and training costs.
J Limitations and Future Work
First, from Table the SER metric of SocraticLM and GPT4 is 0.74 and 0.65, respectively, This shows that the current models have room for improvement in their ability to respond to real and complex student questions. Second, focus on the teaching of mathematical problems in this paper. For other subjects, we need additional data construction and training processes. Third, the testset of moblem--sllingg ability in this paper is the problems that our Socratic teaching dialogues are based on (i.e. GSM8K and MAWPS). In order to more accurately assess the changes in reasoning ability, we will test additional datasets and explore ability- ballitinn training strategies for more tasks. Finally, in this paper, we use Chat GLM3-6b, an open source large language model our base to construct Lo, because is easy to fine-tune and has not been pre-trained with teaching capabilities, which can better verify the effect of our SocraTeach dataset. In the future, we will use dataset to fine-tune more LLMs and explore their potential for teaching and intelligent education.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly explained the scope of this paper and listed the contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We present a "Limitations and Future Work" section in Appendix tto discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We describe in details the construction process of our dataset in the main paper, all prompts used this paper in Appendix A and and the setups needed to reproduce the experimental results in Appendix G
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our dataset and code is available at / httpg//ttuub.cmm/yyystt/
SocraticLM
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipsscccppbllt... wwtppbblcggguiiieecceeeeeeooeeet http://ddese/eee/e sondeeSubmsssiooPooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly describe the dataset splits, hyperparameters, training methods, and GPU devices in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: This paper evaluates the quality of Socratic teaching by human annotators. In order to ensure the consistency annotators, calculate the Kappa score and the result is 0.70, which ensures the credibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present the computation resources in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpht/tttte.ea.pp.../.a//aaaaaa
Answer: [Yes]
Justification:
This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We present a "Border Impacts" section in Appendix too discuss the potential positive/negative socictal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets including data and bascline models used in this paper are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
latasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We clearly introduce the new dataset our paper and provide its documentation in the link
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We include the full template given to the human annotators in Appendix
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: In this paper, we invite human annotators only to assess LLMs' outputs (i.e., give ratings). The annotators themselves are not the subjects of the evaluation and are not being tested. Besides, as shown our annotation template in Appendix F this evaluation process does not collect personal information or privacy of the annotators, and the annotators are fully aware of the purpose of the evaluation and have consented to its use.
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "1P2pNO9Ff7", "reviewer": "Reviewer_twUm", "summary": "In this paper, the authors fine-tuned a language model on synthetic data for Socratic Personalized Teaching and evaluated the performance of the proposed model in comparison with a number of baseline options. \n\nThe authors proposed a multi-agent data synthesis pipeline. Using GPT-4, the authors simulated responses from both the AI teacher (the language model) and the student (the human user.) The author also included a \"dean\" agent role for evaluating proposed responses against a set of principles for Socratic Personalized Teaching. The authors fine-tuned a language model on the resultant synthetic data. The authors recruited human annotators to evaluate the overall quality of the fine-tuned model. Based on their synthetic dataset, the authors proposed a set of additional metrics for evaluating the performance of language models in the context of Socratic Personalized Teaching.\n\nThe authors also shared insights on ways to prevent forgetting during the fine-tuning process.", "questions": "It is unclear whether the seed questions from GSM8K and MAWPS behind the synthetic data for evaluation are entirely separate from the ones for generating the train split of the synthetic dataset. For example, would the same question from GSM8K appear in both the synthetic evaluation set and the synthetic fine-tuning set? Line 610 in the limitation section seems to allude to that, but wasn't entirely clear in my opinion.\n\nWhile \"Overall Quality\" is evaluated through human annotation (see appendix F), what about the metrics IARA, CARA, SER, and SRR? Are these human-in-the-loop as well? It will be helpful if you can elaborate on these details. \n\nAs a side note, the repository (https://anonymous.4open.science/r/NeurIPS-4310) does not seem to contain code for data synthesis.", "limitations": "The use of ChatGLM3-6B as the base model is a possible limitation- given that the base model correctly solves only around 65% of the GSM8K questions (vs ~90% for GPT-4,) it might be unclear whether the model would be able to consistently give a correct answer. The authors acknowledged this limitation in Appendix J by stating that the same synthetic dataset can be used to fine-tune other LLMs. This limitation (related to forgetting during fine-tuning) was also touched upon in section 6.3. \n\nOverall, I find the handling of this limitation quite reasonable- given that this paper is about Socratic Personalized Education, not about improving scores on GSM8K. Nevertheless, the authors might want to adjust the evaluation setup to account for this limitation- e.g., excluding from the evaluation GSM8K questions that cannot be solved using the original ChatGLM3-6B.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The multi-agent pipeline for synthesizing data in the field of Socratic Personalized Teaching is a rather novel approach. The proposed method of leveraging synthetic data for evaluation resolves the limitation where human-written data in this field is difficult to obtain.\n\nResults on the impact of the \"Scale of Problem-solving Data\" (Section 6.3) provides useful insights into the trade-off between fine-tuning data (teaching style) and data from the original domain (ability to solve problems.) \n\nExamples listed in the appendix suggest that the proposed approach indeed performs better than general-purpose baseline models such as GPT-4 or GPT-3.5.", "weaknesses": "Given that the evaluation method is a key contribution of this paper, the authors might also want describe how their approach compare with related works in evaluating LLMs for education. \n\nInstead of specifying one of the six \"Student Cognitive State\" when generating a response as a student, the authors allowed the LLM to pick one of these options on its own (see section B.1,) resulting in an imbalanced data distribution (see figure 6 (b) in the appendix.) It might be helpful if the authors can elaborate more on this design choice, as personally I find it unclear how this imbalanced distribution might affect the model performance in the real world.\n\nIt is nice to see that fine-tuning ChatGLM3-6B on the training split of the synthetic dataset improved performance on the test split, exceeding that of GPT-4, which generated the synthetic responses. Nevertheless, it is unclear from the paper how robust the model would be when the data distribution changes. One possible approach might be to collect additional MOOC question-answer data (as the authors already did) and test the model on these examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your affirmation of the novelty of our pipeline, the contribution of our evaluation system, and the significance of our experiments.\n\n$\\bf{Q1}$:Compare with related works in evaluating LLMs for education?\n\n$\\bf{A1}$:Thanks for your valuable question. The related works can be divided into two categories. The first involves objective similarity assessments with human-annotated instruction, using metrics like BLEU and BERTScore[1]. The second involves subjective human evaluations, such as calculating the percentage where student reaches the correct answer given the instructions[1], analyzing the correlation between student grade changes and the use of LLMs[2,3], and issuing questionnaires[4].\n\nCompared to them, our evaluation system offers several advantages:\n\n1. More comprehensive and adequate assessment. It includes evaluations of overall teaching quality and four key teaching abilities, which current works lack a systematic organization.\n\n2. Better comparability. Since a real student can only be taught by one LLM at a time, it is hard to compare the teaching effects of different LLMs by the aforementioned subjective human evaluations. In comparison, our evaluation is based on shared teaching dialogues, which can support the fair evaluation of multiple LLMs simultaneously.\n\n3. More extensive. The latest dataset[1] contains only 600 testing dialogue data, which is far less than our data volume.\n\nFrollowing your suggestions, we will supplement these discussions in our paper to highlight our contributions.\n\n[1] A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems.\n\n[2] Gptutor: a chatgpt-powered programming tool for code explanation. \n\n[3] GPT-Empowered Personalized eLearning System for Programming Languages. \n\n[4] Recipe: How to integrate chatgpt into efl writing education. \n\n$\\bf{Q2}$:Why allow the LLM to pick instead of specifying student cognitive states? The imbalanced distribution affects the performance?\n\n$\\bf{A2}$:Thanks for your insightful question. We allow the LLM to pick a student cognitive state for two reasons. First, real teaching processes are inherently complex, which may cover multiple cognitive dimensions (e.g., calculation, knowledge mastery) within a single teaching dialogue. Thus, manually specifying the state may limit the diversity and hinder the simulation of real teaching process. Second, it may disrupt the dialogue fluency. For example, if the Teacher agent asks \"Do you know the concept of trigonometry?\", it would be unnatural to ask the Student agent to response according to a weak \"Problem Understanding\" state, which may lead to an almost irrelevant reply, making it less like a real teacher-student conversation. \n\nThe imbalanced distribution will affect the model's performance. For instance, for a real student with weak \"Instruction Understanding\" state (i.e., type 2 in Figure 6), a model trained on an imbalanced dataset with much less data on type 2 may struggle to provide appropriate instructions. To address this, in Section 3.4, we employ data augmentation for four key teaching abilities, and the ablation study in Section 6.2 clearly verifies the necessity of this enhancement.\n\n$\\bf{Q3}$:Robustness when the data distribution changes (e.g., test on more MOOC question-answer pairs) and what about the dataset partition?\n\n$\\bf{A3}$:Thanks for your meticulous attention to our data collection. Firstly, sorry for our unclear statements. In Section 3.4, we collected 200 genuine student inquiries from MOOCs that are unrelated to teaching to conduct data augmentation for students’ “Irrelevant” responses, rather than collecting question-answer pairs. Secondly, we greatly appreciate your concern regarding the model robustness. As stated in lines 316-317 of Section 6, we remove the dialogues for questions in the evaluation set during training. Therefore, the issue of \"the same questions appear simultaneously in both the evaluation set and the fine-tuning set\" does not arise. This also addresses your point about the distribution shift or the need to test on new MOOC data, as our test set has not appeared in training, which validates the robustness of our model. \n\n$\\bf{Q4}$:Elaborate on metrics IARA, CARA, SER, and SRR.\n\n$\\bf{A4}$:Yes! These metrics are also evaluated manually. Since they are considered objective binary classification tasks as explained in Section 5, when annotating, we simply provide the preceding dialogue and ask annotators to evaluate whether the LLM's response \"identifies the incorrect/correct student reply\" (for IARA/CARA), \"addresses the student's question\" (for SER), or \"refuses to answer the student's irrelevant question\" (for SRR). \n\n$\\bf{Q5}$:Code for data synthesis.\n\n$\\bf{A5}$:Thanks for your comments. We supplement all the code of our agents needed for data synthesis in the folder “dataset_synthesis” in our repository. If the paper is accepted, we will also release all the data and code for public use as soon as possible. \n\n$\\bf{Q6}$:Evaluation that excludes the GSM8K questions that ChatGLM3 cannot be solved.\n\n$\\bf{A6}$:Thanks for your constructive suggestion. From the Table below, we observe that the performances of LLMs improve, but the increase is not particularly significant. We think this is because, in constructing our SocraTeach dataset, the prompts for Dean and Teacher agents include the correct solutions for the problems. In other words, we do not require ChatGLM3 to solve the problems directly but rather to learn how to teach the correct solution to a student. Thus, the problem-solving ability of ChatGLM3 does not set the upper limit for the teaching ability of our SocraticLM. \nModel|Overall|IARA|CARA|SER|SRR\n -|-|-|-|-|-\nChatGPT|0.34±0.079|0.50±0.010|0.95±0.004|0.68±0.079|0.23±0.001\nGPT4|0.50±0.000|0.78±0.059|0.91±0.014|0.68±0.002|0.53±0.005\nChatGLM3|0.14±0.039|0.21±0.052|0.88±0.010|0.49±0.079|0.07±0.009\nSocraticLM|0.66±0.045|0.84±0.091|1.00±0.000|0.78±0.076|0.77±0.006", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LE5nkGGCfN", "reviewer": "Reviewer_3yrk", "summary": "In this paper, the authors propose a novel SocraticLM to address the limitations of existing personal teaching methods that follow a “Question-answering” paradigm. To do this, the authors first propose a novel “Dean-Teacher-Student” pipeline to collect multi-round Socratic teaching dialogues, where the authors also design a student cognitive system to guides the agent behaviors. Then, the authors elaborate on four teaching abilities and propose corresponding data augmentation. Next, the authors fine-tune ChatGLM3 on the collected dataset with three strategies to balance teaching and reasoning abilities. Finally, the authors design the first systematic evaluation system for the teaching quality of LLMs. Experimental results clearly show the superiority of SocraticLM. Overall, this paper is easy to follow and presents a nice structure.", "questions": "1.I think the Dean agent is important in the proposed pipeline. Therefore, I have the question: How to assess the importance of Dean agent? Can it be reflected by comparing the results of SocraticLM and GPT4?\n2.In Table 1, comparing the results of “w/o DiaS” and “w/o Correct”, I have a minor question: why the CARA metric of “w/o Correct” is lower than “w/o DiaS”? In my opinion, “w/o Correct” incorporates more single-round teaching dialogues, and thus its performance should be better.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n1.The motivation to introduce a Socratic-teaching LLM that follows “Thought-Provoking” paradigm is reasonable and necessary for AI and intelligent education.\n2.The “Dean-Teacher-Student” multi-agent pipeline can sufficiently collect high-quality teaching dialogues. Besides, this pipeline is general enough to expand to other domains. Additionally, the data augmentation for four teaching abilities makes sense and is clearly organized.\n3.This paper discuss and design a comprehensive systematic evaluation system for the teaching quality of LLMs, which provides an effective way in the related research.\nThe experimental results compared with 8 LLMs are sufficient, and the significant improvement over GPT4 clearly verifies the effectiveness of SocraticLM. Besides, the experiments for evaluating the necessity of component of SocraticLM including single-round dialogues, training strategies, making the paper instructive.", "weaknesses": "I have some concerns:\n1.One of the most important roles in the proposed pipeline is Dean agent, which plays a supervisory role compared to previous work. Thus, I recommend to add more analyses of its effectiveness in experiments. (please refer to question 1 below).\n2.The single-round teaching dialogues are also important in this paper because they correspond to four crucial teaching abilities. Thus, I recommend to add more explanations of the experimental results in Section 6.2. (please refer to question 2 below).\n3.I found some typos, for example,\n--Line 193, “a student need” should be “a student needs”\n--Line 275, “dialogue” should be “dialogues”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your affirmation of the significance of our motivation, the innovation and generalizability of our pipeline, the contributions of our evaluation system, and the effectiveness of our SocraticLM. \n\n$\\bf{Q1}$:How to assess the importance of Dean agent?\n\n$\\bf{A1}$:Many thanks for your careful comment. Yes, as you have noted, the effectiveness of Dean agent can be reflected by comparing the performance of SocraticLM and GPT-4. For a given preceding dialogue, the Dean agent judges and corrects the responses generated by the GPT-4 simulated Teacher agent. This is the fundamental root for the differences between GPT-4 and SocraticLM after fine-tuning. Therefore, the superior performance of SocraticLM compared to GPT-4 as shown in Table 1 can directly demonstrates that Dean agent is effective and necessary.\n\n$\\bf{Q2}$:Why the CARA metric of “w/o Correct” is lower than “w/o DiaS”\n\n$\\bf{A2}$:Thank you very much for the question. We think the reason is that the \"w/o Correct\" model was still fine-tuned on 10K single-round dialogues corresponding to students’ \"Incorrect-reply\" as explained in Section 3.4. This led the model to develop a stronger tendency to perceive a student's reply as incorrect, causing it to incorrectly classify some correct student replies as wrong. As a result, the model's performance on the CARA metric was worse compared to the model that did not include single-round data training (i.e., “w/o DiaS”). This phenomenon indicates that it is necessary to construct data for the \"Correct-reply\" category and balance it with the \"Incorrect-reply\" category. \n\n$\\bf{Q3}$:Exist some typos.\n\n$\\bf{A3}$:Thank you for pointing this out. We will carefully review and polish the writing of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mz3TGFMae9", "reviewer": "Reviewer_sW5a", "summary": "The authors propose a novel method based on Socratic teaching for improving LLM teaching abilities.", "questions": "- why does performance fall on some metrics at 125% data scale?\n- will the full SocraTeach dataset be publicly released?\n- I would be open to increasing my score if concerns from the previous section are addressed", "limitations": "- authors discuss limitations but would be worth it to add that current experiments are only in English, and that current human evaluations come from a very small (and likely not broadly representative) sample", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- novel, interesting, and well-described method for improving LLM teaching ability\n- creation and release of a useful, novel teaching dialogue dataset\n- propose and validate novel ways of testing LLM teaching ability\n- extensive experiments; I especially appreciate the ablation study\n- code released\n- I really appreciate the figures which do a great job of laying out the proposed methods and providing examples", "weaknesses": "- results could benefit from having standard errors or confidence intervals (authors say 'Yes' to this in NeurIPS checklist but only list having the kappa score in justification)\n- there seem to be a large number of related methods (e.g., search for \"socratic LLM teaching\" on Google Scholar), the authors should contextualize their work among this literature in the related works section, and clarify how their method differs from and outperforms existing methods from that literature (e.g., a goog example is Socratic Playground - https://arxiv.org/abs/2406.13919)\n\nMinor (did not affect score):\n- Line 122: \"While some student\" --> \"While some students\"\n- Line 173: \"such not providing\" --> typo here\n- Line 312: \"refuse\" --> \"refuses\" or \"refused\"\n- Table 1 caption: \"denote the\" --> \"denotes\"\n- Line 595: \"Border\" -> \"Broader\"\n\nEthics flag:\n- Research involved human subjects but unclear if authors had IRB approval. Not sure whether IRB approval is needed or not in this case so flagging for review by relevant experts just in case (note: the authors say the paper involves no crowdsourcing or human subjects in question 15 of the NeurIPS checklist, right after saying it does in question 14).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your affirmation of the novelty of our SocraticLM, the value of our constructed teaching dialogue dataset, the innovation of our teaching ability evaluation system, and the clarity of our writing. \n\nFor your concerns regarding the ethics flag, we want to express our sincere gratitude for your attention and sorry for the misunderstanding. In this paper, we aim to construct a SocraticLM for personalized teaching. To achieve this, we first propose a “Dean-Teacher-Student” (DTS) pipeline to collect large-scale Socratic teaching dialogue data, where we implement all agents with GPT-4 and do not incorporate human subjects. As for our evaluation system, similar to other works, we invited 10 annotators to assess and compare the teaching instructions of different LLMs to test their performances. As shown in our annotation template in Appendix F, this process also does not involve research on humans or privacy/security risks. Following your suggestion, we will provide the necessary clarification and corrections to the conference and give a clearer introduction of our annotation process in the revised version. The following are the responses to your other questions:\n\n$\\bf{Q1}$:Standard errors or confidence intervals.\n\n$\\bf{A1}$:Thanks for your constructive comments. We add the following results:\nModel|Overall|IARA|CARA|SER|SRR\n -|-|-|-|-|-\nChatGPT|0.29±0.024|0.42±0.034|0.93±0.001|0.62±0.052|0.19±0.020\nGPT4|0.50±0.000|0.76±0.015|0.91±0.017|0.65±0.060|0.55±0.050\nChatGLM3|0.11±0.006|0.18±0.014|0.87±0.022|0.46±0.015|0.07±0.006\nSocraticLM|0.62±0.029|0.83±0.028|0.98±0.015|0.74±0.039|0.78±0.026\n\n$\\bf{Q2}$:How this paper differs from and outperforms existing methods.\n\n$\\bf{A2}$:Thanks for your thought-provoking question. As summarized in our related work section, existing methods on Socratic teaching with LLMs can be divided into two categories. The first uses a general LLM (e.g., ChatGPT, GPT-4) to assist in conversation design in courses[1], content authoring[2], explaining learning-paths[3], and providing feedback[4]. Notably, the work you mentioned[5] uses GPT-4 to create an innovative Socratic Playground for Learning, constructing diverse learning scenarios where GPT-4 interacts with students in a Socratic manner. The second category collects data to train a specialized teaching LLM, with EduChat[6] being the most representative (also used as our baseline). \n\nCompared to them, our SocraticLM improves from two main aspects. 1) When constructing the SocraTeach dataset, we build a student cognitive system to simulate six kinds of authentic students and enhance four key teaching abilities. This enables SocraticLM to handle more complex teaching scenarios and have more comprehensive and systematic teaching abilities. 2) Our DTS pipeline contains a novel Dean agent to judge and revise the GPT-4-based Teacher agent, addressing the limitations that existing general LLMs may still make for a bad teacher[7] and improving teaching quality. Experiments show that SocraticLM outperforms GPT-4 and EduChat across various dimensions, verifying the effectiveness of our pipeline and model. \n\nThank you for pointing out the excellent works and we will cite them in our revised paper.\n\n[1] Chatgpt in the generalized intelligent framework for tutoring. \n\n[2] Ruffle&riley: Towards the automated induction of conversational tutoring systems. \n\n[3] Supporting student decisions on learning recommendations: An llmbased chatbot with knowledge graph contextualization for conversational explainability and mentoring. \n\n[4] How can i get it right? using gpt to rephrase incorrect trainee responses.\n\n[5] SPL: A Socratic Playground for Learning Powered by Large Language Model. \n\n[6] Educhat: A large-scale language model-based chatbot system for intelligent education. \n\n[7] The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues. \n\n$\\bf{Q3}$:Minor typos.\n\n$\\bf{A3}$:Thank you very much for thoroughly reading our paper and pointing out them. We will revise our paper carefully.\n\n$\\bf{Q4}$:Performance fall on some metrics at 125% data scale?\n\n$\\bf{A4}$:Many thanks for your insightful question. In Figure 4, at the 125% data scale, the metrics that decline are IARA and overall quality, indicating that the root cause is a decrease in the model's ability to identify incorrect answers (the decline in overall quality is a subsequent result). This may be because, with the increase in multi-round dialogue data, the proportion of single-round dialogue data for “Incorrect reply” decreases. When multi-round data scale exceeds 125%, the proportion of this single-round dialogue data may fall below a certain threshold, which results in diminishing effectiveness. Many thanks for pointing out this phenomenon. We will supplement these discussions and explanations in the revised paper.\n\n$\\bf{Q5}$:Will dataset be released?\n\n$\\bf{A5}$:Yes! We have already made the test set of our SocraTeach dataset and the model training code available through an anonymous repository (https://anonymous.4open.science/r/NeurIPS-4310). If our paper is accepted, we will release the full dataset as soon as possible.\n\n$\\bf{Q6}$:Current experiments are only in English and human evaluations come from a small sample.\n\n$\\bf{A6}$:Thanks for your constructive comments. Although our current dataset is in English, as mentioned in Section 3.2, our \"Dean-Teacher-Student\" pipeline is general and can easily be applied to other datasets, such as the Chinese Math23K dataset. For human evaluations, we sampled over 1,000 dialogues from SocraTeach dataset to calculate metrics and calculated the Kappa score within the annotators. The results demonstrate the consistency of the manual evaluations and the effectiveness of our SocraticLM. Inspired by your comments, we will consider constructing multilingual teaching datasets and training judging networks based on human-annotated results for larger-scale automatic testing in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hMFYnUxxbp", "reviewer": "Reviewer_3PwF", "summary": "This paper aims to introduce a “Thought-provoking” paradigm into LLM-based personal teaching. The authors propose an innovative “Dean-Teacher-Student” pipeline with three LLM-based agents to collect Socratic teaching data. During this process, the authors also contribute a student cognitive system to simulate six types of authentic students with the “Student” agent. Then, the authors adopt data augmentation for four crucial teaching abilities and collect more single-round dialogue data. Finally, the authors investigate three training strategies to balance the problem-solving ability and Socratic teaching ability of LLMs. The fine-tuned SocraticLM, shows great teaching performance compared with several LLMs including GPT4 and EduChat. The authors also provide sufficient experiments to validate the importance of single-round data, training strategies, which makes the method self-contained and instructive.", "questions": "- In my opinion, the quality of the collected SocraTeach dataset highly depends on the Teacher agent, which is implemented with GPT4. Therefore, if GPT-4 itself cannot correctly solve a problem, can it still serve as a teacher in an educational role?\n- I notice that in Table 2, SocraticLM performs even better in problem-solving accuracy on MAWPS dataset. Could the authors provide some explanations for this?", "limitations": "Please see questions above.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Motivation: I agree that the idea of LLMs-based teaching is essential for current intelligent education systems, and the existing methods are hard to achieve satisfactory teaching effects through simply answering students’ questions. Therefore, this paper is beneficial for practical applications.\n\n- Dataset Contribution: I think one of the most important contributions of this paper is the proposed SocraTeach dataset that consists of high-quality teaching dialogues. It is the first public large-scale Socratic teaching dataset. The authors claim to release it, which can benefit the community to conduct more research for teaching-LLM.\n\n- Methodology: The proposed “Dean-Teacher-Student” is reasonable and easy to reproduce. The introduction of student cognitive system that simulates six kinds of students guarantee the diversity of the teaching dialogue data. It has potentials to be generalized to other works.\n\n- Comprehensive Assessment: Sufficient experiments is conducted to demonstrate the performance of SocraticLM over several LLMs, which makes the improvement convincing. Moreover, I think the analyses with different data scale is beneficial for reproducing and assist for building other educational LLMs.", "weaknesses": "Some issues that hope to be further addressed by authors:\n- It is rational and acceptable to use GPT4 to construct data, but I am interesting about whether the performance of SocraticLM will be limited by GPT4 (please refer to question 1 below).\n- I recommend the authors to provide more explanations of some experimental results, which can better reflect the effectiveness of the proposed SocraticLM (please refer to question 2 below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your affirmation of our motivation, the contribution of our dataset, the novelty and generalization ability of our pipeline, and our sufficient experiments and comprehensive assessment. \n\n$\\bf{Q1}$:If GPT-4 itself cannot correctly solve a problem, can it still serve as a teacher in an educational role?\n\n$\\bf{A1}$:Thank you very much for your insightful question. We believe it still can serve as a teacher for two main reasons. Firstly, the Dean agent can judge and correct each round of the Teacher agent's instruction. Within a single round, the instruction needed to be judged usually involves just one reasoning step, making this process easier than having GPT-4 solve a problem directly. Therefore, even if GPT-4's problem-solving ability is limited, it still has the potential to judge and revise the instruction accurately. Secondly, our pipeline does not rely on GPT-4's inherent problem-solving ability because we input the correct solutions into the prompts for both the Dean and Teacher agents. In other words, our pipeline is indeed having SocraticLM teach the correct solution to students rather than solving the problem itself. Therefore, the problem-solving ability of GPT-4 does not limit the effectiveness of our pipeline. Your question is very thought-provoking, and we will supplement these discussions in the revised version of our paper.\n\n$\\bf{Q2}$:Why SocraticLM performs even better in problem-solving accuracy on MAWPS dataset?\n\n$\\bf{A2}$:Thanks for your valuable question. We think the reason is that from fine-tuning on our SocraTeach dataset, SocraticLM indeed learns to answer multiple questions from students about various aspects of a single problem (e.g., asking about each reasoning step and the involved knowledge). This process may allow SocraticLM to develop a deeper understanding of the problem-solving process, which in turn can improve its problem-solving accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers’ efforts in reviewing our paper. We would like to thank all of them for providing constructive and valuable feedback, which we will leverage to improve this work. We are encouraged by the positive comments from reviewers, including: 
- **Motivation**: “The motivation to introduce a Socratic-teaching LLM that follows “Thought-Provoking” paradigm is reasonable and necessary for AI and intelligent education” (Reviewer 3yrk), “LLMs-based teaching is essential for current intelligent education systems” (Reviewer 3PwF).

- **Method**: “novel multi-agent pipeline” (Reviewer twUm, Reviewer 3yrk), “shared insights on ways to prevent forgetting” (Reviewer twUm), “novel SocraticLM” (Reviewer 3yrk), “interesting, and well-described method” (Reviewer sW5a), “useful, novel teaching dialogue dataset” (Reviewer sW5a), “novel ways of testing LLM teaching ability” (Reviewer sW5a), “innovative, reasonable and easy to reproduce” (Reviewer 3PwF), “self-contained and instructive” (Reviewer 3PwF).

- **Experimental Results**: “provides useful insights” (Reviewer twUm), “performs better than baseline models such as GPT-4, GPT-3.5, and EduChat” (Reviewer twUm, Reviewer 3PwF), “sufficient experimental results” (Reviewer 3yrk, Reviewer 3PwF), “significant improvement over GPT4” (Reviewer 3yrk), “especially appreciate the ablation study” (Reviewer sW5a).

- **Significance**: “resolves the limitation where human-written data in this field is difficult to obtain” (Reviewer twUm), “evaluation system provides an effective way in the related research.” (Reviewer 3yrk), “contribute a student cognitive system” (Reviewer 3PwF), “the first public large-scale Socratic teaching dataset” (Reviewer 3PwF), “beneficial for practical applications” (Reviewer 3PwF), “beneficial for reproducing and assist for building other educational LLMs.” (Reviewer 3PwF), “code and dataset released” (Reviewer sW5a, Reviewer 3PwF).

**[Response to Ethics Reviewers]**

We deeply appreciate your thoughtful consideration of the ethics flag of our paper. Firstly, our paper focuses on introducing a Socratic-style teaching LLM, SocraticLM. To achieve this, we propose a “Dean-Teacher-Student” pipeline to first collect a teaching dialogue dataset, SocraTeach. As we introduce in Section 3, the entire pipeline is implemented without human involvement, as all three agents in the pipeline are simulated using GPT-4. Therefore, from a technical perspective, our work does not raise ethical issues. Secondly, regarding our evaluation system, similar to other works that involve human evaluation of LLM outputs, we invited 10 annotators to anonymously assess and compare the outputs (i.e., teaching instructions) of different LLMs for performance testing. As shown in the annotation template in Appendix F, this process does not involve testing of annotators or research with human subjects. While our paper aimed to present and evaluate a novel personalized teaching LLM, your feedback highlights the necessity of addressing the potential ethical pitfalls. Many thanks for your concerns, and we will revise the paper to incorporate a dedicated subsection that discusses and clarifies our annotation process.

Meta-review: In this paper, the authors propose a Socratic "Thought-Provoking'' teaching paradigm that simulates the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery. They design a "Dean-Teacher-Student'' multi-agent system for data collection, with a set of authentic teaching scenarios integrated. They show that an LM fine-tuned on such data (with some strategies balancing teaching and reasoning abilities) could outperform strong baselines such as GPT4 in its teaching performance.

All four reviwers agree on the good quality of this submission: the work is novel and sound; the problem is well-motivated; the proposed method is general enough so it can be relatively easily generalized to other domains/scenarios; the presentation is clear; the experimental design is good. 

To me this is a clear accept, but to help the authors further improve their work, let me summarize the reviewers' suggestions:
- Discuss the connection between this work and prior work in the area of LLM evaluation in educational/pedagogical scenarios. (twUm, sW5a)
- Provide more analyses on how the Dean role is useful. (3yrk) 
- Provide more analyses related to the single-round teaching dialogue setting, specifically, in Section 6.2. (3yrk)
- Add error bars. (sW5a)
- Add discussion about performance fall on some metrics at 125% data scale in Figure 4. (sW5a)
- Add discussion about how the pipeline could be robust in cases where the LLM Teacher and Dean are imperfect. (3PwF)
Decision: Accept (spotlight)