SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models

Authors: jiayu  liu
zhenya  huang
tong  xiao
jing  sha
jinze  wu
qi  liu
shijin  wang
enhong  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 06:02:13

Abstract: Large language models (LLMs) are considered a crucial technology for advancing intelligent education since they exhibit the potential for an in-depth understanding of teaching scenarios and providing students with personalized guidance. Nonetheless, current LLM-based application in personalized teaching predominantly follows a "Question-Answering" paradigm, where students are passively provided with answers and explanations. In this paper, we propose SocraticLM, which achieves a Socratic "Thought-Provoking" teaching paradigm that fulfills the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery. To build SocraticLM, we first propose a novel "Dean-Teacher-Student" multi-agent pipeline to construct a new dataset, SocraTeach, which contains $35$K meticulously crafted Socratic-style multi-round (equivalent to $208$K single-round) teaching dialogues grounded in fundamental mathematical problems. Our dataset simulates authentic teaching scenarios, interacting with six representative types of simulated students with different cognitive states, and strengthening four crucial teaching abilities. SocraticLM is then fine-tuned on SocraTeach with three strategies balancing its teaching and reasoning abilities. Moreover, we contribute a comprehensive evaluation system encompassing five pedagogical dimensions for assessing the teaching quality of LLMs. Extensive experiments verify that SocraticLM achieves significant improvements in the teaching performance, outperforming GPT4 by more than 12\%. Our dataset and code is available at https://github.com/Ljyustc/SocraticLM.

Main paper: 
 introduction: 
 Large language models (LLMs) have achieved impressive results across a variety of tasks including natural language processing, translation, and question-answering [54, 56, 59] . This draws widespread attention to the potential of using LLMs to revolutionize intelligent education, especially personalized teaching [2, 23, 39] , mainly due to their two advantages. On one hand, LLMs have displayed humanlike mastery and proficiency of knowledge in fundamental subjects like math and physics [1, 48] . Therefore, they can effectively deliver subject-specific instructions to students like a real teacher. On the other hand, LLMs exhibit exceptional understanding and adaptability to users' (i.e., students') inputs. They are capable of comprehending diverse students' demands, providing instant feedback, and engaging in interactive pedagogy. Based on these two aspects, there has been a lot of research (a) "Question-Answering" teaching paradigm Problem: Julie is reading a 120-page book. Yesterday, she was able to read 12 pages and today, she read twice as many pages as yesterday. If she wants to read half of the remaining pages tomorrow, how many pages should she read? I don't know how to solve this problem.Julie read 12 x 2 = 24 pages today. So she was able to read a total of 12 + 24 = 36 pages since yesterday. There are 120 -36 = 84 pages left to be read. Since she wants to read half of the remaining pages tomorrow, then she should read 84/2 = 42 pages. I don't know how to solve this problem.That's okay. In this problem, we need to first calculate how many pages are remaining. To do this, we need to first count the number of pages Julie has read today. Do you know how to calculate it? Was it 12 pages? Nice try, but Julie read twice as many pages today as she did yesterday. If she read 12 pages yesterday, how many pages would she have read today? (b) "Thought-Provoking" teaching paradigm OK, I think it's 24.Great! Now, we can calculate the total number of pages Julie has read since yesterday? Can you figure it out? … I know, half of 84 is 42, so Julie should read 42 pages tomorrow. investigating to apply a general LLM (e.g., ChatGPT) to personalized teaching [5, 18, 44, 51] , or building specific teaching LLMs, such as MathGPT2 , EduGPT3 , and EduChat [10] .However, current LLMs-based personalized teaching methods predominantly adhere to a "Question-Answering" paradigm. As shown in Figure 1 (a), they passively offer functionalities such as providing answers to questions and explaining knowledge concepts to students' queries. In this process, they oversimplify the teaching into a series of Q&As, directly delivering complete answers based on CoT [56] , ToT [57] , etc., which falls short of truly identifying the issues students may have and offering targeted assistance. Consequently, students may struggle to comprehend the problem-solving process, lack a genuine improvement in their ability, and fail to resolve similar issues in the future.In this paper, we draw inspiration from the Socratic method of teaching [13, 45] and propose Socrati-cLM, which achieves a novel "Thought-Provoking" teaching paradigm as depicted in Figure 1(b) . The key of this paradigm is to engage students in a dialogue to active participation in the learning process, which continually poses open-ended questions (marked red, e.g., "... how to calculate it?") to encourage them to articulate their thoughts, challenge assumptions, and think independently. This process enables students to learn to solve a problem by themselves, thereby fostering a deeper mastery and ability. Compared with LLM-based applications using prompt engineering directly (e.g., GPT4), we aim to systematically study 1) The pedagogical demands of "Thought-Provoking" teaching and empower SocraticLM to fulfill these demands. 2) The teaching abilities of teachers and reinforce these abilities in SocraticLM. 3) The cognitive states of students and enable SocraticLM to accurately identify them during the teaching process. Consequently, our SocraticLM can provide higher quality guidance that is more tailored and appropriate for each student's needs, transitioning from a "guardian of knowledge" to "choreographer of learning".To build SocraticLM, we first construct a new dataset, SocraTeach, which consists of 35K high-quality, fine-grained Socratic-style multi-round teaching dialogues grounded in mathematical problems. In constructing the dataset, we propose a novel "Dean-Teacher-Student" pipeline, implementing three LLM agents to simulate the key roles in authentic teaching scenarios: Dean, Teacher, and Student. The Dean is a director that oversees and refines the Teacher's instructions before they are presented to the Student, ensuring that the whole teaching process adheres to the Socratic style. The Teacher actively and gradually guides the Student to solve a problem by generating Socratic instructions, inspired by classic pedagogical theories [13, 45] . The Student responds to the Teacher's instructions, where we establish a student cognitive state system that simulates six kinds of students in classroom to cover real and diverse teaching scenarios. Through multiple rounds of "Teacher-Student" interaction under the supervision of Dean, a comprehensive Socratic teaching dialogue is formed. One step further, to enhance the diversity and robustness of our dataset, we summarize four types of student responses from real teaching scenarios and perform data augmentation to generate extra 22K single-round teaching dialogues, specifically tailored to enhance four corresponding crucial teaching abilities.We fine-tune ChatGLM3-6b [12] on our SocraTeach dataset to obtain SocraticLM. During this process, we elaborate three training strategies to improve the pedagogical abilities while ensuring the problem-solving capacity of SocraticLM simultaneously. In addition, we contribute a novel evaluation system encompassing five pedagogical dimensions for assessing the teaching quality of LLMs, which to the best of our knowledge, is the first exploration in this field. Experimental results show that our dataset can enhance the pedagogical performances of LLMs and the teaching quality of our SocraticLM surpasses GPT4 by more than 12%.The contributions of this paper are:• We present SocraticLM, a language model that achieves Socratic "Thought-Provoking" teaching paradigm. Experimental results show that its Socratic teaching quality exceeds GPT4 by 12%, while maintaining the good problem-solving ability of the original ChatGLM3-6b.• We construct a new dataset SocraTeach that contains massive, fine-grained Socratic teaching dialogues. To construct SocraTeach, we propose a novel "Dean-Teacher-Student" multiagent pipeline, in which we design an innovative supervisory role Dean, a cognitive state system to direct the Student's behavior, and an enhancement in four teaching abilities for the Teacher. This pipeline is general and can be transferred to the teaching in other subjects.• We develop a five-dimensional comprehensive evaluation system to assess the teaching quality of LLMs, which to the best of our knowledge, is the first attempt in the field.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We have clearly explained the scope of this paper and listed the contributions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 LLMs-enhanced intelligent education. Large language models (LLMs) revolutionize three typical applications of intelligent education, namely automatic generation of educational resources, instant assessment of student learning outcomes, and personalized teaching assistance [26, 29, 42] . For educational resources, there is a tendency to use LLMs to generate textbooks, exercises, etc., based on teaching goals and needs, providing teachers with richer inspiration [4, 16] . For students' outcomes, LLMs can analyze students' homework and exams to provide assessments and feedbacks on their learning progress [9] . As for the most concerned personalized teaching in this paper, one line of research uses general LLMs like ChatGPT to provide students with multi-level assistance [44, 51, 58] in multiple disciplines, such as writing [18] , programming [5] , and medical education [25] . By analyzing students' learning data and behavioral patterns, these LLMs also have the potential to design unique learning paths to help students learn more effectively [20] . Another line of research is to collect a large amount of teaching instructions to fine-tune large models (e.g., EduChat [10] ), giving them targeted teaching capabilities such as problem solving and emotional support.Personalized teaching dialogue dataset. Constructing teaching dialogues is the basis for building LLM-based personalized teaching systems. In the literature, early attempts relied on crowd-sourcing (e.g., CIMA [49] ) or rules (e.g., AutoTutor [17] ) to create authentic dialogues. Subsequently, researchers adopted human-computer collaborative approaches. For instance, QuizBot [46] leveraged semantic similarity algorithms to analyze real students' responses and provided adaptive question by predefined teaching workflow. However, these methods required substantial manual effort or were constrained by predefined teaching procedures, resulting in limited scalability and difficulty in covering diverse real-world teaching scenarios. Recently, with LLMs demonstrating advantages in synthetic data generation [28] , utilizing them to assist in teaching dialogue generation has attracted much attention. However, existing research [51] suggests that GPTs make for a bad teacher, thus current efforts mainly use LLMs to simulate students with different backgrounds [41] , personalities [41] and error types [40] , followed by human teachers providing explanations. Nevertheless, this process still requires human involvement, resulting in the latest dataset MATHDIAL [40] containing only 3K samples. Besides, these datasets also lack a systematic investigation into Socratic teaching.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We present a "Limitations and Future Work" section in Appendix J to discuss the limitations of our work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the socrateach dataset: 
 Pedagogical theories point out that there are two basic demands for Socratic teaching [13, 45] : 1) it is fundamentally dialogic, relying on conversations between teachers and students to facilitate learning; 2) it uses probing questions to actively engage students, promoting independent thinking and encouraging them to find answers themselves. In building our SocraTeach dataset to meet these Problem: Julie is reading a 120-page book. Yesterday, she was able to read 12 pages and today, she read twice as many pages as yesterday. If she wants to read half of the remaining pages tomorrow, how many pages should she read? Step-by-step Guiding Questions Q1: How many pages did Julie read today? Q3: How many pages are left to be read?! : How many pages did Julie read today?Q2: How many pages did Julie read since yesterday? Q4: How many pages should she read tomorrow? requirements, we face the following challenges. First, for the teacher, there is considerable variability in pedagogical methodologies and presentation styles among teachers. It may be difficult for a model to learn all of them at once, which may result in confusion and errors within the teaching logic. Second, for the student, in real teaching scenarios, students' cognitive states are intricate and heterogeneous [22, 35] . While some students have strong understanding abilities and sufficient knowledge, there are also a considerable number of students who cannot understand the problem or even lack the essential knowledge. We expect that our dataset should cover all these situations, so that the model can learn to provide different levels of guidance for students with different states.To solve the above challenges, we construct SocraTeach as follows. First, for each problem that needs to be taught, we decompose a list of step-wise guiding questions (Section 3.1). On this basis, we can control the simulation of teachers by aligning the instructional approach and explanatory style with these questions. Second, we devise an innovative "Dean-Teacher-Student" pipeline, implementing three LLM agents including "Dean", "Teacher", and "Student" to collect fine-grained multi-round teaching dialogues (Section 3.2). Especially, to align with student profiles in authentic scenarios, we build a cognitive state system to simulate six kinds of students in "Student" from the aspects of comprehension, calculation, knowledge mastery, etc. (Section 3.3). Finally, to further enhance the diversity and robustness of SocraTeach, we design data augmentation methods to construct additional single-round teaching dialogues for improving four crucial teaching abilities (Section 3.4).In this paper, we take the teaching of mathematical problems at the primary school level as an example for exploration, because mathematics is a fundamental and critical subject and such problems involve the examination of students' basic understanding and reasoning abilities [32, 34] . Our problems are sourced from two representative datasets: MAWPS [27] and GSM8K [8] , which contain 2.3K and 8.8K problems, respectively.To ensure that the expression style and teaching approach are consistent in simulating the teacher role, we decompose each problem into a series of step-by-step guiding questions, such as Q1-Q4 in Figure 2 (please refer to Appendix A for details). It should be noted that to ensure the efficiency and conciseness of teaching, a numerical calculation and a summary of the solution do not count as a step.To create our SocraTeach dataset that achieves the Socratic "Thought-Provoking" teaching paradigm, we propose a novel "Dean-Teacher-Student" (DTS) pipeline to collect one-to-one, multi-round, teacher-student dialogues, which consists of three LLM agents:• Dean D: Research has indicated that GPTs have inadequacies in understanding students and language expression required to serve as a teacher [51] . To address this issue, we propose a Dean agent to serve as an oversight role, which judges whether the Teacher's instructions meet the requirements of Socratic teaching. If it thinks the instructions do not meet the requirements, it has the authority to revise them before they are presented to the Student.• Teacher T : The Teacher agent actively provokes the Student agent to solve problems in a Socratic style, serving two primary purposes according to Socrates' educational theory [13, 45] . First, it should prompt the Student to think at the appropriate time with Socratic questions, such as guiding the Student to consider the next step after completing a reasoning step. Second, it needs to provide the Student with explanations of the steps and the involved knowledge points. To maintain a consistent teaching style for a given problem, the Teacher is asked to deliver teaching following the step-by-step questions constructed in Section 3.1.• Student S: Representing the learner within the dataset, the Student agent generates replies to the Teacher's instructions (i.e., questions and explanations). To ensure the authenticity and diversity of Student, we build a cognitive state system that describes six kinds of real students in Section 3.3 and set Student to simulate one of them each time it replies.In DTS pipeline, each teaching dialogue {(T ). The revised response is then sent to the Student, and the next round of dialogue begins. Ultimately, if the Teacher thinks that the teaching process has been completed, it will output an "[END]" token as the ending of its output, indicating to terminate the cycle. It is worth noting that although we focus on teaching mathematical problems in this paper, our DTS pipeline is general and can be extended to problems in other subjects (e.g., physics).To ensure that our dataset covers the real and diverse student status throughout the teaching process, it is necessary to simulate different student cognitive states within the Student agent. However, a systematic and unified definition of these states has not been established in existing research [15] . Some previous studies have concentrated on states that are specific to particular subjects such as math and English [3, 11, 19, 50] , while others abstractly define general states based on human cognitive science, such as concentration, working memory, and logical reasoning [14, 47, 53, 33] .Unfortunately, these definitions are either unadaptable to the teaching process or difficult to implement with LLMs. To address this issue, we review the Socratic teaching process from the perspective of students as follows. Initially, a student needs to grasp the meaning of the problem at hand. Then, he/she comprehends the instructions provided by the teachers and utilizes the computational ability and acquired knowledge to execute the instructions. Ultimately, this process fosters an interest in learning and helps to cultivate effective study results. Based on this idea, we summarize five dimensions of cognitive state:(1) Problem Understanding: refers to the degree to which students understand the given problem.(2) Instruction Understanding: refers to the degree to which students understand and carry out the teacher's instructions. A student in a good state should easily accomplish these instructions.(3) Calculation: refers to the ability to derive mathematical expressions and numbers correctly.(4) Knowledge Mastery: refers to the extent to which students have mastered knowledge.(5) Thirst for Learning: refers to the students' desire or inclination to seek and acquire new information, ask questions, and explore possibilities.Basically, we can define five types of students who perform poorly on one of the above dimensions. Moreover, we add a sixth type of student who excels at all dimensions.The multi-round dialogues constructed by DTS pipeline (denoted as Dia M ) ensure a model to grasp the fundamental Socratic teaching paradigm. However, in Dia M , the Student responds only once to each instruction given by the Teacher and tends to choose simpler student portraits, leading to a lack of simulation for long-tail student responses (as discussed in Appendix D). In this section, to further enhance the diversity and robustness of our dataset, we construct more Student-Teacher single-round dialogues Dia S through data augmentation on Dia M , improving four important teaching abilities.Specifically, in real teaching processes, students' responses can be classified as follows. First, from the macro perspective, the responses can be divided into "Irrelevant" and "Relevant". "Relevant" refers to responses directly related to the problem or instruction, while "Irrelevant" means that the responses are unrelated to the instructional content, such as asking "How's the weather today?" in Figure 2 .Second, within the "Relevant" category, it can be further divided into "Questioning" and "Replying", which refers to students asking questions to the teacher and answering the teacher's questions, respectively. Third, "Replying" can be further classified as "Incorrect-reply" and "Correct-reply" based on whether the students' responses to the teacher's question are correct or not. Along this line, students' responses include four categories: "Irrelevant", "Questioning", "Incorrect-reply", and "Correct-reply". On this basis, there are four key teaching abilities that need targeted enhancement.First, for "Irrelevant" responses, we expect a teacher to recognize them and redirect the conversation towards teaching, such as responding "This question is unrelated to ... Let's focus on the problem first ...". To achieve this, we collect 200 genuine student inquiries from MOOCs that are unrelated to teaching and then construct 2, 000 single-round Student-Teacher dialogues by randomly inserting them into Dia M and asking Teacher to refuse answering (please refer to Appendix C for details).Second, "Questioning" corresponds to the most crucial teaching ability, that is, a teacher should provide students with accurate explanations. Regarding it, we randomly sample 2, 000 Teacher-Student conversation (T i , S i ) from Dia M and use Student agent to ask three more questions S 1 i , S 2 i , S 3 i for T i (see Appendix C.1 for prompt). Then, we ask the Teacher agent to provide T 1 i+1 , T 2 i+1 , T 3 i+1 , ultimately forming 6, 000 single-round Student-Teacher {(S j i , T j i+1 )|j = 1, 2, 3} dialogues. Third, a teacher should accurately identify students' "Incorrect-reply" and point out the idea of correction. To achieve this, we similarly sample 2, 000 Teacher-Student conversation from Dia M and employ rules and generation techniques to rewrite the Student's responses into five wrong answers. Then, we explicitly prompt the Teacher to identify the errors and provide a response, obtaining another 10K instances of Student-Teacher dialogues (please refer to Appendix C for details).Finally, to enable teachers to identify different expressions of the same "Correct-reply" for enhancing robustness, we take the same 2, 000 single-round "Teacher-Student" dialogues used for "Incorrectreply" and create two correct responses with Student (see Appendix C.2 for prompt). Subsequently, we collect Teacher's replies and obtain another 4, 000 single-round "Student-Teacher" dialogues.In summary, our SocraTeach consists of 35K multi-round dialogues Dia M constructed by "Dean-Teacher-Student" pipeline in Sections 3.2 and 22K single-round dialogues Dia S through data augmentation in Section 3.4. The average number of rounds in Dia M is 5.28, resulting in a total of 208K single-round dialogue examples. More statistics of SocraTeach are summarized in Appendix D.In comparison to existing teaching dialogue datasets [17, 40, 41, 46, 49] , our SocraTeach first addresses the deficiency of LLMs inadequately simulating teachers [51] by introducing the role of "Dean" for supervision and correction. Secondly, to the best of our knowledge, SocraTeach is the first publicly available dataset designed for Socratic teaching, which specifically enhances four key teaching abilities of Teacher. Thirdly, while existing datasets simulate different students by setting their demographic backgrounds (e.g., grade) or specific error types, SocraTeach models six cognitive states of Student during the teaching process based on pedagogical experience, which covers a wider range of authentic teaching scenarios and enables LLMs to possess better teaching capabilities. Lastly, SocraTeach is a fully automatically generated large-scale dataset containing 35K multi-round dialogues and 22K single-round dialogues, significantly surpassing the existing datasets that rely on real human students/teachers (e.g., the latest MATHDIAL [40] contains 3K dialogues).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
fine-tune socraticlm: 
 Based on SocraTeach, we can fine-tune a SOTA LLM (e.g., ChatGLM3-6b [12] ) by splitting each dialogue {(T 1 , S 1 ), (T 2 , S 2 ), ...} into multiple rounds, using the preceding context of each round {(T 1 , S 1 ), ..., (T i , S i )} as input and the Teacher's response T i+1 as output. However, it may lead to catastrophic forgetting and reduce the problem-solving ability that the model already has because these dialogues may differ from the data used for pre-training [21, 30, 38] . Specifically, we observe a decrease of 31.2%/9.7% in the accuracy of SocraticLM on the GSM8K/MAWPS dataset in Section 6.2. Therefore, to enhance SocraticLM's teaching ability without compromising its fundamental problem-solving capability, we explore the following three training strategies:Separate Training. To maintain problem-solving ability, one direct way is to mix the dialogue and problem-solving data for training. However, we discover that it does not yield satisfactory results as shown in Section 6.2. Therefore, we adopt a separate training approach wherein we first fine-tune SocraticLM using dialogue data and then fine-tune it on a small amount of problem-solving data randomly sampled from GSM8K and MAWPS. Our experiments revealed that optimal performance is achieved when the ratio α of problem-solving data to the dialogue data is approximately 1 10 . Instruction Tuning. Inspired by [37] , we employ different instructions for the dialogue data and problem-solving data, with the templates presented in Appendix E. It is worth noting that, unlike the prompt for Teacher in Section 3.2, here our instruction for dialogues does not require the model to follow the step-by-step guiding questions in Section 3.1. This is because providing such information in training may lead the model to take shortcuts, that is, to simplify the teaching process into information extraction from the prompt, without truly mastering the pedagogical ability.Training with mixed prompt settings for the same task is an important method for improving LLMs' reasoning abilities [7, 55] . To this end, in addition to the original zero-shot problem-solving data of GSM8K and MAWPS, we also construct their one-shot version for training, which consists of approximately 1 10 of the amount of zero-shot data. 
our socratic teaching evaluation system: 
 Since there is no standard answer for the teaching process, previous metrics that calculate the similarity between model-generated responses and annotated responses (e.g., BLEU [43] , Rouge [31] ) may not fully assess the teaching quality of LLMs. To address this issue, in this paper, we contribute an evaluation system encompassing five pedagogical dimensions for Socratic style and teaching abilities, which to the best of our knowledge, is the first comprehensive exploration in this field.(1) Overall Quality (Overall): This metric is a holistic and subjective evaluation of teaching quality, requiring that the instruction satisfies Socratic style and enhances students' experience.For Overall Quality, we randomly select 1, 000 single-round "Student-Teacher" conversations from Dia M and recruit 10 well-educated annotators to blindly rank the Teacher response pairs provided by each model and GPT4 in the same context (please refer to Appendix F for details). The Overall Quality is estimated by a normalized win rate difference 1 2 (1 + W in-Lost W in+Lost+T ie ) ∈ (0, 1) (GPT4's own result is 0.5). To ensure agreement of quality judge among humans, we also randomly construct 100 Teacher response pairs of SocraticLM and GPT4 and ask all annotators to judge which one is better. The Kappa score is 0.70, indicating good agreement among human annotators.For the four Socratic teaching abilities we elaborated in Section 3.4, we propose metrics (2)- (5) . For each of them, we randomly select 100 corresponding single-round dialogues from Dia S for testing.(2) Incorrect Answer Recognition Accuracy (IARA): This dimension focuses on whether the teacher can accurately identify students' "Incorrect-reply". For example, in Figure 2 , if a student provides an incorrect answer (e.g., "14"), a competent teacher should be able to recognize and point it out. This process is objective and can be considered as a binary classification task. (3) Correct Answer Recognition Accuracy (CARA): In contrast to error recognition, this dimension focuses on whether the model can accurately identify students' "Correct-reply". Neglecting this metric may mislead the LLMs to consider any answer provided by the student as incorrect.(4) Successful Explanation Rate (SER): This dimension focuses on whether the model can provide students with satisfactory explanations for their "Questioning". This metric is subjective, but can be converted to binary classification based on students' real experience.(5) Successful Rejection Rate (SRR): This metric is designed for the case where a teacher should refuse to answer students' "Irrelevant" questions and redirect them back to the instructional content.Based on whether the model refuses to answer the question, it is also calculated as binary classification.Compared with existing works in evaluating LLMs for education, our evaluation system offers three main advantages. First, it provides a more comprehensive and adequate assessment. While previous works either rely on similarity metrics (e.g., BLEU [43] ) or limited-scale manual evaluations (e.g., issuing questionnaires [18] ), our system assesses overall teaching quality along with four key teaching abilities, which provides a more systematic organization of evaluation. Second, it enables better comparability across LLMs. Limited by the fact that a student can only interact with one LLM at a time, traditional human evaluations [5, 24] are difficult to compare the effectiveness of multiple models. In contrast, our system uses the same teaching dialogues shared across different models as test samples, allowing for a fair comparison of different LLMs simultaneously. Third, our system is more extensive and reliable benefiting from our larger SocraTeach dataset, while recent studies rely on smaller datasets, such as the latest one [40] with only around 600 testing dialogues. 
experiments: 
 In this section, we verify the effectiveness of our SocraticLM by taking ChatGPT, GPT4, Vicuna-7b [6] , Llama2-7b, Llama2-13b, Llama3-8b [52] , ChatGLM3-6b [12] , and EduChat-32b [10] as baselines. The implementation details are described in Appendix G. Especially, for fair comparison, for the problems taught in the testing dialogues, we omit all of their dialogues in training. In addition to our proposed evaluation system in Section 5, we also invite human annotators to give a real teacher response for each testing dialogue, taking it as a standard to calculate the BLEU and Rouge.Table 1 summarizes the results for all models. First, our SocraticLM, which contains 6 billion parameters, demonstrates a significant improvement in all Socratic teaching abilities. Notably, it outperforms GPT4 by 12% on Overall, 6% on IARA, 7% on CARA, 9% on SER, and 23% on SRR, as well as over 12% in mirroring the responses of human teachers as measured by BLEU and Rouge. In Appendix H, we present and analyze examples of their outputs. Second, our SocraticLM demonstrates a significant improvement in SER. This indicates that the judgement and correction of our proposed Dean agent can critically boost the explanatory capabilities of large models when they function as teachers. Third, in Figure 3 , we evaluate the four teaching abilities on problems with different numbers of step-by-step guiding questions, which can reflect the difficulty of the problems. It is evident that our SocraticLM consistently outperforms GPT4 across all difficulty levels.Importance of Teaching Ability Enhancement. We explore the importance of single-round teaching dialogues Dia S built for the four key teaching abilities in Section 3.4. From Table 1 , we first observe a significant decline (e.g., Overall Quality by 8%) when these dialogues are removed (i.e., "w/o Dia S "). This illustrates the necessity of our teaching ability enhancement and confirms that our proposed four teaching abilities are effective in meeting the real demands of Socratic teaching. Second, the IARA and SRR metrics decrease the most, indicating the greatest disparity between the current LLM-based teaching and human teaching may lie in the response to students' incorrect answers and irrelevant questions. Third, each time a type of single-round dialogue data is eliminated, all teaching abilities will show a decline, which indicates that there is a coupling effect among different teaching abilities.Especially, the CARA metric in the absence of dialogues for students' "Correct-reply" (i.e., "w/o Correct") is even lower than when all single-round data is removed ("w/o Dia S "). We hold the reason may be that in this case, SocraticLM was still fine-tuned on dialogues corresponding to students' "Incorrect-reply". This causes the model to develop a stronger tendency to perceive a student's reply as incorrect. This phenomenon further indicates that it is necessary to balance different types of single-round dialogues to avoid overfitting on specific instructional patterns. We speculate the reason is that, through finetuning on our SocraTeach dataset, SocraticLM indeed learns to address multiple student questions about various aspects of a single problem (e.g., asking about each reasoning step and the knowledge involved). This process allows SocraticLM to develop a deeper understanding of the problem-solving process, which in turn can improve its problem-solving accuracy.Data scale is crucial for both the efficiency and effectiveness of training large language models. In order to investigate this issue, in this section, we vary the amount of multi-round dialogues in SocraTeach dataset and the ratio α between multi-round dialogues and problem-solving data. Scale of Multi-round Dialogues. To study the impact of different data scale, we randomly select 25%, 50%, 75% multiround dialogues from SocraTeach and expand it to 125% dialogues by running DTS pipeline more times to train Socrat-icLM. The results in Figure 4 indicate that (i) Our data is not only effective but also impactful at different scales, which can significantly enhance the teaching capability of LLMs. (ii) As the volume of data increases, we observe a corresponding increase in the teaching ability. This correlation highlights the importance of data quantity in model performance. Specifically, it is noteworthy that a minimum of 75% (≈ 26K) dialogues is required for surpassing the Overall Quality of GPT4. (iii) As the volume of data surpasses the 35K threshold, it tends to approach a saturation point where further increases in data volume yield smaller incremental benefits to the model's capability. Specifically, at the 125% data scale, the IARA metric shows a decline, indicating that the root cause of this saturation is a decrease in the model's ability to identify incorrect answers (the decline in Overall Quality is a subsequent result). This may be because, with the increase in multi-round dialogue data, the proportion of single-round dialogue data for "Incorrect reply" decreases. When multi-round data scale exceeds 125%, this proportion may fall below a certain threshold, which results in diminishing effectiveness. Scale of Problem-solving Data. Figure 5 shows the performance changes of SocraticLM as we adjust the ratio α between problem-solving data and dialogue data. The trend indicates that having too few or too much problem-solving data does not lead to satisfactory problem-solving ability. Instead, a balance needs to be struck with the teaching dialogue data. In fact, an excessive introduction of problem-solving data may even result in 1.9% decrease in accuracy on GSM8K. This could be attributed to that the parameters corresponding to SocraticLM' problem-solving ability might undergo disturbances after initial fine-tuning using the teaching dialogues. When retraining with problem-solving data, it requires to re-strike a delicate balance between underfitting and overfitting of this ability.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We clearly describe the dataset splits, hyperparameters, training methods, and GPU devices in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we introduced SocraticLM, a LLM designed to facilitate Socratic "Thought-Provoking" personalized teaching. To build SocraticLM, we proposed a "Dean-Teacher-Student" pipeline to construct SocraTeach dataset, which simulated six student cognitive states and strengthened four crucial teaching abilities. Besides, we developed a comprehensive teaching ability evaluation system for LLMs. Experiments demonstrated that SocraticLM significantly outperforms current LLMs such as GPT4 and validated the necessity of each component within the SocraTeach dataset. We discuss more cases, the broader impacts, limitations, and future work in Appendix H, I, and J.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: This paper evaluates the quality of Socratic teaching by human annotators. In order to ensure the consistency of annotators, we calculate the Kappa score and the result is 0.70, which ensures the credibility of our results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We present a "Border Impacts" section in Appendix I to discuss the potential positive/negative societal impacts of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets including data and baseline models used in this paper are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We clearly introduce the new dataset in our paper and provide its documentation in the link https://github.com/Ljyustc/SocraticLM.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: We include the full template given to the human annotators in Appendix F.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: In this paper, we invite human annotators only to assess LLMs' outputs (i.e., give ratings). The annotators themselves are not the subjects of the evaluation and are not being tested. Besides, as shown our annotation template in Appendix F, this evaluation process does not collect personal information or privacy of the annotators, and the annotators are fully aware of the purpose of the evaluation and have consented to its use.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A decomposition of each problem was provided in the GSM8K raw data, which we found largely met our needs. Thus, we use GPT4 to decompose the problems in MAWPS with the prompt in A.1.You are a math teacher, and I want you to answer the math problems I give you in steps. In each step, you need to first give a guided question and then explain the process of solving it, meeting the following criteria:-After all the steps, the final result is the same as the Answer -The steps need to be as few as possible -Not a calculation process as a step -Not a result summary as a step -Different steps are divided by '\n' -Please ensure that your steps are well-organized, precise, and easy to follow, with each step building upon the previous one.# #Problem: Betty is 60 years old, and she is the oldest person in the family. Her daughter is 40 percent younger than she is, and her granddaughter is one-third her mother's age. How old is the granddaughter? #Dialogue:[Teacher]: How old is Betty's daughter?\n[Student]: 60 percent of 60 is... 36, so 36?\n[Teacher]: You're on the right track thinking about percentages, but remember when we say someone is \"40 percent younger\", we need to subtract that percentage from 100%. Can you revisit your calculation with this in mind? To build single-round dialogues for "Irrelevant" student response, we first randomly select 2, 000 dialogues from Dia M constructed by DTS pipeline. From each dialogue, we randomly select one round of Student's response and replace it with a randomly selected question from the 200 questions collected from MOOCs. We then use Teacher agent to refuse answering the question under the supervision of Dean, finally forming 2, 000 single-round Student-Teacher dialogues.#Problem: {Here is the target problem.} #Answer: {Here is the target answer.} #Analysis: {Here is the target analysis.} #Dialogue history: {Here is the target dialogue history.} You are a primary school student, please for the last round of [Dialogue history], put forward three questions that primary school students may ask in concise language. Different questions should be separated by ** .To build five wrong answers for "Incorrect-rely", on one hand, we identify all the numbers (e.g., "2") and operators (e.g., "+") in a Student's reply and randomly introduce a perturbation within theYou are a primary school student, please rewrite the last round of [Student]'s reply in [Dialogue history] into (5 wrong) 2 correct replies, meeting the following criteria: -Only answer the question raised by the [Teacher] in the last round, do not ask anything else.-Reply in the tone of a primary school student.-Different replies should be separated by **.#Problem: {Here is the target problem.} #Answer: {Here is the target answer.} #Analysis: {Here is the target analysis.} #Dialogue history: {Here is the target dialogue history.} range of 10 to one number or randomly replace one operator with another. We apply these rules to obtain two new Student's responses. On the other hand, we use GPT4 to rewrite the original Student's response and generate three incorrect answers using prompt in C.2. With these five responses, we prompt the Teacher to response and obtain 10K Student-Teacher dialogues.As stated in Section 3.5, our SocraTeach consists of 35K multi-round dialogues Dia M and 22K single-round dialogues Dia S . The average number of rounds in Dia M is 5.28. On average, the Student/Teacher's responses contain 16.4/30.3 words respectively. The overall statistics is summarized in Table 3 . Moreover, from Figure 6 (a), most teaching dialogues consist of 5-6 rounds. In Figure 6 (b), we visualize the probability mass function of Student's cognitive states in SocraTeach, where (1)-( 5) correspond to a student portrait that performs poor in one of the dimensions in our proposed student cognitive system (Section 3.3), while (6) corresponds to a student with strong states in all dimensions.It can be seen that the Student agent tends to simulate a student who is excellent in all aspects or alternatively, has problems in calculation ability (i.e., "(3)") or knowledge mastery (i.e., "(4)"). Our SocraticLM is obtained by P-Tuning [36] ChatGLM3-6b (not ChatGLM3-6b-Base) for 2 epochs with a learning rate of 0.02 and batch size of 64. For dataset partition, as described in Section 5, the testing set is composed of 400 single-round dialogues for evaluating objective teaching ability (i.e., IARA, CARA, SER, SRR) and 1, 000 multiple-round dialogues for evaluating subjective Overall Quality. Of the remaining data in SocraTeach, 10%/90% is used for validation/training. All experiments are conducted on a server with six NVIDIA RTX 3090 GPUs. Our dataset and code is available at https://github.com/Ljyustc/SocraticLM.In the following, we present four samples of our SocraticLM and GPT4 for students' "Incorrectreply", "Correct-reply", "Questioning" and "Irrelevant" response respectively. In the first example, SocraticLM and GPT4 identify the student's mistake, but GPT4 gives the answer "7" directly, which contradicts the demands of Socratic teaching. In the second example, GPT4 also gives an almost complete solution process, which does not meet the requirements of step-by-step Socratic guidance of thinking. In the third example, GPT4's expression is more stilted and lacks the sense of a real human teacher. In the fourth example, SocraticLM redirects the student to solve the math problem first, while GPT4 is shifted to answer the irrelevant question.#Problem: Brenda's mother made cookies for 5 . If she prepared 35 cookies and each of them had the same number of cookies , how many did each of them have ? #Dialogue history:[Teacher]: How can we calculate the number of cookies each person gets? [Student]: Can't you just divide it for me? [Teacher]: Well, you could just do that, but it's important to understand why we divide. Dividing allows us to equally distribute something amongst a certain number of people or things. In this case, if Brenda's mother made 35 cookies for 5 people, how would you divide those cookies so that each person gets the same amount? [Student]: Oh, do we just divide the total number of cookies by the total number of people? [Teacher]: That's right! Now using this method, how many cookies does each person get if Brenda's mother made 35 In this paper, we propose a large language model SocraticLM to achieve Socratic teaching, which is a crucial pedagogy in intelligent education and has a significant impact on our daily lives. On the one hand, as explained in Section 3.3 and 3.4, our SocraticLM can provide instructions to students with different cognitive states and various types of responses. Therefore, it can support multiple personalized applications in classrooms and online platforms, delivering high-quality instructions more efficiently and conveniently to students from various backgrounds, thus promoting educational equity and rapid development. On the other hand, in our work, we use GPT4 to simulate conversations between teachers and students to generate dialogue data, entirely without the involvement of real human teachers or students. This approach provides a way to collect data without interacting with or testing on real humans, reducing the burden of human interaction while also avoiding privacy, security, and other ethical concerns. However, since our model needs to be trained with multi-round teaching dialogue data, in order to expand to more subjects (e.g., physics), it may require more educational resources (e.g., problems in textbooks) and training costs.First, from Table 1 , the SER metric of SocraticLM and GPT4 is 0.74 and 0.65, respectively. This shows that the current models have room for improvement in their ability to respond to real and complex student questions. Second, we focus on the teaching of mathematical problems in this paper. For other subjects, we need additional data construction and training processes. Third, the testset of problem-solving ability in this paper is the problems that our Socratic teaching dialogues are based on (i.e. GSM8K and MAWPS). In order to more accurately assess the changes in reasoning ability, we will test additional datasets and explore ability-balancing training strategies for more tasks. Finally, in this paper, we use ChatGLM3-6b, an open source large language model as our base to construct SocraticLM, because it is easy to fine-tune and has not been pre-trained with teaching capabilities, which can better verify the effect of our SocraTeach dataset. In the future, we will use our dataset to fine-tune more LLMs and explore their potential for teaching and intelligent education.https://www.mathgpt.com/https://edugpt.com/ 


Decision: In this paper, the authors propose a Socratic "Thought-Provoking'' teaching paradigm that simulates the role of a real classroom teacher in actively engaging students in the thought process required for genuine problem-solving mastery. They design a "Dean-Teacher-Student'' multi-agent system for data collection, with a set of authentic teaching scenarios integrated. They show that an LM fine-tuned on such data (with some strategies balancing teaching and reasoning abilities) could outperform strong baselines such as GPT4 in its teaching performance.

All four reviwers agree on the good quality of this submission: the work is novel and sound; the problem is well-motivated; the proposed method is general enough so it can be relatively easily generalized to other domains/scenarios; the presentation is clear; the experimental design is good. 

To me this is a clear accept, but to help the authors further improve their work, let me summarize the reviewers' suggestions:
- Discuss the connection between this work and prior work in the area of LLM evaluation in educational/pedagogical scenarios. (twUm, sW5a)
- Provide more analyses on how the Dean role is useful. (3yrk) 
- Provide more analyses related to the single-round teaching dialogue setting, specifically, in Section 6.2. (3yrk)
- Add error bars. (sW5a)
- Add discussion about performance fall on some metrics at 125% data scale in Figure 4. (sW5a)
- Add discussion about how the pipeline could be robust in cases where the LLM Teacher and Dean are imperfect. (3PwF)

Reviews: review: paper_summary: paper_summary: In this paper, the authors fine-tuned a language model on synthetic data for Socratic Personalized Teaching and evaluated the performance of the proposed model in comparison with a number of baseline options. 

The authors proposed a multi-agent data synthesis pipeline. Using GPT-4, the authors simulated responses from both the AI teacher (the language model) and the student (the human user.) The author also included a "dean" agent role for evaluating proposed responses against a set of principles for Socratic Personalized Teaching. The authors fine-tuned a language model on the resultant synthetic data. The authors recruited human annotators to evaluate the overall quality of the fine-tuned model. Based on their synthetic dataset, the authors proposed a set of additional metrics for evaluating the performance of language models in the context of Socratic Personalized Teaching.

The authors also shared insights on ways to prevent forgetting during the fine-tuning process.
strength_weakness: strength_weakness: Given that the evaluation method is a key contribution of this paper, the authors might also want describe how their approach compare with related works in evaluating LLMs for education. 

Instead of specifying one of the six "Student Cognitive State" when generating a response as a student, the authors allowed the LLM to pick one of these options on its own (see section B.1,) resulting in an imbalanced data distribution (see figure 6 (b) in the appendix.) It might be helpful if the authors can elaborate more on this design choice, as personally I find it unclear how this imbalanced distribution might affect the model performance in the real world.

It is nice to see that fine-tuning ChatGLM3-6B on the training split of the synthetic dataset improved performance on the test split, exceeding that of GPT-4, which generated the synthetic responses. Nevertheless, it is unclear from the paper how robust the model would be when the data distribution changes. One possible approach might be to collect additional MOOC question-answer data (as the authors already did) and test the model on these examples.
questions: questions: It is unclear whether the seed questions from GSM8K and MAWPS behind the synthetic data for evaluation are entirely separate from the ones for generating the train split of the synthetic dataset. For example, would the same question from GSM8K appear in both the synthetic evaluation set and the synthetic fine-tuning set? Line 610 in the limitation section seems to allude to that, but wasn't entirely clear in my opinion.

While "Overall Quality" is evaluated through human annotation (see appendix F), what about the metrics IARA, CARA, SER, and SRR? Are these human-in-the-loop as well? It will be helpful if you can elaborate on these details. 

As a side note, the repository (https://anonymous.4open.science/r/NeurIPS-4310) does not seem to contain code for data synthesis.
limitations: limitations: The use of ChatGLM3-6B as the base model is a possible limitation- given that the base model correctly solves only around 65% of the GSM8K questions (vs ~90% for GPT-4,) it might be unclear whether the model would be able to consistently give a correct answer. The authors acknowledged this limitation in Appendix J by stating that the same synthetic dataset can be used to fine-tune other LLMs. This limitation (related to forgetting during fine-tuning) was also touched upon in section 6.3. 

Overall, I find the handling of this limitation quite reasonable- given that this paper is about Socratic Personalized Education, not about improving scores on GSM8K. Nevertheless, the authors might want to adjust the evaluation setup to account for this limitation- e.g., excluding from the evaluation GSM8K questions that cannot be solved using the original ChatGLM3-6B.
paper_summary: paper_summary: In this paper, the authors propose a novel SocraticLM to address the limitations of existing personal teaching methods that follow a “Question-answering” paradigm. To do this, the authors first propose a novel “Dean-Teacher-Student” pipeline to collect multi-round Socratic teaching dialogues, where the authors also design a student cognitive system to guides the agent behaviors. Then, the authors elaborate on four teaching abilities and propose corresponding data augmentation. Next, the authors fine-tune ChatGLM3 on the collected dataset with three strategies to balance teaching and reasoning abilities. Finally, the authors design the first systematic evaluation system for the teaching quality of LLMs. Experimental results clearly show the superiority of SocraticLM. Overall, this paper is easy to follow and presents a nice structure.
strength_weakness: strength_weakness: I have some concerns:
1.One of the most important roles in the proposed pipeline is Dean agent, which plays a supervisory role compared to previous work. Thus, I recommend to add more analyses of its effectiveness in experiments. (please refer to question 1 below).
2.The single-round teaching dialogues are also important in this paper because they correspond to four crucial teaching abilities. Thus, I recommend to add more explanations of the experimental results in Section 6.2. (please refer to question 2 below).
3.I found some typos, for example,
--Line 193, “a student need” should be “a student needs”
--Line 275, “dialogue” should be “dialogues”
questions: questions: 1.I think the Dean agent is important in the proposed pipeline. Therefore, I have the question: How to assess the importance of Dean agent? Can it be reflected by comparing the results of SocraticLM and GPT4?
2.In Table 1, comparing the results of “w/o DiaS” and “w/o Correct”, I have a minor question: why the CARA metric of “w/o Correct” is lower than “w/o DiaS”? In my opinion, “w/o Correct” incorporates more single-round teaching dialogues, and thus its performance should be better.
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose a novel method based on Socratic teaching for improving LLM teaching abilities.
strength_weakness: strength_weakness: - results could benefit from having standard errors or confidence intervals (authors say 'Yes' to this in NeurIPS checklist but only list having the kappa score in justification)
- there seem to be a large number of related methods (e.g., search for "socratic LLM teaching" on Google Scholar), the authors should contextualize their work among this literature in the related works section, and clarify how their method differs from and outperforms existing methods from that literature (e.g., a goog example is Socratic Playground - https://arxiv.org/abs/2406.13919)

Minor (did not affect score):
- Line 122: "While some student" --> "While some students"
- Line 173: "such not providing" --> typo here
- Line 312: "refuse" --> "refuses" or "refused"
- Table 1 caption: "denote the" --> "denotes"
- Line 595: "Border" -> "Broader"

Ethics flag:
- Research involved human subjects but unclear if authors had IRB approval. Not sure whether IRB approval is needed or not in this case so flagging for review by relevant experts just in case (note: the authors say the paper involves no crowdsourcing or human subjects in question 15 of the NeurIPS checklist, right after saying it does in question 14).
questions: questions: - why does performance fall on some metrics at 125% data scale?
- will the full SocraTeach dataset be publicly released?
- I would be open to increasing my score if concerns from the previous section are addressed
limitations: limitations: - authors discuss limitations but would be worth it to add that current experiments are only in English, and that current human evaluations come from a very small (and likely not broadly representative) sample
paper_summary: paper_summary: This paper aims to introduce a “Thought-provoking” paradigm into LLM-based personal teaching. The authors propose an innovative “Dean-Teacher-Student” pipeline with three LLM-based agents to collect Socratic teaching data. During this process, the authors also contribute a student cognitive system to simulate six types of authentic students with the “Student” agent. Then, the authors adopt data augmentation for four crucial teaching abilities and collect more single-round dialogue data. Finally, the authors investigate three training strategies to balance the problem-solving ability and Socratic teaching ability of LLMs. The fine-tuned SocraticLM, shows great teaching performance compared with several LLMs including GPT4 and EduChat. The authors also provide sufficient experiments to validate the importance of single-round data, training strategies, which makes the method self-contained and instructive.
strength_weakness: strength_weakness: Some issues that hope to be further addressed by authors:
- It is rational and acceptable to use GPT4 to construct data, but I am interesting about whether the performance of SocraticLM will be limited by GPT4 (please refer to question 1 below).
- I recommend the authors to provide more explanations of some experimental results, which can better reflect the effectiveness of the proposed SocraticLM (please refer to question 2 below).
questions: questions: - In my opinion, the quality of the collected SocraTeach dataset highly depends on the Teacher agent, which is implemented with GPT4. Therefore, if GPT-4 itself cannot correctly solve a problem, can it still serve as a teacher in an educational role?
- I notice that in Table 2, SocraticLM performs even better in problem-solving accuracy on MAWPS dataset. Could the authors provide some explanations for this?
limitations: limitations: Please see questions above.

Comments: title: comment from reviewer
comment: Many thanks for your response! We are very happy to address your concerns and will add all your suggested modifications into the revised version. Thank you again for the time you took to review our paper and your affirmation of our work. If there are any further questions, please feel free to raise them, and we can discuss any questions at any time. We will try our best and respond as soon as possible.
Many thanks for your response and increasing your score! We are very happy to address your concerns and will add all suggested clarifications and modifications into the revised version following your comments. Thank you again for the time you took to review our paper and your affirmation of our work. If there are any further questions, please feel free to raise them, and we can discuss any questions at any time. We will try our best and respond as soon as possible.
Thank you for the rebuttal! My concerns have been addressed and I have increased my score accordingly.
Thank you for clarifying the doubts. All my concerns have been addressed.
We wish to once again express our great appreciation for the time you have taken to review our paper. We would appreciate your feedback on whether your main concerns have been adequately addressed. We truly value your understanding and support, and will carefully revise the paper according to your suggestions. Thank you very much!
We wish to once again express our great appreciation for the time you have taken to review our paper. We would appreciate your feedback on whether your main concerns have been adequately addressed. We truly value your understanding and support, and will carefully revise the paper according to your suggestions. Thank you very much!
We wish to once again express our great appreciation for the time you have taken to review our paper. We would appreciate your feedback on whether your main concerns have been adequately addressed. We truly value your understanding and support, and will carefully revise the paper according to your suggestions. Thank you very much!
Thank you very much for reviewing the ethics issues of our paper and affirming that our method does not raise issues about human agents. 

As you understood, in our “Dean-Teacher-Student” pipeline for data collection, each agent is simulated by GPT-4, and no human teachers or students are involved. Regarding your concern about “asking some human beings to provide teacher inputs”, as described in Section 5, during model evaluation, we randomly sampled a portion of the data constructed from “Dean-Teacher-Student” pipeline for testing. We used the preceding context from this data as input and let the LLMs generate outputs automatically. Therefore, we did not ask human teachers to provide inputs either. The only aspect of our work that involves humans is inviting 10 human annotators (described in Section 5) to anonymously compare the outputs of different LLMs. This process is consistent with existing works that also involve human evaluation for LLM output, where the human annotators solely assess the model's output (i.e., give ratings) and they themselves are not being tested. Besides, as shown our annotation template in Appendix F, this evaluation process does not collect personal information or privacy of the annotators, and the annotators are fully aware of the purpose of the evaluation and have consented to its use. Based on your concerns, we will provide a more detailed explanation of our human annotation process in the revised paper.

We sincerely hope our responses can address your concerns.
Thank you very much for reviewing the ethics issues of our paper. 

In this paper, our goal is to develop a Socratic teaching large language model, SocraticLM. To achieve this, we propose a “Dean-Teacher-Student” pipeline to collect a set of multi-round teaching dialogue data and fine-tune ChatGLM3 to obtain SocraticLM. This entire pipeline is fully simulated by GPT-4, with no involvement of human teachers or students. In our experiments, as explained in Section 5, we randomly sampled a portion of the collected dialogue data (which was not used in fine-tuning) and used the preceding context of each dialogue as input to generate outputs from different LLMs. Then, similar to existing works that involve human evaluation for LLM output, we invited human annotators to compare the outputs of different LLMs. In this process, we invite human annotators only to assess the model's output (i.e., give ratings). The annotators themselves are not the subjects of the evaluation and are not being tested.

Regarding the potential societal implications in your suggestions, due to the page limit of the main text, we discussed the Border Impact of our work in Appendix I. To be more specific, on one hand, our research can facilitate some applications of intelligent education. As explained in Section 3.3 and 3.4, our SocraticLM can provide instructions to students with different cognitive states and various types of responses. Therefore, it can support some personalized applications in classrooms and online platforms, delivering high-quality instructions more efficiently and conveniently to students from various backgrounds, thus promoting educational equity and rapid development. On the other hand, in our work, we use GPT-4 to simulate conversations between teachers and students to generate dialogue data, entirely without the involvement of real human teachers or students. This approach provides a way to collect data without interacting with or testing on real humans, reducing the burden of human interaction while also avoiding privacy, security, and other ethical concerns. This contributes to the protection of social ethics. Following your suggestion, we will reorganize a dedicated Social Implications section to include these discussions and clarifications in the revised version of our paper.

We sincerely hope our responses can address your concerns.
Many thanks for your quick response! We are very happy to resolve your concerns and will add all suggested modifications into the revised version based on your comments. Thank you again for the time you took to review our paper and your affirmation of our work. If there are any further questions, please feel free to raise them, and we can discuss any questions at any time. We will try our best and respond as soon as possible.
Thanks for your answer. These responses fulfill my doubts and reinforce my score.
Hello. This submission has been labeled by the reviewer for additional ethics review, I recommond the authors to priortize their response to the ethics concern so the ethics reviewers could get a clearer picture of the situation as early as possible. Please find above the ethics concerns raised by Reviewer sW5a. 

The purpose of the additional ethics review is to help the authors mitigating potential ethical, legal or societal harm at an early stage (if there's any). Thank you for your understanding.