Students Rather Than Experts: A New AI for Education Pipeline to Model More Human-like and Personalised Early Adolescences

Authors: Yiping Ma, Shiyu Hu, Xuchen Li, Yipei Wang, Shiqing Liu, Kang Hao Cheong
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI for Education; Large Language Models; LLM-based Agent; Teacher Training

Abstract:
The capabilities of large language models (LLMs) have been applied in expert systems across various domains, providing new opportunities for AI in Education (AI4Education). Educational interactions involve a cyclical exchange between teachers and students. Current research predominantly focuses on using LLMs to simulate teachers, leveraging their expertise to enhance student learning outcomes. However, the simulation of students, which could improve teachers' instructional skills, has received insufficient attention due to the challenges of modeling and evaluating virtual students. This research poses the question: “Can LLMs be utilized to develop virtual student agents that mimic human-like behavior and individual variability?” Unlike expert systems focusing on knowledge delivery, virtual students must replicate learning difficulties, emotional responses, and linguistic uncertainties. These traits present significant challenges in both modeling and evaluation. To address these issues, this study focuses on language learning as a context for modeling virtual student agents. We propose a novel AI4Education framework, termed SOE (Scene - Object - Evaluation), to systematically construct LVSA (LLM-based Virtual Student Agents).  By curating a dataset of personalized teacher-student interactions with various personality traits, question types, and learning stages, and fine-tuning LLMs using LoRA, we conduct multi-dimensional evaluation experiments that integrate both subjective human evaluations and objective metrics. Specifically, we: (1) develop a theoretical framework for generating LVSA; (2) integrate human subjective evaluation metrics into GPT-4 assessments, demonstrating a strong correlation between human evaluators and GPT-4 in judging LVSA authenticity; and (3) validate that LLMs can generate human-like, personalized virtual student agents in educational contexts, laying a foundation for future applications in pre-service teacher training and multi-agent simulation environments.

Decision: reject
Reviewer scores: 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729715423525

thread_type:
official_review

<thread_object>
creator: reviewer_ywbj

<content>
summary:
++ I have increased the scores after the clarifications from the Authors.

This paper introduces a novel approach to using Large Language Models (LLMs) for simulating virtual students, instead of the more common focus on AI-driven teacher models. The authors propose a framework called SOE (Scene-Object-Evaluation) for generating LLM-based Virtual Student Agents (LVSA). This framework is applied within educational contexts, particularly for teacher training. The key contributions include:

1.Theoretical Framework: A comprehensive model for constructing LVSA based on personalized teacher-student interaction data.
2.Human and GPT-4 Evaluations: The integration of human subjective evaluation with GPT-4's capabilities to assess the authenticity of virtual student agents.
3.Experimental Validation: Extensive experiments confirm the feasibility of generating human-like, personalized virtual student agents for improving teacher training.

strengths:
Originality: The paper presents a novel shift from teacher-centric AI systems to virtual student simulations, which can greatly impact teacher training and educational AI systems. By focusing on virtual students, it provides a fresh and creative problem formulation, opening new avenues in AI4Education research.

Quality: The paper demonstrates a well-structured methodology, from developing a dataset of teacher-student interactions to fine-tuning LLMs and validating their outputs with human and GPT-4 evaluations. The theoretical and operational frameworks are well thought out, ensuring rigorous scientific contribution.

weaknesses:
Limited Real-world Data Application: While the framework is well-developed, the fine-tuning process relies on datasets sourced from controlled environments, which may not fully capture the complexity of real-world classroom dynamics. Future work should consider using more diverse and naturalistic data for better generalization.


Evaluation Focus: The evaluation primarily centers on language comprehension and emotional simulation. However, cognitive challenges like problem-solving or more complex reasoning skills, which are critical to student behaviors, are underexplored. This may limit the scope of the virtual students' realism in broader educational contexts.

questions:
Handling Cognitive Diversity: Can the authors expand on how their system handles more complex cognitive behaviors beyond basic language learning? For example, how well can LVSA simulate problem-solving tasks or deal with creative thinking in open-ended questions?

Mitigating LLM Hallucinations: What specific strategies can be employed to minimize hallucinations in virtual student responses? Would incorporating additional real-world classroom data or using more advanced fine-tuning methods help in this regard?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730050158981

thread_type:
official_review

<thread_object>
creator: reviewer_3qky

<content>
summary:
This work presents an AI4Education framework, termed SOE (Scene - Object - Evaluation), to construct LVSA (LLM-based Virtual Student Agents). It contributes a new Chinese education dataset and evaluates four existing LLMs which are fine-tuned in this dataset in simulating virtual students with specific traits. A Turing test is used to evaluate whether LLMs can generate such human-like virtual students. Specifically, GPT-4 is used for large-scale automatic evaluation and a small group of real humans are recruited for small-scale evaluation.

strengths:
1. This work presents a LLMs-based agent evaluation pipeline including data preparation, LLM fine-tuning, and virtual student evaluation.

2. A new large dataset is contributed to study the virtual student agents mimicked by LLMs.

3. A comprehensive experiment is conducted to evaluate the performance of student agents powered by LLMs.

4. This paper presents sufficient supplemental materials for the readers to understand the prompts.

5. Overall, it is easy to read and understand this paper, with clear arguments.

weaknesses:
W1. One of my main concerns is the novelty in model/framework design. The proposed SOE framework looks more like a dataset processing and LLM testing pipeline. However, this is no specific model design or new prompt structure design. The authors simply instruct the LLMs to mimic students with different traits based on the big five rationale. The LLMs used from existing work are directly fine-tuned on a new dataset. There are no unique insights in prompt design or fine-tuning strategies, compared with previous work. Moreover, the incorporation of big five theory for agent personality is also widely utilized in existing work such as [1]. Therefore, this is not a novelty nor contribution.

W2. Another main concern is that I think this work cannot really support its claim in line 532 "resembled real student behavior" and line 533 “simulating realistic student roles”. I agree with line 053 “virtual student agents must replicate a broader range of human behaviors”. But such replication needs labels/ground truth to demonstrate and quantify the accuracy of such replication. A simple Turing test cannot really support that, and lots of existing work has explored LLMs in passing Turing test [2] and a new Turing experiment [3]. That said, Turing test can show that your agent simulation is human-like, but that does not necessarily mean that your agent simulation is realistic among students’ real behaviors, which needs more convincing ground truth for comparison. But this work does not have a real ground truth for comparison. 

Specifically, I also do not understand why the five kinds of agents (HN, HA, LO, HE, LC) are combined with real students as the six one together for evaluation in appendix figure A18. Do the authors mean that human evaluators are asked to distinguish one of the six types? I’m confused because at first the authors say that this work aims to simulate student traits. So I thought real students serve as the labels instead of an additional type in addition to the five traits. Such Turing test setting can only show that humans cannot distinguish whether the expressions are from agents or real students. This is not surprising considering existing work showing the performance of LLMs in the Turing test such as [2]. However, it cannot show that such simulation is realistic because there is no ground truth for comparison. 


W3. I’m also concerned regarding the limited generalization and evaluation. This work only uses one Chinese dataset from the authors. No public datasets or English datasets are used. It is uncertain whether the performance of this framework can have good generalization.

W4. Moreover, another important concern is the lack of a real baseline. The evaluation simply compares pre-finetuning with post-finetuning. It is not surprising that finetuning in the targeted dataset can result in better performance. But there is not a real baseline to show the unique advantage of the propose SOE framework, compared with existing approaches.

W5. Furthermore, using GPT-4 for automatic evaluation is not convincing. It looks like the authors are using one LLM to evaluate other LLMs’ performance. No ground truth is used for more convincing evaluation. Line 373 mentioned the high inter-coder reliability just proves that the two students have consistent opinions in annotation. But it cannot prove that GPT 4 is a convincing automatic evaluation tool to replace real human annotators to perform automatic evaluation.

W6. In Appendix D.5.3 and figure A18, since GPT4 can “effectively simulate and assess nuanced student behaviors” (from line 4236), why not directly use GPT4 for student trait simulation? Why do you use other LLMs to perform such simulation and use GPT 4 to serve as the evaluator? Furthermore, in line 4244, “GPT-4 and human evaluators was found to be 0.6806, which falls within the range of substantial agreement”. Why 0.6806 is acceptable and why 0.6806 is within the range of substantial agreement? Who defines the agreement? Is there a standard rule for the value range?

W7. I’m really confused about what the evaluation score means in section 5.4. What the score refers to? What does 100% mean in line 416? Does it refer to the accuracy for human evaluators or GPT 4 to distinguish different types of agent traits? Similarly, for different learning stages in line 463 and different question types in line 482, what does the percent % mean? Does it refer to the accuracy or any metrics, to measure what?

W8. Line 478: “These findings suggest that LVSA effectively adapts to different learning stages, offering comprehensive support for pre-service teacher training. Virtual students enable teachers to practice and refine instructional strategies across all teaching phases, enhancing skill development throughout the entire instructional process.” These assertions are over-claimed. If you want to show its effectiveness to help teachers, then you need to conduct a real teaching experiment to show it, which is more convincing. Otherwise, such assertions sound weak.

W9. Line 334: No ethic information (such as IRB approval) about recruiting human participants for evaluation and releasing collected datasets to the public. Potential ethic concerns.

W10. 115 samples are too limited for human evaluation. The number of questions used for human evaluation in Appendix Figure A11 and table A3 is too limited. It is also unclear why you only use 40 samples or 35 samples per setting, which is quite confusing. Why all five fatigue test items come from the real student responses? Why not use 2 items from each group of fine-tuned models, direct inference, and real students?

W11. For section 5.2 human Turing test, simply showing the average results of all human evaluators is not a standard nor convincing approach. A formal statistical analysis such as ANOVA is needed. 

Reference:

[1]. CGMI: Configurable General Multi-Agent Interaction Framework, Shi et al. 2023

[2]. Does GPT-4 pass the Turing test? Jones et al. 2023

[3]. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies, ICML 2023

questions:
All questions are listed in the weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658794718

thread_type:
official_review

<thread_object>
creator: reviewer_rtkz

<content>
summary:
The paper introduces a novel AI framework, the "Scene-Object-Evaluation" (SOE) pipeline, designed to model Large Language Model-based Virtual Student Agents (LVSA) that mimic the behaviors and personalities of early adolescent students in educational settings. Unlike previous work that uses large language models (LLMs) to simulate expert teaching roles, this study focuses on creating realistic, student-like agents that reflect common learning challenges, emotional reactions, and personality differences.

The SOE pipeline operates in three stages: Scene, which defines the educational scenarios and content; Object, which constructs virtual student agents with distinct personality traits through LoRA fine-tuning; and Evaluation, which assesses these agents' performance using human and GPT-4 evaluations. The virtual students are evaluated based on traits derived from the Big Five Personality Traits framework to ensure a range of realistic, individualized behaviors. Experiments show that these fine-tuned virtual agents provide human-like and personalized responses, validating their potential to support pre-service teacher training by offering realistic classroom simulations.

This research contributes a new approach to AI4Education, focusing on developing student agents that improve teachers’ instructional skills through realistic interactions, and lays groundwork for future applications in teacher training and multi-agent simulations.

strengths:
This paper brings a high degree of originality by shifting the focus from expert or teacher-oriented AI simulations to student simulations, addressing a significant gap in AI-driven teacher training tools. The introduction of the Scene-Object-Evaluation (SOE) pipeline to model Large Language Model-based Virtual Student Agents (LVSA) represents a creative approach to personalized student simulation, providing an adaptable framework that mirrors the complexity of real students. Using the Big Five Personality Traits for developing distinct virtual student personalities adds further novelty, offering a method for producing varied and realistic student responses that go beyond standard, pre-scripted models.

The research methodology is robust, leveraging the SOE pipeline to guide each stage of student agent construction. The use of LoRA fine-tuning to adapt the LLMs to personality-based response styles is well-executed, supported by a well-defined dataset and clear criteria for generating realistic dialogues. The authors conduct subjective evaluations through human raters and GPT-4, lending credibility to the LVSA models’ authenticity in simulating student behaviors. This multi-dimensional evaluation demonstrates thoughtful experimentation, allowing for both quantitative and qualitative assessments of the LVSA’s effectiveness. The use of the Big Five traits adds psychological rigor, enhancing the methodological depth and grounding the work in established personality theory.

The paper is generally clear in its presentation, with each step of the SOE pipeline outlined in a logical and accessible manner. Figures, particularly the pipeline overview, provide helpful visual support, clarifying the process of constructing and evaluating virtual student agents. The experimental setup, while complex, is conveyed effectively, making it easier for readers to understand the key steps and outcomes. However, more detailed examples of LVSA responses across personality types could further improve the clarity and transparency of the results. Expanding on specific evaluation metrics and providing examples of dialogue interactions for different personality types would also enhance readability.

The significance of this work is substantial, as it addresses an important gap in pre-service teacher training by providing a tool for realistic, student-like simulations. In traditional teacher training, access to diverse student interactions is often limited, and this work offers a scalable solution to that problem. The LVSA model provides a unique contribution by capturing the variability of student responses and behaviors, allowing teachers to practice and adapt to different learning personalities and styles. By simulating more realistic classroom scenarios, the LVSA approach has the potential to improve teacher preparedness, adaptability, and effectiveness. These contributions align well with the ICLR community’s interest in novel applications of AI, particularly in education and social simulation, and could stimulate further research into AI-driven, personalized learning simulations.

weaknesses:
Issue: The evaluation primarily relies on subjective assessments from human raters and comparisons with GPT-4, which introduces variability in interpretation and limits reproducibility.
Suggestion: Including additional objective metrics—such as measuring linguistic coherence, response diversity, or sentiment alignment—could strengthen the evaluation process and provide quantitative insights into LVSA’s effectiveness. Moreover, using established psychometric tools to assess the authenticity of the personality-based responses could enhance credibility. Adding these metrics would improve the benchmark’s robustness and offer more reproducible, quantitative insights into model performance.


Issue: The evaluation primarily relies on subjective assessments from human raters and comparisons with GPT-4, which introduces variability in interpretation and limits reproducibility.
Suggestion: Including additional objective metrics—such as measuring linguistic coherence, response diversity, or sentiment alignment—could strengthen the evaluation process and provide quantitative insights into LVSA’s effectiveness. Moreover, using established psychometric tools to assess the authenticity of the personality-based responses could enhance credibility. Adding these metrics would improve the benchmark’s robustness and offer more reproducible, quantitative insights into model performance.


Issue: The paper lacks detailed examples of LVSA responses across different personality types, making it challenging to assess the realism and nuances of virtual student interactions. A deeper qualitative analysis of the generated responses could reveal specific strengths and limitations in simulating various personality traits.
Suggestion: Adding representative examples of LVSA responses across personality traits, question types, and learning stages would clarify how the model adapts to different scenarios. A qualitative analysis of these responses could provide actionable insights into areas where the model succeeds and where it may struggle to simulate certain personalities. This addition would also make the results more transparent and accessible to readers interested in the specifics of the student interactions.



Issue: The paper notes difficulties in modeling low-conscientiousness (LC) virtual students and instances of hallucination, where generated responses lack coherence. However, these challenges are not fully explored, and there is little discussion on specific mitigation strategies.
Suggestion: Including a more comprehensive analysis of failure cases, especially regarding LC personality traits, would improve understanding of the model’s limitations. Discussing methods to mitigate hallucinations, such as incorporating stricter training constraints or filtering responses, could add value to the paper and inform future improvements. Identifying potential approaches to handle “undesirable” traits like low conscientiousness could also clarify how LVSA might be adapted for broader educational use.


Issue: The SOE pipeline's adaptability to varying classroom contexts, cultural backgrounds, or different educational levels is not addressed, which may affect its broader applicability and restrict generalization across diverse educational settings.
Suggestion: Discussing the adaptability of the SOE pipeline to various contexts, such as different age groups, subject areas, or cultural backgrounds, would enhance the relevance of the work. Exploring how the model could be fine-tuned or adjusted to reflect different educational environments would increase its generalizability and appeal. Additionally, expanding the fine-tuning dataset to include a wider array of scenarios or educational subjects could improve model robustness across diverse contexts.

questions:
Question: Beyond human and GPT-4 evaluations, did you consider incorporating objective metrics such as linguistic coherence, response diversity, or sentiment analysis to evaluate the LVSA’s responses?
Suggestion: Adding these metrics would enhance the rigor and reproducibility of the evaluation process. Objective measures could also help reduce variability in subjective assessments and provide additional quantitative insights into the effectiveness of the LVSA.


Question: Do you have plans to test the LVSA in real-world educational settings or with pre-service teachers? If so, what are your anticipated challenges in transitioning from simulation to actual classroom interactions?
Suggestion: Conducting field tests or real-world studies with pre-service teachers interacting with LVSA could provide valuable insights into the model’s practical value and usability. Understanding how LVSA performs in diverse, real-life scenarios would bolster its credibility and relevance for educational applications.


Question: Could you provide specific examples of how LVSA responses differ across personality types, particularly with the Big Five traits? Are there qualitative differences in the way different personality modes handle similar questions or challenges?
Suggestion: Including examples of LVSA responses across a variety of personality traits and educational scenarios would enhance transparency and allow readers to better understand the nuances of the virtual student interactions. This would also make it easier to assess the model’s adaptability and realism in simulating diverse student behaviors.

Question: You mention difficulties with modeling low-conscientiousness traits and hallucination issues. Could you elaborate on the specific challenges faced and any steps taken to mitigate these problems?
Suggestion: Providing a detailed analysis of failure cases, especially in simulating LC traits, would clarify the model’s limitations. Discussing potential strategies to reduce hallucinations (such as response filtering or stricter fine-tuning constraints) would improve the framework’s robustness and help readers understand how LVSA might be improved for complex or nuanced personalities.


Question: How adaptable is the SOE pipeline to different classroom contexts, age groups, or cultural backgrounds? Could it be easily fine-tuned for students in different educational systems or with varying cultural characteristics?
Suggestion: Discussing how the SOE pipeline could be generalized or adjusted to accommodate diverse educational contexts would enhance its broader applicability. Exploring fine-tuning or dataset adjustments that capture different cultural and age-based contexts would clarify how adaptable the LVSA could be in global educational settings.

Question: Have you considered conducting longitudinal studies to assess whether interactions with LVSA lead to measurable improvements in teachers’ skills, particularly in classroom management or personalized instruction?
Suggestion: If feasible, a longitudinal study tracking skill development in pre-service teachers over time would provide a deeper understanding of LVSA’s educational impact. This type of evaluation could offer insights into whether sustained interaction with diverse virtual students has lasting benefits on teachers’ adaptability and instructional skills.

Plans for Scaling and Efficiency Optimization
Question: Given the computational demands of generating varied personality-based responses, do you have plans to optimize the SOE pipeline for efficiency, perhaps through model distillation or response sampling techniques?
Suggestion: Discussing strategies for scaling or optimizing the model’s computational efficiency would improve the framework’s usability for larger educational deployments. Efficiency improvements could also make LVSA more accessible to institutions with limited computational resources.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689955087

thread_type:
official_review

<thread_object>
creator: reviewer_wxdr

<content>
summary:
The paper tries to simulate student behaviors (as opposed to teachers behavior) using LLMs and in order to do that, they have proposed SOE (Scene-Object-Evaluation) framework. They evaluate these virtual students using multi-dimensional experiments.
The paper has three contributions: 
1) A theoretical framework for LVSA
2) The integration of human subjective evaluation metrics into GPT-4
3) An empirical demonstration that virtual student agents can closely emulate real student behaviors

strengths:
- Applying virtual students for teacher training can be potentially impactful.
- Paper is easy to follow

weaknesses:
- Reliance on subjective metrics and GPT-4 assessments could introduce biases. Personality-based biases (relying on the Big Five traits -> lead to stereotypical behaviors) and cultural biases (focusing on Chinese students could limit generalizability).
Use a more diverse range of evaluators. 

- Limited realism. Not supporting multimodal inputs limits its usefulness. Incorporating visual and audio elements, at a minimum, could better reflect real-world classroom dynamics.
Real classroom settings often rely on visual and auditory cues.

- Needs more varied context. The model should have the capability to simulate more subjects like math and science. This would allow for a  more relevant model.
The need for a model that is capable of advanced reasoning and interpretation.

questions:
Some important questions could be how virtual students could be validated in settings beyond junior high language tasks? How might the model be modified to incorporate multimodal inputs? What steps are being taken to address biases that might occur in the evaluations? Did the authors explore additional evaluation metrics? Could the authors elaborate on the ethical implications of simulating student behaviors (data privacy could be a concern)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732548802968

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern 1：Generalizability of the SOE pipeline

comment:
We sincerely thank the reviewers for their concerns regarding the generalization and adaptability of SOE pipeline. In the discussion section, we have further elaborated on the adaptability of the SOE pipeline across multiple disciplines, age groups, and cultural contexts. The SOE framework proposed in this study is both general and modular. It is designed to flexibly accommodate the needs of various disciplines, providing a clear and actionable research pathway for multidisciplinary applications. 

Below is our detailed response to this issue.

**1. Design and Adaptability of the SOE Pipeline**

The core goal of the SOE pipeline is to provide a highly generalizable and extensible framework to address the challenges of modeling and evaluating virtual students in educational contexts. Its modular design allows for flexible adaptation to various educational systems and teaching needs. This structure not only supports adjustments tailored to specific disciplinary requirements but also optimizes for the characteristics of students across different age groups and cultural backgrounds.

- **Scene Module**: Enables the flexible definition of subject contexts based on different classroom scenarios and teaching objectives. For example, it validates and ensures that base models have the requisite knowledge and reasoning capabilities for mathematics, thus constructing appropriate virtual students.
- **Object Module**: Generates personalized virtual students through data fine-tuning, capable of simulating student behaviors across various age groups and cultural contexts. For instance, it supports designing simplified language interactions for younger students and adding complex reasoning tasks or open discussions for older students by constructing fine-tuning instruction datasets.
- **Evaluation Module**: Supports the adjustment of evaluation standards based on the scenario to ensure that the evaluation results align with the objectives of specific educational contexts.

**2. Validation of SOE Pipeline Adaptability:**

Language studies, with their openness and diversity, provide an ideal environment for simulating interaction needs across different personality traits and subject teaching content. This makes them well-suited for validating the framework’s generalizability across various teaching stages. In this study, we have preliminarily validated the applicability of the SOE pipeline in specific language-based scenarios, laying a solid foundation for expanding the SOE pipeline into other disciplines, cultures, and age groups.

- **Adaptability Across Teaching Stages**: Experiments demonstrated that virtual students can adapt to multiple teaching stages (e.g., lesson introduction, knowledge instruction, and review) and exhibit behavior characteristics aligned with the goals of these scenarios.
- **Adaptability to Different Question Types**: Virtual students can accommodate both open-ended questions (e.g., problem-solving and creative thinking) and closed-ended questions (e.g., fixed-answer knowledge responses), addressing diverse cognitive needs.

**3. Next Steps for Exploring Applications in Other Disciplines:**

- **Cultural and Age Adaptability**: By adjusting training datasets to include examples from different cultural backgrounds, educational practices, and language styles, LVSA can enhance its applicability in multicultural and multilingual educational contexts. For example, in multilingual education, training data can include scenarios of bilingual or multilingual teaching interactions.
- **Adaptation to Diverse Educational Systems**: To meet the educational objectives and teaching methodologies of different countries and regions, datasets covering a wide range of disciplines and teaching tasks will be designed.
- **Refined Fine-Tuning and Data Augmentation**: By leveraging data augmentation techniques to expand the coverage of small-sample datasets and applying fine-tuning to improve the model’s adaptability to new tasks, LVSA can better align with specific educational contexts.

We are deeply grateful for reviewers' valuable suggestions, which have greatly advanced our understanding of the adaptability and universality of the SOE pipeline. Moving forward, we will continue to expand and deepen this research direction to ensure that LVSA can deliver maximum value across global educational systems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549026149

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern 2：Supplementary Experiments on Objective Evaluation Metrics

comment:
We sincerely thank the reviewers for their concerns regarding the evaluation methods in this study. We fully agree that incorporating objective metrics can enhance the scientific rigor of the evaluation process. Accordingly, we have conducted additional experiments, and the results align with the conclusions drawn from subjective evaluations (**see Sec 5.5 & Appendix D.6**). These findings validate the personalization and effectiveness of LVSA modeling while also revealing the limitations of objective metrics, further supporting the core motivation of this study: addressing the challenge of "difficult evaluation of virtual students." 

Below is our detailed response:

**1. Selection of Objective Evaluation Metrics**

We reviewed commonly used objective evaluation metrics in the field of text generation. Due to the open-ended and diverse nature of student language expression in educational scenarios, traditional reference-based evaluation metrics (e.g., BLEU, ROUGE) are not suitable. Therefore, we selected reference-free objective metrics that better meet the needs of educational scenarios, including:
- **Text Token**: Measures the length of expressions across students with different personalities.
- **Perplexity**: Evaluates the fluency of language generation and model adaptability.
- **Type-Token Ratio (TTR)**: Reflects the diversity of language expression.
- **Sentiment Analysis**: Indicates the emotional tendencies in language expression.

The last three objective evaluation metrics correspond to the aspects you mentioned: **linguistic coherence, response diversity, and sentiment analysis**.

**2. Objective Evaluation Results and Limitations**

Observations from objective metrics yielded results consistent with subjective evaluations, demonstrating that the virtual students effectively simulated student behaviors associated with different personality traits. Using min-max normalization to visualize the experimental results, we found that the HE (High Extraversion) personality performed prominently in clarity and positive sentiment, reflecting its outgoing nature and fluency in expression. The HA (High Agreeableness) personality exhibited high lexical richness and positive sentiment, indicative of rich expression and strong cooperativeness. In contrast, the LO (Low Openness) and LC (Low Conscientiousness) personalities scored lower in clarity and lexical richness, consistent with their concise, conservative, or casual language styles. (Detailed data analysis results can be found in **Sec 5.5 & Appendix D.6**).

However, we also identified limitations in objective evaluation, which further underscore the importance of subjective evaluations in capturing the complexity of student language and emotional expression.
- **Bias in TTR**: TTR is highly dependent on text length, which may lead to overestimation of shorter responses (e.g., "It's from the Song Dynasty." has a lexical richness of 1).
- **Specificity of Perplexity**: Perplexity can increase due to the presence of classical Chinese or technical terms in the text, obscuring the actual fluency of the student's expression (e.g., "It's a narrative text." has a perplexity of 119.8).
- **Directional Bias in Sentiment Analysis**: Sentiment analysis can be influenced by keywords from the teaching content, leading to deviations from the true emotional state (e.g., "The theme of this poem is filial piety and loyalty. It praises Mulan's devotion and longing for a peaceful life." is merely a description of textbook content but was misclassified as positive sentiment).

**3. Importance of Subjective Evaluation**

By combining subjective evaluation with GPT-4 automated evaluation, we have achieved a good balance between scientific rigor and efficiency. Subjective evaluation captures the emotional and cognitive complexity in students' language, while GPT-4 automated evaluation provides greater consistency and rapid verification capabilities. This dual approach makes the evaluation both comprehensive and efficient, offering a highly practical solution for future virtual student modeling.

We sincerely thank the reviewers' valuable suggestions regarding objective evaluations. These recommendations not only provided important references for the evaluation system in this study but also broadened our perspective on virtual student modeling and evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549292769

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern 3：The Cognitive Levels and Corresponding Ability Evaluation of Virtual Students

comment:
We sincerely thank the reviewers for their concerns regarding the evaluation of the cognitive abilities of virtual students (e.g., problem-solving, reasoning, creative thinking), which are essential to assessing the realism of virtual students and their applicability across diverse educational contexts. We fully agree that these abilities are critical and welcome this opportunity to clarify and expand on how our study addresses these aspects.

**1. Cognitive Ability Modeling Is Reflected in the Experimental Design:**

In this study, we designed the SOE pipeline with an emphasis on the importance of cognitive development in areas like problem-solving and creative thinking. Consequently, we included question types as a key evaluation dimension to reflect the virtual students' ability to operate at different cognitive levels (**see Sec 4.1 & Appendix B.2.2**):
- **Lower-Order Cognition Based on Closed-Ended Questions**: Focused on students’ understanding and memory (e.g., explanatory abilities).
- **Higher-Order Cognition Based on Open-Ended Questions**: Focused on application and analysis (e.g., problem-solving and reasoning) as well as creation and evaluation (e.g., creative thinking). We can take an example of our experiment scene (see more examples in **Appendix D.2.2**). 
  - **Problem-Solving Ability**: This aspect primarily focuses on cognitive levels of application and analysis. We designed task-based questions closely related to the content of the lesson, requiring virtual students to engage in reasoning and analysis. For instance, when discussing reading material, HE students were able to provide clear and logical answers based on the text.
      - **Example Question**: “Duanmu Hongliang used many vivid words to express his feelings for his hometown. Can you find some examples and discuss them?”
      - **Example Response**: “Sure, teacher. For example, “My hometown is in the northeast, with its dark, fertile soil, golden grains, red maple leaves, and white snow.” These descriptions make me feel his deep love for his hometown.”
  - **Creative Thinking Ability**:  This aspect evaluates students at the evaluation and creation levels of cognition. We designed open-ended questions to encourage students to express creative ideas, such as reinterpreting the theme of a passage or continuing a story. HE virtual students demonstrated proactive responses.
      - **Example Question**: “Now, let’s see whose imagination is the most creative. Based on the text, can you describe the image of the Yellow River in your own words?”
      - **Example Response**: “Sure, Teacher. In the poem, the Yellow River is depicted as a great mother who nourishes the entire Chinese land. Its surging waves symbolize the resilience and perseverance of the Chinese people, and its grandeur reflects their greatness and pride.”

**2. Research Focus on Modeling More Human-Like and Personalized LVSA**:

In educational scenarios, students display diverse traits and behaviors, with varying cognitive abilities. **Rather than focusing solely on advanced capabilities, we prioritize modeling and assessing human-like, diverse virtual student behaviors. The priority lies in evaluating whether the base model can support virtual student modeling in the whole teaching stage and give effective feedback for pre-service teachers.** Although comprehensive analysis of LVSA abilities is not the main focus at this stage, our experimental data suggests that LVSA already shows representative potential in complex cognitive behaviors, such as problem-solving and creative thinking.

In future work, we plan to further expand this aspect through the following steps:
- **Refining the Cognitive Evaluation Framework**: Drawing on cognitive taxonomy theories (e.g., Bloom's taxonomy), we aim to design more fine-grained question types to comprehensively cover virtual students' performance in complex cognitive behaviors.
- **Analyzing the Relationship Between Personality and Cognitive Abilities**: Investigate how virtual students with different personality traits perform in problem-solving and creative tasks, to further improve the modeling accuracy of LVSA.
- **Expanding Multidisciplinary Scenarios**: Extend evaluation tasks to other disciplines (e.g., solving math problems, designing scientific experiments) to test LVSA cognitive simulation capabilities across subject domains.

We greatly appreciate the reviewers' valuable suggestions regarding complex cognitive tasks. Through the evaluation dimension of question types, this study has preliminarily validated the potential of virtual students in problem-solving and creative thinking. In the future, we will conduct further in-depth analysis to continually enhance the simulation capabilities and educational value of LVSA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549747292

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We would like to express our heartfelt gratitude to all reviewers and the Area Chair (AC) for their thorough evaluation and thoughtful feedback on this study, which have greatly enhanced the academic value of our work and encouraged critical reflection. The reviewers have highlighted the following key contributions of our paper:

- **Research contributions align with ICLR's focus**: This study aligns with ICLR’s interest in innovative AI applications in education and social simulation, showcasing potential to inspire future research (Reviewer rtkz).
- **Originality and innovation**: By shifting from teacher-centered AI to virtual student simulations, this study pioneers a new direction in AI4Education and introduces a novel approach to personalized student modeling (Reviewers rtkz, yWBj).
- **Significant educational value**: The application of virtual students in pre-service teacher training addresses the lack of student interaction in traditional training, enhancing teachers' readiness, adaptability, and effectiveness (Reviewers wxDr, rtkz, yWBj).
- **Comprehensive and rigorous research methodology**: The SOE (Scenario-Object-Evaluation) pipeline integrates psychological theories and AI fine-tuning to systematically tackle virtual student modeling and evaluation with scientific rigor (Reviewers rtkz, 3QkY, yWBj).
- **Robust experimental validation**: A large-scale dataset and comprehensive experiments validate virtual student performance, with supplementary materials providing deeper insights (Reviewer 3QkY).
- **Clear and comprehensible writing**: The visualization of the SOE framework and structured experimental design make the research accessible and easy to understand (Reviewers wxDr, rtkz, 3QkY).
- **Distinct interdisciplinary characteristics**: This study integrates education, psychology, and AI, adding depth to AI applications in education (Reviewers rtkz, yWBj).

Based on the reviewers' invaluable suggestions, we have made extensive revisions to the paper, with all changes highlighted in blue. The key revisions are as follows:

**1. Expanded Experiments and Evaluations**

- **Objective Evaluation Experiments:** To complement subjective evaluation, four objective metrics—Text Token, Perplexity, TTR, and Sentiment Analysis—were introduced and added in Sec 5.5 and Appendix D.6, along with visualizations of the objective evaluation results (fig. 7, fig. A25-A28). These experiments collectively validate the personalization and human-like attributes of LVSA modeling.
- **ANOVA Experiments:** One-way and two-way ANOVA analyses were added to Sec 5.2, Appendix D.4.2, Table A7, and fig. A22. The results confirm significant statistical differences before and after fine-tuning, with no significant differences between virtual students and real students in human evaluations.
- **Fine-Grained Analysis of Virtual Student Responses Across Teaching Stages:** Specific examples of responses from five types of virtual students across different teaching stages and question types were added to Sec 4.3 and Appendix D.2. Using content analysis, these responses were examined in depth to reveal how they differentiate and reflect the unique personality traits of virtual students.

**2. Supplemented Details in the Manuscript**

- **Refined Abstract and Introduction:** Added description of objective evaluation: "we conduct multi-dimensional evaluation experiments that integrate both subjective human evaluations and objective metrics."
- **Refined Discussion on LC Personality:** In Sec 5.6 and Appendix D7.1, further detailed analyses of the LC personality are provided. 
- **Adjustments to the Discussion Section:** In Sec 6, descriptions of the SOE framework’s generalization capability have been added, along with future directions.
- **Supplemented Description of Evaluation Metrics:** In Sec 5, a clarification was added: “The metrics are based on the probability of virtual responses being judged by humans or GPT-4 as resembling real student responses.”
- **Clarification of Real Students’ Role in the Turing Test:** In Sec 5.2, it was emphasized that “Instead of using a fixed ground truth, real students served as a control group to assess the human-like characteristics and role-playing effectiveness of LVSA in educational interactions.”
- **Clarification of Experimental Conclusions and Their Boundaries:** In Sec 5.4 and Sec 6, it was clarified that the current results are based on controlled experimental settings and human feedback. 

**3. Introduction of Ethical Protection**

An ethical protection and data privacy statement was added in Appendix D.8.

We sincerely thank the reviewers for their valuable suggestions, which greatly improved our paper’s quality and impact. We believe the revised manuscript is now more rigorous and compelling. Below, we address three common concerns first, followed by point-by-point responses to specific comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to model LLM-based Virtual Student Agents (LVSA) that mimic the behaviors and personalities of early adolescent students in educational settings. Reviewers recognized that this paper studies a quite interesting problem and the new framework is well motivated. However, reviewers also raised some major concerns on contributions, technical details, model justifications, evaluations, etc. Although some of the concerns have been addressed by the authors during the rebuttal and discussion stage, reviewers still found that some issues cannot be easily addressed by a minor revision, such as novelty and generalization ability. Overall, the current version of this work is not ready for publication at ICLR.