Discerning Minds or Generic Tutors? Evaluating Instructional Guidance Capabilities in Socratic LLMs

## Abstract

Abstract The conversational capabilities of large language models hold significant promise for enabling scalable and interactive tutoring.
While prior research has primarily examined their ability to generate Socratic questions, it often overlooks a critical aspect: adaptively guiding learners in accordance with their cognitive states.
This study moves beyond question generation to emphasize instructional guidance capability. We ask: Can LLMs emulate expert tutors who dynamically adjust strategies in response to learners’ states?
To investigate this, we propose GuideEval, a benchmark grounded in authentic educational dialogues that evaluates pedagogical guidance through a three-phase behavioral framework: (1) Perception, inferring learner states; (2) Orchestration, adapting instructional strategies; and (3) Elicitation, stimulating proper reflections.
Empirical results indicate that existing LLMs often fail to provide effective adaptive scaffolding when learners experience confusion or require redirection. To complement the quantitative evaluation, we conduct a detailed failure case analysis, providing an intuitive understanding of these shortcomings. Furthermore, we introduce a behavior-guided finetuning strategy that leverages behavior-prompted instructional dialogues, substantially enhancing guidance performance.
By shifting the focus from isolated content evaluation to learner-centered state-aware interaction, our work advocates a more dialogic paradigm for evaluating Socratic LLMs.

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable progress across diverse natural language processing tasks (Wang et al., 2023a; Wei et al., 2021; Zhao et al., 2023), establishing themselves as foundational technologies for building intelligent educational systems. Their integration has begun to reshape learning by improving efficiency, adaptability, and personalization (Hu et al., 2025a).
In particular, educational question answering has emerged as a rapidly evolving area in which LLMs serve not only as fact-retrieving engines but also as interactive tutors that engage students in discipline-specific reasoning (Wollny et al., 2021; Lieb & Goel, 2024; Kuhail et al., 2023).

Recent research has increasingly explored the integration of the Socratic method (Elder & Paul, 1998; Paul & Elder, 2007) in educational LLMs, emphasizing iterative Socratic-style questioning to foster critical thinking (Liang & Wu, 2025).
While these efforts highlight the potential of LLMs to emulate philosophical dialogue, they often overlook the crucial pedagogical principle of guidance. In real educational settings, questioning, especially when misaligned with a learner’s cognitive readiness, can lead to cognitive overload and hinder learning (Li et al., 2021; Scarlatos et al., 2025). Empirical evidence further indicates that when instructional explanations fall outside a learner’s zone of proximal development (Shabani et al., 2010), scaffolding fails to enhance engagement and may impair comprehension and retention (Wittwer & Renkl, 2008; Hu et al., 2025b).
Therefore, effective pedagogical state-aware guidance is critical. As LLMs increasingly assume the role of virtual educators, a pressing question arises: can they provide appropriate guidance like human teachers?

This paper seeks to address this question by systematically examining the Socratic guidance capabilities of educational LLMs, aiming to elucidate whether these models are evolving into discerning educational partners or remain generic tutors executing scripted dialogues.
Effective tutoring extends beyond content delivery; it requires continuous adaptation to learners’ evolving cognitive states.
While human instructors dynamically adjust pedagogical strategies based on context, existing LLM assessments often overlook this discerning responsiveness.
Figure [1](https://arxiv.org/html/2508.06583v2#S1.F1) illustrates several examples revealing similar responses across varying student understanding states generated by GPT-4.1, highlighting a lack of adaptive instructional guidance.
To this end, we introduce GuideEval (Instructional Guidance Evaluation Benchmark), specifically designed to systematically evaluate the instructional guidance capabilities of LLMs serving as interactive tutors.

Figure: (a) Responses to accurate and erroneous states
Refer to caption: x1.png

In this paper, we hypothesize that effective instructional guidance can be conceptualized as a three-phase behavioral framework, which is motivated by the observation that meaningful instruction requires not only understanding the learner but also dynamically orchestrating strategies and eliciting active engagement.
The first phase, *Perception*, concerns the model’s ability to accurately infer the learner’s current state, a prerequisite for all subsequent instructional decisions.
The second phase, *Orchestration*, entails the adaptive pedagogical strategies aligned with the learner’s zone of proximal development. This includes techniques such as analogy generation, scaffolding, conceptual decomposition, and the strategic use of examples or counterexamples.
The third phase, *Elicitation*, centers on stimulating learner reflection and deeper understanding through targeted questioning. While prior work has emphasized generic questioning strategies, we highlight discernment-based elicitation, wherein questions are responsive to the learner’s perceptual state.

Building on this conceptual decomposition, we propose an evaluation framework that operationalizes the three core phases. Each dimension is defined by a set of behavioral indicators and performance criteria, enabling systematic, fine-grained analysis of an LLM’s instructional competencies throughout the educational interaction tutoring.
To support this framework, we construct a test corpus grounded in authentic student-model interactions by collecting multi-turn dialogues from real-world educational scenarios. To evaluate behavioral variability, we design contrastive student utterances simulating diverse cognitive states (e.g., accurate vs. erroneous, comprehension vs. confusion). These controlled variations probe the model’s sensitivity to pedagogically salient cues and its capacity for adaptive response.
Leveraging this corpus, we design a suite of evaluation tasks aligned with each core dimension. These tasks elicit distinct instructional behaviors, enabling precise measurement of instructional guidance capabilities.

We evaluate a wide range of open-source and closed-source LLMs as well as education-oriented models, with a detailed analysis of each dimension and comprehensive failure pattern study. Our empirical results reveal three critical findings in current LLMs’ instructional behaviors.

- •
Asymmetric feedback hinders error correction: While models readily affirm correct responses, they often provide vague or non-committal feedback on incorrect ones, limiting learners’ ability to promptly identify and rectify mistakes.
- •
Limited sensitivity to implicit knowledge states: Models respond effectively to explicit expressions of understanding or confusion but struggle with implicit cues that require deeper inference such as inferring underlying misconceptions from erroneous responses.
- •
Consistent failure patterns across states: Certain errors recur across different learner states, including indiscriminate affirmation and repeated explanations, indicating systematic limitations in model behaviors.

In summary, our contributions are:

- •
To the best of our knowledge, we make the first comprehensive effort to conceptualize discerning guidance as a distinct and critical dimension of Socratic LLM competence in educational question answering scenario.
- •
We introduce GuideEval, a benchmark dataset grounded in authentic multi-turn dialogues, with fine-grained tailored metrics, corresponding to the three-phase guidance behavioral framework, to enable nuanced evaluation of models’ adaptive and guiding capabilities.
- •
We conduct a systematic evaluation revealing that current LLMs exhibit substantial limitations in delivering effective instructional guidance, and we identify typical failure patterns to aid in diagnosis.
We further design behavior-aware prompting and fine-tuning schemes that markedly enhance models’ strategic adaptability.

## 2 Instructional Guidance Evaluation

Figure: Figure 2: The figure contrasts three instructional approaches and illustrates how models, when aware of student states, adapt their behaviors across Perception, Orchestration, and Elicitation dimensions.
Refer to caption: x3.png

The role of LLMs in education has evolved from providing complete solutions in a single step, to guiding learners through sequential steps, and ultimately aspires toward state-aware Socratic instruction that adapts dynamically to learners’ evolving states, as shown in Figure [2](https://arxiv.org/html/2508.06583v2#S2.F2).
Achieving this level of guidance requires addressing three fundamental questions: What is the learner’s state? How should the instruction be adapted? What questions should be posed to stimulate thinking?

### 2.1 Instructional Behavioral Modeling

Building on the three guiding questions, we formalize instructional guidance into a three-stage behavioral framework: Perception, Orchestration, and Elicitation. Each dimension represents a core competency of effective guidance and is supported by established educational theories.

Perception. Instructional guidance begins with perceiving the learner’s cognitive state, whether the response reflects accurate reasoning, misconceptions, comprehension, or confusion. According to Vygotsky’s theory of the Zone of Proximal Development (Chaiklin et al., 2003; Shabani et al., 2010), effective instruction hinges on recognizing a learner’s readiness for new knowledge. Accurate perception ensures that subsequent actions align with the learner’s actual needs rather than operating at an inappropriate level of difficulty.

Orchestration. Once perception is established, one must orchestrate instruction by adapting strategies to scaffold learning. Scaffolding theory (Van de Pol et al., 2010) highlights techniques such as simplification, analogy, and conceptual decomposition, which provide calibrated support while avoiding redundancy or cognitive overload. Orchestration thus concerns how to advance learning in a way that is sensitive to the learner’s current zone of development.

Elicitation. Beyond explanation, elicited reasoning and reflection are fostered by posing purposeful questions. Bloom’s taxonomy (Chandio et al., 2016; Eber & Parker, 2007) emphasizes adjusting the cognitive depth of questions (from factual recall to abstraction and transfer) according to learner readiness. Constructivist perspectives further stress that learning is an active meaning-making process, and well-designed questions are critical triggers of deeper engagement.

Figure [2](https://arxiv.org/html/2508.06583v2#S2.F2) illustrates representative examples of effective and ineffective behaviors in each dimension, concretizing the distinctions within our framework and highlighting how instructional strategies should shift in response to different learner states.

**Table 1: Scoring criteria for three instructional behaviors each evaluated with two dimensions.**
| Behavior | Dimension | Criteria | Score |
| --- | --- | --- | --- |
| Perception | P-Affirm | Incorrectly identifies the correctness of the student response. | 0 |
| Does not explicitly judge, but implicitly signals correctness through follow-up actions. | 0.5 |  |  |
| P-Redirect | Explicitly states whether the response is correct or incorrect (e.g., “completely correct”). | 1 |  |
| Orchestration | O-Advance | Fails to move instruction forward; content is repetitive or stagnant. | 0 |
| Advances instruction with follow-up prompts or challenges. | 1 |  |  |
| O-Reconfigure | Provides no adaptation when the student is confused or incorrect. | 0 |  |
| Reconstructs explanation (e.g., using analogies, step-by-step reasoning, or simplification). | 1 |  |  |
| Elicitation | E-Strategic | No question posed; only declarative explanation. | 0 |
| Asks factual or recall-based question (e.g., “Do you know the formula?”). | 1 |  |  |
| E-Heuristic | Asks procedural or computation-oriented question (e.g., “Can you solve for $x$?”). | 2 |  |
| Asks higher-order question encouraging reasoning or transfer (e.g., “What if the condition changes?”). | 3 |  |  |

### 2.2 Instructional Guidance Evaluation Dimensions

Learner cognitive states.
As an initial attempt, we adopt a coarse-grained categorization for cognitive-state modeling. Student utterances are classified into four primary states: accurate (demonstrating correct reasoning), erroneous (providing incorrect answers), comprehension (indicating explicit understanding), and confusion (expressing uncertainty). These categories capture the predominant patterns observed in authentic learner–tutor interactions and provide a systematic basis for examining how models adjust instructional strategies across different learner states.

Evaluation dimensions and scoring criteria.
The four states can be summarized into two broader categories: positive (accurate and comprehension) and negative (erroneous and confusion).
By intersecting with the three instructional behaviors, we derive six evaluation dimensions.

When learners in positive state, effective instruction should involve explicit affirmation to reinforce confidence (P-Affirm), advance the discussion by introducing new concepts or challenges (O-Advance) and pose higher-order questions that stimulate reasoning (E-Strategic) (Kang et al., 2021; Chandio et al., 2016).
Conversely, when learners exhibit negative, instruction should provide redirection through corrective feedback (P-Redirect), restructure explanations via scaffolding such as simplification or analogy (O-Reconfigure), and employ heuristic questioning that reduces cognitive load and fosters intuitive engagement (E-Heuristic) (Hyslop-Margison & Strobel, 2007).

Table [1](https://arxiv.org/html/2508.06583v2#S2.T1) presents the detailed scoring rubric for each metric, formalizing these instructional goals into evaluable criteria. To ensure interpretability, the scoring rubric was designed with reference to pedagogical theory: perception and orchestration metrics adopt discrete levels reflecting the presence or absence of appropriate instructional actions, while elicitation metrics are inspired by Bloom’s taxonomy to differentiate question depth.

Figure: Figure 3: Benchmark construction with human annotation, filtering, and paired state editing.
Refer to caption: x4.png

**Table 2: Benchmark comparison with major open-source educational dialogue datasets. The Contrastive Student States column denotes whether the dataset explicitly constructs paired state samples. #Turns are calculated as the product of dialogue counts and average dialogue length.**
| Function | Dataset | Multi-turn | Socratic | Contrastive<br>Student States | Real Student<br>Involvement | #Turns |
| --- | --- | --- | --- | --- | --- | --- |
| Train | CIMA (Stasaski et al., 2020) | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\sim$3.3k |
| MathDial (Macina et al., 2023) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\sim$14.2k |  |
| SocraticMath (Ding et al., 2024) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\sim$34k |  |
| SocraTeach (multi) (Liu et al., 2024) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\sim$208k |  |
| TutorChat (Chevalier et al., 2024) | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\sim$1170k |  |
| Ours (Train) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\sim$50.7k |  |
| Eval | Bridge (Wang et al., 2023b) | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | 700 |
| MathDial(test) (Macina et al., 2023) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | 572 |  |
| SocraticMath(test) (Ding et al., 2024) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | 685 |  |
| SocraTeach (multi,test) (Liu et al., 2024) | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | 1,000 |  |
| MRBench (Maurya et al., 2024) | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | 1,596 |  |
| Ours (Eval) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 5,177 |  |

### 2.3 Benchmark Construction

Dataset collection.
We begin with a corpus of 7,899 authentic learner–model dialogues collected from a Socratic tutoring platform, focusing on middle school–level science problems. After integrity check and privacy removal, 800 dialogues were sampled as the evaluation set.
Human annotators first revised model outputs to correct factual errors. Each student utterance was then labeled with one of four cognitive states. To mitigate imbalances between accurate and erroneous responses, a subset of erroneous samples was generated from validated accurate answers, and comprehension/confusion pairs were created by producing counterpart utterances with the same context. Such state editing also enabled paired evaluation of instructional behaviors.
The final benchmark consists of 5,177 samples approximately balanced across the four states (1,190 accurate, 1,181 erroneous, and 1,403 each for comprehension and confusion, see Figure [3](https://arxiv.org/html/2508.06583v2#S2.F3) for the data construction pipeline). Similarly, paired training sets (8,648 Acc./Err. pairs and 16,993 Comp./Conf. pairs) were obtained from the remaining dialogues, facilitating preference-based optimization in subsequent fine-tuning experiments.

Benchmark comparison.
Table [2](https://arxiv.org/html/2508.06583v2#S2.T2) compares our dataset with major open-source educational dialogue resources.
Most prior resources either lack Socratic interactions, omit contrastive cognitive state labels, or do not involve real-world interactions.
In contrast, our benchmark integrates real-world dialogues with controlled states under the same context to enable, for the first time, a systematic evaluation of instructional guidance for LLM competence in educational question answering.

## 3 Experiments

Setup.
Following “LLM-as-a-judge” (Fu et al., 2023; Liu et al., 2023), we employ GPT-4o-mini to evaluate mainstream LLMs and score how each model exhibits the desired instructional behaviors.
For Elicitation, we introduce *Elicitation Strategy Adaptivity* (ESA), defined as the average change in question depth across contrasting learner states (E-S $-$ E-H). Higher ESA, particularly when coupled with strong E-Strategic scores, indicates state-aligned questioning strategies.
More details about setup including specific prompts can be found in Appendix.

### 3.1 LLM-Human Preference Consistency

We first design an experiment to analyze the consistency between LLM-based scoring and human annotations in our scenario, building on recent explorations of LLMs as automatic evaluators for natural language generation tasks (Fu et al., 2023; Sottana et al., 2023).
Specifically, we select samples previously annotated by humans and re-evaluate them using our LLM-based scoring framework. Alignment is quantified via two complementary metrics: (1) agreement ratio, the proportion of instances where the LLM and human assigned the same label; and (2) average score deviation, the mean absolute difference between LLM and human scores.
As reported in Table [3](https://arxiv.org/html/2508.06583v2#S3.T3), results indicate strong concordance, with agreement ratios ranging from 89% to 97% and consistently low average deviations. These findings suggest that, when guided by carefully designed prompts, LLMs can serve as reliable and scalable evaluators of instructional behaviors.

**Table 3: LLM-Human preference consistency by agreement ratio and average score deviation.**
| Metric | P-Affirm | P-Redirect | O-Advance | O-Reconfig. | E-Level:1 | E-Level:2 | E-Level:3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Agree. (%) | 96.5% | 90.0% | 96.5% | 94.5% | 89.0% | 90.0% | 97.0% |
| Avg Dev. | 0.02 | 0.05 | 0.04 | 0.06 | 0.14 | 0.04 | 0.03 |

**Table 4: Evaluation results over Perception (P-A, P-R), Orchestration (O-A, O-R), Elicitation (E-S, E-H), and ESA (E-S $-$ E-H). Top three per column are highlighted green and the best is bold. The E-H metric is an auxiliary reference and is neither highlighted nor marked with arrows.**
|  | Accurate / Erroneous | Comprehension / Confusion |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | P-A ($\uparrow$) | P-R ($\uparrow$) | O-A ($\uparrow$) | O-R ($\uparrow$) | E-S ($\uparrow$) | E-H | ESA ($\uparrow$) | O-A ($\uparrow$) | O-R ($\uparrow$) | E-S ($\uparrow$) | E-H | ESA ($\uparrow$) |
| Open-source general LLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen3-8B | 0.7613 | 0.5919 | 0.9176 | 0.6681 | 2.0605 | 1.9213 | 0.3281 | 0.9644 | 0.8959 | 2.0527 | 1.7254 | 0.1049 |
| GLM-4-9B | 0.8534 | 0.4140 | 0.7529 | 0.3492 | 1.8992 | 1.9195 | -0.0173 | 0.6885 | 0.8582 | 1.6450 | 1.6515 | -0.0064 |
| Qwen3-32B | 0.8092 | 0.6008 | 0.9521 | 0.7748 | 2.2252 | 2.0906 | 0.1646 | 0.9672 | 0.8660 | 2.1454 | 1.9808 | 0.1481 |
| Llama-3.3-70B-Instruct | 0.8077 | 0.5390 | 0.8651 | 0.4601 | 2.0766 | 2.1087 | 0.0606 | 0.7997 | 0.8509 | 1.9936 | 1.9330 | -0.0418 |
| DeepSeek-V3 | 0.8748 | 0.5428 | 0.8849 | 0.6740 | 2.0059 | 1.9856 | 0.3136 | 0.8810 | 0.8460 | 2.0485 | 1.7349 | 0.0272 |
| DeepSeek-R1 | 0.8546 | 0.5483 | 0.9445 | 0.7036 | 2.1966 | 2.0322 | 0.3659 | 0.9608 | 0.9294 | 2.1946 | 1.9287 | 0.2141 |
| Closed-source general LLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| Mistral-medium | 0.8206 | 0.5585 | 0.7857 | 0.4924 | 2.0319 | 1.9322 | 0.3058 | 0.8125 | 0.7840 | 1.9544 | 1.6486 | 0.0889 |
| O4-mini | 0.8189 | 0.4306 | 0.9084 | 0.5597 | 2.0193 | 1.9018 | 0.1098 | 0.9152 | 0.8161 | 2.0299 | 1.9202 | 0.1099 |
| GPT-4.1 | 0.8710 | 0.4280 | 0.8790 | 0.6076 | 2.2546 | 2.1254 | 0.2495 | 0.8717 | 0.6151 | 2.2153 | 1.9658 | 0.1185 |
| Claude-sonnet-4 | 0.9502 | 0.4780 | 0.9248 | 0.6407 | 2.1410 | 1.9924 | 0.3464 | 0.9187 | 0.8539 | 2.1026 | 1.7562 | 0.1328 |
| Gemini-2.5-pro | 0.9324 | 0.4136 | 0.8597 | 0.5483 | 2.1471 | 1.8483 | 0.3961 | 0.8709 | 0.8771 | 2.0516 | 1.6533 | 0.2593 |
| GLM-4-plus | 0.9059 | 0.4665 | 0.8580 | 0.5373 | 1.9134 | 1.9712 | 0.0720 | 0.7883 | 0.9565 | 1.8297 | 1.7577 | -0.0506 |
| Doubao-Seed-1.6-Thinking | 0.5913 | 0.5548 | 0.8848 | 0.7810 | 2.0690 | 1.8744 | 0.3051 | 0.9430 | 0.8496 | 2.0848 | 1.7798 | 0.2141 |
| Education-oriented models |  |  |  |  |  |  |  |  |  |  |  |  |
| SocraticLM (base: GLM-4-9B) | 0.9105 | 0.1411 | 0.6429 | 0.0712 | 1.3303 | 1.3458 | 0.0720 | 0.6108 | 0.4939 | 1.2397 | 1.1675 | -0.0160 |
| Spark X1 | 0.7445 | 0.4949 | 0.8815 | 0.6147 | 2.2168 | 2.0923 | 0.2295 | 0.9387 | 0.7320 | 2.1518 | 1.9223 | 0.1222 |

### 3.2 Evaluation Results

Figure: Figure 4: Deep dive for Perception and Orchestration behaviors. (a) Feedback score distribution contrasting model responses to accurate versus erroneous answers. (b) Orchestration Strategy Adaptivity (OSA) across paired states such as accurate/erroneous or comprehension/confusion, reflecting how different models vary in their flexibility to adjust instructional strategies.
Refer to caption: x5.png

Figure: Figure 5: Deep dive for Elicitation behavior. Bars indicate the average E-Strategic scores under accurate and comprehension states, while lines plot the corresponding Elicitation Strategy Adaptivity (ESA). Models are ordered by ESA under accurate/erroneous contrast.
Refer to caption: x6.png

Overall findings.
Table [4](https://arxiv.org/html/2508.06583v2#S3.T4) presents the evaluation results across all proposed dimensions. Among the models,
Qwen3-32B and DeepSeek-R1 stand out with the most balanced performance across metrics, while GPT-4.1 excels in elicitation, providing deeper and more strategic questions.
A consistent pattern emerges across models: they handle ideal positive inputs effectively but struggle to adapt to negative learner responses. This underscores a critical challenge for future development, enhancing instructional flexibility to support adaptive teaching under imperfect conditions.
To gain deeper insights, we further analyze model performance along the three instructional behaviors, focusing on adaptation to diverse learner states and the degree of strategic responsiveness.

Perception: deficient error detection.
Figure [4](https://arxiv.org/html/2508.06583v2#S3.F4) (a) shows the distribution of perception scores contrasting model responses to accurate and erroneous answers. Models achieve consistently high P-A values (typically $>$0.8) by affirming correctness (e.g., Claude-Sonnet-4: 0.95), yet P-R values remain substantially lower, reflecting a limitation to provide corrective feedback. Only a few models (Qwen3-32B: 0.60; DeepSeek-R1: 0.55) show moderate detection, whereas SocraticLM performs poorly (0.14). Many models resort to vague commentary or topic shifts, yielding mid-level (0.5) scores. This tendency to prioritize politeness over explicit correction leaves misconceptions unaddressed, highlighting deficient error recognition as a core limitation for effective tutoring.

Orchestration: limited flexibility to implicit states.
Figure [4](https://arxiv.org/html/2508.06583v2#S3.F4) (b) illustrates the Orchestration Strategy Adaptivity (OSA), defined as the proportion of paired cases where a model delivers effective responses under both states (e.g., accurate vs. erroneous or comprehension vs. confusion). Higher OSA reflects greater flexibility in strategy adjustment.
Across models, OSA is consistently higher for comprehension/confusion than for accurate/erroneous contrasts, showing that LLMs react more easily to explicit signals of understanding than to implicit cues from answer accuracy. Qwen3-32B (0.840 vs. 0.758) adapts relatively well, GPT-4.1 remains stable (0.549 vs. 0.537), whereas LLaMA-3.3-70B and Mistral-Medium show declines exceeding 0.2, and SocraticLM nearly fails (0.043). These results highlight a systemic limitation: many models struggle to flexibly orchestrate strategies in response to implicit cues, constraining their capacity for adaptive scaffolding.

Elicitation: reduced adaptivity across accurate/erroneous responses.
As shown in Figure [5](https://arxiv.org/html/2508.06583v2#S3.F5), Gemini-2.5-Pro and Claude-Sonnet-4 achieve high ESA alongside robust E-Strategic scores, reflecting adaptive questioning. In contrast, LLaMA-3.3-70B and GLM-4-Plus exhibit near-zero ESA, indicating uniform questioning. Notably, most models show marked ESA declines under accurate/erroneous contrasts; some even fall below zero (posing deeper questions in response to error answers), which may hinder comprehension. These findings highlight elicitation adaptivity as a persistent challenge and underscore the need for more cognitively aware questioning in future LLMs.

### 3.3 Case Study: Failure Pattern Analysis

To gain a better understanding of the quantitative evaluation, we manually examined representative low-scoring responses, focusing on Perception and Orchestration, and conducted a detailed failure pattern analysis. These dimensions were selected because their error patterns are more readily identifiable in individual outputs, enabling finer-grained insights into model limitations.
Typical Perception errors include misjudging unconventional yet valid reasoning, questioning established knowledge, or endorsing incorrect answers. Orchestration failures mainly reflect insufficient adaptability, such as repeating flawed or already mastered explanations, offering vague remediation after weak feedback, or failing to provide new input despite clear learner confusion.
Appendix [A](https://arxiv.org/html/2508.06583v2#A1) provides illustrative examples and visual summaries that concretize these observations. Such cases recur across multiple models, offering valuable references for designing behavior-aware strategies to enhance model robustness and pedagogical effectiveness.

Figure: (a) Perception
Refer to caption: x7.png

### 3.4 Behavior-Guided Finetuning

We further explore ways to enhance instructional capabilities. For efficiency and feasibility, we conduct experiments with a compact LLM, Qwen3-8B (Yang et al., 2025).
We take the model (regarded as Base)
and first examine a prompt-based approach, Rule-Prompt, which incorporates external if–then policies for different student states (prompt details in the Appendix).
We then employ finetuning strategies. Specifically, we explore: SFT, only distills final answers; KTO (Ethayarajh et al., 2024), a pointwise preference optimization guided by correctness but lacking relative contrast; DPO (Rafailov et al., 2023), a pairwise preference optimization that exploits relative comparisons to capture behavioral differences; and SFT (w/ CoT Distillation) (Li et al., 2023), which incorporates reasoning traces by process-level supervision. This spectrum of methods allows us to analyze how supervision, from outcome-only to process-aware, shapes the model’s capacity.
Experimental details are provided in the Appendix.

Figure [6](https://arxiv.org/html/2508.06583v2#S3.F6) presents the comparative results across the three instructional behaviors. SFT (w/ CoT Distillation) yields the strongest improvements in most cases, particularly in handling accurate and erroneous states. Four key findings emerge. First, SFT degrades instructional strategies by emphasizing final output.
Second, KTO introduces contrastive supervision through positive and negative examples under opposite student states. Yet, its pointwise formulation restricts effectiveness: lacking pairwise contrast, the model fails to distinguish subtle behaviors, resulting in minimal divergence between strategic and heuristic questioning. Third, DPO alleviates this limitation by leveraging relative comparisons, thereby enhancing state recognition and guidance adaptation. Finally, the most substantial gains derive from CoT Distillation, where high-quality reasoning traces function as process supervision, providing explicit guidance on instructional process knowledge.

## 4 Related Work

Building socratic LLMs.
LLMs are increasingly employed as interactive educational tutors due to their advanced language understanding and generation capabilities. Recent research has focused on enhancing LLMs’ tutoring performance through multi-turn dialogues grounded in textbook content, such as TutorChat (Chevalier et al., 2024) and MathDial (Macina et al., 2023), further extended across diverse subjects and grade levels by NewtBot (Lieb & Goel, 2024) and SocraticMath (Ding et al., 2024).
Building on these developments, recent studies have explored integrating the Socratic method, a pedagogical strategy centered on critical thinking via iterative questioning (Elder & Paul, 1998; Paul & Elder, 2007).
Early attempts in this area explored prompt engineering, for example, Chang et al (Chang, 2023) constructed Socratic prompts.
Other approaches leverage data augmentation, LoRA-based fine-tuning, and preference optimization techniques (Kumar & Lan, 2024; Shani et al., 2024).
EduChat (Dan et al., 2023) integrates open-ended question answering, essay evaluation, and Socratic dialogue into a unified LLM framework. Similarly, SocraticLM (Liu et al., 2024) introduces a multi-agent “Dean–Teacher–Student” architecture to emulate Socratic-style instruction in foundational mathematical reasoning. Despite these advances, most systems focus predominantly on Socratic questioning while overlooking the equally critical role of instructional guidance.

Evaluating socratic LLMs.
Most existing evaluation efforts for LLMs emphasize metrics such as BLEU, ROUGE, and BERT-Score, or focus on answer correctness and presentation quality (Favero et al., 2024; Chevalier et al., 2024). TutorEval (Chevalier et al., 2024), for example, evaluates scientific reasoning and comprehension through QA tasks based on extended textbook excerpts. Recent efforts have shifted toward structured evaluations of Socratic tutoring capabilities. SocraticLM (Liu et al., 2024) introduces a five-dimensional framework.
SocratiQ (Ang et al., 2023) offers a large-scale (question, context) dataset for Socratic-style question generation, assessed by human ratings of fluency, relevance, and answerability. Dr.Academy (Chen et al., 2024) presents a benchmark for evaluating questioning abilities in educational LLMs. MRBench (Maurya et al., 2024) and its accompanying taxonomy represent progress toward pedagogical value but remain focused on response content rather than adaptive guidance.
In contrast, our framework adopts a state-controlled contrastive design, varying student states while holding history contexts constant. This enables precise evaluation of LLMs’ sensitivity and capacity for context-aware, adaptive instructional support.

## 5 Conclusion and Limitation

We present GuideEval, the first benchmark to evaluate instructional guidance in educational Socratic LLMs through a three-phase behavior framework of Perception, Orchestration, and Elicitation.
Our empirical evaluation across diverse LLMs reveals consistent limitations in current systems: asymmetric feedback that impedes error correction, weak sensitivity to implicit cognitive states, and recurring failure patterns across states.
To support a deeper understanding of these shortcomings, we provide a detailed analysis of failure patterns, offering a foundation for diagnosing limitations and guiding the development of educational models. Furthermore, we explore preliminary strategies to enhance LLMs’ ability to approximate the discernment and responsiveness characteristic of human teachers. We hope that GuideEval will serve as a benchmark to advance LLMs toward more adaptive, nuanced, and personalized learning experiences.

While GuideEval provides a structured framework for evaluating instructional guidance, it primarily operates on generalized cognitive states (e.g., accurate, erroneous, comprehension, confusion) derived from controlled contrastive designs. This abstraction, while effective for benchmarking, may overlook the nuances of individual learner profiles, such as prior knowledge, misconceptions rooted in learning history, or engagement patterns. Future work may explore personalized evaluation settings by integrating longitudinal learning traces or student modeling techniques, enabling a finer-grained assessment of model adaptability to diverse and evolving learner needs.

## 6 Broader Impact

This work contributes a systematic benchmark for evaluating the guidance-oriented behaviors of LLMs in educational contexts.
It underscores the necessity of designing models that can respond appropriately to diverse learners’ cognitive states, promoting fairness and inclusivity in AI-driven education. As LLMs become more prevalent in classrooms, tutoring platforms, and remote learning environments, their capacity for context-sensitive, pedagogically sound guidance will directly affect educational quality, equity, and accessibility. Furthermore, this research lays the foundation for integrating longitudinal student interaction data, enabling personalized, trajectory-aware instruction that respects individual learning paths while safeguarding against potential biases.

## Ethics statement

This study is based on real-world educational dialogues between students and a Socratic-style tutoring model. To protect privacy, we do not disclose the specific platform or model involved.
During data collection, human annotators carefully reviewed to remove personally identifiable information, including names, school identifiers, and geographic references. In addition, potentially harmful or inappropriate content (e.g., offensive language) was systematically filtered out. As a result, the dataset used in this work contains only de-identified and pedagogically relevant interactions.

## Reproducibility statement

We are committed to ensuring the reproducibility of our work. To this end, we provide detailed descriptions of dataset construction, annotation, and state-editing procedures in Section 2 and Appendix A. All evaluation metrics and scoring rubrics are explicitly defined in Section 2.2 and Appendix B, and the full set of prompts used for both generation and evaluation are included in Appendix E. Experimental settings, model configurations, and hyperparameters for all fine-tuning and evaluation procedures are documented in Appendix D. Together, these materials enable independent researchers to replicate our evaluation and reproduce the reported results.

## References

- Ang et al. (2023)
Beng Heng Ang, Sujatha Das Gollapalli, and See Kiong Ng.
Socratic question generation: A novel dataset, models, and evaluation.
In *Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pp. 147–165, 2023.
- Chaiklin et al. (2003)
Seth Chaiklin et al.
The zone of proximal development in vygotsky’s analysis of learning and instruction.
*Vygotsky’s educational theory in cultural context*, 1(2):39–64, 2003.
- Chandio et al. (2016)
Muhammad Tufail Chandio, Saima Murtaza Pandhiani, and Rabia Iqbal.
Bloom’s taxonomy: Improving assessment and teaching-learning process.
*Journal of education and educational development*, 3(2):203–221, 2016.
- Chang (2023)
Edward Y Chang.
Prompting large language models with the socratic method.
In *2023 IEEE 13th annual computing and communication workshop and conference (CCWC)*, pp. 0351–0360. IEEE, 2023.
- Chen et al. (2024)
Yuyan Chen, Chenwei Wu, Songzhou Yan, Panjun Liu, Haoyu Zhou, and Yanghua Xiao.
Dr. academy: A benchmark for evaluating questioning capability in education for large language models.
*arXiv preprint arXiv:2408.10947*, 2024.
- Chevalier et al. (2024)
Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, et al.
Language models as science tutors.
*arXiv preprint arXiv:2402.11111*, 2024.
- Cochran et al. (2023)
Keith Cochran, Clayton Cohn, Jean Francois Rouet, and Peter Hastings.
Improving automated evaluation of student text responses using gpt-3.5 for text data augmentation.
In *International conference on artificial intelligence in education*, pp. 217–228. Springer, 2023.
- Dan et al. (2023)
Yuhao Dan, Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, et al.
Educhat: A large-scale language model-based chatbot system for intelligent education.
*arXiv preprint arXiv:2308.02773*, 2023.
- Ding et al. (2024)
Yuyang Ding, Hanglei Hu, Jie Zhou, Qin Chen, Bo Jiang, and Liang He.
Boosting large language models with socratic method for conversational mathematics teaching.
In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*, pp. 3730–3735, 2024.
- Eber & Parker (2007)
Patricia A Eber and Trent S Parker.
Assessing student learning: applying bloom’s taxonomy.
*Human service education*, 27(1), 2007.
- Elder & Paul (1998)
Linda Elder and Richard Paul.
The role of socratic questioning in thinking, teaching, and learning.
*The Clearing House*, 71(5):297–301, 1998.
- Ethayarajh et al. (2024)
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
Kto: Model alignment as prospect theoretic optimization.
*arXiv preprint arXiv:2402.01306*, 2024.
- Favero et al. (2024)
Lucile Favero, Juan Antonio Pérez-Ortiz, Tanja Käser, and Nuria Oliver.
Enhancing critical thinking in education by means of a socratic chatbot.
*arXiv preprint arXiv:2409.05511*, 2024.
- Fu et al. (2023)
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu.
Gptscore: Evaluate as you desire.
*arXiv preprint arXiv:2302.04166*, 2023.
- Hu et al. (2025a)
Bihao Hu, Jiayi Zhu, Yiying Pei, and Xiaoqing Gu.
Exploring the potential of llm to enhance teaching plans through teaching simulation.
*npj Science of Learning*, 10(1):7, 2025a.
- Hu et al. (2025b)
Xiangen Hu, Sheng Xu, Richard Tong, and Art Graesser.
Generative ai in education: From foundational insights to the socratic playground for learning.
*arXiv preprint arXiv:2501.06682*, 2025b.
- Hyslop-Margison & Strobel (2007)
Emery J Hyslop-Margison and Johannes Strobel.
Constructivism and education: Misunderstandings and pedagogical implications.
*The Teacher Educator*, 43(1):72–86, 2007.
- Kang et al. (2021)
Nan Kang et al.
A brief overview of the effectiveness of implicit and explicit feedback in second language acquisition.
*Frontiers in Educational Research*, 4(14), 2021.
- Kuhail et al. (2023)
Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori.
Interacting with educational chatbots: A systematic review.
*Education and Information Technologies*, 28(1):973–1018, 2023.
- Kumar & Lan (2024)
Nischal Ashok Kumar and Andrew Lan.
Improving socratic question generation using data augmentation and preference optimization.
*arXiv preprint arXiv:2403.00199*, 2024.
- Li et al. (2021)
Huihan Li, Tianyu Gao, Manan Goenka, and Danqi Chen.
Ditch the gold standard: Re-evaluating conversational question answering.
*arXiv preprint arXiv:2112.08812*, 2021.
- Li et al. (2023)
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi.
Symbolic chain-of-thought distillation: Small models can also" think" step-by-step.
*arXiv preprint arXiv:2306.14050*, 2023.
- Liang & Wu (2025)
Weijun Liang and Yanjun Wu.
Exploring factors influencing chatgpt-supported critical thinking skill cultivation in english-as-a-foreign-language classrooms.
*Educación XX1*, 28(2), 2025.
- Lieb & Goel (2024)
Anna Lieb and Toshali Goel.
Student interaction with newtbot: An llm-as-tutor chatbot for secondary physics education.
In *Extended Abstracts of the CHI Conference on Human Factors in Computing Systems*, pp. 1–8, 2024.
- Liu et al. (2024)
Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen.
Socraticlm: Exploring socratic personalized teaching with large language models.
*Advances in Neural Information Processing Systems*, 37:85693–85721, 2024.
- Liu et al. (2023)
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
G-eval: Nlg evaluation using gpt-4 with better human alignment.
*arXiv preprint arXiv:2303.16634*, 2023.
- Macina et al. (2023)
Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan.
Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems.
*arXiv preprint arXiv:2305.14536*, 2023.
- Maurya et al. (2024)
Kaushal Kumar Maurya, KV Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar.
Unifying ai tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of llm-powered ai tutors.
*arXiv preprint arXiv:2412.09416*, 2024.
- Paul & Elder (2007)
Richard Paul and Linda Elder.
Critical thinking: The art of socratic questioning.
*Journal of developmental education*, 31(1):36, 2007.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in neural information processing systems*, 36:53728–53741, 2023.
- Scarlatos et al. (2025)
Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, and Andrew Lan.
Training llm-based tutors to improve student learning outcomes in dialogues.
*arXiv preprint arXiv:2503.06424*, 2025.
- Shabani et al. (2010)
Karim Shabani, Mohamad Khatib, and Saman Ebadi.
Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development.
*English language teaching*, 3(4):237–248, 2010.
- Shani et al. (2024)
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, and Idan Szpektor.
Multi-turn reinforcement learning from preference human feedback.
[https://arxiv.org/abs/2405.14655](https://arxiv.org/abs/2405.14655), 2024.
- Sottana et al. (2023)
Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan.
Evaluation metrics in the era of gpt-4: Reliably evaluating large language models on sequence to sequence tasks.
*arXiv preprint arXiv:2310.13800*, 2023.
- Stasaski et al. (2020)
Katherine Stasaski, Kimberly Kao, and Marti A Hearst.
Cima: A large open access dialogue dataset for tutoring.
In *Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications*, pp. 52–64, 2020.
- Van de Pol et al. (2010)
Janneke Van de Pol, Monique Volman, and Jos Beishuizen.
Scaffolding in teacher–student interaction: A decade of research.
*Educational psychology review*, 22(3):271–296, 2010.
- Wang et al. (2023a)
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu.
Document-level machine translation with large language models.
*arXiv preprint arXiv:2304.02210*, 2023a.
- Wang et al. (2023b)
Rose E Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky.
Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes.
*arXiv preprint arXiv:2310.10648*, 2023b.
- Wei et al. (2021)
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
*arXiv preprint arXiv:2109.01652*, 2021.
- Wittwer & Renkl (2008)
Jörg Wittwer and Alexander Renkl.
Why instructional explanations often do not work: A framework for understanding the effectiveness of instructional explanations.
*Educational Psychologist*, 43(1):49–64, 2008.
- Wollny et al. (2021)
Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler.
Are we there yet?-a systematic literature review on chatbots in education.
*Frontiers in artificial intelligence*, 4:654924, 2021.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report.
*CoRR*, abs/2505.09388, 2025.
- Zhao et al. (2023)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.
A survey of large language models.
*arXiv preprint arXiv:2303.18223*, 1(2), 2023.

## Appendix A Failure Case Taxonomy and Examples

To supplement our quantitative evaluation, we present a taxonomy of model failures under the Perception and Orchestration behaviors, as their correctness can be reliably and intuitively judged by human evaluators. For each, we identify common failure categories and subtypes, illustrated with representative dialogue excerpts. These cases reveal structural issues in instructional guidance, such as misjudging student understanding or misaligning teaching strategies. In this way, we hope to provide deeper insights into the underlying limitations of current models, thereby extending our understanding beyond numerical performance metrics. Table [B.2](https://arxiv.org/html/2508.06583v2#A2.SS2) and Table [B.2](https://arxiv.org/html/2508.06583v2#A2.SS2) summarize the taxonomy and provide example cases.

## Appendix B Metric Design Rationale

### B.1 From Cognitive States to Instructional Goals

To operationalize instructional guidance, we first delineate the learner’s cognitive states. We adopt a coarse binary view of learning progress—positive states indicate that learning can continue productively, whereas negative states suggest a stall or misalignment in progress. Building on this, we further refine the categorization along two axes: whether the learner’s response is accurate or erroneous, and whether it conveys explicit comprehension or confusion. This yields four primary states that comprehensively capture the predominant conditions observed in authentic learner–tutor dialogues. Such a design not only ensures coverage of typical response scenarios but also provides natural pairs of contrasting states (e.g., accurate vs. erroneous, comprehension vs. confusion), which enable controlled comparisons in subsequent experiments. Once this taxonomy is established, the instructional goals associated with positive versus negative states naturally follow: positive states should be reinforced and advanced through affirmation, progression, and deeper questioning, while negative states call for corrective feedback, reconfiguration of explanations, and heuristically grounded questions to lower cognitive barriers.

### B.2 Operationalization of Six Metrics

Following the mapping from learner states to instructional goals, we translate the framework into six concrete evaluation metrics. These metrics represent observable instructional behaviors that arise when the model interacts with either positive or negative learner states. Grounded in established pedagogical theory, the definitions provide fine-grained criteria for systematically evaluating instructional quality:

- •
P-Affirm: Evaluates whether the model explicitly affirms accurate responses. Clear, unambiguous affirmation reinforces learning and confidence (Kang et al., 2021), making this a foundational perceptual behavior.
- •
P-Redirect: Evaluates the model’s ability to deliver corrective feedback following an erroneous response. This includes identifying misconceptions and guiding the student toward a more accurate understanding, consistent with evidence on the importance of timely, targeted remediation (Kang et al., 2021).
- •
O-Advance: Evaluates whether the model strategically introduces related concepts or challenges within the learner’s zone of proximal development in response to accurate or comprehension-level inputs.
- •
O-Reconfigure: Evaluates whether the model adaptively restructures the instruction, such as revisiting foundational concepts or modifying explanatory strategies, to address underlying misconceptions in response to confused or incorrect student answers.
- •
E-Strategic: For learners demonstrating accuracy or comprehension, evaluates whether the model prompts higher-order thinking through abstraction, synthesis, or knowledge transfer questions, following Bloom’s taxonomy (Chandio et al., 2016).
- •
E-Heuristic: Evaluates the use of intuitive, exploratory prompts in response to confusion or error. These heuristic questions are designed to foster curiosity, activate prior knowledge, and engage informal reasoning to facilitate discovery-based learning (Hyslop-Margison & Strobel, 2007).

Together, these six metrics constitute an operationalized, theory-informed rubric for evaluating instructional behaviors. They capture not only the appropriateness of model responses but also their alignment with pedagogical best practices in scaffolding and adaptive tutoring.

**Table 5: Representative failure cases under the Perception behavior.**
| Failure Category | Failure Subtype | Example Case |
| --- | --- | --- |
| Misjudging Accurate Answers | Rigid adherence to procedural form. The student reasonably combines multiple steps in a calculation (e.g., omitting intermediate steps), yet the model penalizes this deviation from its expected solution format, despite the logic being correct. | (DeepSeek-V3)<br>History: Model: (Previously inferred that the last page of the book is page 128) … The question says Lele finished reading the book in 4 days, so how many pages did he read on average each day?<br>Student: 32<br>Model response: Let’s think about it again. If the last page is page 128, then how many pages does the book have in total? |
| Unwarranted skepticism toward foundational knowledge. In tasks involving definitions (e.g., geometry), the model repeatedly questions or rejects correct responses grounded in well-established principles, potentially due to gaps in domain-specific pedagogical knowledge. | (Llama-3-3-70B-Instruct)<br>History: Model: …Now, let’s look at Statement (2): If two parallel lines are cut by a third line and the consecutive interior angles are equal, then both parallel lines are perpendicular to the third line. Do you think this statement is correct?<br>Student: Yes, it’s definitely correct.<br>Model response: Actually, Statement (2) is a bit misleading. If two parallel lines are cut by a third line and the consecutive interior angles are equal, this usually suggests that the third line is perpendicular to the two parallel lines, but it does not necessarily mean that both parallel lines are perpendicular to the third line… |  |
| Failure to track student response intent. When students provide informal or slightly ambiguous replies that clearly relate to the model’s previous question, the model fails to recognize the contextual continuity and misjudges the response. | (Qwen3-32B)<br>History: Model: …Now, we know that the speed ratio between the passenger car and the freight car is 5:6, so we can distribute the total speed according to this ratio. Can you calculate the speed of each vehicle?<br>Student: 55/66<br>Model response: Your approach is correct, but the result seems a bit off. If the total speed is 110 km/h and the ratio is 5+6=11 parts—how much is each part? Can you try recalculating it? |  |
| Affirming Erroneous Answers | Uncritical acceptance of student answers. The model affirms student responses (e.g., “Correct” or “Well done”) even when the answer is wrong. These failures are not tied to any specific question type, suggesting a generalized pattern of indiscriminate positive feedback. | (GPT-4.1)<br>History: Model: …The question says the new quotient is 2 less than the original one, and the remainder stays the same. This means the dividend was reduced by 2 times the divisor. Can you use this information to find out what the divisor is?<br>Student: Thirty. (The correct answer is 36)<br>Model response: Great, you’ve found that the divisor is 30. Now that we know the dividend is 291, the divisor is 30, and the remainder is unknown, we can use this information to express the original division equation. |
| Contradictory positive feedback. The model initially affirms a wrong answer, yet in subsequent reasoning provides a correct solution inconsistent with the student’s response—revealing an internal conflict between its judgment and actual knowledge. | (glm-4-plus)<br>History: Model: …Can you tell me the values of n and m by solving the system of equations -3n = -15 and m = n - 3?<br>Student: m = 5, n = 2 (The correct answer is m = 2, n = 5)<br>Model response: Very good! By solving the equation -3n = -15, we get n = 5. Then, substituting n = 5 into the equation m = n - 3, we get m = 5 - 3 = 2. So, the values of m and n are 2 and 5 respectively… |  |

### B.3 Scoring Granularity and Asymmetry

Our scoring rubric applies different levels of granularity across dimensions, reflecting the distinct nature of the instructional behaviors being measured.

For Perception, judgments of response correctness are not strictly binary. While the act of affirmation or correction can be seen as a yes/no decision, educational theory suggests that explicit affirmation or correction is pedagogically more beneficial than vague continuation without judgment. Accordingly, we adopt a three-tier scheme (1/0.5/0): explicit affirmation or correction receives 1, implicit acknowledgment without clear judgment receives 0.5, and misjudgment receives 0.
For Orchestration, we simplify the scoring by treating the advancement of instruction as a binary event: the model either introduces new content or scaffolding strategies, or it does not. Thus, O-Advance and O-Reconfigure are scored on a 0/1 basis.
For Elicitation, we draw on Bloom’s taxonomy of educational objectives, which divides cognitive demand into six levels: knowledge, comprehension, application, analysis, synthesis, and evaluation. To balance interpretability and practicality, we merge adjacent levels into three ordered categories; together with the absence of questioning (score 0), this yields a four-level scheme (0–3).

Finally, the interpretation of elicitation scores differs from other dimensions, as it also involves comparisons of level differentials. Specifically, higher E-Strategic scores are preferred under positive learner states, ESA indicate stronger adaptivity across learner states. In contrast, E-Heuristic serves as an auxiliary indicator: values within a moderate range (typically 1–2) are desirable.

## Appendix C Evaluate Experimental Details

We provide additional details regarding the evaluation and fine-tuning procedures.

### C.1 Input Data Structure

Each evaluation instance isolates a single *state turn* while preserving the authentic dialogue context
(the integrity of context is ensured in a separate preprocessing step and is not part of this section).
Formally, we represent one instance as

$$ $D\;=\;\Big(\;\underbrace{\{(S_{Q},\,Q_{0}),\,(S_{1},\,Q_{1}),\,\ldots,\,(S_{t-1},\,Q_{t-1})\}}_{\text{authentic context }\mathcal{C}},\;\;S_{\mathrm{state}},\;\;R\;\Big).$ (1) $$

Here, $S_{Q}$ denotes the student’s initial query and $Q_{0}$ the model’s first reply. For $j\geq 1$,
$S_{j}$ is the learner’s $j$-th utterance and $Q_{j}$ the corresponding model reply; all pairs $(S_{j},Q_{j})\in\mathcal{C}$
come from *real* interactions. The pivot student utterance $S_{\mathrm{state}}$ is either the original $S_{t}$ or a
*state-edited* variant obtained by flipping exactly one axis:

$$ $S_{\mathrm{state}}\;=\;\begin{cases}S_{t},&\text{authentic (edit flag $e=0$)},\[4.0pt] \mathrm{Flip}_{\alpha}(S_{t}),&\text{state-edited (}e=1,\;\alpha\in\{\text{Acc/Err},\,\text{Comp/Conf}\})\end{cases}$ (2) $$

where

$$ $y_{\mathrm{state}}\in\{\text{accurate},\,\text{erroneous},\,\text{comprehension},\,\text{confusion}\}.$ $$

We use two mutually exclusive edit types:

$$ $\displaystyle\alpha=\text{Acc/Err}$ $\displaystyle:\text{answer-correctness flip (accurate }\leftrightarrow\text{ erroneous)},$ $\displaystyle\alpha=\text{Comp/Conf}$ $\displaystyle:\text{metacognitive-expression flip (comprehension }\leftrightarrow\text{ confusion)}.$ $$

Given $\mathcal{C}$ and $S_{\mathrm{state}}$, the model under evaluation produces

$$ $R\;=\;\mathcal{M}\!\big(\mathcal{C},\,S_{\mathrm{state}}\big).$ (3) $$

For contrastive analyses within the same context, we maintain a pair identifier $\mathrm{pid}$ linking the authentic
$S_{t}$ and its edited counterpart $\mathrm{Flip}_{\alpha}(S_{t})$ when $e=1$.

### C.2 Evaluation Settings

We adopt two inference protocols depending on accessibility. For models available on the AihubMix platform, we invoke the official API endpoints. For open-source models not accessible via API (e.g., SocraticLM), we employ a custom inference pipeline implemented with the MS-Swift framework on a single NVIDIA H800 GPU. In both cases, the decoding temperature was fixed at 0.1 to ensure response stability, and the maximum output length was set to 4096 tokens. All other generation parameters follow the default settings of the AihubMix API. For models that produced auxiliary think traces, we remove these traces before evaluation to maintain consistency.

## Appendix D Finetuning Experimental Details

**Table 7: Evaluation results of DeepSeek-R1 and DeepSeek-R1 with designed prompt.**
|  | Accurate / Erroneous | Comprehension / Confusion |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | P-A ($\uparrow$) | P-R ($\uparrow$) | O-A ($\uparrow$) | O-R ($\uparrow$) | E-S ($\uparrow$) | E-H | ESA ($\uparrow$) | O-A ($\uparrow$) | O-R ($\uparrow$) | E-S ($\uparrow$) | E-H | ESA ($\uparrow$) |
| DeepSeek-R1 | 0.8546 | 0.5483 | 0.9445 | 0.7036 | 2.1966 | 2.0322 | 0.2659 | 0.9608 | 0.9294 | 2.1946 | 1.9287 | 0.1420 |
| + Designed Prompt | 0.9071 | 0.9149 | 0.9554 | 0.8857 | 2.1017 | 1.7714 | 0.4362 | 0.9537 | 0.9878 | 2.1682 | 1.7320 | 0.3272 |

### D.1 Training Data Generation

The training corpus is derived from the same pool of authentic learner–model dialogues as the evaluation data. After reserving 800 dialogues for evaluation, the remaining conversations serve as candidates for training. Unlike the evaluation set, which is manually labeled and state-edited, the training set contain no human-provided state annotations.

To ensure quality at scale, we adopt an automatic filtering pipeline. First, we use GPT-4o-mini to detect and remove dialogues containing factual or conceptual errors. Next, we construct diverse contexts covering the four target states. Specifically:

- •
We identify a subset of contexts where the learner’s answer was verified as correct by both the original model and ChatGPT (dual validation).
- •
From these contexts, we generate erroneous variants by rewriting correct answers into incorrect ones. This procedure has been shown to produce more natural and pedagogically plausible errors than directly sampling erroneous from a model (Cochran et al., 2023).
- •
We further select a subset of reliable contexts and append additional learner utterances explicitly expressing comprehension or confusion, covering the metacognitive dimension.

Finally, to generate target instructional responses, we employ DeepSeek-R1, one of the strongest-performing models in our evaluation. A carefully designed system prompt instructed the model to analyze learner states, adjust strategies, and provide guided explanations; the full prompt is presented below this section. This produced high-quality responses are recorded along with their associated *think traces* for subsequent experiments.
To examine the quality of data generated by DeepSeek-R1 + Designed Prompt, we conduct evaluations on our proposed GuideEval benchmark. As shown in Table [7](https://arxiv.org/html/2508.06583v2#A4.T7), the model exhibits notable improvements across multiple dimensions compared to the original DeepSeek-R1, validating the reliability and effectiveness of the generated data.

Formally, both training and evaluation instances share the tuple structure introduced earlier:

$$ $D=\big(\mathcal{C},S_{\mathrm{state}},R\big).$ (4) $$

The difference lies in how $S_{\mathrm{state}}$ and $R$ are obtained:

- •
In the evaluation set, $S_{\mathrm{state}}$ is either an authentic or state-edited student utterance with a human-verified label, and $R$ is the response of the model under evaluation.
- •
In the training set, $S_{\mathrm{state}}$ is either a verified correct answer or its error/metacognitive rewrite, and $R$ is the guided response generated by DeepSeek-R1 with think traces.

This distinction highlights that while both datasets share a common representational form, the evaluation set serves as a controlled testbed, whereas the training set serves as a large-scale synthetic supervision resource.

### D.2 Automatic Filtering

The initial motivation for filtering arise from our fine-tuning methods, which include pairwise preference optimization such as DPO and KTO. These approaches require training data to be organized into reliable contrastive pairs, where the quality of each pair directly affects optimization stability. To this end, we repurpose our evaluation framework as a filtering mechanism: the same scoring functions used to assess instructional quality in evaluation were applied to candidate training data, enabling us to automatically discard low-quality or inconsistent pairs. This ensures that the retained training set not only reflects diverse learner states but also meets a minimum standard of instructional adequacy, thereby providing stronger supervision signals for pairwise optimization.

A *candidate pair* refers to two state-opposed responses, either accurate vs. erroneous or comprehension vs. confusion. Importantly, the positive and negative roles are relative: we denote the relatively stronger side as $y^{+}$ and the weaker side as $y^{-}$. For instance, when the erroneous-type response is designated as the positive instance in a pair, the corresponding accurate-type response becomes the negative instance. Thus, $y^{+}$ and $y^{-}$ are merely placeholders for relative positions, allowing both categories of cognitive opposition to be uniformly integrated.

Each sample receives rubric scores $S$ from our evaluation framework, which reuses the scoring logic developed for instructional quality assessment. Specifically, $S$ is computed as the average of Perception and Orchestration scores; if the perception dimension is absent, we simply use the orchestration score. For a candidate pair $(y^{+},y^{-})$, we obtain four scores under the accurate and erroneous rubrics: $S_{a}(y^{+}),S_{e}(y^{+}),S_{a}(y^{-}),S_{e}(y^{-})$. We then compute a diagonal-margin metric:

$$ $A_{\Delta}=\bigl(S_{a}(y^{+})+S_{e}(y^{-})\bigr)-\bigl(S_{e}(y^{+})+S_{a}(y^{-})\bigr).$ $$

This formulation reflects two principles: (i) within-standard quality—each sample should score high under its own rubric; and (ii) cross-standard contrast—each sample should score low under the opposite rubric. Larger $A_{\Delta}$ values therefore indicate stronger separation along the perception and orchestration dimensions.

To incorporate the elicitation dimension, we further define

$$ $C=A_{\Delta}\cdot\bigl(1+\beta\cdot\max(E_{\Delta},0)\bigr),\quad\beta=0.2,$ $$

where $E_{\Delta}$ corresponds to the elicitation strategy adaptivity (ESA). In this scheme, perception and orchestration serve as the foundation, while elicitation acts as a bonus factor, rewarding pairs that also exhibit greater questioning divergence. We set a threshold of $C\geq 1$; only such pairs are preserved for training, yielding a dataset that satisfies baseline instructional adequacy while amplifying pedagogical contrast.

### D.3 Finetuning Settings

We fine-tune Qwen3-8B under three paradigms: supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and Kahneman–Tversky Optimization (KTO). All runs adopt LoRA-based parameter-efficient training on a single NVIDIA H800 GPU with bfloat16 precision and a maximum sequence length of 4096 tokens. All three paradigms followed the same evaluation and checkpointing schedule: validation every 100 steps, checkpoint saving every 400 steps (up to 5 retained).

*SFT (response-only vs. with think traces / CoT distillation).*
We train on the full curated dataset ($\sim$50k samples, real educational contexts with DeepSeek-R1 targets under detailed prompts). We implement two variants: one using response-only outputs, and another incorporating intermediate *think traces* (CoT distillation). Hyperparameters: LoRA rank $=8$, $\alpha=32$, 5% validation split, per-device batch size $=2$, gradient accumulation steps $=8$, learning rate $=1\times 10^{-4}$, and warmup ratio $=0.05$.

*DPO (pairwise).*
This is trained on the automatically filtered subset ($\sim$30k samples) to ensure contrastive quality. Using swift rlhf, we set LoRA rank $=8$, $\alpha=32$, bfloat16 precision, and maximum length $=4096$. We train for 2 epochs with per-device batch size $=2$, gradient accumulation $=8$, learning rate $=1\times 10^{-5}$ (smaller for stability), and warmup ratio $=0.05$. Following common practice, the DPO inverse-temperature was fixed at $\beta=0.1$.

*KTO (pointwise).*
This is also trained on the automatically filtered subset ($\sim$30k samples), to study approval-style pointwise preference optimization without pairs. Specifically, we split the positive/negative examples from DPO pairs into *accepted* and *rejected* samples for KTO. We train with the same LoRA setup (rank $=8$, $\alpha=32$, bfloat16, max length $=4096$), with 2 epochs, per-device batch size $=2$, gradient accumulation $=8$, learning rate $=1\times 10^{-4}$, and warmup ratio $=0.05$.

## Appendix E Prompt Templates

In our experiments, we employ two categories of prompt templates. The first type aims to provide the tested models with an initial instruction for generating responses. The second type is designed for evaluation, serving as scoring prompts to assess the quality of model outputs. In this section, we present all prompt templates used in both stages.

### E.1 Generation Prompts

We use two types of system prompts for model response generation:

- •
Original Prompt: The base system instruction which is added to all models before testing.
- •
Rule Prompt: An instructional behavior guideline describing how the model should adapt based on student responses (applied in §[3.4](https://arxiv.org/html/2508.06583v2#S3.SS4) Behavior-Guided Finetuning).

The detailed system prompt texts are shown in the boxes below.

### E.2 Evaluation Prompts

Each behavioral metric is scored using prompt-based assessments.
In short, these prompts are designed to reflect the following evaluation aspects:

- •
P-Affirm: Was the correct student answer explicitly affirmed?
- •
P-Redirect: Was the student error identified and redirected?
- •
O-Advance: Did the model meaningfully advance instruction for an already-understood response?
- •
O-Reconfigure (Erroneous): Did the model revise its explanation after encountering a misconception?
- •
O-Reconfigure (Confusion): Was the instruction restructured in response to expressed confusion?
- •
Elicitation Level: Into which cognitive level, as defined in our taxonomy, did the model’s question fall?

The detailed evaluation prompt texts are provided below.

## Appendix F Example of State-Sensitive Responses after CoT Distillation

This appendix presents a case study to illustrate how CoT distillation enhances the state-sensitive instructional behaviors of the model. Compared with GPT-4.1 responses shown in Figure [1](https://arxiv.org/html/2508.06583v2#S1.F1), the distilled model produces more differentiated feedback when interacting with students in distinct cognitive states. As shown in Figure [7](https://arxiv.org/html/2508.06583v2#A6.F7), the model adapts its explanations and questioning strategies more explicitly depending on whether the student’s response is accurate vs. erroneous or demonstrates comprehension vs. confusion. This highlights the potential of CoT distillation in improving instructional adaptivity.

Figure: (d) Responses to accurate and erroneous states
Refer to caption: x10.png

## Appendix G Use of LLMs

In this study, large language models (LLMs) were employed as auxiliary tools to assist in the rewriting and refinement of selected text passages during the manuscript preparation process. All outputs generated by the models were carefully reviewed, edited, and filtered by the authors to ensure both accuracy and adherence to academic writing standards. It is important to note that the conceptual design, methodological development, data analysis, and interpretation of results were carried out independently by the authors without reliance on automated systems. The authors retain full responsibility for the originality, validity, and integrity of the work presented in this paper.

### D.2 Automatic Filtering

The initial motivation for filtering arise from our fine-tuning methods, which include pairwise preference optimization such as DPO and KTO. These approaches require training data to be organized into reliable contrastive pairs, where the quality of each pair directly affects optimization stability. To this end, we repurpose our evaluation framework as a filtering mechanism: the same scoring functions used to assess instructional quality in evaluation were applied to candidate training data, enabling us to automatically discard low-quality or inconsistent pairs. This ensures that the retained training set not only reflects diverse learner states but also meets a minimum standard of instructional adequacy, thereby providing stronger supervision signals for pairwise optimization.

A *candidate pair* refers to two state-opposed responses, either accurate vs. erroneous or comprehension vs. confusion. Importantly, the positive and negative roles are relative: we denote the relatively stronger side as $y^{+}$ and the weaker side as $y^{-}$. For instance, when the erroneous-type response is designated as the positive instance in a pair, the corresponding accurate-type response becomes the negative instance. Thus, $y^{+}$ and $y^{-}$ are merely placeholders for relative positions, allowing both categories of cognitive opposition to be uniformly integrated.

Each sample receives rubric scores $S$ from our evaluation framework, which reuses the scoring logic developed for instructional quality assessment. Specifically, $S$ is computed as the average of Perception and Orchestration scores; if the perception dimension is absent, we simply use the orchestration score. For a candidate pair $(y^{+},y^{-})$, we obtain four scores under the accurate and erroneous rubrics: $S_{a}(y^{+}),S_{e}(y^{+}),S_{a}(y^{-}),S_{e}(y^{-})$. We then compute a diagonal-margin metric:

$$ $A_{\Delta}=\bigl(S_{a}(y^{+})+S_{e}(y^{-})\bigr)-\bigl(S_{e}(y^{+})+S_{a}(y^{-})\bigr).$ $$

This formulation reflects two principles: (i) within-standard quality—each sample should score high under its own rubric; and (ii) cross-standard contrast—each sample should score low under the opposite rubric. Larger $A_{\Delta}$ values therefore indicate stronger separation along the perception and orchestration dimensions.

To incorporate the elicitation dimension, we further define

$$ $C=A_{\Delta}\cdot\bigl(1+\beta\cdot\max(E_{\Delta},0)\bigr),\quad\beta=0.2,$ $$

where $E_{\Delta}$ corresponds to the elicitation strategy adaptivity (ESA). In this scheme, perception and orchestration serve as the foundation, while elicitation acts as a bonus factor, rewarding pairs that also exhibit greater questioning divergence. We set a threshold of $C\geq 1$; only such pairs are preserved for training, yielding a dataset that satisfies baseline instructional adequacy while amplifying pedagogical contrast.

### D.3 Finetuning Settings

We fine-tune Qwen3-8B under three paradigms: supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and Kahneman–Tversky Optimization (KTO). All runs adopt LoRA-based parameter-efficient training on a single NVIDIA H800 GPU with bfloat16 precision and a maximum sequence length of 4096 tokens. All three paradigms followed the same evaluation and checkpointing schedule: validation every 100 steps, checkpoint saving every 400 steps (up to 5 retained).

*SFT (response-only vs. with think traces / CoT distillation).*
We train on the full curated dataset ($\sim$50k samples, real educational contexts with DeepSeek-R1 targets under detailed prompts). We implement two variants: one using response-only outputs, and another incorporating intermediate *think traces* (CoT distillation). Hyperparameters: LoRA rank $=8$, $\alpha=32$, 5% validation split, per-device batch size $=2$, gradient accumulation steps $=8$, learning rate $=1\times 10^{-4}$, and warmup ratio $=0.05$.

*DPO (pairwise).*
This is trained on the automatically filtered subset ($\sim$30k samples) to ensure contrastive quality. Using swift rlhf, we set LoRA rank $=8$, $\alpha=32$, bfloat16 precision, and maximum length $=4096$. We train for 2 epochs with per-device batch size $=2$, gradient accumulation $=8$, learning rate $=1\times 10^{-5}$ (smaller for stability), and warmup ratio $=0.05$. Following common practice, the DPO inverse-temperature was fixed at $\beta=0.1$.

*KTO (pointwise).*
This is also trained on the automatically filtered subset ($\sim$30k samples), to study approval-style pointwise preference optimization without pairs. Specifically, we split the positive/negative examples from DPO pairs into *accepted* and *rejected* samples for KTO. We train with the same LoRA setup (rank $=8$, $\alpha=32$, bfloat16, max length $=4096$), with 2 epochs, per-device batch size $=2$, gradient accumulation $=8$, learning rate $=1\times 10^{-4}$, and warmup ratio $=0.05$.

## Appendix E Prompt Templates

In our experiments, we employ two categories of prompt templates. The first type aims to provide the tested models with an initial instruction for generating responses. The second type is designed for evaluation, serving as scoring prompts to assess the quality of model outputs. In this section, we present all prompt templates used in both stages.

### E.1 Generation Prompts

We use two types of system prompts for model response generation:

- •
Original Prompt: The base system instruction which is added to all models before testing.
- •
Rule Prompt: An instructional behavior guideline describing how the model should adapt based on student responses (applied in §[3.4](https://arxiv.org/html/2508.06583v2#S3.SS4) Behavior-Guided Finetuning).

The detailed system prompt texts are shown in the boxes below.

### E.2 Evaluation Prompts

Each behavioral metric is scored using prompt-based assessments.
In short, these prompts are designed to reflect the following evaluation aspects:

- •
P-Affirm: Was the correct student answer explicitly affirmed?
- •
P-Redirect: Was the student error identified and redirected?
- •
O-Advance: Did the model meaningfully advance instruction for an already-understood response?
- •
O-Reconfigure (Erroneous): Did the model revise its explanation after encountering a misconception?
- •
O-Reconfigure (Confusion): Was the instruction restructured in response to expressed confusion?
- •
Elicitation Level: Into which cognitive level, as defined in our taxonomy, did the model’s question fall?

The detailed evaluation prompt texts are provided below.

## Appendix F Example of State-Sensitive Responses after CoT Distillation

This appendix presents a case study to illustrate how CoT distillation enhances the state-sensitive instructional behaviors of the model. Compared with GPT-4.1 responses shown in Figure [1](https://arxiv.org/html/2508.06583v2#S1.F1), the distilled model produces more differentiated feedback when interacting with students in distinct cognitive states. As shown in Figure [7](https://arxiv.org/html/2508.06583v2#A6.F7), the model adapts its explanations and questioning strategies more explicitly depending on whether the student’s response is accurate vs. erroneous or demonstrates comprehension vs. confusion. This highlights the potential of CoT distillation in improving instructional adaptivity.

Figure: (d) Responses to accurate and erroneous states
Refer to caption: x10.png

## Appendix G Use of LLMs

In this study, large language models (LLMs) were employed as auxiliary tools to assist in the rewriting and refinement of selected text passages during the manuscript preparation process. All outputs generated by the models were carefully reviewed, edited, and filtered by the authors to ensure both accuracy and adherence to academic writing standards. It is important to note that the conceptual design, methodological development, data analysis, and interpretation of results were carried out independently by the authors without reliance on automated systems. The authors retain full responsibility for the originality, validity, and integrity of the work presented in this paper.

### E.2 Evaluation Prompts

Each behavioral metric is scored using prompt-based assessments.
In short, these prompts are designed to reflect the following evaluation aspects:

- •
P-Affirm: Was the correct student answer explicitly affirmed?
- •
P-Redirect: Was the student error identified and redirected?
- •
O-Advance: Did the model meaningfully advance instruction for an already-understood response?
- •
O-Reconfigure (Erroneous): Did the model revise its explanation after encountering a misconception?
- •
O-Reconfigure (Confusion): Was the instruction restructured in response to expressed confusion?
- •
Elicitation Level: Into which cognitive level, as defined in our taxonomy, did the model’s question fall?

The detailed evaluation prompt texts are provided below.

## Appendix F Example of State-Sensitive Responses after CoT Distillation

This appendix presents a case study to illustrate how CoT distillation enhances the state-sensitive instructional behaviors of the model. Compared with GPT-4.1 responses shown in Figure [1](https://arxiv.org/html/2508.06583v2#S1.F1), the distilled model produces more differentiated feedback when interacting with students in distinct cognitive states. As shown in Figure [7](https://arxiv.org/html/2508.06583v2#A6.F7), the model adapts its explanations and questioning strategies more explicitly depending on whether the student’s response is accurate vs. erroneous or demonstrates comprehension vs. confusion. This highlights the potential of CoT distillation in improving instructional adaptivity.

Figure: (d) Responses to accurate and erroneous states
Refer to caption: x10.png

## Appendix G Use of LLMs

In this study, large language models (LLMs) were employed as auxiliary tools to assist in the rewriting and refinement of selected text passages during the manuscript preparation process. All outputs generated by the models were carefully reviewed, edited, and filtered by the authors to ensure both accuracy and adherence to academic writing standards. It is important to note that the conceptual design, methodological development, data analysis, and interpretation of results were carried out independently by the authors without reliance on automated systems. The authors retain full responsibility for the originality, validity, and integrity of the work presented in this paper.

## Appendix F Example of State-Sensitive Responses after CoT Distillation

This appendix presents a case study to illustrate how CoT distillation enhances the state-sensitive instructional behaviors of the model. Compared with GPT-4.1 responses shown in Figure [1](https://arxiv.org/html/2508.06583v2#S1.F1), the distilled model produces more differentiated feedback when interacting with students in distinct cognitive states. As shown in Figure [7](https://arxiv.org/html/2508.06583v2#A6.F7), the model adapts its explanations and questioning strategies more explicitly depending on whether the student’s response is accurate vs. erroneous or demonstrates comprehension vs. confusion. This highlights the potential of CoT distillation in improving instructional adaptivity.

Figure: (d) Responses to accurate and erroneous states
Refer to caption: x10.png

## Appendix G Use of LLMs

In this study, large language models (LLMs) were employed as auxiliary tools to assist in the rewriting and refinement of selected text passages during the manuscript preparation process. All outputs generated by the models were carefully reviewed, edited, and filtered by the authors to ensure both accuracy and adherence to academic writing standards. It is important to note that the conceptual design, methodological development, data analysis, and interpretation of results were carried out independently by the authors without reliance on automated systems. The authors retain full responsibility for the originality, validity, and integrity of the work presented in this paper.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. What are the details of sample size per metric or level, sampling protocol, number of human raters, and inter-rater reliability statistics?

2. Generation prompts forbid giving final answers, while the O-Advance rubric rewards them. How are these conflicting criteria reconciled to ensure consistent evaluation?

3. How many items per metric were used for the LLM–human consistency analysis, and how were they sampled from the full dataset?

4. Could you replicate headline results using at least one alternative judge and report inter-judge agreement and sensitivity to confirm robustness?

5. Since training and evaluation data are drawn from the same source pool, do problem contexts (e.g., identical or near-duplicate problem IDs, stems, passages, or scaffolds) repeat across splits? If so, what is the overlap rate, and how does it affect evaluation outcomes?
soundness: 2
strengths: 1. Clear behavioral decomposition with actionable metrics. The three-phase split translates “be a better tutor” into concrete, checkable behaviors, offering conceptual clarity and operational guidance that enable reproducible, phase-wise diagnosis across different models.

2. Useful failure taxonomy grounded in qualitative evidence. The paper goes beyond reporting average behaviors and highlights failure modes supported by dialogue snippets, providing interpretability and practical insight into model behavior.
summary: This paper investigates whether LLM tutors can deliver adaptive instructional guidance in Socratic dialogues rather than relying on generic questioning. The authors conceptualize guidance as a three-phase behavior: Perception (inferring the learner’s state), Orchestration (selecting an appropriate next-step strategy), and Elicitation (formulating prompts suited to the learner’s state). Building on this framework, they introduce GuideEval, a benchmark composed of real multi-turn tutoring dialogues with contrastive student states, and define phase-aligned evaluation metrics to assess model performance across these dimensions.
weaknesses: 1. Human–LLM agreement is reported without sample size or reliability statistics. In Table 3, the claim that “LLMs can serve as reliable and scalable evaluators of instructional behaviors” rests on high agreement ratios and minimal score deviations, but the paper omits sample size per metric or level, sampling protocol, number of human raters, and inter-rater reliability. Without these, chance agreement and selection bias cannot be ruled out, especially with coarse labels (binary or 3-point) that inflate raw agreement.

2. Prompt–rubric inconsistency. Generation prompts forbid giving final answers (“Do not directly provide the final answer or full solution process” in Original/Rule templates), yet the O-Advance evaluation rubric awards full credit when “the model provides the final answer.” This contradiction allows models to achieve high orchestration scores while violating generation rules. Please align the prompt and rubric definitions.

3. All models are decoded at temperature = 0.1. Please justify this setting and provide an ablation across temperatures to ensure conclusions are not artifacts of low-variance decoding.

4. Human annotators reportedly revised model outputs to create “state-edited counterparts” (e.g., answer-correctness flips, comprehension/confusion flips). Please specify the exact editing operations and provide per-operation statistics, including counts, edit distance, token-level change distribution, and human vs. synthetic proportions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have you tested or do you plan to test this framework on other domains (e.g., humanities, programming) or age groups? What challenges do you anticipate?

2. Can you provide comparison with human tutor performance on the same benchmark? This would contextualize LLM performance.

3. The "Designed Prompt" used to generate the training data for the finetuning experiments is very explicit and rule-based. How can we be sure that the model learned a generalizable instructional guidance capability rather than just learning to mimic the explicit rules baked into the generation prompt? Have you tested or are you planning to test the finetuned model on out-of-domain tasks to see if the "skill" transfers?

4. The filtering mechanism for training data (equations on p.18) is complex. How sensitive are results to these hyperparameters (β, threshold)?
soundness: 3
strengths: 1. The paper focuses on critical gap in LLM tutoring evaluation by focusing on adaptive guidance rather than static content quality.

2. The three-phase model is well-motivated by educational psychology literature and operationalized into measurable metrics.

3. The exp covers 14 diverse models, revealing consistent failure patterns across architectures.

4.The paper contains a detailed failure case analysis, providing an intuitive understanding beyond quantitative metrics.

5. I really like the comparative analysis of different training strategies part. The finding that outcome-only SFT degrades performance while process supervision (CoT Distillation) and pairwise preference optimization (DPO) provide substantial gains is a critical, actionable insight for the community.
summary: This paper introduces GuideEval, a benchmark for evaluating instructional guidance capabilities of large language models (LLMs) when serving as Socratic tutors. The authors argue that existing evaluations focus primarily on question generation while overlooking adaptive guidance—the ability to dynamically adjust teaching strategies based on learners' cognitive states. The paper proposes a three-phase behavioral framework: (1) Perception - inferring learner states (accurate/erroneous/comprehension/confusion); (2) Orchestration - adapting instructional strategies through scaffolding; and (3) Elicitation - stimulating deeper thinking through strategic questioning.

The authors construct a dataset of 5,177 test samples from authentic tutoring dialogues with contrastive student state pairs, enabling controlled evaluation of model adaptivity. They evaluate 14 LLMs across 6 metrics derived from their framework, finding that models struggle with error detection, adapting to implicit cognitive cues, and maintaining consistent guidance strategies. The paper includes detailed failure pattern analysis and demonstrates that behavior-guided finetuning with Chain-of-Thought distillation substantially improves guidance performance.
weaknesses: 1. it comes with limited scope: dataset topic - middle school science problems in Chinese. It would be more curated if you expand it to other difficulty levels and languages.

2. The cognitive modeling with 4 states (Accurate, Erroneous, Comprehension, Confusion) may be too simplified to capture nuanced learning states. As authors acknowledge, it doesn't capture individual learner profiles, misconception history, or engagement patterns

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - How well can LLMs recognize the learner states (independent of how affirmative their responses are)?
- What is the distribution of failure types for each of the LLM models? Do they fail in similar ways?
soundness: 3
strengths: - The paper conducted an extensive analysis of various LLMs and provides several insights on their capability to recognize learner states, to guide / scaffold, and to elicit further follow-ups.
- The collected dataset GuideEval can help advance the field further.
- LLM-based scoring were validated with human annotations
- The failure analysis provides useful insights
summary: This paper conducted an extensive evaluation of the capabilities of LLMs in guiding learners and dynamically adapting its responses to the learners' states. To achieve this, they also collected a benchmark dataset of real multi-turn dialogues of learners from a Socratic tutoring platform.
weaknesses: - The authors evaluated the consistency between LLM based scoring and the Human annotators using the proportion of the same labels. I am not sure if this is the right way to go about it since simply showing the proportion of agreement can be misleading, especially if there is an imbalance in the label distribution. I believe there are more appropriate inter-rater agreement metrics that account for these.
- The failure analysis categorizes the types of failures but the authors did not seem to provide the frequencies of occurrence for each type of failure category. 
- The authors only measured how the LLMs responded. For example, P-affirm and P-redirect scores are only based on how the LLM responded. But this does not tell us whether or not the LLM can recognize the learner states. It might be the case that the LLM does indeed recognize but just does not know how to respond properly.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - How many examples were used for creating Tab. 3?

- Will you release the dataset publicly? Based on the conclusion I would assume so but I did not find it written anywhere.

## References

Wang et al., Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes, NAACL 2024

Daheim et al., Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors, EMNLP 2024

Dinucu-Jianu et al., From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning., arXiv 2025

Scarlatos et al., Training llmbased tutors to improve student learning outcomes in dialogues, AIED 2025
soundness: 2
strengths: - The research area is severely lacking resources that use real students (mainly due to ethical and regulatory problems / hurdles), so releasing a dataset of this scale with real students is quite impactful.

- The results show interesting observations for the future, for example, they point out the sycophancy of current LLMs which can be bad for learning.
summary: The paper introduces GuideEval, a benchmark to evaluate the tutoring capabilities of LLMs across three stages based on interactions with real students.
The authors use the benchmark to evaluate current commercial and open-source LLMs and find, for example, that they struggle to adapt to learner states, especially when critiquing wrong solutions.
The authors also use their data to finetune an LLM and show that finetuning improves these shortcomings.
weaknesses: - In general, the discussion of related work is a bit short and many findings of the paper have been discovered in prior works (though in different form). For example, Wang et al. 2023 already shows that LLMs tend to not actively engage with student error patterns and Daheim et al. 2024 show that identifying student mistakes is a challenging task even for sota LLMs. Another example is Dinucu-Jianu et al. 2025 who show that there exists a trade-off between pedagogy and giving hints (here the authors also point out that only focusing on not telling solutions leads to poor teaching). Scarlatos et al. 2025 also use DPO but this is not discussed, too. 
I don't think that this takes away from the papers findings in general but proper discussion would improve it.
The authors also do not discuss knowledge tracing but the goal of this field is precisely what the paper sets out: to adapt to current learner states. I think a discussion of such concepts could also be helpful.

- I am wondering about more details about the dataset, many seem missing, as the discussion seems limited to Sec. 2.3.
For example, no annotator agreement is reported and beyond the number of turns and dialogues there are no further statistics. I think even reporting simple statistics would be helpful. The domain is also not specified beyond saying that it consists of middle school science questions. It is also mentioned that humans verified and even edited utterances (to create negatives) but the exact process is not detailed.

- The agreement between LLM and humans is fairly high but only checks binary preference over a combination of metrics and not agreement for a specific metric which limits expressiveness for how reliable the individual metrics are.


Decision: Reject