## 4 Results and Discussion
### Three-Phase Evaluation
#### 4.2.3 Phase 3: Data Generation Method Comparison  
In this phase, EduDCM is compared against two prominent data generation approaches. The first is Template-Based Generation, which creates synthetic dialogues using predefined templates and expert-defined rules. While structured, this method is labor-intensive and commonly used in early natural language generation tasks. The second approach is Few-Shot GPT Data Generation, which leverages GPT-3.5 in a few-shot setup to generate datasets based on limited annotated examples, relying on the model's capability to extrapolate patterns from minimal inputs. The datasets generated by these methods are evaluated through identical models and examined by domain experts across three dimensions.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Dataset Ratio 3:7**} & \multicolumn{3}{c}{**Dataset Ratio 1:1**} \\ \cline{2-7}  & **F1** & **Recall (R)** & **Precision (P)** & **F1** & **Recall (R)** & **Precision (P)** \\ \hline BERT & 0.646 & 0.686 & 0.611 & 0.739 & 0.775 & 0.707 \\ mREBEL & 0.625 & 0.652 & 0.601 & 0.722 & 0.750 & 0.695 \\ GPT-2 & 0.670 & 0.693 & 0.648 & 0.760 & 0.767 & 0.753 \\ OneRel & 0.747 & 0.766 & 0.729 & 0.818 & 0.836 & 0.801 \\ CasRel & 0.779 & 0.793 & 0.765 & 0.838 & 0.848 & 0.829 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of models trained on EduTalk and tested on mixed EduTalk and TalkMoves test sets.  
The results presented in Table 5 reveal significant differences in dataset quality across the three methods. EduDCM consistently outperforms the other methods, achieving the highest scores in all three dimensions. Specifically, its score of 4.7 in accuracy of annotations highlights its ability to correctly label utterances, thanks to the framework's integration of context-aware label generation and heuristic alignment techniques. Similarly, EduDCM achieves a coverage score of 4.6, demonstrating its capacity to ensure balanced representation across all pedagogical categories. Its consistency score of 4.8 reflects its effectiveness in maintaining a coherent and uniform annotation style, reducing variability that could negatively impact downstream tasks.  
In contrast, the Template-Based Generation method performs the worst across all dimensions. Its accuracy score of 3.2 indicates limited flexibility in capturing the nuanced and context-dependent nature of educational dialogues. The coverage score of 2.8 reveals significant biases in category representation, as predefined templates tend to favor certain dialogue structures while underrepresenting others. Additionally, the consistency score of 3.0, while reflecting a uniform style within the constraints of the templates, highlights its inability to adapt to real-world variations in dialogue.  
The Few-Shot GPT method demonstrates substantial improvements over Template-Based Generation. Its accuracy score of 4.1 indicates that GPT-3.5 can leverage contextual understanding to generate meaningful and accurate annotations. However, its coverage score of 4.0 suggests that certain pedagogical categories, especially those less represented in the prompt examples, may not be adequately annotated. The consistency score of 4.2 shows that Few-Shot GPT maintains a relatively uniform annotation style, though occasional variability due to inherent model randomness is observed.  
In summary, these results highlight the superiority of EduDCM in generating high-quality datasets for educational dialogue classification. While Few-Shot GPT offers a viable alternative with reasonable performance, its reliance on limited prompts restricts its effectiveness in achieving comprehensive category coverage. Template-Based Generation, though structured, lacks the flexibility and adaptability required for the nuanced nature of educational dialogues.