## 4 Results and Discussion
### Three-Phase Evaluation
#### 4.2.2 Phase 2: Combined Dataset Evaluation  
In this phase, we aim to evaluate the generalization capability of models trained on the EduTalk dataset. To achieve this, we created mixed test sets by combining EduTalk and TalkMoves datasets in different ratios (3:7 and 1:1) and assessed the models' performance on these test sets. This approach allows us to determine the extent to which the EduTalk data generated by the EduDCM framework can match the quality of the manually annotated TalkMoves dataset.  
\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Dataset** & **Criteria** & **Mean Score** & **Standard Deviation** & **Cohen’s Kappa** \\ \hline \multirow{3}{*}{EduTalk} & Annotation Accuracy & 4.4 & 0.3 & 0.85 \\  & Category Coverage & 4.3 & 0.4 & 0.83 \\  & Consistency & 4.4 & 0.3 & 0.84 \\ \hline \multirow{3}{*}{TalkMoves} & Annotation Accuracy & 4.5 & 0.3 & 0.86 \\  & Category Coverage & 4.4 & 0.4 & 0.84 \\ \cline{1-1}  & Consistency & 4.5 & 0.3 & 0.86 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Expert evaluation results for EduTalk and TalkMoves datasets.  
The results, as shown in Table 4, reveal several important trends. In the 3:7 dataset ratio, where TalkMoves dominates the test set, all models exhibit lower performance compared to the 1:1 ratio. This decline can be attributed to the smaller size of the TalkMoves dataset, which limits the diversity of the data and reduces the models' ability to generalize effectively. Among the models, CasRel and OneRel demonstrate the highest F1 scores of 0.779 and 0.747, respectively, reflecting their superior ability to adapt to mixed datasets, even with a significant proportion of manually annotated TalkMoves data.  
In the 1:1 dataset ratio, where EduTalk and TalkMoves are equally represented, all models show improved performance. This improvement underscores the role of the high-quality EduTalk data in enriching the diversity of the smaller TalkMoves dataset, addressing the challenges arising from its limited size. CasRel achieves the highest F1 score of 0.838, followed closely by OneRel with an F1 score of 0.818. Both models demonstrate balanced precision and recall, indicating their robust generalization capability in handling diverse test sets.  
Lower-performing models, such as BERT and mREBEL, show similar trends, with better performance in the 1:1 ratio compared to the 3:7 ratio. However, their F1 scores of 0.739 and 0.722 in the 1:1 ratio suggest limitations in handling the variability of manually annotated TalkMoves data. These models rely more heavily on dataset size and quality, which may restrict their adaptability in low-resource scenarios. The performance of GPT-2, with an F1 score of 0.760 in the 1:1 ratio, indicates its competitive ability to generalize when presented with a balanced dataset. However, its slightly lower precision compared to CasRel and OneRel suggests that it struggles with maintaining high accuracy in more diverse data conditions.  
Overall, these results highlight the capability of the EduTalk dataset to enhance model performance when combined with manually annotated TalkMoves data. High-performing models like CasRel and OneRel exhibit robust adaptability to mixed datasets, validating the EduDCM framework's potential to generate high-quality data that effectively support dialogue classification tasks in low-resource educational contexts.