## 4 Results and Discussion
### Three-Phase Evaluation
#### 4.2.1 Phase 1: Individual Dataset Performance Evaluation  
The performance of various models was evaluated on the TalkMoves and EduTalk datasets to establish baseline metrics and assess the quality of annotations. The models used for evaluation include BERT, mREBEL, GPT-2, OneRel, and CasRel. The evaluation metrics include F1 score, precision (P), and recall (R), providing a comprehensive view of each model's effectiveness in handling the different datasets. The results in Table 2 demonstrate that models perform better on the EduTalk dataset compared to TalkMoves. This is largely attributed to the controlled and consistent nature of EduTalk's annotations, which align closely with the structured output generated by the EduDCM framework. The CasRel and OneRel models exhibit superior performance on both datasets, with F1 scores of 0.914 and 0.917 on EduTalk, and 0.778 and 0.742 on TalkMoves, respectively. These results highlight the effectiveness of models specifically designed for joint entity and relation extraction when applied to datasets with high annotation quality. For the TalkMoves dataset, which contains manually annotated data, performance metrics are generally lower. This can be attributed to the inherent complexity and variability of the dataset, as well as the potential for ambiguous or overlapping pedagogical categories. The CasRel model outperforms other models with an F1 score of 0.778, reflecting its robustness in handling diverse and nuanced annotations. OneRel also performs well, achieving an F1 score of 0.742, indicating its adaptability to tasks requiring precise annotation quality. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Category** & \multicolumn{2}{c}{**Sampled Instances**} & \multicolumn{2}{c}{**Positive (\%)**} & \multicolumn{1}{c}{**Negative (\%)**} \\ \hline Learning Community: Keeping everyone together & 175 & 94 & 6 \\ Learning Community: Getting students to relate to another’s ideas & 205 & 91 & 9 \\ Learning Community: Restating & 195 & 89 & 11 \\ Learning Community: Relating to another student & 148 & 81 & 19 \\ Learning Community: Asking for more information & 118 & 79 & 21 \\ Content Knowledge: Pressing for accuracy & 185 & 87 & 13 \\ Content Knowledge: Making a claim & 129 & 77 & 23 \\ Rigorous Thinking: Revoicing & 162 & 85 & 15 \\ Rigorous Thinking: Pressing for reasoning & 142 & 82 & 18 \\ Rigorous Thinking: Providing evidence or reasoning & 101 & 74 & 26 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Expert evaluation of the EduTalk dataset, showing positive and negative annotation rates across 10 categories. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**EduTalk**} & \multicolumn{3}{c}{**TalkMoves**} \\ \cline{2-7}  & **F1** & **Recall (R)** & **Precision (P)** & **F1** & **Recall (R)** & **Precision (P)** \\ \hline BERT & 0.863 & 0.871 & 0.856 & 0.624 & 0.710 & 0.565 \\ mREBEL & 0.879 & 0.868 & 0.890 & 0.596 & 0.580 & 0.613 \\ GPT-2 & 0.892 & 0.844 & 0.947 & 0.635 & 0.642 & 0.628 \\ OneRel & 0.917 & 0.924 & 0.910 & 0.742 & 0.726 & 0.759 \\ CasRel & 0.914 & 0.907 & 0.921 & 0.778 & 0.766 & 0.790 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance comparison of models on EduTalk and TalkMoves datasets. In contrast, the EduTalk dataset, generated via EduDCM, demonstrates competitive results, with models achieving performance metrics comparable to those on manually annotated datasets like TalkMoves. The structured nature of the annotations, combined with the semantic alignment mechanisms employed by the EduDCM framework, likely contributes to the high performance. CasRel and OneRel maintain their leading positions, achieving precision and recall scores above 0.91, which underscores their ability to capitalize on the dataset's quality. Models like BERT and mREBEL exhibit comparatively lower performance on both datasets. This discrepancy is particularly pronounced on TalkMoves, where the variability and complexity of the data pose significant challenges.