# WEREC: An Unsupervised Method for Educational Concept Extraction Based on Word Embedding Refinement
## 4 Experiments
### Experimental Results
#### 4.3.3 Comparisons of the Proposed Method with Baselines  
To compare the performance of WRECE and baselines, 30 bootstrapped samples were generated by resampling the test set from the EDUTECH-DT dataset. The methods' overall performance on the bootstrapped samples is summarized in Table 4. WRECE with optimal parameter settings (\(N=1000,K=18,M=65\)) outperformed the baselines in terms of average precision, recall, and F1 scores. TextRank returned the lowest average F1 score, and its recall was significantly higher than its precision (\(p<0.001\)) at the 99.9% confidence level. Conversely, one-class SVM performed best among the baselines. WRECE's precision and recall demonstrated comparative improvements (from 0.091 to 0.522 and from 0.078 to 0.16, respectively). The F1 score comprehensively represented precision and recall, resulting in improvements ranging from 0.096 to 0.408. The baselines showed unsatisfactory precision, with the K-means method being particularly unstable. This outcome further confirms that capitalizing on WRECE's discriminant function fortified the method's resilience against outliers. The average recall of the isolation forest was 1.000, implying that this technique is limited in its ability to precisely identify domain concepts. Another experimental study considered this model an ideal choice, given its average recall of 100% [64]. This result might be based on an imbalanced distribution of positive and negative samples or the presence of close similarities between negative and positive samples within a dataset.  
A Mann-Whitney U test was performed again to determine if WRECE's improvements over the encouraging one-class SVM method were statistically significant. Both methods involved six lists, each with a set of 30 precision, recall, or F1 scores. Figure 7 portrays WRECE's significant improvement over one-class SVM at the 99.9% confidence level (\(p<0.001\)). This statistical test reinforced that WRECE significantly outperformed one-class SVM as measured by precision, recall, and F1 scores. The comparison in Table 4 and Figure 7 offers compelling evidence that WRECE can substantially enhance the effectiveness of educational concept extraction.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Evaluation Metrics**} \\ \cline{2-4}  & **Precision** & **Recall** & **F1 Score** \\ \hline TextRank & 0.337 \(\pm\) 0.014 & 0.705 \(\pm\) 0.028 & 0.456 \(\pm\) 0.018 \\ \hline TF–IDF & 0.378 \(\pm\) 0.016 & 0.792 \(\pm\) 0.033 & 0.512 \(\pm\) 0.022 \\ \hline Isolation Forest & 0.544 \(\pm\) 0.066 & 1.000 \(\pm\) 0.000 & 0.702 \(\pm\) 0.056 \\ \hline K-means & 0.551 \(\pm\) 0.295 & 0.710 \(\pm\) 0.404 & 0.619 \(\pm\) 0.343 \\ \hline One-Class SVM & 0.768 \(\pm\) 0.015 & 0.770 \(\pm\) 0.037 & 0.768 \(\pm\) 0.017 \\ \hline The proposed Method & 0.859 \(\pm\) 0.022 & 0.870 \(\pm\) 0.037 & 0.864 \(\pm\) 0.023 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of WRECE and baselines on the EDUTECH-DT dataset (Mean \(\pm\) SD).