# WEREC: An Unsupervised Method for Educational Concept Extraction Based on Word Embedding Refinement
## 4 Experiments
### Dataset Preparation  
To the best of our knowledge, no gold-standard datasets are publicly available with which to measure domain concept extraction in education. Thus, a pair of education-related datasets was created to proceed with our experiments based on public data and the literature. The first dataset (EDU-DT) was derived from MOOCCube [53], a large-scale data warehouse covering various domains that has been used for different NLP tasks [54]. When assembling this dataset, 2956 educational concepts deemed as positives were screened out from MOOCCube; 1000 concepts from other domains (e.g., world history, computer science, and management science) were randomly chosen as negatives. The EDU-DT dataset was further separated into two parts: (1) a training set with 2000 seed concepts stochastically selected from the positives and (2) a test set with 1956 candidate concepts comprising the remaining positives and all negatives. This dataset was adopted to explore the feasibility of the proposed WERECE method. In the following experiments, we built a K-means model on seed concepts to verify the proposed approach's effectiveness according to the quality of clustering results. Then, the prediction performance of the proposed discriminant function was compared to that of the K-means model. In essence, based on the strength of the K-means model, we explored the discriminant function's capacity for concept extraction.  
The second dataset, EDUTECH-DT, concerned educational technology (i.e., a secondary matter in education) and comprised a collection of seeds along with many candidates. Seeds were drawn from a thesaurus dataset released in our previous research [55], wherein domain experts manually collected educational concepts to improve the standardization of terminology in educational technology. As for candidate concepts, we organized a corpus of academic literature from 2015 to 2020 on educational technology. Each source's title, abstract, and keywords were obtained from the China National Knowledge Internet, one of the most popular academic databases in China. State-of-the-art NLP techniques were used to develop a text preprocessing pipeline comprising word segmentation, stop word removal, part-of-speech (POS) tagging, and duplicate removal. After preprocessing, concept candidates were identified from a list of POS-based linguistic patterns, which could convey the formation and collocation of domain concepts [11; 56; 57]. Finally, a set of nouns, noun phrases, verbs, and verbal phrases were retained as candidates. Each candidate was further annotated as either positive or negative by two students majoring in educational technology. The Cohen's Kappa score for this annotation step was 0.84, reflecting moderate rater agreement. We referred to the EDUTECH-DT dataset when conducting an initial experiment to explore optimal parameter settings for WERECE. As described in the above section, the proposed method encompassed three key parameters: the dimensions of refined embedding vectors (\(M\)), the number of seed concepts (\(N\)), and the number of clusters (\(K\)). For convenience in implementation, the grid search method was leveraged to determine WERECE's optimal parameters. With optimal parameter settings, the proposed method's effectiveness against baselines was successively verified by means of the bootstrap sampling method. Table 2 lists the basic statistics for our two datasets, including the number of concepts and their length characteristics.