# WEREC: An Unsupervised Method for Educational Concept Extraction Based on Word Embedding Refinement
## 3 Methods
### Manifold Learning-Based Word Re-Embedding for Domain Concepts  
Most distributed word representation models ignore how words' geometric structure affects semantic calculation [41; 42; 43]. Word re-embedding seeks to eliminate this oversight by refining the word representation based on intrinsic geometric information in the original embedding space. Scholars have deployed manifold learning algorithms for this purpose by integrating geometric information between words and their neighbors. To get the most out of semantic information for domain-specific concept extraction, both local and global geometric information were exploited to refine seed concept vectors originating from pre-trained word embeddings through a manifold learning algorithm.  
IsoMap, short for \"isometric mapping\", is a popular approach to manifold learning [44]. This method extends multidimensional scaling by introducing geodesic distance to measure the similarity between all data points. Compared to the locally linear embedding algorithm that has been used for word embedding refinement, IsoMap captures global information; it uses local information to construct a global neighborhood graph that describes the original embedding space's overall structure. This algorithm can hence be characterized by a neighborhood graph and geodesic distance. Given \(N\) seed concept vectors \(V=\{v_{1},\ v_{2},\ v_{3},\ \cdots,\ v_{N}\}\), the IsoMap algorithm was implemented in four steps. To start, K nearest neighbor methods (e.g., Ball-tree, KD-tree) were applied to search for neighbors for each concept vector. Next, an adjacency matrix \(A_{N\times N}\) was created to describe undirected neighborhood relationships among vectors. Each element \(a_{ij}\) in \(A_{N\times N}\) denotes the connection weight between concept vectors \(v_{i}\) and \(v_{j}\). If a neighborhood relationship exists between \(v_{i}\) and \(v_{j}\), then \(a_{ij}\) is initially determined by the Euclidean metric; otherwise, \(a_{ij}\) is set to infinity. Thereafter, each matrix element \(a_{ij}\) was updated to approximate the geodesic distance from \(v_{i}\) to \(v_{j}\). This approximation was based on the Floyd algorithm, a shortest-path algorithm grounded in dynamic programing as shown in Equation (1). Lastly, the adjacency matrix \(A_{N\times N}\) with final distance values was plugged into multidimensional scaling, and eigendecomposition was performed to construct a refined embedding space \(U\) with \(d\) dimensions. This refined space ensured that the intrinsic geometry of the original spatial data was best preserved in low dimensions. Suppose that the seed concept vectors projected onto the refined embedding space can be represented as \(R=\{r_{1},\ r_{2},\ r_{3},\ \cdots,\ r_{N}\}\). Then, multidimensional scaling can yield an optimal projection by minimizing the cost function in Equation (2). The solution to the global minimization problem is identical toEquation (3), where \(\Sigma\) denotes the top \(d\) eigenvalues of \(A^{T}A\), and \(V\) is the eigenvector corresponding to \(A^{T}A\).  
\[a_{ij}=\underbrace{\min\Bigl{(}a_{ij},\,a_{ik}+a_{kj}\Bigr{)}}_{i,\,j,\,k\in\{1, 2,3,\cdots,\,N\}}. \tag{1}\]  
\[J(R)=\sum_{i=1}^{N}\sum_{j=1}^{N}\Bigl{(}\left\|r_{i}-r_{j}\right\|_{2}^{2}-a_{ ij}\Bigr{)}^{2}. \tag{2}\]  
\[U=\sqrt{\Sigma}V^{T}. \tag{3}\]