SlowGuess/NAACL_2021_OCR huggingface dataset:
Chenghao Jia, Yongliang Shen, Yechun Tang, Lu Sun, Weiming Lu*
College of Computer Science and Technology, Zhejiang University, Hangzhou, China
{chjia, syl, tangyechun, sunlu98, luwm} @zju.edu.cn
Prerequisite relations among concepts are crucial for educational applications, such as curriculum planning and intelligent tutoring. In this paper, we propose a novel concept prerequisite relation learning approach, named CPRL, which combines both concept representation learned from a heterogeneous graph and concept pairwise features. Furthermore, we extend CPRL under weakly supervised settings to make our method more practical, including learning prerequisite relations from learning object dependencies and generating training data with data programming. Our experiments on four datasets show that the proposed approach achieves the state-of-the-art results comparing with existing methods.
With the increasing availability of learning resources and the requirement of self-regulated learning, there is a rising need to organize knowledge in a reasonable order. Concept prerequisite relations are essentially considered as the dependency among concepts, and they are crucial for people to learn, organize, apply and generate knowledge (Margolis and Laurence, 1999). For example, if someone wants to learn the knowledge about Conditional Random Fields, the knowledge about Hidden Markov Model should be learned first. Consequently, the concept Hidden Markov Model is a prerequisite concept of the concept Conditional Random Fields. Nowadays, prerequisite relations among concepts have played a crucial role in educational applications, such as curriculum planning (Liu et al., 2016) and intelligent tutoring (Wang and Liu, 2016; Chen et al., 2018).
Recently, several attempts have been made to extract prerequisite relations among concepts from textbooks (Wang et al., 2016; Liang et al., 2018), MOOCs (Massive Open Online Courses) (Pan
et al., 2017), courses (Liang et al., 2015a; Liu et al., 2016; Liang et al., 2017; Li et al., 2019a; Roy et al., 2019) and scientific papers (Gordon et al., 2016). They either proposed a local statistical information, such as reference distance (Liang et al., 2015a) and cross-entropy (Gordon et al., 2016) to measure the prerequisite relations between concepts, or proposed handcrafted features to learn a prerequisite relation classifier (Pan et al., 2017). Liang et al. (2017) proposed CPR-Recover to recover concept prerequisite relations from course dependencies. More recently, Li et al. (2019a) applied variational graph autoencoders to learn concept prerequisite relations from courses. While Roy et al. (2019) developed a supervised learning approach called PRREQ.
However, there are still several challenges to learn the prerequisite relations among concepts. Firstly, there are multiple and complex relations among concepts and learning resources, but they were not fully utilized before. Secondly, labeling training data is enormously expensive and time consuming, especially when domain expertise is required for concept prerequisite relation judgement.
In order to address these challenges, we propose a novel concept prerequisite relation learning approach, named CPRL, which firstly learns concept representation via a relational graph convolutional network (R-GCN) (Schlichtkrull et al., 2018) on a heterogeneous graph, and predicts the concept prerequisite relations with a Siamese network. Then, it is optimized with the learning object dependencies and handcrafted features.
Moreover, we extend CPRL under the weakly-supervised settings to make our approach more practical, including learning prerequisite relation from learning object dependencies and generating training data with data programming paradigm.
- We propose a heterogeneous concept-learning object graph (HCLoG), which can model the
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2036-2047 June 6-11, 2021. ©2021 Association for Computational Linguistics
multiple and complex relations among concepts and learning resources to learn concept representation.
- We propose a novel concept prerequisite relation learning approach, named CPRL, which combines evidences from concept representations via R-GCN on HCLoG, learning object dependencies, and concept pairwise features.
- We extend CPRL under weakly supervised settings to avoid costly training data labeling.
- We conduct extensive experiments on four real-world datasets with different domains: Textbook, MOOC, LectureBank and University Course, and our approach achieves new state-of-the-art performance.
The educational data can be a textbook or a course, which can be modeled as a sequential learning objects (denoted as LO for short), such as book chapters, MOOC videos and lectures. There are concepts in an educational data, and we would like to extract the prerequisite relation among these concepts, as shown in Figure 1.
Figure 1: An example of prerequisite relation learning for concepts in educational data.
For convenience, we will use the following notations:
-  $D = \{o_1, o_2, \dots, o_M\}$  is an educational data, where  $o_i$  denotes the  $i$ -th learning object in  $D$  and is represented as a document. The document can be the text from a book chapter, or the speech script from a MOOC video.
-  $C = \{c_1, c_2, \dots, c_N\}$  is a set of concepts in  $D$ .
Therefore, the problem could be formally defined as: given an educational data  $D$  and its corresponding concepts  $C$ , the goal is to learn a function  $F_{\theta}: C \times C \to \{0,1\}$ , which can predict whether  $c_{i}$  is a prerequisite concept of  $c_{j}$  by mapping the concept pair  $\langle c_i, c_j \rangle$  to a binary class.
3 The CPRL Framework
The overview of our proposed CPRL is shown in Figure 2.
We firstly build a heterogeneous concept-learning object graph from the educational data, and then use a relational graph convolutional network (R-GCN) (Schlichtkrull et al., 2018) to represent the concepts and learning objects. Then, pairwise features for concepts are extracted according to their textual and structural information. Finally, all features are combined to learn the concept prerequisite relations.
It should be noted that the dependencies among learning objects can be viewed as a signal of weak supervision, which are also used to train the model.
3.1 Heterogeneous Concept-Learning Object Graph
We build a heterogeneous concept-learning object graph from an educational data, which contains concepts and learning objects, so the concept cooccurrence and the learning object-concept relations can be explicitly modeled.
The heterogeneous concept-learning object graph is defined as a graph  $G = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  consists of two types of nodes: concept nodes  $V_{c} = \{c_{1},c_{2},\dots,c_{N}\}$  and learning object nodes  $V_{o} = \{o_{1},o_{2},\dots,o_{M}\}$ , and  $\mathcal{E}$  represents the relations among them.
Specifically, we define the following three types of edges in  $G$ .
1. an edge between a concept and a learning object, and the weight is the term frequency-inverse document frequency (tfidf) of the concept in the document, where the term frequency is the number of times the concept appears in the document, while the inverse document frequency is the logarithmically scaled inverse fraction of the number of documents that contain the concept. E.g.,  $e_{co}$  in Figure 2.
2. an edge between two concepts which co-occur in a fixed size sliding window in documents. Point-wise mutual information (PMI) is used to calculate the weight. Formally,  $pmi(i,j) = \log \frac{p(i,j)}{p(i)\cdot p(j)}, p(i,j) = \frac{\#W(i,j)}{\#W}$  and  $p(i) = \frac{\#W(i)}{\#W}$ , where  $\# W(i,j)$  is the number of sliding windows that contain both  $c_i$  and  $c_j$ ,  $\# W(i)$  is the number of sliding windows that only contain  $c_i$ , and  $\# W$  is the
Figure 2: The overview of our proposed CPRL framework.
number of sliding windows in  $D$ . E.g.,  $e_{cc}$  in Figure 2.
3. an edge between two learning objects, and the weight is the normalized distance between these two learning objects in the educational data. Formally,  $dis(i,j) = \frac{|j - i|}{M}$ . E.g.,  $e_{oo}$  in Figure 2.
Thus, the adjacency matrix  $A\in \mathbb{R}^{(M + N)\times (M + N)}$  of the graph  $G$  is defined as:
$$
A _ {i j} = \left\{ \begin{array}{l l} p m i (i, j) & i \text {a n d} j \text {a r e c o n c e p t s} \\ t f i d f (i, j) & i \text {i s a c o n c e p t a n d} j \text {i s a L O} \\ d i s (i, j) & i \text {a n d} j \text {a r e L O s} \\ 1 & i = j \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
3.2 Concept Representation via R-GCN
Since there are different types of relations among the nodes in the heterogeneous concept-learning object graph, we employ R-GCN to learn the representations of concepts and LOs.
We first use pretrained word embeddings GLoVE (Pennington et al., 2014) to represent each concept node in  $G$ . To represent the learning object, we calculate the average word embeddings of concepts in that learning object. Then, we update the node representation with R-GCN by aggregating messages from its direct neighbors as follows:
$$
h _ {i} ^ {l + 1} = \sigma (W _ {0} ^ {l} h _ {i} ^ {l} + \sum_ {r \in R} \sum_ {j \in N _ {i} ^ {r}} \frac {1}{c _ {i , r}} W _ {r} ^ {l} A _ {i j} h _ {j} ^ {l})
$$
where  $N_{i}^{r}$  is the neighbors of node  $i$  of relation  $r\in R$ ,  $W_{r}^{l}\in \mathbb{R}^{d\times d}$  is a relation-specific weight matrix,  $W_0^l\in \mathbb{R}^{d\times d}$  is a general weight matrix,  $h_i^l$  is the hidden state of node  $i$  at  $l$ -th layer,  $\sigma$  is the ReLU function, and  $c_{i,r} = \sum_{j\in N_i^r}A_{ij}$  is a normalization constant.
We stack the networks for  $L$  layers, and the concepts and learning objects can be represented by the hidden state of nodes in the  $L$ -th layer.
3.3 Prerequisite Relation Classification
After representing concepts via R-GCN, a Siamese network is used to predict whether the concept  $c_{i}$  is prerequisite of  $c_{j}$ .
We firstly take the concept representation of  $c_{i}$  and  $c_{j}$  as the input of a Siamese network, as shown in Figure 3, to calculate the likelihood of  $c_{i}$  being a prerequisite concept of  $c_{j}$ . Formally,  $\vec{c}_i = RELU(W_s\cdot h_{c_i}^L +b_s)$ , where  $h_{c_i}^L$  is the output of the R-GCN for concept  $c_{i}$  in  $L$ -th layer. Then, the likelihood  $p_{GCN}(c_i,c_j)$  is calculated as  $\sigma (W^{T}[\vec{c}_{i};\vec{c}_{j};\vec{c}_{i} - \vec{c}_{j};\vec{c}_{i}\otimes \vec{c}_{j}] + b)$ , where  $\sigma$  is the sigmoid function,  $\otimes$  and  $-$  are the element-wise multiplication and subtraction operators, and  $[\cdot ;\cdot ]$  means the concatenation of vectors.
Finally, we use the cross-entropy as the loss function:  $\mathcal{L}_c = \frac{1}{|T|}\sum_{(c_i,c_j,y_{ij})\in T} - [y_{ij}\cdot \log (p_{GCN}(c_i,c_j)) + (1 - y_{ij})\cdot \log (1 - p_{GCN}(c_i,c_j))]$ , where  $T$  is the training dataset, and  $y_{ij}\in \{0,1\}$  is the ground truth of  $(c_i,c_j)$ .
Figure 3: The Siamese network
3.4 Optimized with LO Dependencies
Intuitively, the dependencies among learning objects can reflect the prerequisite relations among concepts, but how can we utilize the learning object dependencies to enhance our model?
In the heterogeneous concept learning object graph, concepts and learning objects are both rep
resented in the same space, so they can be fed to the same Siamese network.
Formally, we feed the representations of learning object  $o_i$  and  $o_j$  to the same Siamese network mentioned in previous section, and obtain the likelihood of the learning object dependency as  $p_{GCN}(o_i,o_j)$ . Similarly, we define the loss function as:  $\mathcal{L}_o = \frac{1}{|T|}\sum_{(o_i,o_j,y_{ij})\in T} - [y_{ij}\cdot \log (p_{GCN}(o_i,o_j)) + (1 - y_{ij})\cdot \log (1 - p_{GCN}(o_i,o_j))]$ , where  $T$  is the training dataset, and  $y_{ij}\in \{0,1\}$  is the ground truth of  $(o_i,o_j)$ .
Predicting the dependencies among learning objects can be considered as an auxiliary task for concept prerequisite relation learning, so the loss function could be:  $\mathcal{L} = \mathcal{L}_c + \mu \mathcal{L}_o$ .
3.5 Fusing Handcrafted Pairwise Features
In order to fully utilize the information of LOs, we also extract concept pairwise features from their textual and structural information.
Liang et al. (2015a) pointed out that when learning concept  $A$ , if one needs to refer to concept  $B$  a lot but not vice versa, then  $B$  is more likely to be a prerequisite of  $A$  than  $A$  of  $B$ . Inspired by this idea, we propose a new generic metric, namely learning object reference distance (LOrd), in a learning object sequence  $D = \{o_1, o_2, \dots, o_M\}$  to measure prerequisite relations among concepts.
For a concept pair  $(c_i, c_j)$ , we propose the reference weight  $(rw)$  to qualify how  $c_j$  is referred by LOs which mention concept  $c_i$ , defined as:
$$
r w (c _ {i}, c _ {j}) = \frac {\sum_ {o \in D} f (c _ {i} , o) \cdot r (o , c _ {j})}{\sum_ {o \in D} f (c _ {i} , o)}
$$
where  $f(c_{i},o)$  indicates the frequency of concept  $c_{i}$  appears in the learning object  $o$ , and  $r(o,c_{j})\in \{0,1\}$  denotes whether concept  $c_{j}$  appears in  $o$ . Then, the LOrd is defined as:  $LOrd(c_i,c_j) =$ $rw(c_j,c_i) - rw(c_i,c_j)$ . Obviously, LOrd can be easily calculated for textbooks, MOOC courses and university courses.
In addition, for MOOCs, we use features as in (Pan et al., 2017). While for textbooks, we extract several pairwise features as in (Pan et al., 2017), including Semantic Relatedness, Wikipedia reference distance and complexity level distance. The details can be referred in the Appendix.
Moreover, we also extract head matching feature and ToC distance (Wang et al., 2016) for concept pairs for textbooks. Head matching feature represents whether two concepts have a common head or
not, which is obtained by suffix matching. Usually, it implies the existence of prerequisite relation, e.g., tree and binary tree. ToC distance measures the distance of concepts in the table of contents in  $D$ .
All the pairwise features are concatenated and fed into a forward neural network, which will generate the prediction result  $p_{F}(c_{i}, c_{j})$  for the concept pair  $(c_{i}, c_{j})$ . The loss function for the pairwise features is:  $\mathcal{L}_{f} = \frac{1}{|T|} \sum_{(c_{i}, c_{j}, y_{ij}) \in T} - [y_{ij} \cdot \log(p_{F}(c_{i}, c_{j})) + (1 - y_{ij}) \cdot \log(1 - p_{F}(c_{i}, c_{j}))]$ .
Therefore, the overall loss function is:  $\mathcal{L} = \mathcal{L}_c + \mu \mathcal{L}_o + \lambda \mathcal{L}_f$ , where  $\mu$  and  $\lambda$  are two hyperparameters.
4 The CPRL with Weak Supervision
In practice, it is expensive to collect massive hand-labeled data for model training. One intuitive way to alleviate the labeling cost is that we can train the model in one domain (e.g. Calculus), and then use it to predict the concept prerequisite relations in other domains (e.g. Data Structure and Physics). However, the idea fails and we will explain it in our experiments.
Therefore, we extend our model under the weak supervision settings in two ways.
We call the first way as learning prerequisite relations from LO dependencies. Since concepts and LOs are embedded into the same space through R-GCN in the heterogeneous graph, our model can implicitly infer the prerequisite relationships between concepts by explicitly learning the dependencies between LOs. This procedure is called  $\mathrm{CPRL}_{lo}$ .
Another way is use the data programming (Ratner et al., 2016) paradigm to create probabilistic training data. Data programming expresses weak supervision strategies or domain heuristics as labeling functions (LFs), and then estimates the label accuracies by fitting a generative model. The process is shown as Figure 4.
Figure 4: The pipeline of probabilistic label generation.
Here, we express some of the concept pairwise
features extracted before as heuristic labeling functions (LF for short):  $\lambda : (c_i, c_j) \to \{-1, 0, 1\}$ , where  $-1$  means the labeling function abstains from providing a label. We define label functions corresponding to the features among concepts, and some examples are shown in Figure 5.
def  $LF\_LOrd(c_i, c_j)$ :
if  $Lord\left(c_i,c_j\right) <   \theta_{Lord}^{min}$  return 1
elseif  $LOrd(c_i,c_j) > \theta_{LOrd}^{max}$  return 0
else return -1
def  $LF\_Suffix(c_i, c_j)$ :
if  $c_i$  matches  $c_j'$ 's suffix string return 1 else return -1
Figure 5: Two LF examples, where  $\theta_{LOrd}^{max}$  and  $\theta_{LOrd}^{min}$  are learned thresholds. Other LFs and the settings of thresholds are listed in the Appendix.
We apply  $m$  such LFs to the unlabeled concept pairs  $\{(c_{t_i}, c_{t_j})_{t=1}^n\}$  to generate a label matrix  $\Lambda \in \{-1,0,1\}^{n \times m}$ . Then, we use the weak supervision framework Snorkel (Ratner et al., 2019a) to train a probabilistic model. The probabilistic model takes the label matrix  $\Lambda$  as input, and generates the probabilistic training labels  $\tilde{Y} = p(Y|\Lambda)$  for each concept pair. The generated labels could be used to train our model.
With the probabilistic training data,  $\mathcal{L}_c$  and  $\mathcal{L}_f$  are changed to the noise-aware variants:  $\mathcal{L}_c = \sum_{(c_i,c_j)\in T}\mathbb{E}_{y_{ij}\sim \tilde{Y}}[-[y_{ij}\cdot \log (p_{GCN}(c_i,c_j)) + (1 - y_{ij})\cdot \log (1 - p_{GCN}(c_i,c_j))]]$  and  $\mathcal{L}_f = \sum_{(c_i,c_j)\in T}\mathbb{E}_{y_{ij}\sim \tilde{Y}}[-[y_{ij}\cdot \log (p_F(c_i,c_j)) + (1 - y_{ij})\cdot \log (1 - p_F(c_i,c_j))]]$ . This procedure is called  $\mathrm{CPRL}_{dp}$ .
In order to validate the efficiency of our model, we conducted experiments on four datasets with different domains.
- Textbook: we selected six Chinese textbooks in each of the three domains: Calculus, Data Structure, and Physics, and then extracted 89, 84 and 139 concepts, and labeled 449, 439 and 623 prerequisite relations for each domain respectively. The datasets will be publicly available later.
- MOOC: we used MOOC data<sup>1</sup> mentioned in (Pan et al., 2017), which involves two domains: Data Structure and Algorithms (DSA) and Machine Learning (ML).
- LectureBank: This dataset $^2$  (Li et al., 2019a) contains 1,352 English lecture files collected from university courses, and the annotations of prerequisite relations on 208 concepts.
- University Course: This dataset<sup>3</sup> (Liang et al., 2017) has 654 courses with 861 course prerequisite edges from various universities in USA, and 1008 pairs of concepts with prerequisite relations are manually annotated.
The set of concepts and prerequisite relations among them was annotated by experts, and released with the datasets. The statistics of the datasets are listed in the appendix.
We used the following state-of-the-art approaches as baselines.
Binary classifiers: We compared our model with the binary classifiers as in (Pan et al., 2017), including Naïve Bayes classifier (NB), Support vector machine (SVM), Logistic Regression (LR) and Random Forest classifier (RF).
RefD: RefD (Liang et al., 2015b) is a simple link-based metric for measuring the prerequisite relations among concepts.
GAE: GAE denotes graph autoencoder, which encodes a graph with GCN, and predicts links through the adjacency matrix reconstruction. Li et al. (2019a) used GAE for concept prerequisite relation learning.
VGAE: VGAE is an extension to GAE, which was also used in (Li et al., 2019a) for concept prerequisite relation learning.
PRERQ: PRERQ (Roy et al., 2019) obtains latent representations of concepts through the pairwise-link LDA model, and identifies concept prerequisite relations through a Siamese network.
We also compared our weakly-supervised variants with CPR-Recover (Liang et al., 2017), which is an unsupervised approach, and can recover concept prerequisite relations from course dependencies.
Consistent with many methods, we mainly used  $\mathrm{F}$  -score  $(\mathrm{F}_1)$  to evaluate the performance of CPRL with all the baselines. We also compared precision(P) and recall(R) against other methods.
<sup>1</sup>http://keg.cs.tsinghua.edu.cn/jietang/software/ac117-prerequisite-relation.rar
$^{2}$ https://github.com/Yale-LILY/LectureBank
<sup>3</sup>https://github.com/suderoy/PREREQ-IAAI-
19/tree/master/datasets/University%20Course%20Dataset
In all datasets, only concept prerequisite pairs are manually annotated, and we split the positive samples into train and test sets. In order to fairly compare with the previous researches,  $90\%$  samples of LectureBank were used for training while the rest  $10\%$  for testing. For other datasets, the proportions changed to  $70\%$  and  $30\%$ . Then, we generated negative samples by sampling random unrelated pairs of concepts from the vocabulary in addition to the reverse pair of original positive samples. In order to address the imbalance problem, we oversampled 3.5 and 1.5 times the number of the positive examples in the training and testing sets for Textbook dataset and other datasets respectively. The results are averaged over 5 train-test splits.
The parameters were initialized randomly from a Gaussian distribution with zero mean and standard deviation  $\sigma = 0.3$ . The initial learning rate  $\gamma$  is 0.5 for Textbook and 0.1 for other datasets. Besides, the learning rate annealed every 50 epochs by  $0.99\gamma$ . We trained CPRL using the Stochastic Gradient Descent method and stopped training if the train loss did not decrease for 30 consecutive epochs. For baseline models, we used default parameter settings as in their original implementations, and also used 300-dimensional GloVE as the pre-trained word embeddings.
For R-GCN, we set the number of R-GCN layers  $L = 2$  and set the embedding size of the first convolution layer as 256 and the second convolution layer as the number of concepts in each dataset. We experimented with other settings and found that small changes did not influence the result much. In addition, we set  $\lambda = 0.2$  and  $\mu = 0.1$ , since they made the best performance. The influence of parameters  $L$ ,  $\lambda$  and  $\mu$  can be referred to the Appendix.
5.4 Performance Comparison
Table 1 shows the precision, recall and F-score on four datasets with different domains.
From the table, we find that (1) CPRL achieves the best performance with F-score against all baselines on all datasets, except for DSA domain of the MOOC dataset. (2) CPRL performs best in LectureBank and University even without pairwise features and dependencies among learning objects. It tells that HCLoG can effectively model the multiple and complex relations among concepts and learning resources to learn better concept representation. (3) RefD can indeed measure the prerequisite relations
among concepts, and obtains a higher precision, but a lower recall. (4) GAE and VGAE utilize GCN for adjacency matrix reconstruction, but they perform worse than CPRL. The reason is that CPRL utilizes the heterogeneous concept learning object graph to learn the concept representation, which can fully utilize the complex relationships among concepts and learning objects, while GAE and VGAE only use the graph among concepts.
In order to prove the effects of pairwise features and LO dependencies, we conducted ablation experiments on Textbook and MOOC datasets. The results are shown in table 2
Dataset	Metric	CPRL	\( CPRL_f \)	\( CPRL_c \)
Textbook	DS	P	0.795	0.793	0.811
R	0.809	0.802	0.749
F1	0.802	0.797	0.779
PHY	P	0.778	0.779	0.778
R	0.798	0.799	0.716
F1	0.788	0.789	0.746
CAL	P	0.770	0.772	0.769
R	0.825	0.809	0.755
F1	0.797	0.790	0.762
MOOC	DSA	P	0.640	0.659	0.562
R	0.619	0.615	0.565
F1	0.630	0.636	0.563
ML	P	0.800	0.788	0.767
R	0.642	0.628	0.598
F1	0.712	0.699	0.672
Table 2: Ablation Study on CPRL. Row-wise best results are in bold.  $\mathrm{CPRL}_f$  and  $\mathrm{CPRL}_c$  are the models which minimize  $\mathcal{L}_c + \lambda \mathcal{L}_f$  and  $\mathcal{L}_c$  respectively.