Venue: NAACL 2021

# Heterogeneous Graph Neural Networks for Concept Prerequisite Relation Learning in Educational Data

 Chenghao Jia, Yongliang Shen, Yechun Tang, Lu Sun, Weiming Lu\({}^{*}\)

College of Computer Science and Technology, Zhejiang University, Hangzhou, China

{chjia, syl, tangyechnu, sunlu98, luwm}@zju.edu.cn

###### Abstract

Prerequisite relations among concepts are crucial for educational applications, such as curriculum planning and intelligent tutoring. In this paper, we propose a novel concept prerequisite relation learning approach, named CPRL, which combines both concept representation learned from a heterogeneous graph and concept pairwise features. Furthermore, we extend CPRL under weakly supervised settings to make our method more practical, including learning prerequisite relations from learning object dependencies and generating training data with data programming. Our experiments on four datasets show that the proposed approach achieves the state-of-the-art results comparing with existing methods.

+
Footnote †: corresponding author

_Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2036–2047

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

With the increasing availability of learning resources and the requirement of self-regulated learning, there is a rising need to organize knowledge in a reasonable order. Concept prerequisite relations are essentially considered as the dependency among concepts, and they are crucial for people to learn, organize, apply and generate knowledge Margolis and Laurence (1999). For example, if someone wants to learn the knowledge about _Conditional Random Fields_, the knowledge about _Hidden Markov Model_ should be learned first. Consequently, the concept _Hidden Markov Model_ is a prerequisite concept of the concept _Conditional Random Fields_. Nowadays, prerequisite relations among concepts have played a crucial role in educational applications, such as curriculum planning Liu et al. (2016) and intelligent tutoring Wang and Liu (2016); Chen et al. (2018).

Recently, several attempts have been made to extract prerequisite relations among concepts from textbooks Wang et al. (2016); Liang et al. (2018), MOOCs Massive Open Online Courses) Pan et al. (2017), courses Liang et al. (2015); Liu et al. (2016); Liang et al. (2017); Li et al. (2019); Roy et al. (2019) and scientific papers Gordon et al. (2016). They either proposed a local statistical information, such as _reference distance_Liang et al. (2015) and _cross-entropy_Gordon et al. (2016) to measure the prerequisite relations between concepts, or proposed handcrafted features to learn a prerequisite relation classifier Pan et al. (2017). Liang et al. (2017) proposed CPR-Recover to recover concept prerequisite relations from course dependencies. More recently, Li et al. (2019) applied variational graph autoencoders to learn concept prerequisite relations from courses. While Roy et al. (2019) developed a supervised learning approach called PREREQ.

However, there are still several challenges to learn the prerequisite relations among concepts. Firstly, there are multiple and complex relations among concepts and learning resources, but they were not fully utilized before. Secondly, labeling training data is enormously expensive and time consuming, especially when domain expertise is required for concept prerequisite relation judgement.

In order to address these challenges, we propose a novel concept prerequisite relation learning approach, named CPRL, which firstly learns concept representation via a relational graph convolutional network (R-GCN) Schlichtkrull et al. (2018) on a heterogeneous graph, and predicts the concept prequisite relations with a Siamese network. Then, it is optimized with the learning object dependencies and handcrafted features.

Moreover, we extend CPRL under the weakly-supervised settings to make our approach more practical, including learning prerequisite relation from learning object dependencies and generating training data with data programming paradigm.

Our contributions can be summarized as follows:

* We propose a heterogeneous concept-learning object graph (HCLoG), which can model the

multiple and complex relations among concepts and learning resources to learn concept representation.
* We propose a novel concept prerequisite relation learning approach, named CPRL, which combines evidences from concept representations via R-GCN on HCLoG, learning object dependencies, and concept pairwise features.
* We extend CPRL under weakly supervised settings to avoid costly training data labeling.
* We conduct extensive experiments on four real-world datasets with different domains: _Textbook_, _MOOC_, _LectureBank_ and _University Course_, and our approach achieves new state-of-the-art performance.

## 2 Problem Formulation

The educational data can be a textbook or a course, which can be modeled as a sequential learning objects (denoted as LO for short), such as book chapters, MOOC videos and lectures. There are concepts in an educational data, and we would like to extract the prerequisite relation among these concepts, as shown in Figure 1.

For convenience, we will use the following notations:

* \(D=\{o_{1},o_{2},...,o_{M}\}\) is an educational data, where \(o_{i}\) denotes the \(i\)-th learning object in \(D\) and is represented as a document. The document can be the text from a book chapter, or the speech script from a MOOC video.
* \(C=\{c_{1},c_{2},...,c_{N}\}\) is a set of concepts in \(D\).

Therefore, the problem could be formally defined as: given an educational data \(D\) and its corresponding concepts \(C\), the goal is to learn a function \(F_{\theta}:C\times C\rightarrow\{0,1\}\), which can predict whether \(c_{i}\) is a prerequisite concept of \(c_{j}\) by mapping the concept pair \(\langle c_{i},c_{j}\rangle\) to a binary class.

## 3 The CPRL Framework

The overview of our proposed CPRL is shown in Figure 2.

We firstly build a heterogeneous concept-learning object graph from the educational data, and then use a relational graph convolutional network (R-GCN) Schlichtkrull et al. (2018) to represent the concepts and learning objects. Then, pairwise features for concepts are extracted according to their textual and structural information. Finally, all features are combined to learn the concept prerequisite relations.

It should be noted that the dependencies among learning objects can be viewed as a signal of weak supervision, which are also used to train the model.

### Heterogeneous Concept-Learning Object Graph

We build a heterogeneous concept-learning object graph from an educational data, which contains concepts and learning objects, so the concept co-occurrence and the learning object-concept relations can be explicitly modeled.

The _heterogeneous concept-learning object graph_ is defined as a graph \(G=(\mathcal{V},\mathcal{E})\), where \(\mathcal{V}\) consists of two types of nodes: concept nodes \(V_{c}=\{c_{1},c_{2},...,c_{N}\}\) and learning object nodes \(V_{o}=\{o_{1},o_{2},...,o_{M}\}\), and \(\mathcal{E}\) represents the relations among them.

Specifically, we define the following three types of edges in \(G\).

1. an edge between a concept and a learning object, and the weight is the term frequency-inverse document frequency (tfidf) of the concept in the document, where the term frequency is the number of times the concept appears in the document, while the inverse document frequency is the logarithmically scaled inverse fraction of the number of documents that contain the concept. E.g., \(e_{co}\) in Figure 2.
2. an edge between two concepts which occur in a fixed size sliding window in documents. Point-wise mutual information (P-MI) is used to calculate the weight. Formally, \(pmi(i,j)=\log\frac{p(i,j)}{p(i)\cdot p(j)}\), \(p(i,j)=\frac{\#W(i,j)}{\#W}\) and \(p(i)=\frac{\#W(i)}{\#W}\), where \(\#W(i,j)\) is the number of sliding windows that contain both \(c_{i}\) and \(c_{j}\), \(\#W(i)\) is the number of sliding windows that only contain \(c_{i}\), and \(\#W\) is the

Figure 1: An example of prerequisite relation learning for concepts in educational data.

 

number of sliding windows in \(D\). E.g., \(e_{cc}\) in Figure 2.
3. an edge between two learning objects, and the weight is the normalized distance between these two learning objects in the educational data. Formally, \(dis(i,j)=\frac{|j-i|}{M}\). E.g., \(e_{oo}\) in Figure 2.

Thus, the adjacency matrix \(A\in\mathbb{R}^{(M+N)\times(M+N)}\) of the graph \(G\) is defined as:

\[A_{ij}=\left\{\begin{array}{ll}pmi(i,j)&i\text{ and $j$ are concepts}\\ tfidf(i,j)&i\text{ is a concept and $j$ is a LO}\\ dis(i,j)&i\text{ and $j$ are LOs}\\ 1&i=j\\ 0&\text{otherwise}\end{array}\right.\]

### Concept Representation via R-GCN

Since there are different types of relations among the nodes in the heterogeneous concept-learning object graph, we employ R-GCN to learn the representations of concepts and LOs.

We first use pretrained word embeddings GLoVE Pennington et al. (2014) to represent each concept node in \(G\). To represent the learning object, we calculate the average word embeddings of concepts in that learning object. Then, we update the node representation with R-GCN by aggregating messages from its direct neighbors as follows:

\[h_{i}^{l+1}=\sigma(W_{0}^{l}h_{i}^{l}+\sum_{r\in R}\sum_{j\in N_{i}^{r}}\frac {1}{c_{i,r}}W_{r}^{l}A_{ij}h_{j}^{l})\]

where \(N_{i}^{r}\) is the neighbors of node \(i\) of relation \(r\in R\), \(W_{r}^{l}\in\mathbb{R}^{d\times d}\) is a relation-specific weight matrix, \(W_{0}^{l}\in\mathbb{R}^{d\times d}\) is a general weight matrix, \(h_{i}^{l}\) is the hidden state of node \(i\) at \(l\)-th layer, \(\sigma\) is the ReLU function, and \(c_{i,r}=\sum_{j\in N_{i}^{r}}A_{ij}\) is a normalization constant.

We stack the networks for \(L\) layers, and the concepts and learning objects can be represented by the hidden state of nodes in the \(L\)-th layer.

### Prerequisite Relation Classification

After representing concepts via R-GCN, a Siamese network is used to predict whether the concept \(c_{i}\) is prerequisite of \(c_{j}\).

We firstly take the concept representation of \(c_{i}\) and \(c_{j}\) as the input of a Siamese network, as shown in Figure 3, to calculate the likelihood of \(c_{i}\) being a prerequisite concept of \(c_{j}\). Formally, \(\vec{c}_{i}=ReLU(W_{s}\cdot h_{c_{i}}^{L}+b_{s})\), where \(h_{c_{i}}^{L}\) is the output of the R-GCN for concept \(c_{i}\) in \(L\)-th layer. Then, the likelihood \(p_{GCN}(c_{i},c_{j})\) is calculated as \(\sigma(W^{T}[\vec{c}_{i};\vec{c}_{j};\vec{c}_{i}-\vec{c}_{j};\vec{c}_{i} \otimes\vec{c}_{j}]+b)\), where \(\sigma\) is the sigmoid function, \(\otimes\) and \(-\) are the element-wise multiplication and subtraction operators, and \([\cdot;\cdot]\) means the concatenation of vectors.

Finally, we use the cross-entropy as the loss function: \(\mathcal{L}_{c}=\frac{1}{|T|}\sum_{(c_{i},c_{j},y_{ij})\in T}-[y_{ij}~{}\cdot \log(p_{GCN}(c_{i},c_{j}))+(1-y_{ij})\cdot\log(1-p_{GCN}(c_{i},c_{j}))]\), where \(T\) is the training dataset, and \(y_{ij}\in\{0,1\}\) is the ground truth of \((c_{i},c_{j})\).

### Optimized with LO Dependencies

Intuitively, the dependencies among learning objects can reflect the prerequisite relations among concepts, but how can we utilize the learning object dependencies to enhance our model?

In the heterogeneous concept learning object graph, concepts and learning objects are both rep

Figure 3: The Siamese network

Figure 2: The overview of our proposed CPRL framework.

 

resented in the same space, so they can be fed to the same Siamese network.

Formally, we feed the representations of learning object \(o_{i}\) and \(o_{j}\) to the same Siamese network mentioned in previous section, and obtain the likelihood of the learning object dependency as \(p_{GCN}(o_{i},o_{j})\). Similarly, we define the loss function as: \(\mathcal{L}_{o}=\frac{1}{|T|}\sum_{(o_{i},o_{j},y_{ij})\in T}-[y_{ij}\cdot\log( p_{GCN}(o_{i},o_{j}))+(1-y_{ij})\cdot\log(1-p_{GCN}(o_{i},o_{j}))]\), where \(T\) is the training dataset, and \(y_{ij}\in\{0,1\}\) is the ground truth of \((o_{i},o_{j})\).

Predicting the dependencies among learning objects can be considered as an auxiliary task for concept prerequisite relation learning, so the loss function could be: \(\mathcal{L}=\mathcal{L}_{c}+\mu\mathcal{L}_{o}\).

### Fusing Handcrafted Pairwise Features

In order to fully utilize the information of LOs, we also extract concept pairwise features from their textual and structural information.

Liang et al. (2015a) pointed out that when learning concept \(A\), if one needs to refer to concept \(B\) a lot but not vice versa, then \(B\) is more likely to be a prerequisite of \(A\) than \(A\) of \(B\). Inspired by this idea, we propose a new generic metric, namely _learning object reference distance_ (LOrd), in a learning object sequence \(D=\{o_{1},o_{2},...,o_{M}\}\) to measure prerequisite relations among concepts.

For a concept pair \((c_{i},c_{j})\), we propose the reference weight (_rw_) to qualify how \(c_{j}\) is referred by LOs which mention concept \(c_{i}\), defined as:

\[rw(c_{i},c_{j})=\frac{\sum_{o\in D}f(c_{i},o)\cdot r(o,c_{j})}{\sum_{o\in D} f(c_{i},o)}\]

where \(f(c_{i},o)\) indicates the frequency of concept \(c_{i}\) appears in the learning object \(o\), and \(r(o,c_{j})\in\{0,1\}\) denotes whether concept \(c_{j}\) appears in \(o\). Then, the LOrd is defined as: \(LOrd(c_{i},c_{j})=rw(c_{j},c_{i})-rw(c_{i},c_{j})\). Obviously, _LOrd_ can be easily calculated for textbooks, MOOC courses and university courses.

In addition, for MOOCs, we use features as in Pan et al. (2017). While for textbooks, we extract several pairwise features as in Pan et al. (2017), including _Semantic Relatedness_, _Wikipedia reference distance_ and _complexity level distance_. The details can be referred in the Appendix.

Moreover, we also extract _head matching feature_ and _ToC distance_Wang et al. (2016) for concept pairs for textbooks. _Head matching feature_ represents whether two concepts have a common head or not, which is obtained by suffix matching. Usually, it implies the existence of prerequisite relation, e.g., _tree_ and _binary tree_. _ToC distance_ measures the distance of concepts in the table of contents in \(D\).

All the pairwise features are concatenated and fed into a forward neural network, which will generate the prediction result \(p_{F}(c_{i},c_{j})\) for the concept pair \((c_{i},c_{j})\). The loss function for the pairwise features is: \(\mathcal{L}_{f}=\frac{1}{|T|}\sum_{(c_{i},c_{j},y_{ij})\in T}-[y_{ij}\cdot\log( p_{F}(c_{i},c_{j}))+(1-y_{ij})\cdot\log(1-p_{F}(c_{i},c_{j}))]\).

Therefore, the overall loss function is: \(\mathcal{L}=\mathcal{L}_{c}+\mu\mathcal{L}_{o}+\lambda\mathcal{L}_{f}\), where \(\mu\) and \(\lambda\) are two hyper-parameters.

## 4 The CPRL with Weak Supervision

In practice, it is expensive to collect massive hand-labeled data for model training. One intuitive way to alleviate the labeling cost is that we can train the model in one domain (e.g. _Calculus_), and then use it to predict the concept prerequisite relations in other domains (e.g. _Data Structure_ and _Physics_). However, the idea fails and we will explain it in our experiments.

Therefore, we extend our model under the weak supervision settings in two ways.

We call the first way as _learning prerequisite relations from LO dependencies_. Since concepts and LOs are embedded into the same space through R-GCN in the heterogeneous graph, our model can implicitly infer the prerequisite relationships between concepts by explicitly learning the dependencies between LOs. This procedure is called CPRL\({}_{lo}\).

Another way is use the _data programming_Ratner et al. (2016) paradigm to create probabilistic training data. _Data programming_ expresses weak supervision strategies or domain heuristics as labeling functions (LFPs), and then estimates the label accuracies by fitting a generative model. The process is shown as Figure 4.

Here, we express some of the concept pairwise

Figure 4: The pipeline of probabilistic label generation.

 

features extracted before as heuristic labeling functions (LF for short): \(\lambda:(c_{i},c_{j})\rightarrow\{-1,0,1\}\), where \(-1\) means the labeling function abstains from providing a label. We define label functions corresponding to the features among concepts, and some examples are shown in Figure 5.

We apply \(m\) such LFs to the unlabeled concept-t pairs \(\{(c_{t_{i}},c_{l_{j}})^{n}_{i=1}\}\) to generate a label matrix \(\Lambda\in\{-1,0,1\}^{n\times m}\). Then, we use the weak supervision framework Snorkel Ratner et al. (2019) to train a probabilistic model. The probabilistic model takes the label matrix \(\Lambda\) as input, and generates the probabilistic training labels \(\tilde{Y}=p(Y|\Lambda)\) for each concept pair. The generated labels could be used to train our model.

With the probabilistic training data, \(\mathcal{L}_{c}\) and \(\mathcal{L}_{f}\) are changed to the _noise-aware_ variants: \(\mathcal{L}_{c}=\sum_{(c_{i},c_{j})\in T}\mathbb{E}_{y_{ij}\sim\tilde{Y}}[-[ y_{ij}\cdot\log(p_{GCN}(c_{i},c_{j}))+(1-y_{ij})\cdot\log(1-p_{GCN}(c_{i},c_{j}))]\) and \(\mathcal{L}_{f}=\sum_{(c_{i},c_{j})\in T}\mathbb{E}_{y_{ij}\sim\tilde{Y}}[-[ y_{ij}\cdot\log(p_{F}(c_{i},c_{j}))+(1-y_{ij})\cdot\log(1-p_{F}(c_{i},c_{j}))]]\). This procedure is called CPRL\({}_{dp}\).

## 5 Experiments

### Datasets

In order to validate the efficiency of our model, we conducted experiments on four datasets with different domains.

* _Textbook_: we selected six Chinese textbooks in each of the three domains: _Calculus_, _Data Structure_, and _Physics_, and then extracted 89, 84 and 139 concepts, and labeled 449, 439 and 623 prerequisite relations for each domain respectively. The datasets will be publicly available later.
* _MOOC_: we used MOOC data1 mentioned in Pan et al. (2017), which involves two domains: _Data Structure and Algorithms (DSA)_ and _Machine Learning (ML)_. Footnote 1: [http://keg.cs.tsinghua.edu.cn/jietang/software/acl17-prerequisite-relation.rar](http://keg.cs.tsinghua.edu.cn/jietang/software/acl17-prerequisite-relation.rar)
* _LectureBank_: This dataset2 Li et al. (2019) contains 1,352 English lecture files collected from university courses, and the annotations of prerequisite relations on 208 concepts. Footnote 2: [https://github.com/Vale-LLLY/LectureBank](https://github.com/Vale-LLLY/LectureBank)
* _University Course_: This dataset3 Liang et al. (2017) has 654 courses with 861 course prerequisite edges from various universities in USA, and 1008 pairs of concepts with prerequisite relations are manually annotated. Footnote 3: [https://github.com/suderoy/PREREQ-IAAI-19tree/master/datasets/University%20Course%20Dataset](https://github.com/suderoy/PREREQ-IAAI-19tree/master/datasets/University%20Course%20Dataset)

The set of concepts and prerequisite relations among them was annotated by experts, and released with the datasets. The statistics of the datasets are listed in the appendix.

### Baselines

We used the following state-of-the-art approaches as baselines.

**Binary classifiers**: We compared our model with the binary classifiers as in Pan et al. (2017), including Naive Bayes classifier (NB), Support vector machine (SVM), Logistic Regression (LR) and Random Forest classifier (RF).

**RefD**: RefD Liang et al. (2015) is a simple link-based metric for measuring the prerequisite relations among concepts.

**GAE**: GAE denotes graph autoencoder, which encodes a graph with GCN, and predicts links through the adjacency matrix reconstruction. Li et al. (2019) used GAE for concept prerequisite relation learning.

**VGAE**: VGAE is an extension to GAE, which was also used in Li et al. (2019) for concept prerequisite relation learning.

**PREEQ**: PREEQ Roy et al. (2019) obtain-s latent representations of concepts through the pairwise-link LDA model, and identifies concept prerequisite relations through a Siamese network.

We also compared our weakly-supervised variants with **CPR-Recover**Liang et al. (2017), which is an unsupervised approach, and can recover concept prerequisite relations from course dependencies.

Consistent with many methods, we mainly used F-score(F\({}_{1}\)) to evaluate the performance of CPRL with all the baselines. We also compared precision(P) and recall(R) against other methods.

Figure 5: Two LF examples, where \(\theta^{max}_{LOrd}\) and \(\theta^{min}_{LOrd}\) are learned thresholds. Other LFs and the settings of thresholds are listed in the Appendix.

 



### Implementation Details

In all datasets, only concept prerequisite pairs are manually annotated, and we split the positive samples into train and test sets. In order to fairly compare with the previous researches, 90% samples of _LectureBank_ were used for training while the rest 10% for testing. For other datasets, the proportions changed to 70% and 30%. Then, we generated negative samples by sampling random unrelated pairs of concepts from the vocabulary in addition to the reverse pair of original positive samples. In order to address the imbalance problem, we oversampled 3.5 and 1.5 times the number of the positive examples in the training and testing sets for _Textbook_ dataset and other datasets respectively. The results are averaged over 5 train-test splits.

The parameters were initialized randomly from a Gaussian distribution with zero mean and standard deviation \(\sigma=0.3\). The initial learning rate \(\gamma\) is 0.5 for _Textbook_ and 0.1 for other datasets. Besides, the learning rate annealed every 50 epochs by \(0.99\gamma\). We trained CPRL using the Stochastic Gradient Descent method and stopped training if the train loss did not decrease for 30 consecutive epochs. For baseline models, we used default parameter settings as in their original implementations, and also used 300-dimensional GloVE as the pre-trained word embeddings.

For R-GCN, we set the number of R-GCN layers \(L=2\) and set the embedding size of the first convolution layer as 256 and the second convolution layer as the number of concepts in each dataset. We experimented with other settings and found that small changes did not influence the result much. In addition, we set \(\lambda=0.2\) and \(\mu=0.1\), since they made the best performance. The influence of parameters \(L\), \(\lambda\) and \(\mu\) can be referred to the Appendix.

### Performance Comparison

Table 1 shows the precision, recall and F-score on four datasets with different domains.

From the table, we find that (1) CPRL achieves the best performance with F-score against all baselines on all datasets, except for DSA domain of the MOOC dataset. (2) CPRL performs best in LectureBank and University even without pairwise features and dependencies among learning objects. It tells that HCLoG can effectively model the multiple and complex relations among concepts and learning resources to learn better concept representation. (3) RefD can indeed measure the prerequisite relations among concepts, and obtains a higher precision, but a lower recall. (4) GAE and VGAE utilize GCN for adjacency matrix reconstruction, but they perform worse than CPRL. The reason is that CPRL utilizes the heterogeneous concept learning object graph to learn the concept representation, which can fully utilize the complex relationships among concepts and learning objects, while GAE and VGAE only use the graph among concepts.

### Ablation Study

In order to prove the effects of pairwise features and LO dependencies, we conducted ablation experiments on _Textbook_ and _MOOC_ datasets. The results are shown in table 2

As shown in Table 2, CPRL performs better than CPRL\({}_{f}\) and CPRL\({}_{c}\) on most of the datasets, so pairwise features and learning object dependencies can both contribute to the performance. Besides, even CPRL\({}_{c}\) obtains a better performance than the baselines in Table 1, which proves the effectiveness of the heterogeneous graph.

### Effectiveness of Weak Supervision

In order to evaluate our weakly supervised prerequisite relation learning approaches, we compared our two variants CPRL\({}_{lo}\) and CPRL\({}_{dp}\) with CPRRecover [1] in _Textbook_ dataset, and the results are shown in Table 3.

From the table, we find that CPRL\({}_{lo}\) and CPRL\({}_{dp}\) outperform CPR-Recover in all metrics, and CPRL\({}_{dp}\) achieves the best performance. It proves that the knowledge of learning object dependencies can be transferred to learn the concept prerequisite relations through the concept learning object graph. In addition, the data programming with our designed label functions can generate help

\begin{table}
\begin{tabular}{|r|l|c|c|c|c|} \hline \multicolumn{2}{|c|}{Dataset} & Metric & CPRL & CPRL\({}_{f}\) & CPRL\({}_{c}\) \\ \hline \multirow{8}{*}{Testbook} & \multirow{3}{*}{DS} & P & 0.795 & 0.793 & **0.811** \\  & & R & **0.809** & 0.802 & 0.749 \\  & & F\({}_{1}\) & **0.802** & 0.797 & 0.779 \\ \cline{2-6}  & \multirow{3}{*}{PHY} & P & 0.778 & **0.799** & 0.778 \\  & & R & 0.798 & **0.799** & 0.716 \\  & & F\({}_{1}\) & 0.788 & **0.789** & 0.746 \\ \cline{2-6}  & \multirow{3}{*}{CAL} & P & 0.770 & **0.772** & 0.769 \\  & & R & **0.825** & 0.809 & 0.755 \\  & & F\({}_{1}\) & **0.797** & 0.790 & 0.762 \\ \hline \multirow{8}{*}{MOOC} & \multirow{3}{*}{DSA} & P & 0.640 & **0.659** & 0.562 \\  & & R & **0.649** & 0.615 & 0.565 \\ \cline{1-1}  & & F\({}_{1}\) & 0.630 & **0.636** & 0.563 \\ \cline{1-1} \cline{2-6}  & \multirow{3}{*}{ML} & P & **0.800** & 0.788 & 0.767 \\ \cline{1-1}  & & R & **0.642** & 0.628 & 0.598 \\ \cline{1-1}  & & F\({}_{1}\) & **0.712** & 0.699 & 0.672 \\ \hline \end{tabular}
\end{table}
Table 2: Ablation Study on CPRL. Row-wise best results are in bold. CPRL\({}_{f}\) and CPRL\({}_{c}\) are the models which minimize \(\mathcal{L}_{c}+\lambda\mathcal{L}_{f}\) and \(\mathcal{L}_{c}\) respectively.



ful training data, and achieve comparable performance with the supervised CPRL.

### Verification of Domain Transfer Ability

In order to explore the transfer ability of our model between different domains, we conducted an experiment on _Textbook_ dataset.

Specifically, for CPRL, we firstly trained the model in one domain, and then used the model to predict prerequisite relations between concepts in another domain. While for CPRL\({}_{dp}\), we obtained the best thresholds such as \(\theta^{max}_{LOrd}\) and \(\theta^{min}_{LOrd}\) in LFs in one domain and then used them to other domains. The results are shown in Table 4

We observe that (1) F-scores drop severely in CPRL, so we cannot simply transfer the model across domains due to the difference among concepts and LOs. (2) CPRL\({}_{dp}\) is more stable and can be used in practice since we only need to label a small amount of training data in one domain.

### Effectiveness of Ensemble

Our approach can learn the concept prerequisite relations from one learning object sequence, such as a textbook. While the concepts in textbooks in the same domain are basically the same, so the prerequisite relations among them can be aggregated.

Here, we used a simple majority voting strategy for aggregation, and the results are shown in Figure 6. From the table, we see a significant improvement for the ensemble results.

## 6 Related Work

### Prerequisite Relation Learning

Learning prerequisite relations between concepts has attracted much recent work, and can be classified into three categories: local statistical information based approaches, recovery based approaches and learning based approaches.

\begin{table}
\begin{tabular}{|c|l|c|c|c|c|c|c|c|c|} \hline Dataset & Metric & SVM & LR & RF & NB & febnl & GAE & VOAE & PRLRRQ & CPRL \\ \hline  & P & 0.818 & 0.852 & 0.755 & 0.481 & **0.920** & 0.404 & 0.226 & 0.795 \\  & DS & R & 0.632 & 0.590 & 0.685 & 0.897 & 0.244 & **0.900** & 0.570 & 0.369 & 0.809 \\  & F\({}_{1}\) & 0.713 & 0.697 & 0.718 & 0.626 & 0.355 & 0.597 & 0.493 & 0.280 & **0.820** \\ \cline{2-11}  & \multirow{3}{*}{PHY} & P & 0.806 & 0.853 & 0.748 & 0.990 & **0.900** & 0.505 & 0.460 & 0.432 & 0.770 \\  & PHY & R & 0.655 & 0.588 & 0.752 & 0.922 & 0.409 & **0.943** & 0.649 & 0.423 & 0.825 \\  & & F\({}_{1}\) & 0.723 & 0.699 & 0.750 & 0.557 & 0.652 & 0.657 & 0.538 & 0.427 & **0.779** \\ \cline{2-11}  & \multirow{3}{*}{CAL} & P & 0.839 & 0.800 & 0.746 & 0.404 & **0.950** & 0.435 & 0.414 & 0.391 & 0.778 \\  & & R & 0.637 & 0.570 & 0.715 & **0.995** & 0.302 & 0.900 & 0.558 & 0.506 & 0.798 \\  & & F\({}_{1}\) & 0.724 & 0.686 & 0.730 & 0.574 & 0.458 & 0.587 & 0.475 & 0.441 & **0.788** \\ \hline  & \multirow{3}{*}{DOAC} & P & 0.705 & 0.808 & 0.344 & 0.613 & **0.920** & 0.294 & 0.269 & 0.492 & 0.641 \\  & & R & 0.624 & 0.168 & **0.715** & 0.696 & 0.252 & **0.715** & 0.657 & 0.462 & 0.619 \\  & & F\({}_{1}\) & **0.662** & 0.278 & 0.464 & 0.652 & 0.396 & 0.417 & 0.382 & 0.476 & 0.630 \\ \cline{2-11}  & \multirow{3}{*}{ML} & P & 0.668 & 0.748 & 0.375 & 0.577 & 0.784 & 0.293 & 0.266 & 0.448 & **0.800** \\ \cline{2-11}  & & R & 0.577 & 0.27 & 0.669 & 0.623 & 0.188 & **0.733** & 0.647 & 0.592 & 0.542 \\ \cline{2-11}  & & F\({}_{1}\) & 0.619 & 0.397 & 0.481 & 0.599 & 0.303 & 0.419 & 0.377 & 0.510 & **0.712** \\ \hline  & \multirow{3}{*}{LectureBank} & P & 0.857 & 0.744 & 0.855 & 0.670 & 0.666 & 0.462 & 0.417 & 0.590 & **0.861** \\ \cline{2-11}  & & R & 0.692 & 0.744 & 0.681 & 0.640 & 0.228 & 0.811 & 0.575 & 0.502 & **0.858** \\ \cline{2-11}  & & F\({}_{1}\) & 0.766 & 0.744 & 0.758 & 0.655 & 0.339 & 0.589 & 0.484 & 0.543 & **0.860** \\ \cline{2-11} \cline{2-11}  & \multirow{3}{*}{University Course} & P & 0.796 & 0.595 & 0.797 & 0.478 & **0.591** & 0.450 & 0.470 & 0.468 & 0.689 \\ \cline{2-11}  & & R & 0.635 & 0.546 & 0.480 & 0.649 & 0.415 & 0.866 & 0.694 & **0.916** & 0.760 \\ \cline{2-11}  & & F\({}_{1}\) & 0.707 & 0.569 & 0.582 & 0.550 & 0.572 & 0.597 & 0.560 & 0.597 & **0.723** \\ \hline \end{tabular}
\end{table}
Table 1: The performance of CPRL on four datasets with different domains. Row-wise best results are in bold. CPRL is the model which minimizes \(\mathcal{L}_{c}+\mu\mathcal{L}_{o}+\lambda\mathcal{L}_{f}\), while the models for _LectureBank_ and _University Course_ only minimize \(\mathcal{L}_{c}\) since the LOs have no prerequisite relations in them and we cannot extract structural features.

Figure 6: The ensemble results which are aggregated from six textbooks in each domain.

 

As local statistical information, _reference distance_(Liang et al., 2015) and _cross-entropy_Gordon et al. (2016) were proposed to measure the concept prerequisite relations. CPR-Recover Liang et al. (2017) is a recovery based approach, which recovers prerequisite relations from course dependencies. The learning based approaches are the most popular. For example, Pan et al. (2017) proposed contextual, structural and semantic features for concept prerequisite relation classification. Roy et al. (2019) applied the pairwise-link LDA model to represent concept, and trained a Siamese network to identify prerequisite relations. Li et al. (2019) trained variational graph autoencoders to predict concept prerequisite relations. However, these approaches didn't model the mutiple and complex relations among concepts and learning resources. Meanwhile, they also need a large set of training data, which is costly to obtain.In order to reduce the amount of training data required, active learning was investigated in (Liang et al., 2018) and (Liang et al., 2019) for concept prerequisite learning.

### Weakly Supervised Learning

One of the most significant bottlenecks for machine learning is the need for a big training data set. Nowadays, it is very promising to use weakly supervised learning techniques to reduce the amount of human intervention needed. For example, _distant supervision_ can produce noisy training data by aligning unlabeled data with an external knowledge base, e.g. relation extraction in (Smirnova and Cudre-Mauroux, 2018). _Crowdsourcing_(Yuen et al., 2011) and _heuristic rules_(Sa et al., 2016) can also generate noisy training data.

However, these weakly supervised data is incomplete, inexact and inaccurate, so it is important to integrate multiple noisy labeling data to produce more accuracy data. _Data programming_(Ratner et al., 2016) provides a simple and unifying framework for the creation of training sets, which expresses weak supervision strategies as labeling functions, and then uses a generative model to denoise the labeling data. Snorkel4 Ratner et al. (2019) is a system built around the data programming paradigm for rapidly creating, modeling, and managing training data. Several works have been explored to use _data programming_ for training data creation. For example, SwellShark Fries et al. (2017) was proposed for quickly building biomedical named entity recognition taggers using lexicons, heuristics, and other forms of weak supervision instead of hand-labeled data. GWASkb with thousands of genotype-phenotype associations was created by using Snorkel in Kuleshov et al. (2019). Snorkel was also used for chemical reaction relationship extraction (Mallory et al., 2020), discourse structure learning Badene et al. (2019) and medical entity classification Fries et al. (2020).

Footnote 4: [https://www.snorkel.org/](https://www.snorkel.org/)

In addition, _data programming_ was further improved under different situations. For example, MeTaL Ratner et al. (2019) was proposed for modeling and integrating weak supervision sources with different unknown accuracies, correlations, and granularities. Cross-modal data programming was proposed in Dunnmon et al. (2020). FlyingSquid Fu et al. (2020) speeded up weak supervision with triplet methods.

## 7 Conclusion

In this paper, we propose a novel concept prerequisite relation learning approach, named CPRL, which combines both concept representation learned from a heterogeneous graph and concept pairwise features. Furthermore, we extend CPRL under weakly supervised settings to make our method more practical. The experiments on four datasets show that our method achieves state-of-the-art performance. In addition, we also prove the effectiveness of our weakly supervised prerequisite relation learning variants.

In future, we plan to design more effective label functions or employ more reliable weakly supervised learning approaches Li et al. (2019); Guo et al. (2019) to further improve the performance. Moreover, we will also introduce concept prerequisite relations into curriculum planning and intelligent tutoring applications, e.g. organizing learning resources into a reasonable order and incorporating prerequisite relations into knowledge tracing technologies.

## Acknowledgments

This work is supported by the National Key Research and Development Project of China (No. 2018AAA0101900), the Zhejiang Provincial Natural Science Foundation of China (No. LY17F020015), the Chinese Knowledge Center of Engineering Science and Technology (CKCEST) and MOE Engineering Research Center of Digital Library 



## References

* Badene et al. (2019) Sonia Badene, Kate Thompson, Jean-Pierre Lorre, and Nicholas Asher. 2019. Data programming for learning discourse structure. In _Association for Computational LInquistics (ACL)_.
* Chen et al. (2018) P. Chen, Yu Lu, V. Zheng, and Yang Pian. 2018. Prerequisite-driven deep knowledge tracing. _2018 IEEE International Conference on Data Mining (ICDM)_, pages 39-48.
* Dunnmon et al. (2020) Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L Rubin, et al. 2020. Cross-modal data programming enables rapid medical machine learning. _Patterns_, page 100019.
* Fries et al. (2020) Jason A Fries, Ethan Steinberg, Saelig Khattar, Scott L Fleming, Jose Posada, Alison Callahan, and Nigam H Shah. 2020. Trove: Ontology-driven weak supervision for medical entity classification. _arXiv preprint arXiv:2008.01972_.
* Fries et al. (2017) Jason Alan Fries, Sen Wu, A. Ratner, and Christopher Re. 2017. Swellshark: A generative model for biomedical named entity recognition without labeled data. _ArXiv_, abs/1704.06360.
* Fu et al. (2020) Daniel Y. Fu, M. F. Chen, F. Sala, Sarah Hooper, K. Fatahalian, and Christopher Re. 2020. Fast and three-rious: Speeding up weak supervision with triplet methods. _ArXiv_, abs/2002.11955.
* Gordon et al. (2016) Jonathan Gordon, Linhong Zhu, Aram Galstyan, Prem Natarajan, and Gully Burns. 2016. Modeling concept dependencies in a scientific corpus. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 866-875.
* Guo et al. (2019) Lan-Zhe Guo, Yu-Feng Li, Ming Li, Jingfeng Yi, Bowen Zhou, and Z. Zhou. 2019. Reliable weakly supervised learning: Maximize gain and maintain safeness. _ArXiv_, abs/1904.09743.
* Kuleshov et al. (2019) V. Kuleshov, J. Ding, C. Vo, B. Hancock, A. Ratner, Yang I. Li, C. Re, S. Batzoglou, and Michael T. Snyder. 2019. A machine-compiled database of genome-wide association studies. _Nature Communications_, 10.
* Li et al. (2019a) Irene Li, Alexander R. Fabbri, Robert R. Tung, and Dragomir R. Radev. 2019a. What should i learn first: Introducing lecturebank for nlp education and prerequisite chain learning. In _AAAI_.
* Li et al. (2019b) Y. Li, Lan-Zhe Guo, and Z. Zhou. 2019b. Towards safe weakly supervised learning. _IEEE transactions on pattern analysis and machine intelligence_.
* Liang et al. (2015a) Chen Liang, Z. Wu, W. Huang, and C. Lee Giles. 2015a. Measuring prerequisite relations among concepts. In _EMNLP_.
* Liang et al. (2015b) Chen Liang, Zhaohui Wu, Wenyi Huang, and C. Lee Giles. 2015b. Measuring prerequisite relations among concepts. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Liang et al. (2018) Chen Liang, Jianbo Ye, Shuting Wang, B. Pursel, and C. Lee Giles. 2018. Investigating active learning for concept prerequisite learning. In _AAAI_.
* Liang et al. (2017) Chen Liang, Jianbo Ye, Z. Wu, B. Pursel, and C. Lee Giles. 2017. Recovering concept prerequisite relations from university course dependencies. In _AAAI_.
* Liang et al. (2019) Chen Liang, Jianbo Ye, H. Zhao, B. Pursel, and C. Lee Giles. 2019. Active learning of strict partial orders: A case study on concept prerequisite relations. _ArXiv_, abs/1801.06481.
* Liu et al. (2016) Hanxiao Liu, Wanli Ma, Yaming Yang, and J. Carbonell. 2016. Learning concept graphs from online educational data. _J. Artif. Intell. Res._, 55:1059-1090.
* Mallory et al. (2020) Emily K Mallory, Matthieu de Rochemoteix, Alex Ratner, Ambika Acharya, Chris Re, Roselic A Bright, and Russ B Altman. 2020. Extracting chemical reactions from text using snorkel. _BMC bioinformatics_, 21(1):1-15.
* Margolis and Laurence (1999) Eric Margolis and Stephen Laurence. 1999. _Concepts: core readings_. Mit Press.
* Pan et al. (2017) Liangming Pan, C. Li, Juan-Zi Li, and J. Tang. 2017. Prerequisite relation learning for concepts in moocs. In _ACL_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.

* Ratner et al. (2019b) A. Ratner, B. Hancock, Jared Dunnmon, F. Sala, Shreyash Pandey, and C. Re. 2019b. Training complex models with multi-task weak supervision. _Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence_, 33:4763-4771.
* Ratner et al. (2016) Alexander J. Ratner, C. D. Sa, Sen Wu, Daniel Selsam, and C. Re. 2016. Data programming: Creating large training sets, quickly. _Advances in neural information processing systems_, 29:3567-3575.
* Roy et al. (2019) S. Roy, Meghana Madhyastha, Sheril Lawrence, and V. Rajan. 2019. Inferring concept prerequisites relations from online educational resources. In _AAAI_.
* Sa et al. (2016) C. D. Sa, A. Ratner, Christopher Re, Jaeho Shin, Feiran Wang, Sen Wu, and Ce Zhang. 2016. Deepdiev: Declarative knowledge base construction. _SIGMOD record_, 45 1:60-67

M. Schlichtkrull, Thomas Kipf, P. Bloem, R. V. Berg, Ivan Titov, and M. Welling (2018)Modeling relational data with graph convolutional networks. In ESWC, Cited by: SS3.
* A. Smirnova and P. Cudre-Mauroux (2018)Relation extraction using distant supervision. ACM Computing Surveys (CSUR)51, pp. 1- 35. External Links: Document, ISSN 0947-69XX Cited by: SS1.
* S. Wang and L. Liu (2016)Prerequisite concept maps extraction for automaticassessment. In WWW, Cited by: SS1.
* S. Wang, A. Ororbia, Z. Wu, K. Williams, C. Liang, B. Pursel, and C. Giles (2016)Using prerequisites to extract concept maps from textbooks. In Proceedings of the 25th acm international on conference on information and knowledge management, pp. 317-326. Cited by: SS1.
* M. Yuen, I. King, and K. Leung (2011)A survey of crowdsourcing systems. 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing, pp. 766-773. Cited by: SS1.

 

Appendix

### Datasets Statistics

We conducted experiments on four datasets with different domains and the statistics are detailed in Table 5.

### Concept Pairwise Features

For MOOC dataset, we used the features as in Pan et al. (2017). While for _Textbook_ dataset, we also extract several pairwise features as in Pan et al. (2017), including _Learning object reference distance_, _Semantic Relatedness_, _Wikipedia reference distance (Wrd)_ and _complexity level distance (Cld)_.

_Semantic Relatedness_ measures the relatedness of two concepts in the semantic space, which can be calculated by: \(sr(c_{i},c_{j})=\frac{1}{2}(1+\frac{v_{c_{i}}.v_{c_{j}}}{||v_{c_{i}}||\cdot|v_ {c_{j}}||})\), where \(v_{c_{i}}\) is the semantic vector of concept \(c_{i}\).

_Wikipedia reference distance_ is defined based on the structure of Wikipedia. Specifically, for a concept \(c_{i}\), we can obtain a set of top \(M\) most related concepts according to the semantic relatedness between two concepts, denoted as \(R_{c_{i}}\). Then, the Wikipedia reference weight can be defined as: \(Wrw(c_{i},c_{j})=\frac{\sum_{e\in R_{c_{i}}}Erw(e,c_{j})\cdot sr(e,c_{i})}{ \sum_{e\in R_{c_{i}}}sr(e,c_{i})}\), where \(Erw(e,c_{i})\) is a binary indicator, in which \(Erw(e,c_{i})=1\) if the Wikipedia article of \(e\) refers to any concept in \(R_{c_{i}}\), and \(Erw(e,c_{i})=0\) otherwise. Thus, the Wikipedia reference distance is defined as: \(Word(c_{i},c_{j})=Wrw(c_{j},c_{i})-Wrw(c_{i},c_{j})\).

_Complexity level distance_ is defined based on the observation that if a concept covers more chapters in a textbook or it survives longer time in the textbook, it is more likely to be a basic concept rather than an advanced one. Besides, the larger the difference between the distribution of two concepts, the more likely they are to have a prerequisite relationship. Therefore, we can measure complexity level distance from two aspects. On the one hand, \(Cld_{Frequency}(c_{i},c_{j})=ava(c_{i})-ava(c_{j})\), where \(ava(c_{i})=\frac{|I(D,c_{i})|}{|D|}\). On the other hand, \(Cld_{Distribution}(c_{i},c_{j})=D(P(c_{i})||P(c_{j}))\) where \(P(c_{i})\) means the distribution of concept \(i\) in \(D\) and \(D(P(c_{i})||P(c_{j}))\) represents the _KL Divergence_ between the distribution of concept \(i\) and concept \(j\).

Moreover, we also extract _head matching feature_ and _ToC distance_ for concept pairs in _Textbook_ dataset. _Head matching feature_ represents whether two concepts have a common head or not, which is obtained by suffix matching. Usually, it implies the existence of prerequisite relation, e.g., _tree_ and _binary tree_. _ToC distance_ measures the distance of concepts in the table of contents in \(D\).

### Labeling Functions

We use the concept pairwise features mentioned before as heuristics labeling functions, and the labeling functions used in _Textbook_ dataset are listed in Figure 7.

The optimal thresholds of the labeling functions can be obtained by grid search with a small amount of training data. Some empirical values are given in Table 6 for the _Textbook_ dataset.

### Influence of Parameters

In order to determine the parameters \(\lambda\) and \(\mu\) in the loss function, we conducted the experiments on Textbook dataset in Physics domain with different \(\lambda\)s and \(\mu\)s, and Figure 8 shows the results. Therefore, we chose \(\lambda=0.2\) and \(\mu=0.1\) in our experiments, which made the best performance.

In addition, we also evaluated our approach with different number of GCN layers (\(L\)), and the result

\begin{table}
\begin{tabular}{|r|l|l|l|l|l|} \hline  & \multicolumn{3}{|c|}{\(Wrd\)} & \multicolumn{3}{|c|}{\(LocT\)} & \multicolumn{3}{|c|}{\(Tc\)} \\ \cline{4-7}  & 0 & 0.1 & 0.2 & 0.5 & 1 \\ \hline \multirow{9}{*}{\(\lambda\)} & 0 & **0.778** & **0.783** & **0.781** & **0.792** & **0.783** \\ \cline{2-7}  & 0.1 & 0.789 & **0.793** & 0.789 & 0.789 & 0.785 \\ \cline{1-1} \cline{2-7}  & 0.2 & **0.796** & **0.797** & 0.786 & **0.779** & **0.782** \\ \cline{1-1} \cline{2-7}  & 0.5 & **0.790** & 0.784 & 0.787 & **0.790** & **0.787** \\ \cline{1-1} \cline{2-7}  & 1 & **0.783** & 0.786 & 0.786 & 0.785 & 0.784 \\ \hline \end{tabular}
\end{table}
Table 6: The thresholds of label functions for the _Textbook_ dataset.

Figure 8: The F-score of CPRL with different \(\lambda\)s and \(\mu\)s on Textbook dataset in Physics domain.

Figure 7: The labeling functions used in _Textbook_ dataset.

 

is shown in Figure 9. From the figure, we can see that the F-score increases gradually and then drops finally. Thus, we chose \(L=2\) in our experiments.

### Impact of Training Set Size

To compare with the previous research, we used 90% positive samples of _LectureBank_ and 70% positive samples of other datasets to train the model.

In order to further explore the ability of our model, we train our model with with different number of training data, and show the result in _Physics_ domain in the _Textbook_ dataset in Figure 10.

It is shown that, when we use more positive samples to train the model, it can reach a higher F1-Score. Besides, it could outperform the baselines with only about 30% positive samples, which implies our model's ability to fully utilize the training samples.

Figure 10: The impact of the size of training set.

\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline \multicolumn{2}{|c|}{Dataset} & \multicolumn{2}{|c|}{\# Learning Object} & \multicolumn{1}{l|}{\# Concepts} & \multicolumn{1}{l|}{\# Pairs(+)} & \multicolumn{1}{l|}{\# Pars(-)} & \multicolumn{1}{l|}{\# Pars(-)} \\ \cline{4-6} \multicolumn{2}{|c|}{} & DS & 102 & 89 & 449 & 673 & 1861 \\ \hline Textbook & CAL & 134 & 84 & 439 & 658 & 1608 \\ \hline  & PHY & 113 & 139 & 623 & 934 & 3745 \\ \hline MOOC & DSA & 148 & 175 & 354 & 1239 & 5285 \\ \hline ML & 271 & 216 & 1446 & 5061 & 1893 \\ \hline LectureBank & 923 & 208 & 913 & 1369 & 3240 \\ \hline University Course & 654 & 407 & 1007 & 1510 & 60 \\ \hline \end{tabular}
\end{table}
Table 5: Statistics of the Datasets. In University Course, each course is described only using its brief introduction, so the average number of tokens in the learning objects is limited.

Figure 9: The F-score of CPRL with different \(L\)s on Textbook dataset in different domains.