SlowGuess/NAACL_2021_OCR huggingface dataset:
As shown in Table 2, CPRL performs better than  $\mathrm{CPRL}_f$  and  $\mathrm{CPRL}_c$  on most of the datasets, so pairwise features and learning object dependencies can both contribute to the performance. Besides, even  $\mathrm{CPRL}_c$  obtains a better performance than the baselines in Table 1, which proves the effectiveness of the heterogeneous graph.
5.6 Effectiveness of Weak Supervision
In order to evaluate our weakly supervised prerequisite relation learning approaches, we compared our two variants  $\mathrm{CPRL}_{lo}$  and  $\mathrm{CPRL}_{dp}$  with CPR-Discover (Liang et al., 2017) in Textbook dataset, and the results are shown in Table 3.
From the table, we find that  $\mathrm{CPRL}_{lo}$  and  $\mathrm{CPRL}_{dp}$  outperform CPR-Recover in all metrics, and  $\mathrm{CPRL}_{dp}$  achieves the best performance. It proves that the knowledge of learning object dependencies can be transferred to learn the concept prerequisite relations through the concept learning object graph. In addition, the data programming with our designed label functions can generate help
Dataset	Metric	SVM	LR	RF	NB	RefD	GAE	VGAE	PREREQ	CPRL
Textbook	DS	P	0.818	0.852	0.755	0.481	0.920	0.446	0.434	0.226	0.795
R	0.632	0.590	0.685	0.897	0.244	0.900	0.570	0.369	0.809
F1	0.713	0.697	0.718	0.626	0.385	0.597	0.493	0.280	0.802
PHY	P	0.806	0.863	0.748	0.399	0.900	0.505	0.460	0.432	0.770
R	0.655	0.588	0.752	0.922	0.409	0.943	0.649	0.423	0.825
F1	0.723	0.699	0.750	0.557	0.562	0.657	0.538	0.427	0.797
CAL	P	0.839	0.860	0.746	0.404	0.950	0.436	0.414	0.391	0.778
R	0.637	0.570	0.715	0.995	0.302	0.900	0.558	0.506	0.798
F1	0.724	0.686	0.730	0.574	0.458	0.587	0.475	0.441	0.788
MOOC	DSA	P	0.705	0.808	0.344	0.613	0.920	0.294	0.269	0.492	0.641
R	0.624	0.168	0.715	0.696	0.252	0.715	0.657	0.462	0.619
F1	0.662	0.278	0.464	0.652	0.396	0.417	0.382	0.476	0.630
ML	P	0.668	0.748	0.375	0.577	0.784	0.293	0.266	0.448	0.800
R	0.577	0.27	0.669	0.623	0.188	0.733	0.647	0.592	0.642
F1	0.619	0.397	0.481	0.599	0.303	0.419	0.377	0.510	0.712
LectureBank	P	0.857	0.744	0.855	0.670	0.666	0.462	0.417	0.590	0.861
R	0.692	0.744	0.681	0.640	0.228	0.811	0.575	0.502	0.858
F1	0.766	0.744	0.758	0.655	0.339	0.589	0.484	0.543	0.860
University Course	P	0.796	0.595	0.739	0.478	0.919	0.450	0.470	0.468	0.689
R	0.635	0.546	0.480	0.649	0.415	0.886	0.694	0.916	0.760
F1	0.707	0.569	0.582	0.550	0.572	0.597	0.560	0.597	0.723
Table 1: The performance of CPRL on four datasets with different domains. Row-wise best results are in bold. CPRL is the model which minimizes  $\mathcal{L}_c + \mu \mathcal{L}_o + \lambda \mathcal{L}_f$ , while the models for LectureBank and University Course only minimize  $\mathcal{L}_c$  since the LOs have no prerequisite relations in them and we cannot extract structural features.
Dataset	Approach	P	R	F1
Textbook	DS	CPR-Recover	0.317	0.577	0.409
CPRLlo	0.425	0.504	0.461
CPRLdp	0.570	0.926	0.706
PHY	CPR-Recover	0.291	0.609	0.394
CPRLlo	0.427	0.487	0.455
CPRLdp	0.503	0.856	0.634
CAL	CPR-Recover	0.447	0.624	0.521
CPRLlo	0.470	0.659	0.551
CPRLdp	0.517	0.803	0.629
ful training data, and achieve comparable performance with the supervised CPRL.
5.7 Verification of Domain Transfer Ability
In order to explore the transfer ability of our model between different domains, we conducted an experiment on Textbook dataset.
Specifically, for CPRL, we firstly trained the model in one domain, and then used the model to predict prerequisite relations between concepts in another domain. While for  $\mathrm{CPRL}_{dp}$ , we obtained the best thresholds such as  $\theta_{LOrd}^{max}$  and  $\theta_{LOrd}^{min}$  in LFs in one domain and then used them to other domains. The results are shown in Table 4
Table 3: Comparison our weakly supervised prerequisite relation learning variants with CPR-Recover.
CPRL	\( \mathrm{{CPRL}}_{dp} \)
DS	PHY	CAL	DS	PHY	CAL
DS	0.802	0.393	0.219	0.706	0.621	0.587
PHY	0.640	0.797	0.430	0.692	0.634	0.616
CAL	0.520	0.438	0.788	0.658	0.633	0.629
Table 4: Domain transfer ability verification experiments for CPRL and  $\mathrm{CPRL}_{dp}$ , where each row and column represent the source and target domain respectively, and the values in the cells are  $\mathrm{F}_1$ -scores.
We observe that (1) F-scores drop severely in
CPRL, so we cannot simply transfer the model across domains due to the difference among concepts and LOs. (2)  $\mathrm{CPRL}_{dp}$  is more stable and can be used in practice since we only need to label a small amount of training data in one domain.
5.8 Effectiveness of Ensemble
Our approach can learn the concept prerequisite relations from one learning object sequence, such as a textbook. While the concepts in textbooks in the same domain are basically the same, so the prerequisite relations among them can be aggregated.
Here, we used a simple majority voting strategy for aggregation, and the results are shown in Figure 6. From the table, we see a significant improvement for the ensemble results.
Figure 6: The ensemble results which are aggregated from six textbooks in each domain.
6.1 Prerequisite Relation Learning
Learning prerequisite relations between concepts has attracted much recent work, and can be classified into three categories: local statistical information based approaches, recovery based approaches and learning based approaches.
As local statistical information, reference distance (Liang et al., 2015a) and cross-entropy (Gordon et al., 2016) were proposed to measure the concept prerequisite relations. CPR-Recover (Liang et al., 2017) is a recovery based approach, which recovers prerequisite relations from course dependencies. The learning based approaches are the most popular. For example, Pan et al. (2017) proposed contextual, structural and semantic features for concept prerequisite relation classification. Roy et al. (2019) applied the pairwise-link LDA model to represent concept, and trained a Siamese network to identify prerequisite relations. Li et al. (2019a) trained variational graph autoencoders to predict concept prerequisite relations. However, these approaches didn't model the multiple and complex relations among concepts and learning resources. Meanwhile, they also need a large set of training data, which is costly to obtain. In order to reduce the amount of training data required, active learning was investigated in (Liang et al., 2018) and (Liang et al., 2019) for concept prerequisite learning.
6.2 Weakly Supervised Learning
One of the most significant bottlenecks for machine learning is the need for a big training data set. Nowadays, it is very promising to use weakly supervised learning techniques to reduce the amount of human intervention needed. For example, distant supervision can produce noisy training data by aligning unlabeled data with an external knowledge base, e.g. relation extraction in (Smirnova and Cudré-Mauroux, 2018). Crowdsourcing (Yuen et al., 2011) and heuristic rules (Sa et al., 2016) can also generate noisy training data.
However, these weakly supervised data is incomplete, inexact and inaccurate, so it is important to integrate multiple noisy labeling data to produce more accuracy data. Data programming (Ratner et al., 2016) provides a simple and unifying framework for the creation of training sets, which expresses weak supervision strategies as labeling functions, and then uses a generative model to denoise the labeling data. Snorkel (Ratner et al., 2019a) is a system built around the data programming paradigm for rapidly creating, modeling, and managing training data. Several works have been explored to use data programming for training data creation. For example, SwellShark (Fries et al., 2017) was proposed for quickly building biomed
ical named entity recognition taggers using lexicons, heuristics, and other forms of weak supervision instead of hand-labeled data. GWASkb with thousands of genotype-phenotype associations was created by using Snorkel in (Kuleshov et al., 2019). Snorkel was also used for chemical reaction relationship extraction (Mallory et al., 2020), discourse structure learning (Badene et al., 2019) and medical entity classification (Fries et al., 2020).
In addition, data programming was further improved under different situations. For example, MeTaL (Ratner et al., 2019b) was proposed for modeling and integrating weak supervision sources with different unknown accuracies, correlations, and granularities. Cross-modal data programming was proposed in (Dunnmon et al., 2020). FlyingSquid (Fu et al., 2020) speeded up weak supervision with triplet methods.
In this paper, we propose a novel concept prerequisite relation learning approach, named CPRL, which combines both concept representation learned from a heterogeneous graph and concept pairwise features. Furthermore, we extend CPRL under weakly supervised settings to make our method more practical. The experiments on four datasets show that our method achieves state-of-the-art performance. In addition, we also prove the effectiveness of our weakly supervised prerequisite relation learning variants.
In future, we plan to design more effective label functions or employ more reliable weakly supervised learning approaches (Li et al., 2019b; Guo et al., 2019) to further improve the performance. Moreover, we will also introduce concept prerequisite relations into curriculum planning and intelligent tutoring applications, e.g. organizing learning resources into a reasonable order and incorporating prerequisite relations into knowledge tracing technologies.
This work is supported by the National Key Research and Development Project of China (No. 2018AAA0101900), the Zhejiang Provincial Natural Science Foundation of China (No. LY17F020015), the Chinese Knowledge Center of Engineering Science and Technology (CKCEST) and MOE Engineering Research Center of Digital Library.
<sup>4</sup>https://www-snorkel.org/
Sonia Badene, Kate Thompson, Jean-Pierre Lorre, and Nicholas Asher. 2019. Data programming for learning discourse structure. In Association for Computational Linguistics (ACL).
P. Chen, Yu Lu, V. Zheng, and Yang Pian. 2018. Prerequisite-driven deep knowledge tracing. 2018 IEEE International Conference on Data Mining (ICDM), pages 39-48.
Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L Rubin, et al. 2020. Cross-modal data programming enables rapid medical machine learning. *Patterns*, page 100019.
Jason A Fries, Ethan Steinberg, Saelig Khattar, Scott L Fleming, Jose Posada, Alison Callahan, and Nigam H Shah. 2020. Trove: Ontology-driven weak supervision for medical entity classification. arXiv preprint arXiv:2008.01972.
Jason Alan Fries, Sen Wu, A. Ratner, and Christopher Re. 2017. Swellshark: A generative model for biomedical named entity recognition without labeled data. ArXiv, abs/1704.06360.
Daniel Y. Fu, M. F. Chen, F. Sala, Sarah Hooper, K. Fatahalian, and Christopher Ré. 2020. Fast and three-rious: Speeding up weak supervision with triplet methods. ArXiv, abs/2002.11955.
Jonathan Gordon, Linhong Zhu, Aram Galstyan, Prem Natarajan, and Gully Burns. 2016. Modeling concept dependencies in a scientific corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 866-875.
Lan-Zhe Guo, Yu-Feng Li, Ming Li, Jinfeng Yi, Bowen Zhou, and Z. Zhou. 2019. Reliable weakly supervised learning: Maximize gain and maintain safety. ArXiv, abs/1904.09743.
V. Kuleshov, J. Ding, C. Vo, B. Hancock, A. Ratner, Yang I. Li, C. Ré, S. Batzoglou, and Michael T. Snyder. 2019. A machine-compiled database of genome-wide association studies. Nature Communications, 10.
Irene Li, Alexander R. Fabbri, Robert R. Tung, and Dragomir R. Radev. 2019a. What should i learn first: Introducing lecturebank for nlp education and prerequisite chain learning. In AAAI.
Y. Li, Lan-Zhe Guo, and Z. Zhou. 2019b. Towards safe weakly supervised learning. IEEE transactions on pattern analysis and machine intelligence.
Chen Liang, Z. Wu, W. Huang, and C. Lee Giles. 2015a. Measuring prerequisite relations among concepts. In EMNLP.
Chen Liang, Zhaohui Wu, Wenyi Huang, and C. Lee Giles. 2015b. Measuring prerequisite relations among concepts. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Chen Liang, Jianbo Ye, Shuting Wang, B. Pursel, and C. Lee Giles. 2018. Investigating active learning for concept prerequisite learning. In AAAI.
Chen Liang, Jianbo Ye, Z. Wu, B. Pursel, and C. Lee Giles. 2017. Recovering concept prerequisite relations from university course dependencies. In AAAI.
Chen Liang, Jianbo Ye, H. Zhao, B. Pursel, and C. Lee Giles. 2019. Active learning of strict partial orders: A case study on concept prerequisite relations. ArXiv, abs/1801.06481.
Hanxiao Liu, Wanli Ma, Yiming Yang, and J. Carbonell. 2016. Learning concept graphs from online educational data. J. Artif. Intell. Res., 55:1059-1090.
Emily K Mallory, Matthieu de Rochemonteix, Alex Ratner, Ambika Acharya, Chris Re, Roselie A Bright, and Russ B Altman. 2020. Extracting chemical reactions from text using snorkel. BMC bioinformatics, 21(1):1-15.
Eric Margolis and Stephen Laurence. 1999. *Concepts: core readings*. Mit Press.
Liangming Pan, C. Li, Juan-Zi Li, and J. Tang. 2017. Prerequisite relation learning for concepts in moocs. In ACL.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
A. Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason Alan Fries, Sen Wu, and C. Ré. 2019a. Snorkel: rapid training data creation with weak supervision. The Vldb Journal, 29:709 - 730.
A. Ratner, B. Hancock, Jared Dunnmon, F. Sala, Shreyash Pandey, and C. Re. 2019b. Training complex models with multi-task weak supervision. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 33:4763-4771.
Alexander J. Ratner, C. D. Sa, Sen Wu, Daniel Selsam, and C. Ré. 2016. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29:3567-3575.
S. Roy, Meghan Madhyastha, Sheril Lawrence, and V. Rajan. 2019. Inferring concept prerequisite relations from online educational resources. In AAAI.
C. D. Sa, A. Ratner, Christopher Ré, Jaeho Shin, Feiran Wang, Sen Wu, and Ce Zhang. 2016. Deepdive: Declarative knowledge base construction. SIGMOD record, 45 1:60-67.
M. Schlichtkrull, Thomas Kipf, P. Bloem, R. V. Berg, Ivan Titov, and M. Welling. 2018. Modeling relational data with graph convolutional networks. In ESWC.
Alisa Smirnova and Philippe Cudre-Mauroux. 2018. Relation extraction using distant supervision. ACM Computing Surveys (CSUR), 51:1-35.
Shuting Wang and Lei Liu. 2016. Prerequisite concept maps extraction for automatic assessment. In WWW.
Shuting Wang, Alexander Ororbia, Zhaohui Wu, Kyle Williams, Chen Liang, Bart Pursel, and C Lee Giles. 2016. Using prerequisites to extract concept maps from textbooks. In Proceedings of the 25th acm international on conference on information and knowledge management, pages 317-326. ACM.
M. Yuen, Irwin King, and Kwong-Sak Leung. 2011. A survey of crowdsourcing systems. 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing, pages 766-773.
A.1 Datasets Statistics
We conducted experiments on four datasets with different domains and the statistics are detailed in Table 5.
A.2 Concept Pairwise Features
For MOOC dataset, we used the features as in (Pan et al., 2017). While for Textbook dataset, we also extract several pairwise features as in (Pan et al., 2017), including Learning object reference distance, Semantic Relatedness, Wikipedia reference distance (Wrd) and complexity level distance (Cld).
Semantic Relatedness measures the relatedness of two concepts in the semantic space, which can be calculated by:  $sr(c_i, c_j) = \frac{1}{2} (1 + \frac{v_{c_i} \cdot v_{c_j}}{||v_{c_i}|| \cdot ||v_{c_j}||})$ , where  $v_{c_i}$  is the semantic vector of concept  $c_i$ .
Wikipedia reference distance is defined based on the structure of Wikipedia. Specifically, for a concept  $c_{i}$ , we can obtain a set of top  $M$  most related concepts according to the semantic relatedness between two concepts, denoted as  $R_{c_i}$ . Then, the Wikipedia reference weight can be defined as:  $Wrw(c_{i},c_{j}) = \frac{\sum_{e\in R_{c_{i}}}Erw(e,c_{j})\cdot sr(e,c_{i})}{\sum_{e\in R_{c_{i}}}sr(e,c_{i})}$ , where  $Erw(e,c_{i})$  is a binary indicator, in which  $Erw(e,c_{i}) = 1$  if the Wikipedia article of  $e$  refers to any concept in  $R_{c_i}$ , and  $Erw(e,c_{i}) = 0$  otherwise. Thus, the Wikipedia reference distance is defined as:  $Wrd(c_{i},c_{j}) = Wrw(c_{j},c_{i}) - Wrw(c_{i},c_{j})$ .
Complexity level distance is defined based on the observation that if a concept covers more chapters in a textbook or it survives longer time in the textbook, it is more likely to be a basic concept rather than an advanced one. Besides, the larger the difference between the distribution of two concepts, the more likely they are to have a prerequisite relationship. Therefore, we can measure complexity level distance from two aspects. On the one hand,  $Cld_{Frequency}(c_i, c_j) = \frac{|I(D, c_i)|}{|D|}$ . On the other hand,  $Cld_{Distribution}(c_i, c_j) = D(P(c_i) || P(c_j))$  where  $P(c_i)$  means the distribution of concept  $i$  in  $D$  and  $D(P(c_i) || P(c_j))$  represents the KL Divergence between the distribution of concept  $i$  and concept  $j$ .
Moreover, we also extract head matching feature and ToC distance for concept pairs in Textbook dataset. Head matching feature represents whether
two concepts have a common head or not, which is obtained by suffix matching. Usually, it implies the existence of prerequisite relation, e.g., tree and binary tree. ToC distance measures the distance of concepts in the table of contents in  $D$ .
A.3 Labeling Functions
We use the concept pairwise features mentioned before as heuristics labeling functions, and the labeling functions used in Textbook dataset are listed in Figure 7.
def LF_LOrd(ci, cj):
if LOrd(ci, cj) < θminLord return 1
elseif LOrd(ci, cj) > θmaxLord return 0
else return -1	def LF_Suffix(ci, cj):
if cj matches cj's suffix string return 1
else return -1
def LF_Wrd(ci, cj):
if Wrd(ci, cj) < θminWrd return 1
elseif Wrd(ci, cj) > θmaxWrd return 0
else return -1	def LF_Cld_Freq(ci, cj):
if Cld_Freq(ci, cj) < θminCld_Freq return 1
elseif Cld_Freq(ci, cj) > θmaxCld_Freq return 0
else return -1
def LF_ToC(ci, cj):
if ToC(ci, cj) < θminToC return 1
elseif ToC(ci, cj) > θmaxToC return 0
else return -1	def LF_Cld_Dist(ci, cj):
if Cld_Dist(ci, cj) < θminCld_Dist return 1
elseif Cld_Dist(ci, cj) > θmaxCld_Dist return 0
else return -1
Figure 7: The labeling functions used in Textbook dataset.
The optimal thresholds of the labeling functions can be obtained by grid search with a small amount of training data. Some empirical values are given in Table 6 for the Textbook dataset.
Wrd	LOrd	ToC	CldFreq	CldDist
θmaxFeature	1.3	-0.2	0.2	0.4	0.3
θminFeature	-1.3	-0.8	-0.2	-0.8	-0.5
A.4 Influence of Parameters
In order to determine the parameters  $\lambda$  and  $\mu$  in the loss function, we conducted the experiments on Textbook dataset in Physics domain with different  $\lambda$ s and  $\mu$ s, and Figure 8 shows the results. Therefore, we chose  $\lambda = 0.2$  and  $\mu = 0.1$  in our experiments, which made the best performance.
Table 6: The thresholds of label functions for the Textbook dataset.
μ
0	0.1	0.2	0.5	1
λ	0	0.778	0.783	0.781	0.792	0.783
0.1	0.789	0.793	0.789	0.789	0.785
0.2	0.796	0.797	0.786	0.779	0.782
0.5	0.790	0.784	0.787	0.790	0.787
1	0.783	0.786	0.786	0.785	0.784
Figure 8: The F-score of CPRL with different  $\lambda$ s and  $\mu$ s on Textbook dataset in Physics domain.
In addition, we also evaluated our approach with different number of GCN layers  $(L)$ , and the result
Dataset	# Learning Object	# Concepts	# Pairs(+)	# Pairs(-)	# Tokens per Learning Object
Textbook	DS	102	89	449	673	1861
CAL	134	84	439	658	1608
PHY	113	139	623	934	3745
MOOC	DSA	148	175	354	1239	5285
ML	271	216	1446	5061	1893
LectureBank	923	208	913	1369	3240
University Course	654	407	1007	1510	60
Table 5: Statistics of the Datasets. In University Course, each course is described only using its brief introduction, so the average number of tokens in the learning objects is limited.
is shown in Figure 9. From the figure, we can see that the F-score increases gradually and then drops finally. Thus, we chose  $L = 2$  in our experiments.
Figure 9: The F-score of CPRL with different  $L_{s}$  on Textbook dataset in different domains.
A.5 Impact of Training Set Size
To compare with the previous research, we used  $90\%$  positive samples of LectureBank and  $70\%$  positive samples of other datasets to train the model.
In order to further explore the ability of our model, we train our model with different number of training data, and show the result in Physics domain in the Textbook dataset in Figure 10.
It is shown that, when we use more positive samples to train the model, it can reach a higher F1-Score. Besides, it could outperform the baselines with only about  $30\%$  positive samples, which implies our model's ability to fully utilize the training samples.
Figure 10: The impact of the size of training set.