Distantly Supervised Course Concept Extraction in MOOCs with Academic Discipline
Year: 2023

Abstract:
With the rapid growth of Massive Open Online Courses (MOOCs), it is expensive and time-consuming to extract high-quality knowledgeable concepts taught in the course by human effort to help learners grasp the essence of the course. In this paper, we propose to automatically extract course concepts using distant supervision to eliminate the heavy work of human annotations, which generates labels by matching them with an easily accessed dictionary. However, this matching process suffers from severe noisy and incomplete annotations because of the limited dictionary and diverse MOOCs. To tackle these challenges, we present a novel three-stage framework DSMOCE, which leverages the power of pretrained language models explicitly and implicitly and employs discipline-embedding models with a self-train strategy based on label generation refinement across different domains. We also provide an expert-labeled dataset spanning 20 academic disciplines. Experimental results demonstrate the superiority of DS-MOCE over the state-of-the-art distantly supervised methods (with 7% absolute F1 score improvement). Code and data are now available at https: //github.com/THU-KEG/MOOC-NER.

heading: 1 Introduction
text: Course concept extraction in Massive Open Online Courses (MOOCs) aims to recognize high-quality knowledge concepts and subject terms taught in the course. Automatically extracting course concepts can help students better understand knowledgeable concepts of the course and reduce the burden of teacher workloads (Butt and Lance, 2005). It is a core task in course content analysis and MOOC knowledge graph construction, which is a fundamental step to building AI-driven MOOC systems with various downstream applications such as course recommendation and question answering
∗Corresponding author.
(Song et al., 2021). However, MOOCs’ explosive growth, like the number of online courses, which grew from 13.5k in 2019 to 19.4k in 20211, makes it expensive and tedious to annotate course corpus manually. Therefore, there is a clear need to achieve automatically consistent and accurate course concept extraction in MOOCs to eliminate the heavy work of human annotations.
Early works for course concept extraction in MOOCs include graph propagation (Pan et al., 2017; Lu et al., 2019) and statistical ranking methods (Wu et al., 2022; Albahr et al., 2021). Recently, distant supervision has been proposed for the automatic generation of training labels. As shown in Figure 1, the labeling procedure matches the tokens in the course corpus with concepts in an easily accessed dictionary. However, this matching process suffers from two challenges: (1) noisy annotation where a mention can be low-quality (i.e., the mention of ‘plant’ and ‘species’ of the first instance) or unrelated to the field of the course (i.e., the mention of ‘identification’ from Chemistry but this instance is about Agriculture); and (2) incomplete annotation where a mention can be matched partially (i.e., the mention of ‘cerebrovascular disease’ and ‘neurological event’ of the second instance) or missed completely (i.e., the mention of ‘life-threatening’ )
1https://www.classcentral.com/report/
moocs-stats-and-trends-2021/
13044
due to the limited coverage of dictionary. Several training paradigms have been employed in Distantly Supervised NER (DS-NER), such as reinforcement learning (Yang et al., 2018) and bagging active learning (Lee et al., 2016) to address the noise annotation; concept expansion (Yu et al., 2019, 2020b; Wang et al., 2019) and positiveunlabeled learning (Peng et al., 2019; Zhou et al., 2022) to address the incomplete challenge. Unfortunately, the previous studies assume a high precision and reasonable recall after distantly supervised label generation. However, severe low-precision and low-recall are reported in MOOCs according to pioneer experiments and comparison with other benchmarks in Table 1. It indicates that there are more noise and incomplete annotations in MOOCs, which significantly hurt following model training performance, thus making the advanced DS-NER approaches fail to cope with the two challenges.
Our analysis yields that the limited dictionary and diverse MOOCs lead to more noise and incomplete annotations2. First, the dictionary lacks sufficiently extensive coverage because of MOOCs’ rapid growth and missing criteria. Therefore, the out-of-dictionary, low-quality concepts will consequently render more course concepts unmatched and false-positive noisy annotations during matching. Second, MOOCs can span 20 or even more academic disciplines (Mohd Salamon et al., 2016), producing unrelated noisy annotations across different open domains. Additionally, the uneven concept distribution and semantic differences among varied disciplines are different, imposing significant challenges to training an effective and accurate model.
To address the two challenges, we propose a novel three-stage framework DS-MOCE to distantly supervised extract course concepts in MOOCs across different domains. Our framework
2For more case explanations, see Appendix A.1
consists of (1) Discipline-aware Dictionary Empowerment which employs prompt-based learning to explicitly generate concept distribution over diverse MOOC domains and implicitly enhance the dictionary’s limited capability; (2) Distant Supervision Refinement which removes unrelated noise with much higher precision annotations for model training; and (3) Discipline-embedding Models with Self-training to deal with noise iteratively while finding incomplete mentions based on semantic knowledge and syntactic information of pre-trained language models (PLMs) and positiveunlabeled learning (PUL).
For evaluation, we provide an expert-labeled dataset spanning 20 academic disciplines, which contains 522 expert-annotated sentences from 17 courses with 15, 375 course concepts.
Our contributions include 1) a novel three-stage framework to distantly supervised extract course concepts in MOOCs across different domains to eliminate the heavy work of human annotations; 2) a distant supervision refinement method to discard unrelated field noise and discipline-embedding models with a self-training strategy to remove noise iteratively and address the incomplete challenge based on PUL; 3) an expert-labeled dataset with the excellent performance of our DS-MOCE framework over existing distantly supervised methods, with one implementation report of 7% absolute F1 score improvement.

heading: 2 Problem Formulation
text: Following Pan et al. (2017), we give some necessary definitions and then formulate the problem of distantly supervised course concept extraction.
A course corpus is composed of n courses from different academic disciplines, denoted as D = {Ci}ni=1, where Ci is one course. Each course Ci = {Si, Fi} consists of two parts, where Fi = [fi1, . . . , fiki ] is course related academic disciplines, and Si = {vij}j=1,...,ni is composed of ni course video subtitles, where vij stands for the j-th video subtitles. Finally, we get all academic fields F = {fi}i=1,...,k related to course corpus D, so k is the number of academic disciplines.
A dictionary T = {Ti}i=1,...,k, where Ti = {cij}j=1,...,mi is composed of mi course concepts cij in academic disciplines fi.
Distantly Supervised Course Concept Extraction in MOOCs is formally defined as follows. Given the course corpus D and dictionary T , for
each course Ci in D, the objective is to extract Fi discipline-related and high-quality course concepts from video subtitles Si.

heading: 3 The DS-MOCE Framework
text: Considering the limited dictionary and diverse MOOCs, it is natural not to ignore the academic discipline characteristics for distantly supervised course concept extraction in MOOCs. As shown in Figure 2, we propose a three-stage framework DS-MOCE, which includes 1) Discipline-aware Dictionary Empowerment to transfer the power of PLMs to the dictionary; 2) Distant Supervision Refinement which considers academic disciplines to tackle the unrelated field noise explicitly; and 3) Discipline-embedding Models to fully exploit the power of PLMs with concept distribution to implicitly handle the noise and incomplete challenges, which then can be integrated with two advanced DS-NER implementations. One employs a co-training strategy to deal with the noise iteratively, denoted as DS-MOCE(co). The other employs PUL to deal with the incomplete problem, denoted as DS-MOCE(PUL).

heading: 3.1 Discipline-aware Dictionary Empowerment
text: Before distant supervision, we design a preceding step to conduct discipline classification for each concept in the dictionary with prompt-based learning (Liu et al., 2021b), hoping to transfer semantic knowledge from the language model (LM) to the
dictionary. Formally, taking the input of each concept ci in the dictionary T = {ci}i=1,...,m, the classification returns a ranked list of related disciplines Fci ⊂ F and outputs pj for fj ∈ F = {fj}j=1,...,k to indicate its likelihood to be related to fj discipline:
pj(x ′ ) = LM(ffill(x ′ , fj); θ) (1)
where x ′ = fprompt(ci) is a prompt with the concept ci filled template slot [concept], and function ffill(x ′ , fj) fills in the slot [MASK] with the potential answer fj . For example (Figure 2), in one case of discipline classification where ci =“identification”, the template is designed as “[MASK] including [concept]”. Then x ′ would become “ [MASK] including identification”, and we calculate the probability pj for each fj ∈ F = {fj}j=1,...,k according to Eq. (1).
Additionally, creating manually crafted templates takes time and experience and is possibly sub-optimal, failing to retrieve facts that the LM does know (Jiang et al., 2020). Inspired by relation extraction methods (Hearst, 1992), hand-built Hearst patterns such as “Y including X (Cities including Madrid or Barcelona)”, we create eight more lexico-syntactic templates to improve and stabilize the classification performance3.

heading: 3.2 Distant Supervision Refinement
text: With a discipline-aware dictionary, we can generate distantly supervised labels by matching with the
3See more templates in Appendix A.3
Algorithm 1 Dic-Matching with Academic Discipline Input: Course Corpus D = {Ci}ni=1, where Ci = {Si, Fi}; Dictionary T = {ci}mi=1; K number of Top-K;
for each course Ci = {Si, Fi} in D do for each video subtitles vij in Si = {vij} do Xm = [x1, x2, ...xN ]← tokenize vij BIO tag potential concept using POS,RE: Dpotm = [d1, d2, ..., dN ] for potential concept pci tagged in Dpotm do
if pci ∈ T and (top-K fields of pci) ∩Fi ̸= ∅ then
Tag BI to pci tokens else
Tag O to pci tokens end if
end for Academic discipline related: Dm = [d1, d2, ..., dN ]
end for end for
Output: Distantly supervised labels {(Xm, Dm)}Mm=1
top-K-related disciplines4 in the ranked list from Eq. (1). This way, we can have a much higher precision by explicitly removing unrelated noisy annotations.
The entire Dic-Matching with academic discipline process is described in Algorithm 1. The input subtitles are first tokenized and annotated with part-of-speech (POS) tags. Next, we employ the regular expression (RE) by only keeping nouns to handle the noise challenge and mining more noun phrases to address the incomplete challenge, as illustrated in Appendix A.2. Finally filtering out unrelated disciplines, we have {(Xm, Dm)}Mm=1 as distantly supervised data, where Xm = [x1, x2, . . . , xN ], composed of N tokens, Dm = [d1, d2, . . . , dN ], based on the BIO schema (Li et al., 2012). Specifically, the first token of a concept mention is labeled as B; other tokens inside that concept mention are labeled as I; the non-concept tokens are labeled as O.

heading: 3.3 Discipline Embedding Self-training
text: We adapt the PLMs to the sequence labeling tasks with the distant labels and self-training approach
4K is experimentally set to 2.
to iteratively deal with the noisy annotations meantime training a new integrated embedding based on the concept discipline distribution to implicitly enhance model discipline-aware capability. Then we can employ other advanced DS-NER approaches, such as co-training and PUL.

heading: 3.3.1 Discipline Embedding Model
text: At the pre-process of the dictionary, for each concept ci, we calculate its distribution in all academic disciplines according to Eq. (1), denoted as Uci = [p1, p2, . . . , p|F |]. To introduce the discipline feature, each token xj of the input Xm = [x1, x2, . . . , xN ] is encoded as Ej by adding its discipline distribution to BERT word embedding if xj is labeled as belonging to one of concept ci in the dictionary:
Ej =
{ Encoder(xj) + UciW xj ∈ ci
Encoder(xj) xj /∈ any ci (2)
Where dh is a hidden dimension of the encoder, and W ∈ R|F |·dh is trainable parameters. We use BERT (Devlin et al., 2018) as our Encoder to learn the sequence representation. This way, external academic field features are integrated into the embedding, enhancing model discipline-aware capability (Figure 3).
Straightforward, we use f(·; θ) to denote our model parameterized by θ, which is a token-wise classifier on top of a pre-trained BERT. fn,c(·; ·) denotes the probability of the n-th token in Xm belonging to the c-th class from the BIO schema. The model will be learned by minimizing the cross
entropy loss L(θ) over {(Xm, Dm)}Mm=1:
L(θ) = 1 M 1 N
M∑
m=1
N∑
n=1
− log fn,dm,n(Xm; θ)
(3)

heading: 3.3.2 Teacher-Student Self-training
text: Following Liang et al. (2020); Meng et al. (2021); Zhang et al. (2021b); Liu et al. (2021a), we employ the teacher-student self-training strategy because it selects high-confidence and consistent predictions as pseudo labels from the teacher model and then uses them to guide the training of the student model, which removes the noisy labels iteratively. We adopt two advanced self-training DS-NER approaches. One is based on Zhang et al. (2021b), aimed at high-precision performance, which jointly trains two teacher-student networks and confirms its effectiveness and robustness in dealing with the label noise.
The other is inspired by Peng et al. (2019), aimed at high-recall performance, which introduces PUL as it can unbiasedly and consistently estimate the task loss. We apply the binary label assignment mechanism for using this algorithm by mapping “O” to 0 and “B”, “I” to 1. Finally, we get positive set D+m = [dm,1, ..., dm,|D+|] and unlabeled set D u m = [dm,1, ..., dm,|Du|] from the original distantly supervised labels Dm = [dm,1, dm,2, ..., dm,N ]. The PUL training loss is defined by:
L̂(θ) = γ ·πpL̂+p (θ)+max{0, L̂−u (θ)−πpL̂−p (θ)} (4) where
L̂+p (θ) = 1
M
1 |D+| M∑
m=1
|D+|∑
d=1
− log fd,1(Xm; θ)
L̂−p (θ) = 1− L̂+p (θ)
L̂−u (θ) = 1
M
1 |Du| M∑
m=1
|Du|∑
d=1
− log fd,0(Xm; θ)
and πp is the ratio of positive concept words within Du. A class weight γ is introduced to deal with the class imbalance problem (πp is very small). As a whole, in this training strategy, the parameters of the student model θ∗ are learned by the combination of the cross entropy loss (Eq. (3)) and the PUL loss (Eq. (4)):
θ∗ = argmin θ (L(θ) + β · L̂(θ)) (5)
where a parameter β is introduced to balance these two loss functions.

heading: 4 Experiments

heading: 4.1 Experimental Settings

heading: 4.1.1 Dataset
text: We provide a new dataset spanning 20 academic disciplines, which can be used to benchmark distantly supervised methods for course concept extraction task in MOOCs. Based on MOOCCube (Yu et al., 2020a), the input includes two parts: (1) an expert-checked dictionary with over 100k course concepts from CNCTST5, and (2) a subtitle corpus from 315 courses with 167, 496 unlabeled character sequences on average per course. The test set contains 522 expert-annotated sentences from 17 courses with 15, 375 discipline-related course concepts. All data is from XuetangX6, one of the largest MOOC websites in China, so the dataset is in the Chinese language. More details of the dataset can be found in Appendix A.5.

heading: 4.1.2 Baselines and Evaluation Metrics
text: We compare our method with several competitive baselines from three aspects and use Precision (P), Recall (R), and F1 score as the evaluation metrics. Dic-Matching Methods. We construct different Dic-Matching (DM) methods for comparison, including (i) DM: it is a simple string matching with a greedy search algorithm to find the longest matching strings in sentences; (ii) DM(AD-LM) : it adopts the matching strategy proposed in Algorithm 1; (iii) DM(AD-human): it is a variation of AD-LM that replaces the discipline classification results from GLM with ones from CNCTST expert annotations. Fully-supervised Method. We also construct fullysupervised methods for comparison. FLAT (Li et al., 2020): For Chinese NER, it converts the lattice structure into a flat structure consisting of spans to handle word segmentation in the Chinese language. Distantly-supervised Methods. The state-of-theart self-training DS-NER methods are as follows. (i) SCDL (Zhang et al., 2021b): It explores more helpful information from the mislabeled data by a devised co-training paradigm based on self-training. (ii) RoSTER (Meng et al., 2021): A self-training method that uses contextualized augmentations created by pre-trained language models to improve the model’s generalization ability. (iii) BOND(Liang
5http://www.cnterm.cn/ 6https://next.xuetangx.com/
et al., 2020): A two-stage framework that trains a RoBERTa model on distantly-labeled data with early stopping in the first stage and improves the model fitting with a teacher-student framework to iteratively self-train the model in the second stage.

heading: 4.1.3 Implementation Details
text: For concept classification task, we apply the General Language Model (Du et al., 2022), which is capable of handling variable-length blank. We use the pre-trained BERT-wwm-ext model (Cui et al., 2020) as the backbone for our method and other distantly-supervised baselines. The maximum sequence length of our dataset is set to be 512 tokens. The max training epoch is 30, and the batch size is 4. We use Adam (Kingma and Ba, 2014) as the optimizer, and the learning rate is 10−5. The confidence threshold γ is 0.9 for the co-training strategy while 0.7 for the PUL strategy with the purpose of high-recall performance. More implementation details can be found in Appendix A.4.

heading: 4.2 Experimental Results
text: Overall Results. Table 2 shows the overall results of different methods on our MOOCs test set. Our DS-MOCE framework with two self-training strategies achieves the best performance among distantlysupervised methods. Specifically, (1) the proposed Dic-Matching method with academic discipline refines the distant labels by improving precision significantly; (2) DS-MOCE(co) reports a high precision with 7% absolute F1 score improvement over the best performing baseline model BOND, demonstrating the superiority of our proposed DicMatching with academic discipline method and self-training approach; (3) DS-MOCE(PUL) consistently outperforms other distantly-supervised methods with a higher recall and reasonable precision, showing more robustness to the issue of incomplete labeling.
As we have discussed, the Dic-Matching method suffers from extremely low precision and low recall in MOOCs for its diversity, which dramatically hurts the performance of the distantly supervised baselines and limits the model fitting ability in fully supervision. Discipline Classification Results. Through the comparison of DM(AD-human) and DM(ADLM) in Table 2, we find that the academic discipline classification result from GLM outperforms that from expert annotations during Dic-Matching, showing the robustness of our designed classifica-
tion step for transferring PLMs knowledge to the dictionary. On the contrary, human annotations suffer from missing, incorrect, and out-of-date classifications.
Moreover, we evaluate the pre-process concept classification task using the Mean Average Precision (MAP), a metric in information retrieval for evaluating ranked lists. Table 3 shows some example results using different templates. (See more templates in Appendix A.3). The first example is based on experience and the rest are Hearst patterns, showing better and more stable performance. We finally use the best-performing template in the following parts.
Ablation Study. To evaluate the influence of each component, we conduct the following ablation study for further exploration by removing one component at a time: (1) do not adopt Alg. 1 and use the Dic-Matching method when generating distantly supervised labels. (2) only use BERT Encoder without adding academic discipline embedding; (3) do not perform self-training; (4) do not perform co-training for DS-MOCE(co) and only use crossentropy loss in Eq. (3) without adding PUL loss in Eq. (4) for DS-MOCE(PUL). The results are shown in Table 4. It can be seen that w/o Alg.
1 refinement and w/o embedding for both strategies lead to worse performance than the full model, confirming the necessity of considering discipline features in MOOCs. Removing the self-training or co-training component also reduces performance, showing its importance in DS-MOCE(co) of denoising learning because false-negative labels can be explored via peer model or another network iteratively. Without PUL, the recall value decreases sharply, which validates the effectiveness of introducing PUL to tackle the incomplete challenge. Parameter Study. Before discussing the parameter study of πp defined in Eq. (4), we first calculate the true value of πp = (# of concept words) / (# of words of the training set) in our dataset, with a 0.1002 result. Then we train the proposed model DS-MOCE(PUL) with different estimated πp, and evaluate its performance on the test set. From Figure 4(a), we can see that although the highest recall is achieved by setting πp = 0.1, most closely to the true value, the variation of results across different πp is relatively tiny. This motivates us to use a proper estimated value of πp to deal with the diversity of MOOCs where courses from different disciplines have incongruous and unknown πp values. Therefore, we set πp = 0.01 for DSMOCE(PUL) to achieve a high recall and a higher
F1 score. Besides, we set β = 1 in Eq. (5) throughout our experiments without further illustration, according to Figure 4(b).
Different Discipline Analysis. We analyze that the diversity of MOOCs academic disciplines accounts for more noisy and incomplete annotations in distantly supervised MOOCs. As a result, we select some courses from different disciplines and use DS-MOCE(co) framework to perform prediction on these courses. From Table 5, we discover that (1) the intensive and appropriate terminological concepts in formal and applied science, such as CS and Mathematics, bootstrap the model with its high-recall predictions that benefit the model’s generalization; (2) the sparse distribution (low concept ratio) in Humanities and Social Science Philosophy makes it uncertain about selecting tokens to train a robust model; (3) excessive terminological concepts (nested structure and long formulas) in some professions, such as Chemistry and Medicine, amplify the issue of incomplete annotation, where several concept extraction methods have been developed specifically to handle this problem (Wang et al., 2021; Fu et al., 2020).
Case Study. Finally, we perform a case study to understand the advantage of DS-MOCE(co) with a concrete example in Table 6.
Besides, we select another case study to demonstrate why DC-MOCE(PUL) is provided in our work. The extremely high precision accounts for the F1 score increment of the DS-MOCE(co) framework, but low recall leads to more missing tokens. Consequently, aimed at improving recall, we design the DC-MOCE(PUL) as an alternative option by sacrificing the precision properly. Finally, DS-MOCE(co) with high-precision and DSMOCE(PUL) with high-recall can be applied in different real-world scenarios.
To help our model’s behaviors be understood and applied to real-world applications, we suggest: (1) For DS-MOCE(co) with high-precision performance, it is better to apply it to the downstream tasks that acquire accurate concepts but ignore the coverage, such as course concept recommendation and AI-driven robot assistant; (2) For DSMOCE(PUL) with high-recall performance, it is better to apply it to scenarios where there is surplus human labor available for corrections, and where there is a need to recall as many course concepts as possible, such as in MOOC knowledge graph construction.

heading: 5 Related Work
text: Distantly Supervised NER. Our work is more closely related to distantly supervised NER, where the primary research focuses on coping with the
noise and incomplete annotations problem. Several new training paradigms have been proposed along the denoising line, such as Reinforcement learning (Yang et al., 2018), AutoNER (Shang et al., 2018) with a new tagging scheme “tie or break”, Hypergeometric Learning (Zhang et al., 2021a) and Bagging-based active learning with negative sampling (Lee et al., 2016). Along the incomplete mining line, a direct solution is concept expansion (Yu et al., 2019, 2020b; Wang et al., 2019), which finds new candidates and ranks them to expand the set based on the seed set with figurative elements. AdaPU (Peng et al., 2019) and Conf-MPU (Zhou et al., 2022) are developed to address the incomplete challenge by formulating the task as a positive-unlabeled learning problem. Besides, many studies (Yang et al., 2018; Shang et al., 2018) attempt to modify the standard CRF to partial annotation CRF to consider all possible labels for unlabeled tokens. However, these works do not work well in MOOCs where severe low-precision and low-recall problems have been reported previously. Course Concept Extraction. Our study is also relevant to course concept extraction, which is related to keyphrase extraction (Hasan and Ng, 2014) in the information retrieval domain. The well-known methods such as tf-idf (Ramos et al., 2003), cooccurrence (Mihalcea and Tarau, 2004), and PositionRank (Florescu and Caragea, 2017) are frequently used in unsupervised automatic keyphrase extraction. However, the low-frequency (i.e., ap-
pearing only once or twice in the subtitles) feature of keyphrases in MOOCs makes statistical information less useful (Pan et al., 2017). Therefore, Pan et al. (2017) develop a graph-based propagation algorithm, and Albahr et al. (2021) design a novel unsupervised cluster-based approach to address the low-frequency problem in keyphrases extraction from MOOCs.
DS-MOCE also benefits from distributed representations of words, namely word embeddings (Mikolov et al., 2013) to learn academic discipline representations for concepts from the dictionary, which has been employed in Wang et al. (2018); Wu et al. (2022).

heading: 6 Conclusion and Future Work
text: In this paper, we attribute the increased noise and incomplete challenges of distantly supervised course concept extraction in MOOCs to the limited dictionary and diverse MOOCs. To tackle these challenges, we propose a three-stage framework DS-MOCE, which handles the unrelated noise through Dic-Matching refinement and disciplineembedding model training, and leverages the power of pre-trained language models for dictionary empowerment and incomplete mentions mining. We also provide an expert-labeled dataset spanning 20 academic disciplines. Experimental results show that DS-MOCE is highly effective, outperforming the state-of-the-art distantly supervised methods. Although achieving significant improvement, course concept extraction in MOOCs is still nontrivial. In the future, we plan to design a more robust training method to jointly deal with severe noisy and incomplete issues and apply it to other real-world open domains.

heading: 7 Ethic Consideration
text: We provide an expert-labeled dataset spanning 20 academic disciplines, which contains 522 expertannotated sentences from 17 courses with 15, 375 course concepts. We define the 20 academic disciplines according to Discipline Doctor and Master Degree and postgraduate training, the professional directory issued by the Ministry of Education of the People’s Republic of China7. The course corpus is collected from an open-source database MOOCCube (Yu et al., 2020a)8. The dictionary is col-
7http://www.moe.gov.cn/srcsite/A22/
moe_833/200512/t20051223_88437.html 8http://moocdata.cn/data/MOOCCube
lected from CNCTST9 with expert-checked 100k course concepts. The annotated sentences in the test set are from an expert from the Education Department in our university, which may have limitations but missing criteria in MOOCs means that we can accept this human bias. The annotator is a voluntary participant who was aware of any risks of harm associated with their participation and had given their informed consent. To lighten the burden of the annotator, we first use unsupervised methods, such as tf-idf, to give a rough annotation result for each course, randomly selected from XuetangX10. Then the annotator marks mentions of high-quality course concepts based on that. More details of the dataset can be found in Appendix A.5.

heading: 8 Limitations
text: Although we conducted extensive experiments, the exploration scope of this work has some limitations: (1) All data is from one of the largest MOOC websites in China, so the dataset is in the Chinese language, which limits the linguistic features covered in our analyses. We will add comprehensive corpora from other MOOC platforms with various languages such as English, Japanese, French, and so on to enhance the availability and coverage of our dataset. (2) We present two models with highprecision and high-recall behaviors. The severe noisy and incomplete issues could not be coped with simply by combining two technical methods (i.e. co-training and PUL). A more robust training method should be proposed to jointly achieve better overall performance. We encourage future works to address these limitations and get more comprehensive analysis results.

heading: Acknowledgements
text: This work is supported by a grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003). This work is also supported by the NSFC Youth Project (62006136).

heading: A Appendix
text: A.1 Case Explanations of Limited Dictionary & Diverse MOOCs
The limited dictionary. It is expensive and timeconsuming to expand or tailor a dictionary to every specific domain because of MOOCs rapid growth and criteria missing, ending up with outof-dictionary and low-quality concept problems. We categorize two types of low-quality course concepts in the dictionary. The first type is not specific enough, missing prefixes and suffixes. The second type is unigram concepts with many extended meanings, which end up with false-positive labels.
The diverse MOOCs. Compared with other benchmark datasets, Table 1 illustrates that the number of concept types is inversely proportional to the distantly matching performance. As shown in Table 1, where BC5CDR (Shang et al., 2018) is restricted to the biomedical domain, a domainspecific dictionary with a corpus-aware dictionary tailoring method can achieve higher precision and reasonable recall. MOOCs can span 20 or even more academic disciplines. During label generation, unrelated concept annotations would produce more false-positive noise.
Besides, the characteristics among varied disciplines are different. Most of the time, the concept distribution in humanities and social science is sparse, while in formal science is dense. According to the statistics, the concept proportion of contents in one psychology course is 0.0163, whereas in one computer science course is 0.1. The uneven concept distribution may lead to a matching bias toward the concept-intensive academic discipline. Furthermore, in Chinese, homonyms are more likely to appear in humanities and social science, where words share the same characters and pronunciations but have different meanings. For example, in Philosophy, the debate of right and wrong makes "right" annotations correct. However, "right", as a high-frequency phrase, is easily annotated in other contexts, producing false-positive labels. The ambiguity of homonyms makes it difficult to extract the correct meaning concept in these domains.
A.2 Regular expression in Distant Supervision Refinement
During distant supervision refinement, we employ the following regular expression, introduced by
Luo et al.11, only keeping nouns and noun phrases to remove the apparent incorrect POS noise and mining more incomplete annotations by connecting two nouns with @.
(@(([av]?n[rstz]?)|l|a|v)) ∗ (@(([av]?n[rstz]?)|l))
A.3 Templates Results All eight template results based on Hearst Pattern are shown in Table 7.
A.4 Baselines Settings For fully-supervised methods, we use 3/4 of the test set for model training and the rest for evaluation. Fairly, the following distantly supervised methods use the distantly-labeled training set obtained from Dic-Matching(AD-GLM).
• SCDL. We use the authors’ released code: https://github.com/ AIRobotZhang/SCDL. Because our test set is in the Chinese language, we change the basic model to the same pre-trained BERT-wwm-ext model with our method. We train the model for 30 epochs with a batch size of 8. The other hyperparameters are set to default values.
• RoSTER. We follow the officially released implementation from the authors: https:// github.com/yumeng5/RoSTER. Similarly, we modify the backbone model from RoBERTa-base to the same one with our method. The epoch number is set to 3, 3, and 7 for noise-robust training, ensemble model training, and self-training, respectively. We train five models with 2000 intervals of noiserobust training and 1000 of self-training with
11a parent with number CN201911140653.9
a batch size of 8. The rest hyperparameters are the same as the default values.
• BOND. We use the authors’ released code: https://github.com/cliang1453/ BOND/. Also, we choose the pre-trained BERT-wwm-ext model as the backbone model. The early stopping step of the student model is set to 100k. The other hyperparameters are set to default values.
A.5 Dataset Statistic A.5.1 Test Set Annotation We select 17 courses from the course corpus spanning these disciplines and ask an expert to annotate each sentence as our test sets. The more detailed statistics are shown in Table 8. During analysis of different discipline, we choose Introduction to the Classical Works of Chinese Philosophy for Philosophy; Machine Learning for Big Data for Computer Sciences(CS); Finite Element Analysis and Applications for Mathematics; Pathology for Medicine.
A.5.2 Dictionary Information We created our dictionary with 20 academic disciplines by developing the resource from MOOCCube(Yu et al., 2020a) based on its concept taxonomy from CNCTST. Then according to Discipline Doctor and Master Degree and postgraduate training, the professional directory issued by the Ministry of Education of the People’s Republic of China12, we show the prescribed 20 academic disciplines and the distribution of concepts that filtered and mapped from MOOCCube in Table 9.
A.6 Case studies in English To make the case study more vivid, we highlight the corresponding English word in different colors in Table 10. Considering contextual differences
12http://www.moe.gov.cn/srcsite/A22/
moe_833/200512/t20051223_88437.html
between the two languages, there are some missing tokens in English.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We encourage future works to address these limitations and get more comprehensive analysis results