Auto-req: Automatic detection of pre-requisite dependencies between academic videos
Year: 2023

Abstract:
Online learning platforms offer a wealth of educational material, but as the amount of content on these platforms grows, students may struggle to determine the most efficient order in which to cover the material to achieve a particular learning objective. In this paper, we propose a feature-based method for identifying pre-requisite dependencies between academic videos. Our approach involves using a transcript engine with a language model to transcribe domain-specific terms and then extracting novel similarity-based features to determine pre-requisite dependencies between video transcripts. This approach succeeds due to the development of a novel corpus of K-12 academic text, which was created using a proposed feature-based document parser. We evaluate our method on hand-annotated datasets for transcript extraction, video pre-requisites determination, and textbook parsing, which we have released. Our method for pre-requisite edge determination shows significant improvement (+4.7%-10.24% F1-score) compared to existing methods.

heading: 1 Introduction
text: In many online learning platforms, academic videos that cover specific concepts are included in the curriculum. These videos cover certain "academic concepts," which are key ideas that are conveyed in the learning material. These fine-grained concepts aid students in understanding the learning content more effectively and achieving their core learning objectives. The prerequisite dependencies between these concepts, which pertain to the order in which they should be covered, are crucial for both educators and learners. They assist educators in curriculum planning and creating better learning pathways for students. With the increasing reliance on online learning platforms, there is a vast amount of academic content that requires proper organization into dependency graphs to aid in indexing
for smart search capabilities and providing defined learning paths for students. Research has shown that organizing content in this manner has significant benefits for learning, even in offline settings. A meta-analysis of 55 studies involving over 5,000 participants found that students who use concept maps for their daily studies were able to learn more in the same amount of time (Nesbit and Adesope, 2006).
Although learning content is organized in textbooks and MOOCs, the creation of dependency graphs for academic videos serves to extend this organization, enabling us to identify only the relevant and required content for a specific learning objective based on prerequisite relationships. Such a system allows us to recommend personalized learning pathways to users, fostering efficient and effective coverage of specific academic concepts. This tailored approach enhances students’ educational experiences and promotes better understanding of the subject matter. Moreover, it saves time for the student by ensuring that all required concepts or skills are covered before viewing content related to the desired academic concept. In this study, we propose a two-stage methodology for identifying prerequisite relationships among academic videos. The process begins with transcribing videos utilizing a speech-to-text model, combined with a language model specifically trained on a K-12 domain corpus. Subsequently, we extract innovative similarity-based features from these transcripts to determine the prerequisite connections.
The features employed in our study have been meticulously designed with the guidance of expert educators in the respective domain. These features utilize several similarity-based factors between two videos to identify pre-requisite dependencies. These factors include similarities between titles, content, and taxonomy. We also use keyphrase extraction algorithms to identify the topics covered in the transcripts and then compare the similarity
539
between them. Our work introduces the use of extracted keyphrase-based similarity for this task, contributing a novel approach to this research domain. Once the features are extracted we use models such as LGBM (Ke et al., 2017), Random Forrest (Breiman, 2001), and ExtraTrees (Geurts et al., 2006) to predict prerequisite dependencies. Our approach for identifying prerequisite relationships among educational videos demonstrates superior performance compared to existing benchmarks.
To evaluate our pipeline, we used a hand-labeled dataset of K-12 academic videos with annotated pre-requisite edges. We introduced a novel featurebased PDF document parser that extracts a K-12 text corpus which ensures correct transcription of domain-specific terminologies and extraction of accurate semantic similarity-based features that take into account the contextual meaning of such terms. This tool extracts a hierarchical and well-organized corpus of K-12 academic text from core curriculum textbooks, strengthening the resilience and effectiveness of both pipeline stages when addressing technical vocabulary.
The primary contributions of our research can be enumerated as follows:
• A method to extract transcripts from academic videos by using a text-to-speech model such as Wav2Vec2 (Baevski et al., 2020) along with a language model built from a corpus of K12 academic content.
• A novel set of similarity-based features that can predict prerequisite edges between academic videos.
• A method to parse academic PDF textbooks using novel layout-based features to extract hierarchical learning taxonomies and content.
• We introduce the following datasets:
– A hand-labeled dataset of over 2797 prerequisite edges between academic videos annotated by domain expert teachers.
– Extracted transcripts using various methods and ground truth transcripts for a randomly selected subset of videos available in the public domain.
– Hand-labelled textbooks parsed with all section headers, text body, and chapter names, as well as an object detection textbook page image dataset, with bounding boxes labeled on all instances of section headers.
The datasets are released at https://bit.ly/ 412WkQp and a demo for the generated prerequisite edges can be found at https://bit. ly/3VrzMYL.

heading: 2 Current work
text: Our end-to-end pipeline to identify prerequisite dependencies between academic videos is novel. However, the sub-problems, such as transcript extraction, prerequisite edge detection, and parsing textbook PDFs have been well-studied in the literature.

heading: 2.1 Transcript extraction
text: Speech-to-Text Recognition (STR) technology is widely used in the online learning domain. Previous studies have shown that students, especially those with learning disabilities, can greatly benefit from transcripts of learning content (Leibold and Buss, 2019). With an increase in the availability of large-scale datasets and newer deep-learning algorithms, many different methods have shown great performance on this task. End-to-end sequenceto-sequence (S2S) modeling using RNN-based, Transformer-based, and Conformer based models are often used for this task (Wang et al., 2020). Newer methods such as Wav2Vec2 (Baevski et al., 2020) have achieved great performance by masking speech input in the latent space and solving a contrastive task defined over a quantization of the latent representations which are jointly learned. This model trained on the librispeech automatic speech recognition (ASR) dataset (Panayotov et al., 2015) has found wide adoption for speech-to-text tasks. We augment the Wav2Vec2 speech model with a 5-gram n-gram language model trained on a corpus of K-12 academic textbooks.

heading: 2.2 Pre-req edge identification
text: Identification of prerequisite relations between academic concepts has been a subject of study for decades. Teachers and curriculum planners have extensively utilized this knowledge to determine the order in which chapters are organized in conventional learning textbooks and to guide students in covering the syllabus efficiently (Novak, 1990). However, recent data-driven approaches have facilitated the automated identification of prerequisites, resulting in enhanced performance and the emergence of new research avenues. One example is the information-theoretic approach proposed by
(Gordon et al., 2016). External knowledge bases, such as Wikipedia, have also been extensively employed. Liang et al. (2019) utilizes active learning on hand-crafted features (Liang et al., 2018b), while Sayyadiharikandeh et al. (2019) leverages Wiki click-stream-based features for prerequisite detection. Additionally, incorporating features similar to those employed in (Liang et al., 2018a), along with Long Short-Term Memory (LSTM) networks, has demonstrated strong performance as reported in (Miaschi et al., 2019). However, finding exact Wikipedia articles for domain-specific academic concepts is an error-prone process with poor results from direct search. Therefore, in our method, we avoid this mapping and find relevant features from the videos themselves. Recently, some methods have been developed to explore the determination of prerequisites between any two textual documents from different domains, including video transcripts, Wikipedia, etc. One such method, leverages aggregated fast-text word embeddings (Bojanowski et al., 2017) for effective prediction of prerequisites (Gasparetti, 2022). Furthermore, graph-based deep learning methods have also been explored (Li et al., 2019), but these methods tend to require a large amount of training data and may have limited real-world performance.

heading: 2.3 Parsing Academic Textbook PDFs
text: PDF parsing is a well-researched issue, historically addressed using rule-based techniques to extract data from documents’ layouts (Mao et al., 2003). Many recent tools use Conditional Random Fields (CRFs) which are undirected graphical models trained to maximize a conditional probability that can be used to segment and label sequence data (Singh et al., 2016).
Additionally, it is possible to treat PDFs as im-
ages and perform text detection and extraction to extract the content. Deep learning computer vision methods have been found to be useful in this regard. For example, Siegel et al. (2018) utilized a modified version of the ResNet101 network to extract figures and captions from scientific documents. Architectures such as U-net (Ronneberger et al., 2015) has also been utilized for performing text body identification (Stahl et al., 2018). Deep learning methods are also effective for finding tables, headers, or citations in PDF files, treating it as an object detection problem. Huang et al. (2019) uses Yolo (Redmon et al., 2016) architecture to find tables in PDF files. However, it is important to note that most current work focuses on parsing research papers, and work on academic textbooks is limited.

heading: 3 Methodology
text: In this section, we present a comprehensive explanation of the two-stage pipeline used for identifying prerequisite edges between academic videos as shown in Figure 1. The pipeline comprises a transcript extraction stage, followed by a feature extraction and classification stage for prerequisite edge detection. Additionally, the pipeline requires a corpus of academic text obtained from academic textbooks. To fulfill this requirement, we have developed our own academic textbook parser.

heading: 3.1 Transcript Extraction
text: The first step in this process is to create a language model that can be used alongside the Wav2Vec2 speech model to improve the transcription of domain-specific terminologies. In order to create this language model, we use our corpus of academic K-12 text. This corpus contains parsed data from classes 9th to 12th for science and math subjects. To create a generic academic video tran-
scriber, we use all textual data from this corpus. However, for a specific class and subject video transcription, it is possible to query data for only that use case and train the language model accordingly. We create a 5-gram n-gram language model using the KenLM method (Heafield, 2011). KenLM performs interpolated modified Kneser Ney Smoothing for estimating the n-gram probabilities (Kneser and Ney, 1995). This model is used to form the decoder, which is combined with the processor’s tokenizer and feature extractor to form the Wav2Vec2 processor with language model. We use this processor on the output of the Wav2Vec2 Large 960h model trained on the librispeech ASR dataset (Panayotov et al., 2015) to extract transcripts. The fine tuned language model aids the decoding process in Wav2Vec2 by providing context, which adjusts the prediction of the next token in the sequence based on the sequence of previously predicted tokens, thereby enhancing the linguistic coherence of the transcriptions.
However, in order to process MP4 videos through this pipeline, we must first extract audio in the required format. Audio is extracted and saved as an MP3 file. Then, this MP3 file is re-sampled at 16 kHz (the frequency used by the Wav2Vec2 model). Also, as the model only works well with mono-audio, we check if the audio is in stereo format and convert it into mono-audio if required. We use FFmpeg tool (Tomar, 2006) to perform this processing. Finally, the processed audio is saved as WAV files that can be passed into the model to extract transcripts.

heading: 3.2 Pre-requisite Edge Detection
text: The problem of finding prerequisites between academic videos is formulated as follows. An academic video corpus of an online learning platform can be represented by n videos, denoted as C = {V1, · · · , Vi, · · · , Vn} (1), where each Vi is one academic learning video. Each video Vi can be further represented as Vi = {Transcript, Title, Taxonomy, Extracted Phrases} (2).
Transcript is the document of video text of the form Transcript = (s1 . . . si . . . s|V |) (3) , where si is the ith sentence of the video text.
Title is the heading of the video, which is typically the academic concept that the video covers.
Taxonomy is a tuple associated with each video of the form: (su, cl, ch, to, st) (4) where su ∈ Su, cl ∈ Cl, ch ∈ Ch, to ∈ To, st ∈ St where
the set of all subjects is represented as Su, the set of all classes as Cl, the set of all chapters as Ch, the set of all topics as To, and the set of all subtopics as St. All sets, Su, Cl, Ch, To, and St pertain to the K12 curriculum. Furthermore, in this paper, we use a subset of Su and Cl as follows: Su = {Science, Mathematics, Physics, Biology, Chemistry} and Cl = {x | x ∈ Z, 6 ≤ x ≤ 12}.
Extracted Phrases is an ordered set, denoted as {pi|i ∈ N, 1 ≤ i ≤ m} (5), comprising of phrases extracted from the Transcript of Vi (3) using Textrank (Mihalcea and Tarau, 2004). Here, m represents the total number of extracted phrases, and pi denotes the ith phrase. pi is ranked higher than pj if i < j. We opted for TextRank for keyword extraction due to its unsupervised, graph-based nature, which enables it to effectively capture contextual and semantic relationships within the diverse and complex language used in academic video transcripts. Its simplicity and versatility across domains also ensured it could efficiently handle our broad range of data.
Based on these definitions, the problem of finding prerequisites between academic videos in corpus C (1) can be represented by a function F : C2 → {0, 1}, where :
F (⟨a, b⟩) = { 1 if a is prerequisite of b
0 if a is not prerequisite of b (6)
and where ⟨a, b⟩ is a video pair (7) , a, b ∈ C (1). Given this video pair ⟨a, b⟩, we can extract a set of similarity-based features from their content (2). Let (Tra, T ia, Taa, Ea), (Trb, T ib, Tab, Eb) (8) be the transcripts, titles, taxonomies and extracted phrases of videos a and b, respectively. In order to find similarity-based features between these, we define a set:
content pair = { (x, y) |x ∈ Tra, T ia, Taa, Ea
y ∈ Trb, T ib, Tab, Eb (9)
We prune the set content pair manually to remove repeated and unnecessary pairs, and then define a function S : content pair → R (10) that computes the similarity between each pair of corresponding elements of the two videos. Let fi be one possible value generated by S, we take all these possible values together to form the final feature vector k = (f1, f2, · · · , fn). These features can then be used to learn the function F : C2 → {0, 1} (6) using a supervised learning algorithm.

heading: 3.2.1 Calculating Similarity
text: For calculating the similarity as part of the function S (10) described above, we use the following approach: We employ two fine-tuned models, Word2Vec Skip-Gram (Mikolov et al., 2013), pre-trained on 100B Google News words and finetuned with a lock-factor of 0.2 for 5 epochs on our K-12 corpus, and FastText (FT) (Bojanowski et al., 2017), also fine-tuned on the same corpus. Word2Vec is utilized for phrases with less than 5 words; FT for longer phrases. For Word2Vec, embeddings are averaged to obtain a 300-dimensional vector, while FT directly generates sentence-level embeddings. Cosine similarity is computed between the 300-dimensional vectors to determine similarity scores, with -1 indicating complete dissimilarity and 1 representing identical inputs.
We opted for Word2Vec and FT, over transformer models, for their computational efficiency and simplicity, given our large transcript dataset. Word2Vec was chosen due to its strength in handling common words, while FT was selected for its speed and reduced out-of-vocabulary issue, which is particularly useful for longer phrases. Despite the embeddings being in different spaces, the similarity computation remains consistent as we use Word2Vec for shorter phrases and FT for longer ones, ensuring comparable similarity scores across phrase lengths.

heading: 3.2.2 Features Extracted
text: The following features are extracted for each video pair < a, b > (7):
• Title similarity: the similarity between the titles of the two videos Tia, T ib (8), is expected to be higher if the videos occur in a linked context in the K-12 corpus, suggesting that they have pre-requisite dependencies. • Taxonomy Similarity: Chapter- and subjectbased information is vital for determining the prerequisite order of videos. Hence, we calculate the similarity as described above between the taxonomies of two videos Taa, Tab (8). • Title and Transcript similarity: The title of a video appearing in the transcript of another video can be utilized to find dependencies. Therefore, we find similarity between the Title and Transcript Tia, T rb and Tib, T ra (8):
– Simple count of Title and its subsentences in the Transcript.
– Sum of similarities between Title and all phrases in the Transcript i.e for Tia, T rb we compute
∑|Vb| i=1 ∑phrases(si) j S(Tia, j) (10)
where, phrases(si) represents the word phrases in the sentence si and not the extracted phrases using textrank.
– Cosine similarity between the TF-IDF vectors of Title and Transcript.
Additionally, we apply this process to the first 500 characters of the Transcript, as these initial sentences often contain crucial information that indicates prerequisite relationships (Liang et al., 2018a).
• Title and extracted phrases similarity: The title of one video occurring as an important topic in another video can indicate that it is a prerequisite. Thus, we calculate the similarity between Tia, Eb and Tib, Ea (8):
– ∑|Eb|
i=1 S(Tia, pi) where pi ∈ Eb and∑|Ea| i=1 S(Tib, qi) where qi ∈ Ea. – List of instances where the similarity exceeds specific thresholds: {pi ∈ Eb|S(Tia, pi) > t} and
{qi ∈ Ea|S(Tib, qi) > t}, where t ∈ {0.1, 0.2, . . . , 0.9} (11)
• Title and taxonomy similarity: We compute S(Tia, j) where j ∈ Tab and S(Tib, l) where l ∈ Taa (4) to take into account the relatedness of the video title Ti with the subject, chapter, topic or sub-topics in the taxonomy Ta of the other video.
• Similarity between extracted phrases: For each phrase pi ∈ E′a, where E′a denotes the top 10 extracted phrases in Ea, we find the similarity with the extracted phrases in Eb (5), and then sum these similarities while multiplying with the weight wi:
wi ∑
pj∈Eb S(pi, pj) where wi =
1
λi
and i ∈ N : 1 ≤ i ≤ 10. We obtained the best results when λ = 1.1. The motivation behind the weighting parameter arises from the notion that higher-ranked phrases tend to be of greater importance or relevance for prerequisite determination. By incorporating this weighting scheme, we assign more weight to the phrases that are ranked higher, hence magnifying their influence on the similarity score.
• Similarity between video content: To calculate the overall similarity between the two transcripts, we utilize cosine similarity between their TF-IDF vectors, treating them as two independent textual documents. For calculating similarity between two large video transcripts, we use TF-IDF due to its computational efficiency and its capacity to detect recurring themes. TF-IDF, when combined with cosine similarity, enables us to compute the overall resemblance between transcripts, irrespective of their length. This makes it a practical solution for identifying textual similarities in extensive video transcripts.
The aforementioned features result in a feature vector of size 316. Additionally, we append a 665- length Bag of Words (BOW) vector, representing the combined titles of the two videos in the format "<Title of Video A> <Space> <Title of Video B>". This yields a combined feature vector of size 981, which is used to train our models in a supervised setting. We evaluated the performance of 36 widely-used machine learning models for all supervised tasks in this study, and present the results of the models that demonstrated superior performance.

heading: 3.3 Parsing Academic Textbook PDFs
text: Previously, it was demonstrated that a hierarchically organized and clean K-12 academic corpus is essential for both transcript extraction and prerequisite edge determination. To accomplish this, we have created a collection of academic textbook PDFs that are publicly available1. We have selected PDF textbooks in the science, physics, chemistry, biology, and mathematics domains for classes 9th through 12th. Initially, these PDFs are converted to XML using PDF2XML (Peng and Zhang, 2004). Following this, we classify each font into one of three text classes: chapter names, section or subsection headers, and text body, based on the following features:
• Font frequency and size: Chapter names and section headers use fonts that are larger and occur less frequently than the general text, making their font occurrence frequency and size distinct from the general text. • Font location and page occurrence: Chapter names and section headers are positioned at
1NCERT website
the top of the page, and chapter names occur earlier in the overall text. This allows the use of statistical measures of font average location and page number, to distinguish between different text classes. • Color: Section headers and chapter names frequently use distinct colors. We calculate Euclidean color distance (12) between font color and black and white colors to quantify the font color’s uniqueness compared to the page’s most common colors. dist(C1, C2) = √ (r1 − r2)2 + (g1 − g2)2 + (b1 − b2)2
(12) where C1 and C2 represent RGB color values [r1, g1, b1] and [r2, g2, b2] respectively.
• Line width and section numbers: Section numbers (13) can distinguish section headers from other text classes. Additionally, chapter names tend to have a narrower average line width. Sectionno. = x.y.zorx.y, wherex, y, zϵN
(13) Upon extracting the features, a machine learning model classifies each font into three text classes, assigning a class to each text line based on its font. Following the extraction of academic content, section and chapter names, section numbers in headers are utilized to derive the taxonomy. The extracted textual data and its hierarchical structure are included in the released datasets.

heading: 4 Dataset

heading: 4.1 Transcript dataset
text: To showcase the efficacy of our proposed Wav2Vec2 speech model combined with the language model trained on our K-12 corpus, we assembled a dataset comprising five random academic videos in the science and math domains from YouTube. We provide ground truth subtitles for these videos, alongside subtitles extracted by our algorithm and other benchmarks for comparison.

heading: 4.2 VID-REQ pre-requisite dataset
text: To assess our approach, we introduce Vid-Req, a large-scale video prerequisite edge dataset. We initially gathered over 1,500 animated academic videos covering science, mathematics, chemistry, physics, and biology for grades 6 through 12 from Extramarks a leading EdTech company. On average, each video encompasses 418 words. However, these videos resulted in 1,124,250 distinct
video pairs (1500C2), which was an overwhelming amount for labeling. Consequently, we selectively choose videos based on a specific criterion to reduce the dataset to a more manageable size. For this purpose, we firstly find chapter-level prerequisites and formulate the set CP = {(ch1, ch2)|ch1 is a prerequisite of ch2} where ch1,ch2 are chapters. Using CP , we form the potential video prerequisites set PV P = {(a, b)|a, b ∈ C, (cha, chb) ∈ CP, cha ∈ Taa, chb ∈ Tab} (1,4,9). Then, we prune the set PV P to form PV P ′ = {(a, b)|S(Tia, T ib) > 0.7, (a, b) ∈ PV P}. This set comprises 2,797 edges that we have hand-labeled, of which 1,684 are labeled as 0 (non-prerequisite edges) and 1,113 as 1 (prerequisite edges).
Figure 2 displays the pre-requisite edge statistics for the entire dataset, including label 0 (not pre-requisites) and label 1 (pre-requisite edges) on the left, and only label 1 on the right. The figure shows that science-to-science edges are most frequent in the total dataset (n=1167), but in the label=1 set (n=455), mathematics-to-mathematics edges prevail (n=470). While mathematics appears as a pre-requisite for all subjects in the full edge set, it only acts as an actual pre-requisite for itself and science. Science remains a pre-requisite for other subjects, with most pre-requisite edges leading to physics, biology and chemistry (n=61,23,20).

heading: 4.2.1 Annotation Process
text: Multiple experienced teachers were invited and assigned to their preferred subjects, with at least three teachers per subject. These domain experts annotated video pairs, determining if video "B" had a prerequisite video "A" by assigning binary labels (1: A is a prerequisite of B, 0: A is not a prerequisite of B) and also assigned a unique taxonomy from the set of taxonomies extracted from K12 text-
books parsed using our PDF parser to each video. Teachers viewed the videos thoroughly before annotating and provided well-informed judgments and reasons. The relationship is non-symmetric. After annotating 2797 video edges, Cohen’s Kappa coefficient (0.64) confirmed substantial agreement among annotators. These final annotations served as ground truth labels for model training.

heading: 4.3 Academic textbooks dataset
text: We generated a training dataset for PDF parsing by downloading 26 textbooks from2 and converting them to XML using PDF2XML. These textbooks span various subjects and classes, covering 662 unique fonts for chapter names (n=53), text body (n=563), and section name (n=46) text classes, hand-labeled by expert academicians. The model trained on this dataset was used to parse 189 PDFs for subjects like science, math, chemistry, biology, and physics for classes 9 to 12. Intermediary XML files and extracted text with taxonomical hierarchy and page numbers have been released.
Additionally, we created a dataset of 731 handlabeled textbook pages to test our method with object detection baselines, using an 80:10:10 train, validation, and test split. Pages were converted to 416x416 pixel JPEG images, and three augmentations (horizontal flip, vertical flip, and random crop) were applied which led to the final 1755 images with 1901 total objects.

heading: 5 Experiments

heading: 5.1 Transcript extraction
text: We evaluated the performance of Wav2Vec2 Large 960h (Baevski et al., 2020) trained on the Librispeech ASR dataset (Panayotov et al., 2015), with and without our language model (Wav2Vec2 and Wav2Vec2-LM), using Word Error Rate (WER), Match Error Rate (MER), and Word Information Lost (WIL) metrics. We compared it to the Deepspeech ASR method (Amodei et al., 2016), with Wav2Vec2 outperforming Deepspeech in speed and accuracy. Both models ran on CPU, reporting average run-time per video in seconds. Our language model’s inclusion improved domain-specific word transcription and reduced error rates, as shown in Table 1.
2NCERT Textbooks Webpage

heading: 5.2 Pre-requisite detection

heading: 5.2.1 Performance on VID-REQ dataset
text: Upon evaluation, three models emerge as the topperforming models on our released dataset of 2,797 prerequisite video pairs (VID-REQ). These models—Extra Trees (Geurts et al., 2006), LightGBM (LGBM) (Ke et al., 2017), and Random Forest classifiers with linear SVC feature selection (RFSVC) (Breiman, 2001)—are assessed using 5-fold cross-validation, reporting mean accuracy, precision, recall, and F1-score as shown in Table 2. Hyperparameters for each model were fine-tuned via grid-search from Scikit Learn (Pedregosa et al., 2011). Extra Trees emerged as the best-performing model with an F1-score of 79.08%. Although both Extra Trees and Random Forest employ multiple decision trees, the difference in performance can be attributed to their responses to various feature characteristics. The unique splitting mechanism of Extra Trees, which involves more randomness, lends robustness when dealing with potentially noisy or complex data. This resilience to the inherent complexities of the feature set likely contributed to Extra Trees’ superior performance over the LGBM and RF-SVC classifiers in our study. We employed the F1-score as a reliable metric given its simultaneous consideration of both precision and recall. This is crucial from a learner’s perspective, as it is vital to prevent mislabeling non-prerequisite videos as prerequisites while accurately identifying all essential prerequisite videos. Moreover, the F1 metric effectively addresses the slight class imbalance present in the dataset.
Furthermore, we replicate the approach outlined in Gasparetti (2022) on our dataset as a baseline comparison. This technique utilizes aggregated fast-text word-embeddings input into SVC and RF classifiers to predict prerequisite dependencies between pairs of textual documents. As demonstrated in Table 2, our method surpasses the baseline in all metrics, with an F1-score exceeding by more than 10%.

heading: 5.2.2 Performance on AL-CPL dataset
text: We also compared our features with those of (Liang et al., 2018b, 2019). The dataset released in Wang et al. (2016) is the most widely used Wikipedia prerequisite dataset, which covers data mining, geometry, physics, and pre-calculus subjects. The authors of Liang et al. (2018b, 2019) have pre-processed this data which is released as the AL-CPL dataset. We extract our features from this dataset and quote F1-score performance using 5 fold cross validation of the best performing model i.e., Random Forest with linear SVC feature selection in Table 3. We also compare the results of this model with those of Miaschi et al. (2019) who have used a multimodal architecture that uses LSTM and global features similar to Liang et al. (2018b, 2019) to predict pre-requisites. Both the above mentioned methods quote mean 5-fold cross validation results for the F1 metric. However, Miaschi et al. (2019) has showcased performanced on in-domain and crossdomain prerequisite relationships separately, on 3 variants of their proposed architecture (M1,M2,M3). Thererfore, in order to facilitate direct comparison we choose best results for the F1-score across the models and then take average of the in-domain and cross-domain results. As evident in Table 3 our method surpasses Liang et al. (2018b, 2019) for all subjects and Miaschi et al. (2019) for 3 out of 4 subjects. The average F1-score across subjects of our methods also surpasses that of Miaschi et al. (2019).

heading: 5.2.3 Performance on Meta-Academy dataset
text: We further showcase performance of our method on another Wikipedia pre-requisite dateset that includes pre-requisites extracted from MetaAcademy (Sayyadiharikandeh et al., 2019). Metacademy is a free, open-source platform encompassing 487 machine-learning concepts connected by 7,947 prerequisite pairs. Our top-performing model, RF-SVC, trained on our novel features, demonstrates superior performance compared to the AdaBoost model trained on Wiki-clicks-based features (user navigation patterns on Wikipedia) on this dataset. As exhibited in Table 2, our model surpasses the AdaBoost model across all metrics, with an F1-score exceeding by over 5%.
These experiments showcase the robustness of our features, exceeding benchmarks for Wikipedia prerequisites tasks, even though they were designed for videos. This success can be attributed to our in-depth collaboration with domain expert teach-
ers during feature creation, leading to enhanced effectiveness and performance of our algorithm.

heading: 5.3 PDF Parsing
text: To evaluate performance on the dataset described in Section 4.3, we use an 80:20 train-test split. The LightGBM classifier (Ke et al., 2017) achieves the best classification results as shown Table 4 and is used in the PDF parser to generate our K-12 corpus.
To compare our PDF parsing methods with recent deep learning-based approaches, we treat the extraction of text-classes as an object detection problem, focusing on the crucial section name text class. We use a random subset of textbooks (46 section headers) and extract section headers using both methods. Headers are considered correctly matched if they have distance D (14) less than 0.6 (Doucet et al., 2011; Wu et al., 2013).
D = LevenshteinDist(A,B) ∗ 10
Min(Len(A), Len(B)) (14)
For this experiment, we use the YOLOv5 model (Jocher, 2021) for object detection and EASYOCR (AI, 2021) to extract text from cropped header images. Our font-based classification method outper-
forms the YOLO + OCR approach in both performance and average per-page time as shown in Table 5. The deep learning method’s low precision stems from its reliance on visual features alone, which are inadequate for detecting text-classes. In contrast, our method utilizes text, color, and occurrencebased features for accurate classification, and by labeling only the fonts in PDF textbooks, it achieves faster and more precise performance.

heading: 6 Conclusion
text: In this paper, we present a pipeline for detecting prerequisite dependencies among academic videos using novel similarity-based features. Our approach outperforms existing methods, even surpassing prerequisite detection in domains like Wikipedia. We introduce hand-labeled datasets to discover prerequisite relations across diverse subjects, fostering future research in this area.
Future work will explore additional features and methods, extending our approach to a broader range of educational content such as podcasts, slides, and lecture notes. We also aim to integrate collaborative filtering and recommender systems for personalized learning paths, enhancing students’ educational experience and learning outcomes.

heading: Future Work
text: We introduce hand-labeled datasets to discover prerequisite relations across diverse subjects, fostering future research in this area. Future work will explore additional features and methods, extending our approach to a broader range of educational content such as podcasts, slides, and lecture notes. We also aim to integrate collaborative filtering and recommender systems for personalized learning paths, enhancing students’ educational experience and learning outcomes